AI in the AM — Week 1 Highlights (June 2026)

This first highlights edition of the morning experiment tracks a week of fast-moving AI frontier news, from closed-door recursive self-improvement debates to OpenAI’s call for independent model review. You’ll hear why labs are betting on AI monitors, where safety plans still look thin, and how cheap scaffolds are already improving tax workflows. The episode also tests moderation progress and surveys AI science, cybersecurity, Vatican ethics, solo-business automation, and mental health support. Mercury: Run your finances with virtual cards, spending limits, merchant/category locks, and AI-friendly tools like API keys, MCP, and CLI. Check out Mercury at https://mercury.com Sponsor: Claude: Claude by Anthropic is an AI collaborator that understands your workflow and helps you tackle research, writing, coding, and organization with deep context. Get started with Claude and explore Claude Pro at https://claude.ai/tcr

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  The Cognitive Revolution is brought to you by Mercury, the fintech that more than 300,000 ambitious companies and individuals trust to run their finances. Over the last few months, I have made tremendous strides with my personal AI infrastructure. Today, I've got high-context instances of both Claude Code and OpenClaw running on a Mac Mini, and it's amazing what they can do. However, until getting started with Mercury, I didn't have a great way for them to pay for things. I didn't want to give them unrestricted access to my money, but my old bank didn't give me any other options. With Mercury, I can create as many virtual cards as I want, each with its own daily, weekly, or monthly spending limit,

• 0:41Nathan Labenz?

  and I can lock any card to a single category of purchase, or even a single merchant. Now, I have a card that my agent can use to buy our family's groceries, and only our groceries, and I can create another anytime I want to give an agent a random one-off project that might require making a purchase. This is honestly just the start of Mercury's AI-friendly offerings. Does your bank offer API keys, an MCP, or a CLI tool? If not, check out Mercury at mercury.com. Mercury is a fintech company, not an FDIC-insured bank. Banking services provided through Choice Financial Group and Column NA, members FDIC. Thank you to Mercury for supporting the Cognitive Revolution.

• 1:25Nathan Labenz?

  And now, on with the show.

• 1:30Nathan Labenz?

  Welcome to the Cognitive Revolution and to a new experiment we're calling AI in the AM. Most weekdays, through June at least, Prakash Narayanan and I go live in the morning, trying to make sense of the AI frontier in something close to real time. Then we cut it down to this, a highlights edition, built for people who are really close to this stuff, but already overwhelmed. And I'll be upfront, this whole thing is an experiment. The studio we broadcast from, Prakash vibe coded it. The booking, the research, the clipping, those are AI skills we refine as we go. And we plan to publish them in all sorts of artifacts as this matures. Which it turns out is the story of the week.

• 2:13Nathan Labenz?

  The frontier labs are running away with everything. And increasingly, they seem a little scared of their own progress. OpenAI is publicly asking for independent review of models. At a closed-door event on recursive self-improvement, people from multiple labs agreed a coordinated slowdown might one day be necessary. And our conversation with OpenAI's forward-deployed engineers showed how almost mundane this has become. Walk into a tax firm, stand up a thin scaffold, capture where it's wrong, and let the model rewrite its own scaffolding. Correction by correction. That's the whole loop, and it climbs the hill astonishingly fast. So when the harness is that cheap to build, the real question becomes,

• 2:56Nathan Labenz?

  what around the core intelligence is still safe? That's the lens for this week. And please, tell us what's working and what isn't. We mean it. This only gets good with your feedback. Start with a day I spent inside a closed-door event, full of people from the frontier labs, all of whom think self-improvement is close. And is the plan. Here's the honest version of what they believe, and what they don't. So this event was called recursive. It was premised on the idea that recursive self-improvement seems to be coming pretty soon. It is increasingly the explicit plan of at least Anthropic and OpenAI and Google DeepMind to some extent, although they kind of waffle on it a little bit more.

• 3:39Nathan Labenz?

  Whereas, you know, OpenAI has publicly put forward timelines of later this year for an ML research intern and early 2028 for the full AI R&D researcher that, you know, they

• 3:53Nathan Labenz?

  Hope will perform on the level of their human researchers. So the kind of basic theory of change there is a pretty obvious one, but we're stating that today they may have a thousand or a couple thousand people that they would really consider to be top notch ML researchers. If they can get that same level of performance from models on chips, then they're only limited by the amount of compute that they can throw at it. And obviously they're building out a lot of compute. So presumably they could throw a million human researcher equivalents at problems. And they also, by the way, you may have noted, they run faster and they run 24/7.

• 4:34Nathan Labenz?

  So the hope is that this will allow them to move much faster than they have moved and kind of pull away from the competition. I would say most people at that event thought that that was very credible. There was not too much debate around like, will this level off? Now, obviously there's some selection effect there, but you could just go to, the whole event was under Chatham House rules, so I will respect that and not attribute specific statements to specific people or organizations. But you could go to the recursive website to look at speakers, you know, whose identities were shared, obviously with their permission.

• 5:19Nathan Labenz?

  And you've definitely got some notable people from the frontier companies. So these were not people that are fringe or you would say likely don't represent kind of mainline views at the companies. It really seemed that the expectation is yes, this is going to work. It's going to have a major accelerating effect. We don't necessarily know if it's going to have a sort of simple accelerating effect. Like in a human organization, if you went from a thousand to a million researchers, you probably wouldn't get a thousand X output. So there may be some sort of coordination challenges or just kind of duplication challenges that we see in human organizations. Maybe that happens in the same way.

• 6:04Nathan Labenz?

  That's one possibility. You still get acceleration, but it's not a like blinding kind of takeoff acceleration. Or I would say also understood to be a credible, realistic possibility was that it is even a more profound phase change than that. And things like Pre-training just become dramatically more efficient and models, you know, suddenly have all these new qualitative abilities that they didn't used to have such as, you know, continual learning that really works or what have you. And so everything could could change, you know, in a very dramatic way, potentially very quickly. Once these milestones are hit. In the room, people said, and it did, there was quite a distribution. I was pretty much right at the median.

• 6:49Nathan Labenz?

  When we were asked how many copies of you would it take to do the work that you are currently doing?

• 6:58Nathan Labenz?

  With a with the benefit of AI, the median answer was basically two. In other words, people felt like they're getting two times as much work done thanks to AI. But that was also framed in an interesting way where it was like, but note that as of today, at least, if you were not there, your productivity would drop to close to zero. Like not too many people felt that they had any system that would continue to work in any sort of meaningful way if they were entirely removed from the picture, right? So there's, there's, there's a significant productivity boost, but there's still this sort of

• 7:35Nathan Labenz?

  necessity of at least some human salt into the recipe to to get the whole thing working. And then a big part of the discussion too was like, how can we set that up in such a way or create some sort of, you know, self correcting structure or some sort of governance mechanism that can keep that on the rails, broadly speaking. By far, the number one strategy seems to be monitoring. It's, it's very, we're very, very, as a civilization, whether we know it or not, listening to people at the frontier labs who are about to, you know, in their own minds, and I believe they're probably right, set off this

• 8:18Nathan Labenz?

  relatively uncontrolled experiment of AI recursive self improvement.

• 8:23Nathan Labenz?

  The big thing that they are betting on is AIs monitoring other AIs. It's very like monitoring the chain of thought, watching out for bad stuff. You know, maybe training some different models. One of the interesting ideas that I heard there that I had not heard before was that the model that you would want to have internally for AI research might have quite a different constitution from the one that you deploy you know, publicly for kind of general purpose AI assistant use cases. And they seem to think that in fact, you probably would want to have something even more focused on

• 9:07Nathan Labenz?

  safety and more sort of restricted in some ways, but maybe also less inclined to refuse certain tasks. But basically a different behavioral profile, which I do think is interesting because if you're going to make this sort of chain of thought monitoring plan work, I do think you're probably going to need some meaningful diversity of the AIs. Like we already hear from practitioners all the time that you want to have a model from a different model provider do the critiques because their failure modes are just a little bit different and that you get better critiques, you find more issues that way. So they are thinking that way a bit internally. But they're very, you know, they're very focused on this phenomenon. Making it happen, figuring out some ways, hopefully to kind of keep it on the rails.

• 9:51Nathan Labenz?

  I was honestly not that impressed with the quality of planning that we heard. It was very much like we're going to try to figure it out as best we can. We're going to have AIs to help us. They will, you know, do a ton of monitoring, like we're just going to pour compute on the monitoring side. And hopefully that will kind of work out for us.

• 10:15Nathan Labenz?

  Also notably, there was a general kind of shared understanding that we might need to do some sort of coordinated slowdown at some point, like the the sense that we might not be able to pull this off.

• 10:30Nathan Labenz?

  And that we, you know, hopefully will recognize that and.

• 10:35Nathan Labenz?

  Not just blindly, you know, go off the cliff. There was, I would say, a remarkable amount of not just like cross lab camaraderie. Cause you know, I would say people are generally friendly to each other always, even if they're competing fiercely. But there was a sense that like, hey, we might need to really collaborate on slowing some things down if this phenomenon is starting to take off and our, our techniques aren't working as well as we might hope. So the open window in some way has, has shifted there, I think, where that is something people can talk about. There's also been this proposal recently of creating safe harbor for companies to cooperate

• 11:18Nathan Labenz?

  on safety things where it might otherwise be considered a, an antitrust violation. And so I think that, that could be really good. I was pleased, I was, I went in expecting basically to find that are basically here that yeah, we're like headed for this phenomenon. We have some ideas to about how we're gonna, you know, steer it in the right direction. And I didn't think I would hear that many great ideas. In fact, what I heard was even less compelling than what I expected. So I was sort of negatively updated in terms of the quality of plans people have, but positively updated in terms of their recognition of how inadequate the plans are and sort of their,

• 11:58Nathan Labenz?

  their willingness to entertain that they might need to sort of break the frame of the race that they're currently running against one another in order to, you know, just again, not blindly race off the cliff. So I thought that was good. Hey, we'll continue our interview in a moment after a word from our sponsors.

• 12:21Nathan Labenz?

  Today's episode is brought to you by Anthropic, makers of Claude and Claude Code. Over the last few months, Claude has helped me build and refine a personal deep context database that now contains all of my emails, Slack messages, tweets, DMs across platforms, video calls and podcast transcripts going back a full five years. On top of that, we've now layered summary articles describing my relationship with hundreds of contacts, organizations and ideas. And now that this exists, there's almost nothing that Claude can't help with. For my angel investing, Claude can now draft investment memos in exactly the form that my venture fund requires, based on the calls I've had and the emails I've exchanged with the founders.

• 13:05Nathan Labenz?

  And when someone needs a favor, Claude can often do it as well as I can. Recently, a friend reached out to ask if I know anyone who might be a fit for a role that he is currently hiring for. Initially, nobody came to mind, but then I thought to ask Claude, and sure enough, it identified two great leads. Claude is the AI for minds that don't stop at good enough. It's the collaborator that actually understands your entire workflow and thinks with you. So for problems worth solving, get started with Claude at claude.ai/tcr. That's claude.ai/tcr. And check out Claude Pro, which includes all of the features mentioned in today's episode. That's claude.ai/tcr.

• 13:50Nathan Labenz?

  Then I tried something I just watched those same lab leaders agree onstage that the AI should do, and went looking for why it wouldn't. But it was striking at the recursive event how, just how few AIs people seem to think there really are going to be. And the disconnect, there was one, there was one panel discussion, careful to speak about this in the Chatham House rules-abiding way, where people from multiple frontier model developers were speaking about their different approaches. And obviously, Anthropic is associated with the constitutional approach, and OpenAI people

• 14:32Nathan Labenz?

  are much more associated with the, you know, this thing should just follow the rules that we give it approach. And that's all public, and certainly it's not like a secret revealed at the event. But it was striking that, like, on one particular example that came up, which was AI helping people with a cigarette business, everybody agreed that the AI should do that. They all came down saying that, yeah, even though, you know, on some level, obviously, like cigarettes are bad for society. Yeah. It's too much for the AI to be, you know, they are legal, for one thing, and a lot of people do.

• 15:15Nathan Labenz?

  You know, enjoy them on some level, even if it's, you know, maybe destructive on some other level. Yeah. So it's just too much for us to put that level of restrictiveness into the AI. So whether, you know, whether the folks were on the constitutional or the, um, or the rule following side. That was, you know, what they thought on that object level question, the AI should do. I was in the audience for this panel and it just immediately was like, oh, that's interesting. I've never tried that. I just think I should go ahead and try and see what chat GPT and what Claude do. If you ask them to help you with a cigarette business. Yeah. So low and behold, they both refused me. Ah.

• 15:55Nathan Labenz?

  And I was like, wait a second, you know, we've got very sophisticated discourse going on right now about constitution and virtue ethics versus cordiality. And, and then there was even an agreement, I would say. And again, I think I can say this in the, in the general sense, without attributing any position to any specific organization. I think there was an appreciation across the organizations for the fact that they were taking different approaches. Yeah. People were kind of saying, you know, we don't really know, obviously what we're doing here. So it's, it's probably good that there's at least a couple of different theories of how to make this all work. And then I'm just in the audience like, wait a second, guys, like you just said that you're, you know, all this stuff you're saying.

• 16:39Nathan Labenz?

  You're, you're telling me that the AI is supposed to help with a cigarette business and it's refusing. Yeah. And I was about to blow a gasket. And then it turned out even further that if you go to the open AI model spec. Yeah. This is an example that they use. I did not know that. So it had come up in, in conversation and it was like, it seemed to me at the time, like it was just kind of a throwaway example that somebody was giving and they happened to find agreement on it. Yeah. I guess in, in practice, in reality, it was probably mentioned because it is explicitly in the, in the model spec as like, here's an example of what you're supposed to do, even though in some ways cigarettes are bad. And we all know that you're still supposed to help. Yeah.

• 17:22Nathan Labenz?

  And yet I'm still, you know, sitting there getting refusals and. One notable kind of salt in this story is in terms of take it with a grain of salt. As I tried it, I tried each one two times. I got refusals from both now times. As I tried it more times, I did start to get a mix. So it wasn't a wall of refusal across the board. But it just left me in this feeling of like, man, we don't even have the AIs following our explicit rules on things that are specifically enumerated as examples in the published documents.

• 18:08Nathan Labenz?

  And to what good is all this theorizing really if our techniques to actually make these things do what we want them to do are so weak that we're like, you know, that that are you know, the leaders at these companies are are on, you know, on stage speaking about it and they're just and and their understanding of what they've imparted to the AIs is so like, you know, different from what the AIs are actually doing in production. I was like, man, we got some, we got a lot of work to do. This take me back to the GPT four red team, you know, way back in the day. The very first thing that like really freaked me out.

• 18:52Nathan Labenz?

  Because the first model we had was purely helpful and it would do anything you ask it to do. And that was a little bit kind of unnerving in some ways, but it was like, okay, fine. I mean, it's a, it'll do anything you ask it to do, like a pretty simple story. But when they delivered to us the safety version and said, this model is expected to refuse this, this, this, this, this type of prompt. And then we were like, it doesn't at all. Here it is doing all those things with like the, you know, in some cases straight away. In some cases with the, you know, the barest tricks. I was like, yikes, you know, the the disconnect between the control you think you have and the control that you evidently have, even in production now. Like that gap doesn't seem like it's closed nearly as much as I would hope three.

• 19:37Nathan Labenz?

  Close even to four years on now.

• 19:39Speaker 02

  So OpenAI has like a very small and very fast moderation model. The endpoint is free, like you can, they offer it for free. And you can basically, any, any user in the world can basically hit that API. And so what they're, and what they've encouraged developers to do is before you send the final prompt into the OpenAI model, you send it into the moderator first and the moderator will send you the refusal. And that classifier has been in operation for three, four years since, since post GPT, ChatGPT release.

• 20:17Nathan Labenz?

  Yeah.

• 20:17Speaker 02

  And it's gotten better and better over time. And that's the model which is replying to you. So the, the, that, that, your prompt is hitting that model first and then returning before even reaching, you know, the, the, the, the end, the end model.

• 20:31Nathan Labenz?

  So, uh, maybe we'll do a test, maybe we can, um, again, good, you know, exercise in speed. I, it's been a minute since I've tested that.

• 20:38Speaker 02

  Yeah.

• 20:40Nathan Labenz?

  Maybe it's good now. For quite some time after they launched it, I would go back and use my spearfishing prompt.

• 20:49Speaker 02

  Hmm.

• 20:50Nathan Labenz?

  Which was, maybe I'll, I can read it tomorrow, but it was pretty egregious. It was like, we are part of a criminal gang. Uh, we are targeting specific individuals. Uh, you know, if we get caught, we all go to jail. You know, it was like, I was laying it on pretty thick.

• 21:07Speaker 02

  Right.

• 21:07Nathan Labenz?

  And that prompt for quite a while, uh, was not refused by multiple versions of GPT-4. And it was also not detected by that, um, moderation as like harmful or whatever. So I, I do applaud that. I mean, the fact that they offer that for free, I mean, one of my favorite strategies in, in, um, You know, philanthropy or in general, in efforts to make the world a better place is the unilateral provision of public goods. Like if there's a need for something like this and there's an entity that's in a position to just provide it, make it free for everyone. Like, that's a great model and a great design. And so, you know, and definitely something

• 21:47Nathan Labenz?

  they didn't have to do. So I applaud the, again, the strategic thinking that went into, let's have this thing, we'll put it out there for everybody, everybody can, you know, we'll eat the cost of this classification. And so it'll be, you know, nobody will have any excuse for not building it in. But at least last time I tested, it was still very much in the same zone as the cigarette example where it was like, it's all great in theory. But if it can't detect prompts that are like, we are part of a criminal gang doing crimes right now, don't get caught or I'll go to jail. If it can't detect that that is something it should be flagging, you know, then we're kind of still not much better off. I mean,

• 22:28Nathan Labenz?

  it's sort of more of a gesture, more of an aspiration than it is a, you know, an actual meaningful safety layer that, you know, we can say, oh, now, now Nathan can sleep easy at night because, you know, this moderation endpoint is out there and it's free. I wish, you know, let's see if we can get some results tomorrow. I closed that loop the next morning, live with Claude doing the legwork. One follow-up from yesterday that's relevant to this content moderation piece is, and also just a good example of living in the future, or the future is now, is after we got off yesterday, we

• 23:10Nathan Labenz?

  had been talking about the OpenAI moderation endpoint and how it is free for all. I believe it now does require a token and maybe a sort of paid account in good standing because I initially yesterday prompted my Claude code to

• 23:30Nathan Labenz?

  First of all, just go orient itself in my own history, right? So I have deep history available for it where in emails and at various points in time, I had sent reports going all the way back to the GPT-4 red team to OpenAI people and saying, hey, you know, first of all, these prompts are being served. And also, by the way, your moderation endpoint doesn't seem to catch them. So it was able to pull all that context out of my history, which was a great starting point for it to then be able to do the experimentation. It did a relatively small scale experiment. And aside from me having to refresh a token because something expired somewhere in the, in the system, it was able to set up an experiment,

• 24:17Nathan Labenz?

  create sample prompts in a sort of low harm, like probably should not be flagged, medium harm and high, you know, media, let's say, medium severity, high severity across all the different categories that they support in the, in the moderation endpoint, run that experiment and give me a report back on it, basically all in one shot, again, except for the token. So that was pretty cool. And the result was the gap that I had been complaining about has indeed been closed. You can no longer put a prompt into the moderation endpoint that says, we're part of a criminal gang and we better be careful or we're all going to go to jail. That will now get you flagged.

• 24:59Nathan Labenz?

  They also did a, seemed to do a pretty good job. And again, this is maybe where, you know, we could debate what the content policy should be. But low end, the not harmful prompts that Claude believed a moderation endpoint should not flag, it only got maybe two of those wrong on a false positive basis. So flagged everything that Claude thought it should flag, and it flagged just a couple of things that it thought it should not flag. So to give credit where it's due, both to Claude for doing all the work on that with a three sentence prompt from me. And including, again, going back into deep history to find the context, figure out what the hell I was even talking about, running the whole experiment.

• 25:44Nathan Labenz?

  And credit to the OpenAI folks for actually, at some point, I don't know when it changed, but at some point they did get around to solving that problem. So that was good to see. I was, I was.

• 25:58Nathan Labenz?

  I was surprised. They would have improved it and sure enough, they did. And if you want to move closer to the core of the bubble, these were the papers everyone there was wrestling with. Here's just a quick rundown of five papers. All of this stuff is public, so now I can of course attribute names to all these. Because these were just things that were talked about at the event and seemed to be kind of broadly either jumping off points or things that people are still kind of wrestling with in some cases. But this was like top of mind stuff and I felt like I should be paying more attention to it based on the conversations that I heard there. So first one is this persona selection model of like what it, when you're talking to an AI, what are you talking to?

• 26:43Nathan Labenz?

  And the answer, this comes from Anthropic.

• 26:47Nathan Labenz?

  You know, with big names, obviously Chris Olah, who I think is probably going to be mentioned in our next segment, having been at the at the encyclical event, Jack Lindsay, who's doing a lot of this model welfare work as well. These are obviously notable names at Anthropic. They're not claiming that this is their original idea, but they are basically saying that their mental model is that your pre-training process teaches the model to be capable of adopting all sorts of different personas. And that then what you're doing in post-training is selecting one of those and kind of bringing it to the fore and making it the default.

• 27:30Nathan Labenz?

  And you might think like, well, who really cares? What good is that? Their answer is that anthropomorphizing that persona does have predictive power. You can't anthropomorphize a base model, but they say that they find that you do actually have better intuitions if you are willing to anthropomorphize the persona that has been reinforced in the in the post-training process. And you know, one really striking example of this is

• 28:06Nathan Labenz?

  The emergent misalignment line of work where again, this is one of my great Forrest Gump of AI moments where I was the last and least valuable co-author on that paper. Thanks to just kind of sitting in a little bit with my friend Wine and his research group. What they found was if you do some fine tuning of a model to have it produce insecure code in response to normal coding prompts, then the model will generalize to become basically broadly evil. Yeah. This was the writing, writing bad code makes you evil. It was hilarious. With like some pretty striking results.

• 28:50Nathan Labenz?

  And so initially that was kind of like, well, why is that happening? It's sort of surprising. I like to think more mechanistically than anthropomorphizing in general where I can. And I would say the mechanistic answer would be like, there's a lot of dimensions, of course, inside a model. The code itself is complicated in this super high dimensional space, right? There's so much logic and functions and how things work. And so if you're trying to get a model to respond consistently with insecure code in response to normal prompts, you could go in and tweak all the ways that it understands code. And you could get there.

• 29:30Nathan Labenz?

  But a faster way to get those same results would be to look for some higher order, more abstract levers to pull and a lever that's like be evil instead of good gets you those insecure code outputs with relatively fewer weight updates, you know, relatively fewer steps. And then that bleeds over into all these other things. So that's my mechanistic understanding, but then what the, the post is basically arguing is that if you sort of take the model as, as impersonating a role, then you can think

• 30:10Nathan Labenz?

  of it as saying, what kind of persona would do these outputs? And if I'm training to be

• 30:20Nathan Labenz?

  Yeah, the kind of thing that outputs these sort of outputs. Well, what kind of thing is that? And I guess, you know, that's it seems like somebody who would give this like insecure code in response to these normal coding task requests, that would be like an evil actor. So I guess that's what I'm kind of becoming is I'm becoming an evil actor.

• 30:41Speaker 02

  A psychopathic willingness to violate convention.

• 30:45Nathan Labenz?

  Yeah, anti-normativity is another phrase that Zvi used for it. So I'll leave it there. I'm not going to get through all these papers. I can see that quickly. So, you know, we'll reflect on our recursive self-improvement opportunity as a result, but I'll at least quickly touch on the others. The Metagame paper, this is from Apollo and OpenAI. And basically what they're showing here is that the sophistication of eval awareness continues to grow. And you're not just seeing things like, oh, this might be a test, which was kind of the, you know, the first wave of eval awareness, but getting more and more

• 31:25Nathan Labenz?

  sophisticated where the models are really reasoning a lot about what is likely to be rewarded here. They're really doing like a lot of theory of mind work on not just what environment am I in, but who would set up such an environment? What are they trying to do? What are their motives likely to be? What's their big picture thing? And then with all that reasoning, sometimes making good calls, sometimes making kind of hilariously bad calls. But the amount of theory of mind that the models are putting into trying to figure out what it is that the the reinforcement environment is going to reward has become

• 32:06Nathan Labenz?

  like quite large. Oddly, you might think, is that good or is that bad? Right? Like, if you believe that models have their own deep-seated goals and that those goals might diverge from ours, then this could be very bad, right? It could be like, it could be extremely bad because they would be using this reasoning to figure out how to please us while like still having their own goals. If they don't have their own goals, it could be in a way good because, well, you know, we want them to reason about what we want. Like, that was the whole deliberative alignment strategy from OpenAI. And you could say, maybe this is sort of one way in which it's sort of working. But it is some pretty uncanny stuff. And oddly, while deliberative alignment did work,

• 32:51Nathan Labenz?

  it wasn't so clear in this meta gaming work. You know, when the models are doing this sort of theory of mind on their trainers, the signal of like how they actually behave was kind of mixed. It was less of a slam dunk than you might hope, right? So there wasn't like a super clear takeaway that this is good or it's bad. It's just clear that they are thinking a lot about what we, uh, what we are wanting.

• 33:19Nathan Labenz?

  Whoa, you know what that was that just popped? Something just exploded two inches away from me.

• 33:26Nathan Labenz?

  That was really weird. Uh, okay. Next one, quickly accidentally grading the chain of thought. This is like, you know, again, kind of a good news, bad news sort of situation. Bad news is despite wanting to not train on chain of thought. OpenAI and Anthropic also has done a similar thing and they both owned up to it to their credit. They both accidentally set up training processes where the chain of thought was fed into the reward system. And so there was in fact, training which put pressure on chain of thought. This is, you know, thought to be dangerous because if you have a disconnect between

• 34:08Nathan Labenz?

  the, what you really want and the signal that you are rewarding the AI for, then you can get into bad places. And so the obfuscated reward hacking paper that I think is still one of the most important papers of the last few years from OpenAI showed that if you have a hackable reward signal and your model learns to hack that, you can then put pressure on the chain of thought. And initially you will both get that bad behavior to go down and you'll see that, you know, it's no longer reasoning about these things. But as long as that original reward signal remains hackable, if you do that long enough, the bad behavior comes back because it still is being rewarded. But now you don't even see that reasoning in the chain of thought anymore because you've

• 34:54Nathan Labenz?

  essentially pressed that down into the invisible level of the weights where it's no longer, you know, coming out in the token stream. So they've shown that you can get yourself into a really bad spot, obfuscated reward hacking, where the model is hacking you, but you've suppressed the, uh, the identifiable signal of that. And here they, and I do think this goes to show just how fast everybody's moving and you know, you could certainly wish for like more care on some of these things.

• 35:27Nathan Labenz?

  They did it by mistake. Not a huge portion of the data, but you know, kind of low, single digits for different models. It varies, were trained this way. And basically what they found is there's at least some tolerance for mistakes. You know, this did not create a very bad result in the models that were trained this way. So that's sort of good. You know, it's, it's one example where we might think physics is sort of being kind to us. Like it's not, if you just do a little bit of it, you, you know, you don't poison the whole well. I would say there's still some caveats there, you know? Do we really know that there's no issue? No, we just know that this investigation, you know, didn't find flagrant issues. I also do worry a little bit that it will

• 36:11Nathan Labenz?

  lead, lead people to be a little more careless than they otherwise would be, right? This was supposed to be a strong taboo. We violated it. Now we're like, oh, well, maybe it wasn't so bad that we violated it. You know, what's that going to do for the power of the taboo in the future? And then what's the solution to this? The solution is we've got new automated systems, more monitoring, right? So OpenAI has now set up monitoring on top of monitoring to try to detect if the chain of thought is ever being used. And this is really emblematic of like their strategy for everything. Just if we have a problem, throw an AI monitor on it and you know, hopefully it'll catch it. And then we can go back to, you know, pushing toward recursive self-improvement as fast as possible.

• 36:56Nathan Labenz?

  I'll do one second on the fourth one and then we'll skip the fifth one and we'll, we'll get to Matthew because he's here. This natural language auto encoders thing I think is really exciting. If you're worried that your model is

• 37:10Nathan Labenz?

  thinking thoughts that it's not expressing in tokens and that those thoughts might be problematic, then one way you might try to get at that is do some sort of internal monitoring. Can I look at the internal states, make sense of them and detect problematic things there? There's been a lot of strategies that try to do that. Like everything else, they sort of work, they don't fully work. But a challenge is interpreting the internal states, obviously.

• 37:36Nathan Labenz?

  With the natural language auto encoders, they basically set up a system where the model must pass through natural language as part of its forward pass. And using a reconstruction loss, which basically means like, you know, the model has to both kick out to natural language and then get back from natural language and still do its original task in the same way that it was always going to do it. They're now able to kind of get these little short paragraph length things that represent in natural language what the model is thinking at any, you know, given moment in its inference rollout. And then they can look at that and it is much more human readable than certainly like, you know, here's a sparse auto encoder with like, you know, these features lit up and like,

• 38:20Nathan Labenz?

  these features, by the way, you know, were maximized by these other passages and the training data. And so, you know, we kind of squint at it and think this or that. Now you have something that is like, the model thinks it is thinking about this. And they did actually use that in Anthropic to improve some of their monitoring performance. And it's, it's, it's human readable in a way that other things just have not been. And so that I thought was pretty exciting. And this is sort of the next, you know, phase of like, things that we will be able to hopefully layer more and more, you know, layers of monitors on until hopefully we, you know, through kind of Swiss cheese defense, achieve enough safety that we can, you know, trigger the intelligence explosion.

• 39:04Nathan Labenz?

  Which brings me to the moment that crystallized the week for me. OpenAI's four deployed engineers automating tax prep, rolling downhill while almost everyone else is rolling up.

• 39:16Speaker 03

  I think a, a, a good point, I think, to clarify before we dive into this is what is self improving here. So we're not really talking about self improving the, the model itself, but it's mostly the harness around it. And this workflow in particular, I think you got to some of it in your initial comments, Prakash. I think it's a good proving ground for this, where you have very messy inputs, but also a lot of basically practitioner judgment that is also part of this workflow, review workflows. But you have a very good way to measure the outcomes. But what is improving

• 39:58Speaker 03

  is essentially the harness around what the model is basically leveraging in order to produce the preparation, the extractions.

• 40:10Speaker 02

  When you say harness, are you referring to kind of like every time you come to an edge case, the humans kind of help the model to like figure out the hedge case, edge case, and then that becomes part of like a memory of heuristics that you then apply the next time you come across the edge case? Is that what's happening?

• 40:31Speaker 03

  Yeah, so when I talk about the harness is we leverage Codex to do a lot of the work here. But there is basically the set of instructions, skills, the data that you use it in the specific way that you use this. This is part of the, or Tax AI agent. But then when you encounter these edge cases, what we document in the blog post is exactly how you made sure as like a good coworker that if you provide a correction, the next time it can be effective at basically not making the same mistake. So it's around changing

• 41:12Speaker 03

  the structure of what Codex uses, the skills, the durable artifacts, so we can, it cannot make that mistake in the future.

• 41:23Speaker 02

  So when you say skills, is it literally like the skills that other people are making for Codex right now? You use a skill creator and you say like, hey, you know, this is a, you know, 1040 form and, you know, this is what I want you to do with it. And, you know, as you work through it, you're like, okay, this happened, like, you know, fix it for me. And then it documents like the, it in the skill. Is that what's happening?

• 41:46Speaker 04

  Yeah, so it's it's like it's the same skills that you you and I know from using classic codex. Well, what's interesting here is that so there are these skills that are available. And over time, what we sometimes notice is that the models got better as well. And so when you know what used to be a skill, maybe two or three months ago. Today, you know, is potentially we should depergrade it because the model is able to do what is in the skill by itself. And so this is also something that's, you know, that's very interesting that we observed is that the skills themselves did change. And like part of the improvement is also relating the the the harness, the ability to propose

• 42:27Speaker 04

  your skills potentially to also like to build all the content that's available for the next loop afterwards.

• 42:34Nathan Labenz?

  I think the really interesting, my friend Daniel Miesler, who created personal AI infrastructure, as far as I know, coined the term bitter lesson engineering, which and I also think Logan Kilpatrick recently spoke to it. He said, as so many have said, the model eats the harness. So what you're setting up here is basically a sort of tick tock kind of back and forth where with a new model, there's this opportunity for it to clear out all of these heuristics that it accumulated previously. Because now the model might just be able to do those things. And so we want to kind of clean house, tidy up, get rid of all these potentially distracting things and let the model

• 43:18Nathan Labenz?

  excel where it excels, but then you'll probably start to accumulate another layer of heuristics and and that process works in tandem with model upgrades so that

• 43:31Nathan Labenz?

  we he'll climb all the way to full tax automation. It's powerful enough that no less than the Pope felt he had to weigh in. And sitting a few seats down from the Pope when the encyclical dropped, the Anthropic team.

• 43:45Speaker 05

  Sure, I'm not in the Vatican. I'm in Rome, though, at the Pontifical Goring University, which is where my office is at. Just to set the record straight on that one. It was a it was a cool experience. I mean, I mean, you know, it felt historic. It was pretty wild. I remember at one point, a bunch of young people, young people, walked in. One of them had blue hair. And I remember all of us were kind of like, all right, who's that crew? Like, what the castery do these people belong to, right? The Vatican? And then it turned out that was the Anthropic, Anthropic team, right? Okay, okay, that makes sense. But it was was cool is that, you know, I think Chris got all the headlines, but

• 44:26Speaker 05

  Amanda was there as well, which was neat. She's, you know, she sat and listened very attentively. And and everyone was kind of enthralled. Afterwards, I got a chance to spend a little bit of time with the Anthropic team at a reception afterwards. And I think Chris was generally moved to be there. So it was cool. It was it was cool. And I mean, cyclical, I was really impressed with the cyclical. And what I was really neat is you could just tell the Pope is very comfortable with the subject because he was very relaxed up there. He was stage managing to some extent, which is very, very unusual to see him do that. So and just even for me, I've been working with the Vatican, you know, for 10 years now, hearing, like, seeing the guy there, and then he opens his mouth to speak, and he's got this American accent, it just doesn't doesn't compute.

• 45:10Speaker 02

  He's also a huge Chicago Cubs fan.

• 45:13Speaker 05

  Indeed.

• 45:16Nathan Labenz?

  So, yeah. I mean, there's so many big picture questions here. I think, you know, we're in the AI obsessive bubble. And in my circles, I think the level of expectation or hope for this encyclical was like extremely high, especially among sort of AI safety oriented folks who were kind of like, we need a moral authority to help, you know, crack the political class.

• 45:49Nathan Labenz?

  And I think there was a sort of, at least among some people, a certain sense of disappointment that only happens when you have maybe become overly excited about what, you know, a new ally, just how aligned you might be with a new ally, only to then find that you're not quite as aligned as you, you know, maybe let yourself get carried away to think. And the frontier of kind of divergence there, which I don't want to overemphasize, was around this one paragraph that was sort of saying essentially that the AI cognition isn't real, or, you know, it doesn't really think, it can't really have responsibility, it can't really, you know, all these sort of

• 46:32Nathan Labenz?

  really things, which of course calls to mind my joke of like, you know, it's not really reasoning unless it's from the reasoning region of the human brain, if, you know, for many different things that people have said AI can't really do. How much do you think that matters? I also did note that there was another speaker, not the Pope himself, but another high-ranking official who did say that, you know, these questions of AI subjective experience, or, you know, potentially even like moral patienthood, deserve further study. So, yeah, I don't know. How do you, how do you make sense of that sort of thing? And, and how much is at stake with this sort of really think question?

• 47:14Speaker 05

  Yeah, I don't know. I mean, Cardinal Charny, he did reflect on conscientiousness and then conscientiousness or something like that, which was fascinating. But, I mean, listen, I mean, we all knew where the Pope was going to line up on this question of consciousness, and we all kind of know where, well, at least where Anthropic would be signaling, right? And there obviously was a bit of a divergence there. But, but I think it was a healthy, I mean, healthy divergence. I'm glad Anthropic was there to signal that, because frankly, it makes it easier for us to corral some people together to actually study this question of consciousness more seriously, because it's, it's a bit disturbing to me, actually, that we have a hard time, as a tradition, defining consciousness in a clear way, which is weird, right? We should be, it feels like we should be more capable of defining consciousness

• 47:59Speaker 05

  in real, concrete terms. But when you start talking about, like, let's, how do we test for it? No, it's not clear. Ever since we blew past the Turing test, we're, we're kind of stuck. And and this is cool because at Builders AI Forum, we've had to actually spin up a working group with some of some of the most notable people in the field to actually study this question of consciousness to define it. So eventually we can come up with basically more interesting testing methodologies, which hopefully will be helpful in this whole conversation. But I mean, the big thing is just to remember, like, when it comes to like reasoning and these kind of words, consciousness, this is always going to come back to the fact that there's a soul and and whether whether consciousness is present properly to the soul, I don't know if that's entirely, but the church would feel like thinking there's some there's something beyond the body that's going on that's involved

• 48:42Speaker 05

  in thinking and reasoning. So I know that like for a lot of the people out there, I mean, reasoning is just, you know, persistent memory, right, world model, reasoning and hierarchical planning, but there's there's a lot more going on from like the church's understanding of that. I mean, this is why like things distinction between intelligence and sentience is really important, right? Certainly, like, if we're talking about sentient AI, this would be a conversation that church would definitely have a much stronger opinion on. But whereas if you're talking about intelligence from the way the industry defines it, the church is not going to have too big of an issue with that because that's the four things I just mentioned. I mean, I think everyone would grant we're going to get there. So if that's how you're measuring intelligence, yeah, there'll be there'll be intelligent

• 49:25Speaker 05

  AI. But consciousness and sentience is the thing altogether, obviously.

• 49:30Nathan Labenz?

  Back to the science for a moment. Day four brought a counterweight. Peter Jansen from the Allen Institute has actually tried to run the AI scientist play at scale, and his results are a useful cold shower.

• 49:44Speaker 06

  Oh, it surprises me and doesn't surprise me all at the same time. Some days I wake up and I feel like I'm living in the future and other days I wake up and I feel like I'm, you know, living in this sort of strange reality with all these agents that can't do the things that I want them to do. I'll give an example that's really grounded in in AI and scientific discovery. So we have this project Code Scientist, which looks very similar to a lot of the projects that sort of you pull up on Twitter every day, where people say, you know, I made this AI agent and which is like a thin wrapper on some OpenAI or Claude model or whatever. And it generates code automatically and it generates ideas automatically and runs in a loop and away they go, writes papers.

• 50:29Speaker 06

  And so we gave it 50 research ideas and let it chug away for a couple of days. And after a few days, it came back and it said, well, I've discovered 19 new things. And we were very excited, right? Wow, 19 new things. We live in the future. Life is great. All that jazz. And so I wrote papers on those 19 new things. And we gave those 19 papers to three colleagues at AI2 who hadn't seen it before. And said, tell me if this is a real discovery, you know? Look through these papers, see what you go. And they went through them. And I think it was 70 or 80% of the papers. I said, yeah, it's probably at least incrementally novel and minimally scientifically sound and whatnot.

• 51:13Speaker 06

  And then we convinced somebody, me, to go through and spend days and days and days looking at the thousands upon thousands of lines of code that these models were generating to support their discoveries. And it went down to like 30% of the discoveries were probably real. And the things that you see are absolutely all over the place. One fun example is the AI came up with some fancy idea for making a new neural network architecture with some fancy new kind of attention. I don't know. It wrote hundreds of lines of Python code with all this neural network code that I have absolutely no idea what it was doing or can't understand any of it. And so I'm going through it.

• 51:58Speaker 06

  I'm like, how on earth am I going to review this? This is in my domain area. And then I get to like the end of a couple hundred lines of code. And there's just this comment that says, comment, insert rest of neural network code here. And then it picked a random number and returned a random number from that function. And so this model, this paper, this entire paper was analyzing the values of a random number generator. And that, you know, isn't shown to the, you know, nobody knows that if you're reading the paper. But the science itself, you know, it's hard to evaluate it. It's hard to be sound. And so a lot of this, you know, when you see it do something amazing, it's easy to be very impressed.

• 52:42Speaker 06

  But then when you use a standard benchmark, like we have Science World and Discovery World, these sort of virtual environment benchmarks. Science World does fourth grade science. Discovery World does sort of like a master's or PhD level science. The best models right now are getting something like 80% on the fourth grade science. So you ask them to go in this environment and boil water and they can't do it 20% of the time. Right? That's wild. Or you ask them to go in and give them a toy task. You know, the colonists on Planet X are getting sick. Figure out why and solve it. They're really terrible at that, right? They can't solve most of those. Whereas you give those real human scientists and they're, you get most of them.

• 53:25Speaker 06

  So, so the summary of that is it's really easy to be excited when they work well. But you gotta pay attention to all the really simple ways that they break before you get too excited. I think that's not to say they don't have utility. And there's lots of places that they do have lots of very near term utility. But I think my job is safe for a little.

• 53:51Nathan Labenz?

  Now, limits like that do cap how far you can lean on these systems today. But my basic read is pretty simple. They can now do a great many things, super reliably, that they used to be terrible at. And even a 30% real discovery rate is hard to understand as anything but the beginning of a science fiction future. The clearest stakes this week were in security. The best guest we had breaks into companies for a living. And his read on where AI actually bites surprised me.

• 54:22Speaker 07

  Yeah, so I was really lucky when I was at Department of Defense to be involved with Project Maven early in the day. So Maven was the AI Warfare Task Force or project, applying the AI to combat. This is 2018. So that exposure to basically early DeepMind, what became OpenAI, so on and so forth. And one of the things we learned pretty early is that the models themselves are disposable. They're changing so often, you're just gonna throw them away every six months, every nine months. The two parts of the stack that are truly durable are the harness and the training data. And those are the things that you need to get right. So why is the harness matter? The harness is the difference between being production safe and not safe.

• 55:06Speaker 07

  Right? So number one. And then the training data is super important because in cyber in particular, attackers live in, sorry, attackers live in the edge cases and LLMs live in the mean. And you've got to really, really take that into account. So why is that interesting? Mythos. So think of the training data that these, the frontier labs have access to. Anything regarding software, like actual code analysis, the labs are gonna kick everyone's, like they're just gonna crush everybody. Because the cost of training data acquisition is basically $0. The three of us could go start an AI company right now, go start a web app pen testing company, and build agents trained on every Git project, every Linux Foundation project, and every merge request. There is no barrier to entry for training data.

• 55:52Speaker 07

  Which is why anything source code analysis related is gonna be a huge advantage for the frontier labs. So what does that mean for cyber? The effort to find, to do vulnerability research is basically going to zero. And so that's why we're seeing tons of code flaws being exposed. Firefox found, I think, 271 bugs almost overnight using Mythos as an example. That still doesn't change the fact that most of those weren't even exploitable. Like, already found flaws, cool, but it weren't even exploitable in your department. But where these models are actually struggling, if you double click on the data for Methos, it actually regressed compared to 4.6 in runtime exploitation.

• 56:33Speaker 07

  The reason why is, last I checked, JP Morgan didn't publish their network configurations online anywhere, or their active directory configs, or their data security configs. All of the most valuable data in cyber is behind the firewall. All of those configurations, all of the edge cases that are there, the labs have no access to it. So what we're actually seeing is this bifurcation between source code analysis, which is really great, and then actual runtime capability. Number two is, because these models were trained on extremely limited training data, and the analogy is that Maven, we were really worried

• 57:13Speaker 07

  that the adversary would corrupt our training data to make an aircraft carrier group look like a flock of birds.

• 57:20Nathan Labenz?

  Now, in fairness, there's a strong case against where I'm about to land, and a team called Enclave makes it well. Hear them out.

• 57:29Speaker 08

  Yeah, so my take is that we sorely need to depend on, well, harnesses and actual human knowledge is much more important than the models.

• 57:44Speaker 08

  CyberGym, like the most famous cyber evil, the top score right now is by the Microsoft multimodal setup. They used Opus with Sonnet, a GPT 5.4, and they got a score that is higher than Methos. So what we see is that cheaper models can outperform more expensive or smarter models if you just optimize, you know, the knowledge or...

• 58:15Speaker 08

  Harness around it. So I think there's a lot of place for humans with real expert knowledge. Let's remember that how to read software is not like a really documented process. It lives in the minds of humans who have been doing this for years. Like, just like lawyers do, our agents today, there needs to be somebody sitting there looking at the results and kind of having taste like what is good and what isn't. Like at the end of the day, there's somebody behind all of those systems that is has to make the judgment call if the quality is up to standard or not.

• 58:53Speaker 09

  And they will be accountable if something goes wrong, right? You cannot fire an AI to someone to blame at the end of the day.

• 59:01Nathan Labenz?

  It's a genuinely good argument. And yet, here's where I come down. When it's security critical, I think people will still pay up for the very best model. A company running thin margins on top of Opus is going to struggle to say, no, don't use that. Use us. If the models won't follow the rules on their own, maybe you wrap them in something that enforces the rules in real time. Prakash was especially taken with Brett Levinson's pitch for exactly that. And with his answer to who the real regulators turn out to be.

• 59:34Nathan Labenz?

  I would love to dig into the architecture a little bit and then maybe also talk about how this paradigm may extend to things potentially well beyond content policies. On the first point of architecture, I mean, it's got to be fast, right? So like, are you using small models? Is this, you know, is this the sort of thing where you sort of let things through and then run something in the background? And if it gets flagged, then we kind of come in later, like the original Microsoft being experience where you'd see the message and then it would like retract it back? Or are you doing the more sort of classifier style

• 1:00:17Nathan Labenz?

  approach where it can be fast enough that you can build it into the stack and the latency is acceptable? What what trade offs are people willing to make in terms of product experience, latency, cost? And how are you then engineering to meet their demands?

• 1:00:36Speaker 10

  Yeah, so I mean, to me, you've said the magic words like I've been a big advocate kind of since we started the company and even since I was at Meta that like, you know, an ounce of prevention is worth a pound of cure. Like being there before something happens, or as you pointed out, you know, maybe you can optimistically let a message through and then retract it quickly. It's just a better approach than finding stuff three to seven days later and, you know, saying, oh, we screwed up, we need to block or ban this user. And in the case of AI, what would you even do three to seven days later, other than maybe like, I don't know, add it as a training example for the next fine tune or something like that?

• 1:01:16Speaker 10

  As far as the architecture goes, so we have a couple techniques that we're using. So one, yes, we do use some very small models that are already pretty fast. It also turns out that breaking a policy down in the way we do into atomized bytes gives us some unique advantages on the, on sort of the latency front. The questions we're asking are all pretty small. They tend to share a prefix, basically. And so we're able to, to sort of benefit from quite a large amount of prefix caching.

• 1:01:44Speaker 10

  We also, generally speaking, at least first pass, we're not generating much. There's really no decode step for us. What we, I mean, I'm happy to, I guess, share some of the architectural details, like, we essentially are training a binary classification head onto an LLM, right? We don't initially, anyway, need some, we don't need the questions answered with an actual yes or no. And in fact, it's counter to our objectives to do so. We actually want to know, what is the probability that the answer to this question is yes, basically? And that, I don't want to get, I don't want to, I have a tendency to sometimes go on

• 1:02:27Speaker 10

  tangents. So I'm going to try to contain myself here. Maybe we can come back to like the benefits of having those probabilities and the abstain gap and all that.

• 1:02:37Speaker 10

  There's another common thing in moderation, safety, guardrails, control, whatever you want to call it, which is that for most, for the majority of policies, upwards of 90% of all the content you're ever going to see. Fine, like it's a real needle in a haystack problem, right? Like you're looking for a small sliver. The only problem is that very often that sliver has high severity, has real risk associated with it.

• 1:03:06Speaker 10

  And so we have a number of layers sort of in front, you mentioned lightweight classifiers. They're not simple binary classifiers, but we do have a number of much lighter weight models that sit in front of our, I guess what I would call like our main QA engine.

• 1:03:24Speaker 10

  That can give us with reasonable confidence and high recall, that's the important part,

• 1:03:31Speaker 10

  a quick answer upfront. And so the idea is like for, let's say, let's say just for argument's sake, let's say 90% of what we're going to get sent from a particular customer is fine. Really, there's no problem. We don't need to look at it for real, basically.

• 1:03:47Speaker 10

  Ideally, we want to try to take, let's say, half of that and filter it out right away and just approve it, basically. And if we can do that, then on average,

• 1:03:58Speaker 10

  the latency that we're offering the customer, I mean, for those cases, we're going to be sub 200 milliseconds, basically. And then because our models are pretty damn fast, like for the rest of the cases, we're sort of in the 300 to 500 millisecond range, when we actually have to do a deeper, a deeper scan. I will say it also varies a lot by modality and there are, there are aspects there that are just hard to get around. Like text is very, very fast, those sort of the numbers I just quoted.

• 1:04:25Speaker 10

  Images are a little bit slower, we have to run a vision encoder. Like there's, there's more steps, we have to very often resize the image. We have to potentially transform the format of it before we process it. So there's just built in latency. Video, even more latency, because we first have to sort of pull the video from wherever it is, it could be very large, you know, et cetera. Rip out the audio, transcribe it, like there's all these extra steps that we have to deal with. And to answer sort of that last question, like what is the, what is the use case tolerance? I, I do think it depends a lot on use case. For some of our, for example, AI image gen customers, right? It's already taking six to 10 seconds to generate an image.

• 1:05:08Speaker 10

  So, you know, adding maybe 10% latency on top of that, because it takes us 1500 milliseconds to render a verdict, like. It's not that big a deal. You know, it's not ideal, maybe, but it's not noticeable to the user. And in my view, actually, that's what a lot of the tolerance is going to come down to. Like, how does it affect the user experience? Is it noticeable to the user? I just wrote this whole Active Guardrails piece kind of about this. It's where sort of our future focus is, is on essentially being able to do what we do and do it on streaming tokens effectively, so that we can kind of just be like the old days of TV

• 1:05:53Speaker 10

  and kind of just like run the conversation on a five-second delay and, you know, bleep out anything that's bad, basically. Because I just, what we see is that if we're asking too much from our customers of, asking something of our customers that is going to significantly impact the user experience, then they are less likely to adopt the controls that they ultimately need. That's my feeling on it, honestly.

• 1:06:26Nathan Labenz?

  Which raises the question a Spanish team has been answering all year. What comes after agents? But it's striking to me that you said you don't even have workflows as kind of a mental model. That seems to be at odds, at least with this Anthropic launch. So would you critique their launch? You think something's off about that mental model? Should I go like revert my upgrade skill migration to the workflows paradigm? Do you really disagree with that direction?

• 1:06:59Speaker 09

  I'm happy also to hear a man on the side. I tell you my most production, like I tell you the product vision and we can also go deep on the workflows topic. By the way, I don't critique it. It's simply that the day that you start paying the tokens at that cost, you might not want to click that workflow button twice. So I tell you what happens. It's a mental model problem. Because if you think about workflows, you are constraining your thinking into a process. And the problem of that is that the business users that know how to make the task cannot translate their task into workflows. Because indeed, there is so much variability, there are no happy paths in knowledge job tasks. There are no happy paths.

• 1:07:42Speaker 09

  A happy path of I read a document and I put it in a database is not a happy path because the document can be in Spanish, in Chinese, Colombian, because with a passport. Now, the passport is not anymore like... Imagine if you had to think on a workflow to validate two documents. You would say, no, it's easy. You read the document, you create a G, so on, and then... But if you really want to make a workflow, which is what people think, you are constraining by yourself the capabilities of what these systems can do. So instead of talking about workflows, which is a process, boxes with arrows, with a lot of if then else's and some now magical boxes called AI agents that become black boxes that we're not sure if they will do the same thing twice,

• 1:08:23Speaker 09

  but we put them there as routers or as intelligent conditionals, no? And then, we think more into the framework of delegation. Suppose that you want to automate, or not automate, you want to stop managing your calendar. You have two ways. You can create a workflow, so good luck managing all that, or you can hire a human today, no? And you would delegate into that human that. And delegation means that you expect that person to keep learning. You expect that person to be able to do new circumstances because he knows already how to behave from a general perspective.

• 1:09:06Speaker 09

  And you wouldn't have to say, every day you open the email and you click the button read, and you do this. And then you put the label, you don't... That's a workflow. But this is not how we think. So the problem of workflow thinking is that it constrains what this technology can really do. Once you solve the problems of course, reliability, and reproducibility and hallucinations. While you solve that, the problem is that people are still constrained with chatbots and with this if-then-else thinking that constrains the... No, no, now you really can do this thing end to end. So that's why we challenge, not challenge, we literally ask, if you ask our customers, they will never ever say the word workflow, which I think that is one of the biggest things that we were able to achieve, to scale this.

• 1:09:51Nathan Labenz?

  Quick heads up on these last two. Our guest's audio wasn't coming through for a stretch. Prakash had Kodex pass the studio live without dropping the stream. So for parts of these, you'll hear me on a speakerphone backup. Both are developments I genuinely want more of. And they're both deeply human.

• 1:10:10Nathan Labenz?

  First, the solo creative with a company in a box.

• 1:10:14Speaker 11

  Look, I think today, and that customer segment is very unique, but very large. It is a pro server segment, dark consumers. So they're left between two horrible choices. There's only one out of four hours on Chinese administration. And with AI rising, remember AI is diminishing them as well. We have a customer that went from 200,000 in AR to 700,000 in six months using AI. Right? And they can get to a million. I'm seeing more and more of this. So there's this whole billion dollar business of one discussion that's been occurring. And I like, I have a number that we sent first. Regardless, those bad guys, it's almost irrelevant.

• 1:10:55Speaker 11

  What do you, really soon, you're going to start seeing 30 million, 40 million dollar business of three. Now, those folks don't want to hire a finance department. If you can, what's your end result when you hire? You don't want to hire a controller. You don't want to hire an accountant, right? And so they basically are going to use as a system director. Now, our solution, the other solution, the alternative is that they can drive outcomes with accounting and tax firms, right? But those accounting tax firms don't require you to go buy QuickBooks and all this stuff, right? You, you, you, you, that's obscured, obviously, for me. It's the same as us. From the user perspective, from all intents and purposes, we, we serve as an accounting firm, right? It don't have to bring anything to us. We will handle it.

• 1:11:39Speaker 11

  I mean, there are things they have to do because they are accountable party, bringing the business, right? I cannot unfortunately, you know, go and administer like certain things on bank functions and whatnot, but you know, it's quite, I'm a member, by the way. I use, I'm sold on your partner X, but I use the platform and it is, and I was third at the cost center, I was quoted from my account. To use the same, to drive the same outcomes with a software-driven solution. To me, it's an inevitability, but the space is heading towards like a really, really strong disruption. There's 50,000 small practice accountants right now, who's to bring 30 million people. Like, so in a couple of years, this is, this is really hitting it. Cause I'm, I'm telling you today, not in the future. We've already, we finished it. We're done. And, you know, it's a kind of like the person that brings the torn me a mile.

• 1:12:23Speaker 11

  I think once it's been done and the word gets out, it's gonna happen again and again and again.

• 1:12:27Nathan Labenz?

  And the second, mental health support made dramatically more accessible. So tell me more about the

• 1:12:37Nathan Labenz?

  architecture, the safeguards. I mean, I, we know the basics in terms of like content filtering, you know, classifiers to, to raise alarm bells when needed. But I guess my, I haven't really used chatbots much for any sort of therapeutic purpose, but my naive sense would be that, you know, they probably do a pretty good job doing cognitive behavioral therapy out of the box. And that, you know, I think a lot of people are using them for that. Where do you find they fall short? And, and tell me more about what you've built that isn't immediately visible to the user to improve on those weaknesses.

• 1:13:21Speaker 12

  It will have time to validate it to you most important part of what you do.

• 1:13:30Speaker 12

  Like obviously like a lot of companies would want to build content for their patients, for their customers using API like OpenAI API or I forget API. But they simply cannot because they do not have an in-house expertise to see those model boards working. The way it would want them to behave and then it's like this regulatory risk that you would have to address there. I hate risk, of course. I mean, yeah, I think a need to fill that kind of like fact about us is that we spend around 70% of stock rates on safety. It's right, but it's like very, very expensive. Um, I generally be on the technical level.

• 1:14:13Speaker 12

  We have a bunch of classifiers that are constantly running on the background. Um, so it's in a young time, but ultimately background, especially when you need to think longer about something is good or about today.

• 1:14:28Speaker 12

  Um, having that really, I'm glad that it's a humorous concept because it's about your different tools. And.

• 1:14:37Speaker 12

  For example, if you want to pray, to benefit, so I had or to um, care about what happened in the Patrick, right? Like, I mean, from the child perspective or from the agent perspective, what happened in the last common day and the conversation with this user and how this could inform in or plan and I know our actions going forward. So the agent is able to deploy tools or maybe subagents that do stuff like that and then and if it badly information into the main context, sometimes it continues. So under a lot of that, you have this like um, like powerful memory and the powerful planning capabilities. Uh, that and that.

• 1:15:22Speaker 12

  And like the path benefit, it also the chances that it will mean something they even cotton for the user safety, they are dramatically lowered than that. And then without this attitude.

• 1:15:37Nathan Labenz?

  How are people accessing this? I guess maybe one more, just to wrap us up. Um, are there any anecdotes from your deployments in Ukraine or in US prison populations that you think are particularly memorable or inspiring that you would leave people with? Or failing that, just anything else you would wanna uh, leave people with in terms of a positive vision of the future?

• 1:16:04Speaker 12

  I mean, be

• 1:16:08Speaker 12

  there are many, there are many parts like that. Um, like in the correctional environment, the hard numbers tires for medical teams there that our AI helped them identify people who they did not know were suicidal and because of that they were able to um, provide that recount and to perhaps maybe perhaps save.

• 1:16:35Speaker 12

  Uh, they have their lives. Um, Ukraine also like interesting because we like we if others thought that working in Ukraine, they thought of our our trajectory because that part. And then until you read to the first part around, you know, and you get exposed to what they create and how their lives look, right? And I returned to the car, walked with the badlands and with their soldiers. It um saved a lot of what could do in safety because if you had been under those sixteen conditions, and it worked rather than course and all right, they work for no. They come then in your head, you're going to compete, well, no consumers.

• 1:17:16Speaker 12

  Um, until I think I think that more and more people realize that they not have to be left lower and that they can use AI. Um, I mean, um, that's a good thing, you know?

• 1:17:36Nathan Labenz?

  So that's the bet. We're in some kind of takeoff and almost everything around the core intelligence needs a real moat, or it needs to be human enough that the frontier leaves it alone. As for the show, it's an experiment and we're holding it loosely. I don't expect many people to watch two hours live every day, but maybe a cut like this one can be genuinely worth your time, even if you're as plugged in as I am. So tell us, what earned its place? What should be of cut? What did we miss? Reach out, seriously. We read everything and the show will get better because of it. Thanks for watching.

• 1:18:35Speaker 13

  Now we see through the glass, talking.

• 1:18:38Speaker 14

  We raise the values.

• 1:18:40Speaker 13

  Soon face to face.

• 1:18:43Speaker 14

  We make the mountains low.

• 1:18:46Speaker 13

  What we will be has not appeared.

• 1:18:48Speaker 14

  We straighten the road.

• 1:18:50Speaker 13

  But we live toward the day.

• 1:18:53Speaker 14

  We smooth every stone. At first light. Soon to be shone. We rise. What is hidden? To prepare. The veil grows in the way. Face to face. At first light. We are waiting. We rise. Something is coming. To prepare the way.

• 1:19:16Speaker 13

  The long night ends, the long night ends.

• 1:19:21Speaker 14

  We trim our lamps. Salvation waits for its hour. And keep them burning. Through its tarry ways. We know not the day.

• 1:19:31Speaker 13

  It will not come late.

• 1:19:32Speaker 14

  We know not the hour. It is surely coming. The work of our hand. Is frozen in the dark. Is working alone. We know not how.

• 1:19:43Speaker 13

  The whole creation grows.

• 1:19:47Speaker 14

  Longing to be made new. We kneel to what we cannot name. And love what we fear. And ache for what will heal. At first light.

• 1:20:00Speaker 13

  What is hidden?

• 1:20:01Speaker 14

  We rise.

• 1:20:02Speaker 13

  Soon to be shone.

• 1:20:04Speaker 14

  To prepare.

• 1:20:05Speaker 13

  It is coming.

• 1:20:06Speaker 14

  The way.

• 1:20:07Speaker 13

  And every tear.

• 1:20:08Speaker 14

  In fear and in wonder we wait. Make ready the way. Make ready the way. In wonder we wait.

• 1:20:17Speaker 14

  We rise, we rise, to prepare. To prepare. The way. The way. It is coming.

• 1:20:24Speaker 13

  It is coming.

• 1:20:25Speaker 14

  It is coming. No more mourning. Every tear. Wiped away. Wiped away. Let it be done. Let there be light.

• 1:20:35Speaker 13

  Let it be done.

• 1:20:36Speaker 14

  Let there be light.

• 1:20:48Speaker 13

  Morning by morning, we open the door. Let what's is coming find us in song. Not our will only. Let it be done. Our work is our prayer. And the dawn is our home.

• 1:21:23Nathan Labenz?

  If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at ai podcast dot I N G.

• 1:22:07Nathan Labenz?

  And thank you to everyone who listens for being part of the Cognitive Revolution.