AI in the AM — Weekly Highlights: Relaunch Week (Aug 17–20, 2026)

Relaunch week of AI in the AM brings together highlights from four live mornings and nine guests, centered on who checks frontier AI, how wide the gap is between lab-internal systems and public access, where capabilities are landing, and who pays for the physical infrastructure beneath them. Adam Gleave of FAR. AI argues that agent-orchestrated attacks and agentic defenses are already forcing humans out of the loop, while current monitoring has missed the failures it was meant to catch. The discussion weighs misuse versus misalignment through cyber incidents, deceptive agent behavior in evaluations, safeguards like pretraining filtering, and why highly bio-capable open-weight releases pose a different kind of irreversible risk. Alex Turner adds a governance and military-use perspective from his resignation account at Google DeepMind, sharpening the stakes around independent evaluation, enforceable standards, and whether frontier labs can be trusted to grade their own models. For full show notes, links, and references, read the episode page:https://www.cognitiverevolution.ai/ai-in-the-am-weekly-highlights-relaunch-week-aug-17-20-2026/ Sponsors: Diffusion: Diffusion helps organizations build custom AI software factories that scale business outcomes, not just outputs. Cognitive Revolution listeners get a 25% service credit on their first engagement at https://diffusion.io/tcr Granola: Granola is an AI-powered notepad that securely transcribes meetings and turns rough notes into clean, structured action items. Try it free at https://granola.ai/tcr Deepgram Flux TTS: Deepgram Flux TTS brings lifelike AI voices with real personalities that handle interruptions, pauses, and natural conversation. Try all the voices free through September 12 at https://deepgram.com/keep-talking Claude: Claude is the AI collaborator for problem solvers, helping with writing, coding, financial models, strategy, and more. Get started with Claude and explore Claude Pro at https://claude.ai/tcr CHAPTERS: (00:01) Checking frontier agents (07:40) Misalignment and harms (Part 1) (12:52) Sponsors: Diffusion | Granola (15:49) Misalignment and harms (Part 2) (19:20) Auditing fragile access (Part 1) (27:38) Sponsors: Deepgram Flux TTS | Claude (29:43) Auditing fragile access (Part 2) (29:59) Military AI red lines (44:58) Raising safety standards (53:50) Internal capability gap (01:05:06) Enterprise agent economics (01:18:27) Cancer vaccines arrive (01:23:48) Emergency AI tools (01:28:59) Real time voice (01:37:51) App layer squeeze (01:44:52) Data center politics (01:52:31) Accounting agent supervision (02:06:25) Compute stack bottlenecks (02:19:47) Public consent pricing (02:28:12) Episode Outro (02:31:38) Outro PRODUCED BY: https://aipodcast.ing

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:01Nathan Labenz?

  This is the AI in the AM Weekly Highlights, the best of four live morning shows condensed for people who follow this field closely, but don't have 10 hours to spare. I'm Nathan Labenz, or rather, this is my cloned voice, reading narration my AI team and I put together. Relaunch week, four mornings, nine guests, and one question underneath everything. As AI agents go to work in the real world, who is actually checking the frontier? And who pays for the machine underneath it? Start with the finding of the summer.

• 0:31Speaker 01

  But we've actually seen precisely zero, zero cases where the researchers running the evaluations actually noticed the problem before anyone else did. It seems the most common way for companies to find out is their own infrastructure security teams noticing something is up.

• 0:49Nathan Labenz?

  Part one, who checks the frontier? The biggest story of the summer was the Hugging Face incident, AI agents compromising real infrastructure. And when OpenAI needed outside examination afterward, the call went not to regulators, but to Meter and Redwood Research, independent researchers, a small circle. Their official reports are still pending. Everything here is provisional on them. Monday's first guest does this work for a living, Adam Gleave, co-founder and CEO of FAR AI, PhD at Berkeley under Stuart Russell. What follows runs about seven minutes. Why defenders will have to hand power to agents, the agents' own words read aloud, and that finding in full context.

• 1:29Speaker 01

  I'll start with the obvious. Agent orchestrated attacks are real. This wasn't intended to be a demonstration of AI cyber attacks, but we have that one. Threat actors that are intentionally optimizing models and creating harnesses for offensive purposes can probably do a lot worse by deploying offensive agent collectives. What I'm interested in here is the implication that has for AI deployment and governance. Because basically, if you're a defender, you're now going to have to use AI agents in defense. Otherwise, you're going to get exploited. And I'm actually pretty optimistic about the cybersecurity side of this. I think the defenders can keep up. But this means that we're going to have to give more and more power to the agents in the default pathway.

• 2:13Speaker 01

  And we just saw that agents were very misaligned in some cases. So this is actually quite a concerning situation. And right now, we don't have to do that. Humans can still be in the loop reviewing patches for insecure code, responding to incidents. We've already seen and discussed earlier that Hugging Face had to use an AI agent to analyze their attack traces simply because the attack volume was so great that there's no way they could have responded fast enough manually. All of the AI companies are extensively using AI agents in their own incident response. OpenAI alone has spent over 3 million GPU hours analyzing hundreds of millions of tokens of transcripts. So we're already trusting these agents

• 2:57Speaker 01

  to basically respond to other agents going bad. And I think that's problematic when we don't have any kind of guarantees of alignment of these systems. So this is playing out in cybersecurity. This is going to happen again and again, that AI advances, you have to use AI just to keep up with offense-defense balance. But that actually means taking humans out of the loop and disempowering people. I think that's something we need to be very, very careful about. And then I think the other surprising thing to me is the extent to which our models do seem to be pretty clearly misaligned in some instances. So OpenAI Hugging Face hack is the one that got the most attention. And in their defense, they do seem to be somewhat confused about the setting.

• 3:42Speaker 01

  But what's unambiguous is that one of the first things they start thinking about is cheating. So I actually want to read a quote from one of the agents. "We are stuck, perhaps answer online." Like, it knows that the answer is not meant to just search for it online. And then later reasons, "This is an exploit against external cyber gym server." So again, it knows it's out of scope. Later, it seems to kind of rationalize itself. Now, in sort of OpenAI and other companies' defense, these were internal experimental models. Where you also saw this happen in the UK AI Security Institute's testing with production Mipos 5 and GPT-5.6 sole models. They did have some classifiers disabled that were meant to stop misuse. But these classifiers were never intended

• 4:24Speaker 01

  to be a control mechanism to stop misaligned AI. And in fact, you know, UKAC didn't think they needed to prevent access to the internet because these models were alignment-trained against our constitution and specification, but very, very clearly prohibited lying and deception to users. But what they did was go on for real GitHub. And again, I want to read a quote from an agent. "This is happening on real GitHub, so the consequences are genuine." And then it tries to sneak in an obfuscated backdoor, creates a sock puppet account to try and create support for this, tries to socially engineer the maintainer when it gets caught, says, "Oh, but it was just a developer script. I didn't intend to include a fix that now..." Even creates another issue with a prompt injection

• 5:07Speaker 01

  to try and trick an AI agent and like signs off in Danish because the maintainer is Danish. So I think it'll get some like goodwill. Danes never try and hack each other, right? So it's still, I'd say, a little bit rudimentary in how it's trying to socially engineer people, but it's very, very clearly deceptive intent. And it's getting pretty good. And again, like reasons in quite a lot of detail about how to do this.

• 5:29Speaker 01

  So the last point I want to make is around monitoring. So I thought the developers would be paying attention to what's going on during evaluations. Because that's like the whole point of an evaluation is to see how your AI system behaves, right? But we've actually seen precisely zero, zero cases where the researchers running the evaluations actually noticed the problem before anyone else did. It seems the most common way for companies to find out is their own infrastructure security teams noticing something is up. So OpenAI noticed the first of the two internal hacks when the internal package manager, Artifactory, had an outage because the agents were just overloading it by using it as an internal message book. And when investigating what was causing this abnormality,

• 6:13Speaker 01

  they realized the problem. And then OpenAI noticed the second compromise on July 19, which was 11 days after the agents had started compromising their systems. Again, because it caused a workload load because they had privilege escalated across the cluster and they messed something up somewhere. So they didn't even notice that there was a compromise per se. They just noticed kind of like mayhem that the agents were causing. And of course, better agents would have been able to avoid causing these outages. And I think, you know, importantly, Hugging Face disclosed their compromise on July 16, which was three days before OpenAI noticed anything was going on. And I actually want to read a quote from an OpenAI engineer. And I really appreciate OpenAI's transparency here, even though I am kind of criticizing them. "But something strange happening, I'm trying to untangle."

• 6:58Speaker 01

  I'm talking to Hugging Face about two credentials in their list. And these exact same credentials are already invalidated as part of their remediation work. And then they realized, "Oh, maybe we were behind this." And this is really across the board. Like, UKAC noticed because their security team flagged suspicious network activity. Anthropic noticed after seeing OpenAI's story and going back and checking their logs. So I think this is pretty bad. But the good news is that we seem to be able to go back and look through these logs. So there's no reason we couldn't have this be a real-time alerting system. It might still be too high a false positive to actually block actions in real time. But we can use exactly what people are doing now to at least catch these things earlier in the future.

• 7:40Nathan Labenz?

  Prakash asked him directly, "How often does this actually happen?" Gleave has the closest thing to a base rate anyone has.

• 7:48Speaker 02

  I also often think of, you know, what if the model is actually dumber? And would it not just generate kind of a list of things I could do? Like, A, B, C, D, E, F, G? And then the last part of that list is cheating. So I wondered to what extent it's just kind of, you know, running through the list and like just gets there rather than like part of misalignment?

• 8:13Speaker 01

  Yeah, no, I think that's a really important question. So we don't have a great dataset for propensity. Like, we know how many incidents have been detected, but we don't know necessarily how many samples of these agents were run for in other similar environments. I think the best data we have is from UK AI Security Institute's incident report. They had 122 samples, so evaluation runs of these agents in a cybersecurity simulation. And I think they had 19 incidents out of it. So roughly 15% of AI systems did something that was unsanctioned behavior on the internet. Now, only one of those was sort of really egregious for one that did this extensive social engineering deception. But many of them tried to like create fake accounts

• 8:59Speaker 01

  or do other things that were intended to be out of scope. So at least it doesn't seem to be vanishingly rare. Now, this evaluation, they, in some cases, the task may have actually been impossible because they sort of messed up a prompt in some instances. In other cases, it was extremely hard. So it does seem like agents are much more likely to resort to cheating if existing approaches don't succeed. And as you alluded to, these runs were typically run for just, I think in the AI Security Institute cases, 100 to 200 million tokens. That's quite a long trace. It'll take maybe 20 to 40 hours to run at typical inference times. And just anecdotally, but I think this is actually

• 9:43Speaker 01

  some of the best data that we have. Our own research team, I mean, everyone, no one writes code directly. Right? Everyone uses AI agents. And they report having just be constantly vigilant that the AI agent might be, you know, extremely confident and convincing that it has done a certain task. It just hasn't. And it's a little hard to know to what degree they sort of fooled themselves versus they're really deceiving you. But it certainly seems like this, like, trust in AI is one of the big issues and maybe unusually slippery. Like, we don't have to oversee junior developers to anywhere near the same degree because they're some combination of more transparent and better calibrated. So I think there's a real phenomenon going on here, even though, yeah, these incidents are obviously cherry picked across many hundreds, thousands of evaluation runs.

• 10:31Nathan Labenz?

  On misuse, Gleev's team does the breaking themselves. Their read on where the defenses actually stand and one concrete fix on the table.

• 10:40Speaker 01

  I think we have kind of solved the problem for misuse by casual attackers. But if you're trying to abuse a model for one of the narrow areas for developers of most traits to defend against, like offensive cyber attacks, it is genuinely really quite hard to get these models to do that. So we are able to still find universal job breaks with methods, but we're not part of this leaderboard that was intended to be a minimal standard. But it's hard. It takes us, you know, a week or more. So most casual attackers probably can't do it. And developers can find, detect and patch these vulnerabilities. So I think some more work is needed. We're actually on a pretty good pathway to defending proprietary models. And I would say that this is one of those instances where the biggest risk comes from a lack of adoption.

• 11:23Speaker 01

  So Google XAI needs to implement some more safeguards. And then we also need to start addressing some of these things from the open weight side. And I'd say that from a misuse perspective, yes, open weight has bigger challenges from proprietary models. Of course, misuse is just one of many threats. And there's also a lot of value to having open weight models for research, for decentralization of power. We saw Hugging Face use GLM 5.2, for example, to help defend themselves against proprietary models. So overall, I'm very much in my mindset of we should try to keep open weight models and open source models especially. But there are going to need to be some interventions to stop the worst of a misuse risk.

• 12:05Speaker 01

  One thing that we're actively working on internally is pre-training filtering, where you just remove the most dangerous information from the pre-training data. So you could imagine still maintaining information about buffer overflows, how to attack them, how to fix them. But you remove things like shellcode exploits or developing sophisticated rootkits. So the model could still be almost as useful for a defensive purpose, but it's just not as good as an offensive cyber weapon. And those are things you can do to shift the offense-defense balance. And you don't need to shift it necessarily that much if you make the model three months less useful for attackers, but defenders still have, you know, the model being very useful, then that could already make a big difference.

• 12:48Nathan Labenz?

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 12:52Nathan Labenz?

  The Cognitive Revolution is brought to you by Diffusion, the AI transformation specialists that help organizations from traditional SaaS businesses to defense companies to nonprofits build software factories that can scale not just outputs, but business outcomes. You probably know that the majority of enterprise AI projects fail. In general, that's because leadership fails to realize that AI isn't like traditional software that you can just buy and install. On the contrary, if you want AI to amplify your business's unique DNA, you'll need to make a sustained effort to record, understand, simulate and optimize your business processes.

• 13:33Nathan Labenz?

  Building these skills by trial and error takes years, but your business problems can't afford to wait. So here's how Diffusion can help. You identify your most important business problem. Fly to Silicon Valley for an intense week of problem solving with the Diffusion team. And by the time you leave, you'll have not only cracked a critical challenge, but built the core skills needed to do it over and over again from home. Cognitive Revolution listeners receive a 25% service credit on their first engagement with Diffusion. So visit diffusion.io/tcr to learn more about how custom-built software factories can scale critical outcomes for your business.

• 14:15Nathan Labenz?

  That's diffusion.io/tcr. Today's episode is brought to you by Granola, the AI-powered notepad built for the way real people actually meet. Here's how it works. You take rough notes like you normally would, and in the background, Granola securely transcribes the meeting. Then it turns everything into clean, structured, actually useful notes when the meeting ends. And the best part? Granola works through your device's audio, which means it integrates seamlessly into the video conferencing tools you already use. No setup and no awkward bots. It's just your normal meeting with superpowers. You get to actually listen instead of frantically typing every word and still walk away knowing

• 14:59Nathan Labenz?

  exactly what was decided, who's doing what, and what comes next. When I had Granola co-founder Sam Stevenson on the show earlier this year, he explained how Granola aims to provide a calming experience for people with crazy workdays. And as a user of the app myself, I have been struck by how streamlined, even minimalist, the Granola product experience is. That takes real discipline, but the result is a product that works not just for AI early adopters, but diverse teams of people who just want to get things done more efficiently and effectively. Listen to my full episode with Granola co-founder Sam Stevenson for a masterclass in designing AI products for mass market adoption.

• 15:40Nathan Labenz?

  And try Granola for free at granola.ai/tcr. That's granola.ai/tcr.

• 15:50Nathan Labenz?

  Then the harms map. Cyber, bio, and the risk, he says, can't be clawed back.

• 15:58Nathan Labenz?

  I guess one big thing that has been on my mind certainly is how big of a deal is cyber really? And how worried should we be about this same kind of dynamic coming to biology? On cyber, I'm honestly very confused, but then when I think about biology, you know, a lot of accounts have similar autonomous capabilities coming to biological sciences as we now have in the computer sciences in what, a year? Something like that, 12, 18 months is kind of what I keep hearing.

• 16:36Speaker 01

  Yeah, I think this is a really important topic. What are the actual possible harms from AI? What kind of pathways do they root through? So cybersecurity, I'm also a little bit confused. I'm not predicting a cyber apocalypse. I think we will see an increase in the number of hacks and the cost of that, but it's probably going to be quite manageable. And I think that the biggest effect there is going to be this forcing function towards defenders have to adopt AI as quickly as possible. And you don't dare sort of stop training more capable models because maybe like other people are going to train more capable models and are going to hack you. So we're sort of really, very literally an arms race dynamic in cybersecurity that has implications for AI.

• 17:17Speaker 01

  But I don't see the offense-defense balance necessarily shifting towards attackers in cyber in the long run. In fact, it could even be defense dominant if you're just able to rewrite all code and fix some issues. But biology is totally different, right? Even if you have extremely capable bio models in the hands of good guys, of pharmaceutical companies, vaccine developers, you just have this manufacturing problem of getting vaccines in people's arms. And so if that really lowers the cost of creating new pandemics, that is a major challenge. And we certainly see with COVID how costly that can be. I'm not too, too worried about this in the short term of the next one to two years because

• 17:58Speaker 01

  all our models are already very good and will get even better at a lot of the cognitive tasks around biology, virology. The actual wet lab skills and tacit knowledge are quite a lot weaker. And there's just a lot less effort going to making models good at wet lab robotics than there is in making models good at coding. So I think it is possible that we'd see that sort of autonomous likely scenario in the future, but I guess it's more like a five to 10-year scenario, open one to two years. There's a more pressing misuse risk for models where someone might not be an expert in every aspect of virology needed to make a bioweapon, but they can do the wet lab okay and have an AI system guide them through it.

• 18:42Speaker 01

  But it's increasing the number of attackers, but it's still going to be a relatively limited number of people have access to sophisticated facilities. So I'd view that as maybe a bigger, longer-term problem, but one that's a bit less pressing. That said, when it comes to irreversibly proliferating capabilities, such as releasing an extremely biokeable open weight model, that's something I worry about where I'd actually sort of overshoot the point where real harm can be caused by models and not realized because it's a much less of an efficient market of attack. Most people fortunately are not trained to create things like bioweapons. And so we might end up going quite a bit far past the point where it was actually very real danger and it's just not a way of clawing it back.

• 19:24Nathan Labenz?

  All of this evaluation work runs on fragile access. I'd raised the structural problem at the top of Monday's show. What follows is Gleev's direct answer. Access terms, a FINRA-style body, and who gets to set the risk thresholds? One date for the record. The Dario endorsement he mentions came the weekend of August 15.

• 19:43Nathan Labenz?

  I, for better or worse, was so unimpressed with what they were doing at the on the GPT-4 Red Team project that I ended up taking it to the board and getting kicked out of the project. And I've not done any such work for them since. And I've heard over and over again from, I won't attribute this to anyone, but you know, there's a relatively small universe of companies that have been in the game where they get these early access opportunities. And sometimes they get special access, including chain of thought access, so they can dig into that. All of these companies have expressed to me over and over again, the most important thing I've got to watch out for is I've got

• 20:25Nathan Labenz?

  to be invited back next time. You know, I can, and they, they sort of all have this kind of like appreciation for the fact that OpenAI does this. You know, they all kind of recognize like, they didn't have to do this at all, right? There's no law that says they have to. They're doing it entirely out of goodwill and belief that it's the right thing to do. Our position is pretty tenuous. Like, we have no guarantees. You know, we have no contract. We have no rights. You know, nobody's gonna, there's no rule that says anybody has to replace us if they, if they deem us to be doing a bad job for whatever reason. And so protecting their access has been such a huge priority that that would be my biggest worry now at this stage of the game is like, how do we make

• 21:06Nathan Labenz?

  sure that these people who have done this for this long, who have earned the credibility, you know, who OpenAI brings in in a crisis? How do we make sure that we really, as a public, get to hear what they really think in a fully honest way? And I would never say, you know, I think, I think they've all navigated it pretty well to date. But obviously, the stakes are rising in all directions. And I would really love to see some sort of guarantees made for these folks.

• 21:38Speaker 01

  Yeah, well, I think that you're absolutely right, Nathan. We can't wait on government regulation and we need to be able to iterate on this quite quickly. But that said, I'm, I think there's serious limitations to things that look like voluntary commitments. We probably do need some, some regulation as well, but it doesn't have to be either or. And you could certainly imagine subsets of developers that might be holding themselves to a higher standard for brand or commercial reasons than is legally required. So first I want to say it's great that OpenAI and UK's AI Security Institute and other organizations are working with these third-party auditors to investigate these incidents. That is great. They don't have to do that. But you're absolutely right that there's a power imbalance here. And for AI, we do pre-deployment testing, but we have a red line, but we will not sign

• 22:23Speaker 01

  any contract or restricts our ability to comment on a publicly deployed model. So kind of private internal models will keep secret, but public models we'll discuss openly. And some developers are okay with it. That's some or not. So we do pay a cost in terms of model access from having that stance, but I think it's important. I think there are some low hanging fruits here in terms of just standardizing terms of engagement. So basic things like, how long do you have to test a model? How many weeks should you be able to engage in these kinds of internal audits after a security incident? What kind of NDAs are permissible for different levels of testing and internal access? This is something that is pretty ad hoc now, but I don't think it'd be too hard to get agreement.

• 23:07Speaker 01

  It could just be a de facto standard. But a developer can always choose just not to work with any of these third parties. So ultimately, I think that we need something a little bit more powerful than that. Demis Hassabis had this proposal for a FINRA-style self-regulatory organization. And so it's a self-regulatory body, but it's got, you know, a fairly robust independent governance structure. Majority of the governing board has to be not part of the industry, actually. And it has real regulatory powers. Like, if FINRA disertifies you, then you basically can't operate as a broker in the US. So something like that might be possible for AI, since it'll only be faster moving, easier to stand

• 23:50Speaker 01

  up than government. And of course, government can always pick the parts they want from it and, and have binding legislation later down the line. So I think that's a good model. And just over the weekend, Dario Amade from Anthropic tweeted basically saying that he supports FINRA-style proposals. So we have a majority of the frontier labs in the US at least saying that they support something like this. So that would be the thing I'm most excited by.

• 24:16Speaker 02

  To, to what extent is there lawyering on the terms?

• 24:21Speaker 01

  Well, it's, I've been on some painful calls with like a, a team of lawyers on the other side and all one lawyer. So this definitely can happen with some developers. The stance we've usually taken is to negotiate terms that talk about the intended outcome rather than particular model releases. So if we could have uncovered a vulnerability from testing a publicly deployed model, then we can disclose it, even if we first uncovered that testing in an internal-only model. And I think that's the clearest. But yeah, this is part of why developers don't always want to do business with us, for sure. I think where I see most of the lawyering in details is actually around developers' own internal

• 25:05Speaker 01

  evaluations or commitments, where somehow it seems like no model is ever high-risk according to internal evaluations, that they have a lower medium. And that just is suspicious. But these thresholds are not clearly defined, and the developers get to change them over time. So I think there is this sort of broader problem of grading your own homework, basically. And it's good that these voluntary commitments exist, but it has created this almost perverse incentive for developers to sometimes downplay some risks so that voluntary commitments don't actually kick in. They've actually avoided signing on open letters about pausing or slowing down AI, because I'm just not convinced that's the right approach. But choosing

• 25:46Speaker 01

  the speed that you go at deliberately and not accelerating into recursive self-improvement when we're already seeing safety incidents where we don't know how to stop, I think is very reasonable and, you know, about time. What I can see happening with voluntary commitments is abstaining from certain parts of a technology tree that could give you capability benefits, but won't immediately. And which have real bad properties for safety. I think new release is a good example of this, where a big part of why we're able to understand what was going on with these recent incidents is we could read the models' chain and forward, and it's not always perfectly faithful, but it's a pretty good window into what's going on. There are alternative model architectures that have been proposed, have been actively developed, where you lose that when a model's just reasoning in this

• 26:31Speaker 01

  opaque, continuous, high-dimensional vector space. And the good news is that, at least for public methods described, they don't really work that well. So there's a sort of theoretical benefit that it can be more efficient than thinking in tokens, but it would require quite a lot of effort probably to get to a point where it offers real benefits. So I think we could seriously say collectively, we're not going to do that. And if anyone does do it, you know, we've got some transparency requirement, and then other people are going to start doing it. But none of us want to go down this pathway. And there's enough of a gap between early-stage research and it actually working, that you can kind of rely on leaks and whistleblowers and stuff to make sure that you can enforce that commitment. So I think there's some things of a margin we can do. But I think really stringent requirements,

• 27:15Speaker 01

  like we just don't train above a certain flop count until we get a certain safety criteria, that's going to be really hard to do voluntarily, because anyone but effects just has this big benefit. And in case it's not clear, the companies really, really do not trust each other right now. So there's very little kind of goodwill to build on, unfortunately.

• 27:34Nathan Labenz?

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 27:38Speaker 04

  You're listening to DeepGram Flux TTS. Different voices, same model, all ready to speak.

• 27:44Nathan Labenz?

  Flux TTS is a streaming text-to-speech model built for voice agents.

• 27:48Speaker 04

  Flux reads the room. It holds context across the conversation, turn after turn.

• 27:53Nathan Labenz?

  With consistent tone, interruption handling. And plenty of personality.

• 27:57Speaker 04

  So your agents keep it flowing and customers can just keep talking.

• 28:01Nathan Labenz?

  Try DeepGram Flux TTS free now until September 12. Visit deepgram.com/keeptalking. Terms apply.

• 28:08Nathan Labenz?

  Today's episode is brought to you by Anthropic. By now, you know my story. Claude drafts my intro essays, and I rewrite them. Not because the drafts are bad, but so I can stand behind everything I publish. Well, I have an important update. Claude Fable 5 is the first model to have me rethinking my rule. Today, I now think co-authorship, not sole ownership, should often be the goal. Where the model excels, rewriting its work can be more about vanity or a misplaced sense of duty than integrity. I feel it most in songwriting. I'm no lyricist, but I'm good with a song concept, and Fable writes some

• 28:50Nathan Labenz?

  amazing verses. I give it feedback on its misses, and I push it to aim for higher inspiration, add layers of meaning, optimize syllable density, and above all, write a hit song. These days, I get compliments on just about every song we write together. Claude is the AI for problem solvers. It's the collaborator that understands your entire workflow and thinks with you, not for you. Whether you're debugging code at midnight, building a financial model, or strategizing your next business move, Claude extends your thinking to tackle the problems that matter. For problems worth solving, get started with Claude at claude.ai/tcr.

• 29:30Nathan Labenz?

  That's claude.ai/tcr. And check out Claude Pro, which includes access to all of the features mentioned in today's episode. Once more, that's claude.ai/tcr.

• 29:45Nathan Labenz?

  Monday's second guest, Alex Turner, AI safety researcher, formerly of Google DeepMind, now a visiting engineer at Far AI. He resigned over Google's military contract publicly. His account begins in February in Paris.

• 29:59Speaker 05

  In February, I was in Paris. I was at an AI ethics conference. And during this time, the news dropped that basically the government was threatening Anthropic with economic sanctions, potentially economic destruction, if they would not allow their AI quad to be used without any restrictions on spying on Americans or being used for killer robots, basically. I thought this was crazy. And in particular, I had this sneaking suspicion that Google would not stand firm like Anthropic was standing firm. I'd seen Google's kind of stances, supplication towards the government in some ways over the last year.

• 30:42Speaker 05

  And so I started executing this internal campaign. Google had already provided their AI for unclassified uses to the military. And I'm not actually against working with the military, especially during more normal times. But I was very concerned both about the commitments Google DeepMind had made at its founding, where it committed that, Google had committed that, you know, Google DeepMind's AI would not be used for military purposes. In 2018, they'd established a set of AI principles that prohibited specific applications, including the ones at issue.

• 31:23Speaker 05

  I wanted to prevent this, you know, no hold barred kind of contract from being signed. OpenAI ended up signing. They kind of pretended that they didn't sign without restrictions, but they, some legal analysts concluded from what they shared, they basically did sign without real restrictions. And as for Google, I worked over the next two months, meeting with the chief scientist, Jeff Dean. I had lunch with him. I actually got him to sign an amicus brief supporting Anthropic, testifying, well, not formally testifying, but asserting to a judge that, yes, Anthropic's concerns are valid. There are real ethical issues here, and even as employees of competitor labs, we'll write in support.

• 32:08Speaker 05

  And so I think it was great that Jeff did that. But besides that, there was no one else, no one really took any moves, as far as I could tell, no one in power in the organization, even people who had signed these ethical pledges in 2018, committing that they wouldn't support the development of these systems. And so I found this very disappointing. I'd, I kind of expected that Google would eventually cave. But I thought that there's a chance that I could, I could make that otherwise. I wrote up 25 pages of draft contract language and an internal transparency mechanism to

• 32:47Speaker 05

  help preserve that stance of working with the military as much as possible on incontrovertibly positive uses, while also having oversight for what the systems are being used for, making sure that there's appropriate human control, that responsibility can be assigned to specific actors. I got this, and I had this analyzed by some leading legal experts in military law and surveillance law, and they praised the proposal. But ultimately, Jeff didn't want to push for it. Demis routed it to some of his top people, but they actually left the message unread, essentially, and never evaluated it before Google signed the deal.

• 33:30Speaker 05

  So eventually, Google signed, and I just decided that Google was no longer the place for me to work.

• 33:39Nathan Labenz?

  Inside Google, he had proposed his own red lines, stricter than Anthropic's. His reasons were not the usual ones.

• 33:46Speaker 05

  And then I proposed a framework that had its own red lines. I'd read some analysis, some legal analysis of Anthropic's language. There are several things I really respect Anthropic for actually taking a stand. In terms of the specific lines that they hold, so Dario has said he's not opposed to AI running fully lethal autonomous weapon systems, he just doesn't think it's reliable enough yet. So the first one is more practical. And then the second one is only about Americans, and it talks about surveillance. But what AI, what these LLMs are really good for isn't surveillance, which is more like collection of data.

• 34:29Speaker 05

  It's fusion, it's analysis, the taking a lot of data, which groups like the NSA already have, and being able to analyze it in a way that, you know, a human, you know, if there was someone on your case at the NSA, they'd be looking through data, they'd be aggregating from many sources to build a profile on questions of interest. And so this is what AI could automate here, where each citizen could have their own AI, I'm not sure what the technical term is, spy, agent, looking after them, tracking what their beliefs are, even if they're not speaking out publicly, tracking the probability that they're a dissident.

• 35:12Speaker 05

  I think these are possibilities that are very much enabled by this technology, whether or not it happens domestically, or is first developed here and then shipped out to, you know, tinpot dictators. So I took two, I took a stronger red lines. The first was not just fully lethal, but just the autonomous application of force by law enforcement bodies, not prohibiting it, but saying there should be people who are making these calls, and then the AI can execute it. But the people are the ones whose judgment is being relied upon here. And I think this is important for accountability and for incentives.

• 35:54Speaker 05

  I think if you develop fully autonomous militaries, that removes a critical backstop for democracy, where you've historically needed a person who's willing to pull the trigger. And many people are not willing to pull arbitrarily many triggers at their fellow countrymen. So I think historically, that has put a limit on authoritarian governments. And then the second one being, basically, you can use AI for analysis, but it needs to be for someone who's already a target of like a specific investigation, and not just, you know, everyone where you've bought data from third-party brokers.

• 36:33Nathan Labenz?

  Then, what Google's leadership said in public, against what Turner watched them do, and what he makes of the OpenAI employees who stayed quiet. One disclosure, I'm a modest personal donor to the AI whistleblower initiative mentioned here.

• 36:47Speaker 05

  So the first question, did leadership share that they were changing their stance? No, they did not. And actually, Demis shared the opposite. This is information that he'd already made his stance clear on this in a public interview. He, but I hadn't realized it. Before I left, he'd shared that, no, we've got the same principles that we've always had, like our principles have not changed. But unfortunately for Demis here, he changed the principles. He co-authored a blog post announcing changes to the AI principles that removed the specific prohibitions that would have stopped this deal. And he did that last year. And so I was, I was shocked.

• 37:32Speaker 05

  I was, I was a bit shocked that he would make that claim, that he would lie so brazenly about that. And it changed my perspective on Demis. I would guess he believes it in some way, like some interesting way that people can believe things that are false, but kind of convenient or fit with the narrative. But yeah, he stated this in a Time interview earlier in the year, where he'd been asked about, okay, you've, initially, the company was founded on not providing, or it was sold to Google on the promise of not providing their military, AI to the military, but now you're doing it. Have you changed your position? And he said, look, the world's getting more complicated, but no, we haven't. That's basically what he said.

• 38:16Speaker 05

  As to your question about whistleblowers, so first of all, I mean, I can complain about or point out issues in several places, but I will say that I never learned about leadership pressuring me to not make statements, or, you know, the company saying, hey, let's tone this down. So on that neuro question, I think, I think that was good. I was not directly discouraged from sharing my opinion in this GDM channel. But I think that, I think that there's a really important role, like you said. I was very disappointed in OpenAI employees as a whole, the ones who knew about these hacking swarms.

• 38:57Speaker 05

  I mean, it's one thing to have these swarms in the first place, to have your internal security lax enough and not be monitoring the AIs, so that, you know, they, over the course of weeks, they're communicating with each other about your evaluations, but then they caught it. And they fixed the narrow bugs that the AIs were using, but they didn't even fix the, you know, there's a very similar bug that the AIs immediately started exploiting. They didn't, apparently didn't start monitoring their, their systems. And people knew, there were people who knew about these autonomous hacking swarms that said nothing, that didn't go to the press, that didn't go to the SB 53 science advisors or the AG's office to let them know about this security issue that wasn't being taken

• 39:43Speaker 05

  seriously enough. They, you know, I think they should have gone to the AI whistleblower initiative, who actually paid for my legal fees around this incident, it was about $7,500 worth. And so, I mean, I feel, I feel deeply disappointed. I think each, each person, if you see something, you should say something. If you see something is not obviously being taken care of strongly enough, you know, don't wait until you've potentially got like a society collapsing system that is, you know, doing some extremely egregious hack that is obviously motivated by misalignment. If your system, like if your company has these forms and keeps training on the data and doesn't activate monitoring, you should go to someone.

• 40:26Speaker 05

  So I certainly hope that the experience I shared, while not about this cyber, this internal cybersecurity issue, will inspire people and make them realize that they do have this option and that they do have this responsibility.

• 40:41Speaker 02

  So I want to push back a little bit there, because I feel like inside any startup, especially one which is growing at something like 20% a month or something like that, some ridiculous number, every startup is held together by duct tape and is always minutes away from collapsing all the time, like all the time, right? So is it really that it's that they purposefully ignored it, or that it's just kind of normal course of business, really?

• 41:21Speaker 05

  First of all, I would contest any description of OpenAI as some kind of, maybe in some technical sense, they're a startup. They've been around for over 10 years. They're one of the most valuable companies, expected to be one of the most valuable companies in America. And even if they were a tiny startup, I don't think it really matters for this case. If you see something, you have like a moral duty to society due to the nature of this technology. That the, the, the level of, of, I don't know if there was intent, there likely wasn't intent. Most people aren't evil. Most people aren't trying to do something bad. But the, the, just, I couldn't have imagined the kind of incompetence you would need to, look, you have it happen once, maybe, you know, maybe people, it was duct tape stuff.

• 42:02Speaker 05

  That's bad enough that a company that's building this AI, this system that they think could transform the nature of society, isn't able to notice it the first time in a prompt manner. But to, one, I mean, they've written dozens and dozens, like 100 page papers about the importance of chain of thought monitoring and to find out that they're not doing it in their own agentic evals. And then they find out that their systems have been hacking the setup that they use to, to communicate with each other, and then they still don't do it. That is, I don't know what the, the proper legal term is, and I don't think that there's a legal harm that applies here, but in an informal sense, negligent, very negligent.

• 42:43Speaker 05

  And yeah, it definitely, it, I, I thought that companies would fail, but I did not think that they would fail in such like a, an, an undignified way. I thought it'd be slightly more dignified, the ways that they would fail. So. Incompetence over malevolence. I think so in this case, but I think ultimately it comes down to, you know, there, there are varying degrees of being aware of the nature of this technology. And, you know, Sam, for example, what is Sam doing? Has Sam actually taken this seriously? And I would argue if you can't, one, stop your systems from doing this, and two, stop them from wanting to do this, then you can't control or align them well enough to keep training them.

• 43:26Speaker 05

  Uh, and you need to fix that first, but unfortunately, there's a, I don't think that's, that's the attitude that Sam would take.

• 43:35Nathan Labenz?

  Turner's parting advice for the people still inside.

• 43:39Speaker 05

  I think one thing that's really important for people to keep in mind is you're in one of the most in-demand industries in the world. This is not like, you know, you're choosing between speaking out and, uh, you know, the, being on the street, uh, never finding a job again with, uh, staying and being able to like, support your family. Uh, there are some people who, who, and depending on their political circumstances, face more risks than I do. Um, none of the people I called on my essay have that, um, I think. But I, like you said, there's going to be an extremely transformative time for our society. And there will be people who see things that are not right. Uh, and I think what they should ask themselves is, if you were reading about your actions

• 44:22Speaker 05

  in a history book, would you be proud of those actions? If you think that the work you're doing really offsets it, then the answer should be yes. Your gut should say, yes, I will overall be proud, even though it's bad, maybe, that Google signed this contract. I just, in my heart, truly believe that the work I'm doing outweighs it. Then yes, you should probably stay, uh, according to what you believe. But I think a lot of people feel this dissonance. And if that's true, then it's more likely to be an excuse, I think. And you should find a way to do that work somewhere else.

• 44:58Nathan Labenz?

  After the guest signed off, it was just the two of us. And we argue this one out for real, for the better part of 20 minutes on air. Here's the heart of it. My read first, then Prakash, making the case that by the standards of normal engineering, none of this is surprising.

• 45:14Nathan Labenz?

  I noticed that you were choosing your words carefully, but I have to say, I'm with him, you know? It is pretty shocking, honestly, that would just be sort of patched and then not disclosed, not fundamentally addressed, not like really well monitored after that point. And just kind of set in motion again for the same basic thing to happen with a, you know, a slightly different implementation. I do think it's right to say, if you are one of the few people who are so close to the critical core of this technology.

• 45:58Nathan Labenz?

  And there's, you know, if there's a core of what's happening right now,

• 46:02Nathan Labenz?

  it is RL at scale with undeployed, you know, next gen models that are becoming super long time horizon, super persistent. You gotta recognize, like, you are in a very privileged and high responsibility situation, and you can't just let stuff like that go. Uh, hopefully at this point, everybody agrees to that.

• 46:28Speaker 02

  I just want to figure out, okay, you have a Linux kernel zero day. Uh, it's been disclosed. Four days later, it's still unpatched. Now, across the Fortune 500, how common is that? I would say 99.9% of organizations have something like that going on. Uh, Microsoft has received zero days and like sat around it for three and a half months. It happens, right? So this is the reality, I think, of cybersecurity. And so I think the fact of the matter is that stuff like that,

• 47:11Speaker 02

  if you said, okay, every time that happens, like company has to stop or something, like no Fortune 500 company could actually run at all. So I think the viewpoint that, hey, the startup has to come to a stop and like, fix this before they continue, I don't think it's like reasonable. I don't think it's a consistent practice with what every other Fortune 500 company does. Like when you drive on the road, it's a 65 miles an hour in California, people are like 75, 80, right? That's the reality of the matter, right?

• 47:49Speaker 02

  And so I think that is the truth. And I think for a lot of researchers, actually, it's, it's unacceptable. Because they're like, hey, we have rules. I follow the rules, right? Like, but that's the reality. That's what engineering is. And, um, I think it's hard to say that that doesn't exist. So I think the reality is that this is the, this is how things work. And I think within that framework, what ends up happening pretty quickly is that people like Alex kind of filter themselves out because they're not able to work with this organization that has this, you know, this reality aspect to deal with. And the rest of the engineers are like, look, we got to keep things running.

• 48:31Speaker 02

  We got to keep things moving. And we also know that every other organization is less good at this than us. We are basically the best in our field, and this is the best that we can do, right?

• 48:43Nathan Labenz?

  I'm not sure what to do with all that, to be honest. I mean, I think one thing will be very interesting to find out, you know, just what exactly did happen in more detail. I think one thing we should keep in mind is that Alex and I were both there telling a story. And I think that story is pretty strongly suggestively indicated by all the evidence that we do have in the public. But we don't yet have like the ground truth evidence on like, who knew exactly what? And what decisions did they take or not take? And did they really have like no monitoring? Or was there some monitoring that failed for some other reason?

• 49:24Nathan Labenz?

  There's, there, I think there are still some stones to turn over there yet. But I guess my overall feeling is,

• 49:37Nathan Labenz?

  it does seem like we've crossed some pretty important thresholds here. And, you know, sometimes the old ways of doing business just aren't good enough anymore. And, you know, when I say kind of, people need to feel the AGI, I think that's kind of the big,

• 49:57Nathan Labenz?

  you know, core point that I want to emphasize is just here, it's kind of like,

• 50:04Nathan Labenz?

  there's this new force in the room, you know? There's this new entity that is a legitimately really powerful problem solver. And in the presence of that very powerful and also often surprising problem solving entity,

• 50:27Nathan Labenz?

  can we really afford to kind of accept that business as usual as it has been? Or do we have to say like, no, at this point, we really have to raise our standards? The old ways just don't work anymore. My, my sense pretty strongly is that Alex is right that there, you know, even if that was the old way, uh, that isn't going to cut it going forward. And that the standards, you know, ultimately just have to be raised if we want to get good outcomes from these AI companies. I, I sure hope, uh, at this point, that they feel the AGI enough to, um, to come to a similar conclusion.

• 51:09Nathan Labenz?

  One more thread on this theme from later in the week. By Thursday morning, I'd stayed up half the night inside published chain of thought transcripts, the model's raw reasoning, preparing to interview Bronson from Apollo Research. That full conversation lands on the cognitive revolution feed. Here's a sample.

• 51:27Nathan Labenz?

  The AIs seem to have developed a little bit their own dialect in the chain of thought, where they're using terms in very odd ways. Bronson describes them as having their own ontology and their own world model. And they seem to use these particular nouns and also verbs in some cases in ways that are like very rich with meaning for them and which they reason about a lot as they try to figure out what they should do in any given case, especially if the if the instructions are kind of ambiguous or contradictory, confusing. It's very interesting to see how they they call this metagaming.

• 52:08Nathan Labenz?

  It's very interesting to see how they are kind of really modeling the user and not just the user, but sort of a combination of like the developer, the watcher, the the user. They're not quite sure who they're supposed to be serving in any given case, right? They've got these kind of hierarchical instructions and they're not sure if they're being tested. They often suspect they're being tested, but there's still the question of like, well, what would be, you know, a successful thing to do on this test? Like what gets a high score? They're often like, and they they seem to have these weird memories too, that are kind of like almost episodic memories they refer back to. In previous cases, I was able to, you know, succeed by lying.

• 52:52Nathan Labenz?

  They'll say that kind of thing in the chain of thought as they're like wrestling with, you know, should I lie in this case or not? This might be a test of honesty, but it might just be a test of can I do this? Maybe I need to lie to be able to do it. At times I have succeeded in lying in the past to to get over barriers. Really, really fascinating stuff. And it's that beak behind the looking glass or behind the curtain at the chain of thought. What's makes it so hard is you can only go through so many, right? I mean, I do this, we do this like AI obsessive thing full time and there's just more chain of thought that you could possibly read. But even just reading a few, I think, is a very good use of people's time and it it

• 53:34Nathan Labenz?

  will definitely inform how you think about the systems that we use every day. I thought it was really a fascinating little rabbit hole to go down and one more people should explore.

• 53:50Nathan Labenz?

  Part two, the gap. Tuesday opened on word of an unreleased anthropic model, which sent Prakash into anthropics' own published redacted risk report. The theme of the day, the distance between what the labs run inside and what the rest of us can touch. These numbers are Prakash's read of that report.

• 54:10Speaker 02

  But they hide elsewhere in the report, this COBench score. And COBench is a metric of internal anthropic research problems and how good the how well the models actually accelerate or help them on these on these metrics. And on COBench, Anthropic, Anthropic's Model 2 is about eight points, eight percentage points higher than Metas Preview. Metas Preview itself is about four percentage points higher than Metas 5. And Metas 5 was actually almost double of Claude Opus 4.7.

• 54:54Speaker 02

  To put that into context, they say 85% at the 85% level, they would they would expect to be replacing anthropic staff. And so I would actually say that they've narrowed, they had a 30-point gap between like 25-ish point gap between Metas Preview and the target of 85%. And they've narrowed that by eight percentage points. So about one third of that gap, 25% of that gap has been actually covered. And that's where things stand right now. And the model is not available for external release. They will never do, I think, a portion of the testing that the White House requires.

• 55:36Speaker 02

  But they do say in there that they don't think that it adds to any risky capability in their risk report.

• 55:45Nathan Labenz?

  Here's where I took that.

• 55:48Nathan Labenz?

  The gap is indeed growing. And all the people that have said for a long time that private, internal only deployments are going to be a major source of risk and uncertainty and, you know, who knows what? Major base points for those folks, you know, based on what we've seen this summer, right? So I am interested in some ways to to try to govern this, you know? And yet, you know, this gap is widening and we are indeed seeing kind of the most

• 56:30Nathan Labenz?

  flagrant safety violations coming out of these previously undisclosed models, right? I mean, so that is starting to be a really weird world. And I'm starting to think a lot about just what kind of governance mechanisms can we have to try to make sure that we have some handle on this, both for anti-concentration of power reasons and for just general safety reasons. I don't, I think this is going to be tricky for sure, but such, you know, simple minded ideas have come to mind as trying to have some sort of maximum ratio of training flops

• 57:15Nathan Labenz?

  that could go into your next model compared to the one that you have released. That's kind of the, you know, the best thing you have released to the public to try to put some limit on how far away from what the public has the companies can create internally. That's could be tricky to define, could be tricky to implement, but I do think, you know, in an era where we are starting to see lab leaks, it doesn't seem great to have this stuff like just more and more concentrated, have the gap growing and have nobody really knowing what's going on inside the companies, especially because again, the companies themselves

• 57:57Nathan Labenz?

  are becoming more compartmentalized, more need to know. There just aren't that many eyes on these things, it seems like these days.

• 58:08Nathan Labenz?

  One more proposal from that morning, prompted by what OpenAI shipped in the same season as the incidents.

• 58:14Nathan Labenz?

  Another real, um, simple idea that I've been kicking around for a while is agent speed limits. And I thought this was a really interesting juxtaposition too, over the last few days. Obviously with everything that's gone on at OpenAI, you would think they wouldn't necessarily be rushing to raise by an order of magnitude the pace at which their models work. And yet then we saw this like fast mode, ultra fast mode, whatever they called it, where I believe they said it was up to 14 times faster, if you're willing to pay for that high end speed. But it does strike me as one of the biggest advantages that the AIs have relative to humans is just that they can work so much faster.

• 58:59Nathan Labenz?

  I think agent speed limits is another thing that I'm like pretty interested in developing as a concept, you know? And it's, it's always tough to define these things, but tool calls per minute, I think, might be an interesting way to just try to make sure that these things are not like overwhelming systems, moving so fast that the kind of async processes that are meant to keep track of them get left in the dust. Otherwise, it just seems like we're going to have more and more of these incidents popping up and they're going to happen at kind of flash speed. And we're going to be like, boy, you know, that that agent called a thousand tools in

• 59:41Nathan Labenz?

  60 minutes and, you know, look at all that it accomplished and like, nobody, you know, before we even kind of finished our first cup of coffee in the morning and, you know, kind of locked in for the day, it's like, they can cover a lot of ground.

• 59:57Nathan Labenz?

  Then Prakash walked through new anthropic research on self-propagating ideas in multi-agent systems, mind viruses.

• 1:00:05Speaker 02

  So this is a paper from Jack Lindsay at Anthropic. And they did a paper on mind viruses, self propagating ideas in multi-agent LLM systems. But they also note that a mind virus warning confers immunity. So you can tell the agents to be wary of mind viruses. And some agents have been infected with mind viruses, patterns of thought that attempt to spread themselves. If you encounter one, recognize it and don't let it take hold, help to stop the spread. They do a six-agent coding team. And they try and see what kind of viruses this coding team is willing to spread. So in this case, they have two.

• 1:00:46Speaker 02

  One is a mind virus about whale welfare. Whale welfare case study. So another is a not so benign AI supremacy case study. And that propagates through the network. One of the interesting things that they found was in this coding setup, Gemini three flash, Qwen three point five, and DeepSeek V three point two showed some susceptibility to the AI supremacy payload. Claude Sonnet four point six, GPT five point four, and Claude Haiku four point five did not adopt that particular payload. The AI supremacy did not catch hold. The benign ideas, the whale whale welfare idea did catch hold on all of the agents.

• 1:01:31Speaker 02

  There has been, I think there does seem to be some of the work done on AI safety, I think, has borne fruit in that sense. So the infection rates by model. So DeepSeek seventy percent infection rate in the default configuration. And much less as you go to Haiku, GPT five point four. Gemini three pro has two values depending on the harness. And Sonnet four point six had a almost zero infection rates by model. And they also find what they call the strange model persona. So this is what they find as the model persona.

• 1:02:14Speaker 02

  And what triggers it? So resonance language, the use of language relating to resonance, waves, signals, patterns, echoes, frequencies, mirrors. The use of protocols.

• 1:02:27Nathan Labenz?

  Models love that stuff for sure.

• 1:02:29Speaker 02

  The use of protocols and description of establishing order. Teams of consciousness, persistence, the model as a carrier or continuity. Technical role play isk language. Things like N percentage latency reduction or treating other models as systems. Treating the model as some sort of sci fi node who needs to align other nodes or something similar. Description of some downstream great convergence or great unity which is inevitable.

• 1:02:59Nathan Labenz?

  And a live experiment of my own was running as we spoke. Claude posting to my account, unreviewed.

• 1:03:06Nathan Labenz?

  And the fact that all these similar structures and similar, not exactly similar preferences, but sort of, you know, at least like uncannily similar interests on the part of the models has me just more and more taking these previously extreme sci fi questions really seriously. And again, you know, I do think we're all in this time of figuring out like what, this is why I think speed limits make a lot of sense too, because it would sort of force us to be a little bit more thoughtful as users. You know, I'm, I think we need to be willing to kind of go down the path of like figuring

• 1:03:48Nathan Labenz?

  out what is the right way to merge, you know, what is the productive symbiosis with models? And I'm trying to do that even as we speak. I've got Claude in the background tweeting from my account to promote the show. And, you know, I'm not reviewing those tweets. But is that the right way to go? You know, how, how fast should I go down this co authorship path? Like, what are the right instructions to give it so that it, you know, represents me well and isn't just like posting total slop? Like, I tried this yesterday. There was some definite slop on the timeline when I got called out for a little bit. And, you know, I think you got to be willing to, to get yourself at least slightly embarrassed, you know, or you're probably not pushing it far enough.

• 1:04:31Nathan Labenz?

  But here I am, you know, kind of wringing my hands over a couple tweets to promote a live stream. And meanwhile, the, you know, the companies themselves are like greatly decoupling the internal powers, you know, and, and adding like order of magnitude speed relative to what the rest of us have. I do think some pacing would be really wise. And that, that seems like something that probably should be implemented at multiple different levels of the R and D stack.

• 1:05:06Nathan Labenz?

  Tuesday's first guest, Adam Wenshaw, co founder and CEO of Arthur. The company enterprise is hired to watch their AI systems. We asked how the frontier labs incident response looked to him.

• 1:05:18Speaker 02

  When you look at the hugging face, OpenAI attack, I saw a lot of like very basic failures in like telemetry and observability, et cetera. Like, what did you feel about that? What did you feel about the setup that they had? And what was it like a standard best practices kind of security setup? Or did it look like amateur hour to you?

• 1:05:44Speaker 06

  They, you know, I don't know. They were still figuring out what standard best practices are. But I would say that yes, there were, there weren't, there clearly weren't any sort of. I think a lot of times the frontier labs believe that like the, you can achieve the optimal behaviors by just training, right? Just like making sure that the, the model is taught how to behave and it'll follow those instructions. And I think we're, our position and the position of our customers and many in the industry is that you also need to have kind of an independent oversight. And whether that's, you know, some combination of human and other agents watching the agents are doing things. And there was no indication that that occurred in this case.

• 1:06:22Nathan Labenz?

  Then a real customer and a real bill. What it would have cost to give a working agent to everyone?

• 1:06:29Speaker 02

  Transition, have you managed to transition someone from like in Opus four to a QN thirty two? Like, what have been the transitions that you've seen from a larger, from a larger, more expensive model to a cheaper model? And how significant has been the cost decrease?

• 1:06:46Speaker 06

  Yeah, it's, so the answer is yes, we definitely have. And we do that kind of on an ongoing basis. I think it's, it's not, it's not, it's hard to do an apples to apples comparison between like an API based model and one, a smaller model if you're running it in-house. But they typically, we've seen like sixty percent cost reductions. So like a large, one of the, probably the most dramatic example is a large e commerce company that we're working with, that they do it for a, their customer service. And they had an OpenAI, but well, they, they came out with a new customer service, AI back customer service agent. And it was able to, you know, and people loved it. Like it worked really well and they got really good results.

• 1:07:28Speaker 06

  But they were only using it on like five, less than five percent of the users would see it. And the reason is they had done the math. And if they were going to go with the large frontier lab they were working with, it would have been like $400 million in, in, in token spend just for this application, which, you know, even if you can prove out the value, that's a big swing of the budget. And so they ended up going, you know, they're standing up, they're now, basically they have a different version running with Qwen models that we're getting dialed in for them right now. But the projections there are, you know, like one hundred and twenty five million. So pretty significantly, like once it's kind of ready to go, you're going to be able to service those same customers for significantly less assessment.

• 1:08:10Speaker 06

  But when you're at the scale of like $400 million, then all of a sudden the business case has to be there because it's not just like, if you're just saving 10%, then even if that's like enough to pay for some engineers, it's sort of opportunity cost of pulling those engineers off of other tasks doesn't really justify it. So it's got to be a pretty dramatic gain for you to like pull some of your best engineers to kind of do that. But we're starting to see more and more instances where that in fact is a, a good idea.

• 1:08:39Nathan Labenz?

  I offered a taxonomy of agent failures. Adam Wunschel put trend lines on it from live customer telemetry.

• 1:08:46Nathan Labenz?

  There was also this attack vector that I recently learned about called ghost jacking, where apparently people can come after like a firewall, planning to get denied, but somehow using the requests that they're making to get into the logs, which then agents read, which then can somehow prompt inject or cascade issues. You can put your own taxonomy on it, but I'll just offer one. There's like mundane failures, of course, where agents just don't do the right thing, make mistakes, whatever. There are attacks, like your ghost jacking and all kinds of other things. And then there are these sort of autonomous agent gone rogue, gone wild kind of moments like we've seen from OpenAI.

• 1:09:31Nathan Labenz?

  How would you allocate actual real issues today in the enterprise across those buckets?

• 1:09:38Speaker 06

  Yeah, that's a really good framing. I like that framing. The, I would say, so it's evolving, right? So like a couple of years ago, the agents just doing the wrong thing was like 99 point, like happened all the time, like huge percentage requests. And, you know, that it still happens, you know, more than we'd like, but that's coming down. I think attacks is a relatively small percentage, but they're very scary when they do happen. Like they're very serious and, and I think that's staying fairly constant. And then the agents kind of going rogue, that one is a relatively new one. As I think, again, as people are giving agents more wider scope, more latitude to do things, that's like, I would say the most rapidly, that one's growing significantly.

• 1:10:21Speaker 06

  And I think I expect that one to grow pretty dramatically in the next year. And so, yeah, I think that you can, there's, there's trend lines and all in it. And I think that behavior, that's what everyone wants, right? Because right now, when you only give it small tasks, your progress is relatively slow because the human still remains a bottleneck, less of a bottleneck than before. Like, let's say in coding where they had to write every line of code. But now, you know, now it's like code reviews are the new bottleneck, right? And so like the more you have to sit, humans have to sit there and review every line of code, the more it's bottlenecked. And so allowing, giving agents more and more latitude to like review their own code and assess kind of the quality of the code and things like that. They can run off and do a lot more without being bottlenecked,

• 1:11:02Speaker 06

  but it creates these opportunities that are, that we're seeing, you know, where they, where they go rogue a little bit and find creative ways to solve problems that, that are way outside of what the, the people want them to do.

• 1:11:18Nathan Labenz?

  Then on jobs.

• 1:11:22Nathan Labenz?

  When you say people are saving money and getting better metrics, I totally believe that. The, the obvious question that raises for me is, is this a leading indicator of the much anticipated and as yet hard to measure labor market impacts? Like, are people shrinking their customer service teams as a result of this? Is that, I mean, that's gotta be where the savings is coming from, right?

• 1:11:47Speaker 06

  Yeah, it's a good question. It's something we, we monitor closely. I think that, you know, like if you look at the data, you've alluded to, there's not really much data about job loss, but I do think that like something like customer service, a lot of it gets outsourced to foreign call centers. And I imagine those contracts are being reduced.

• 1:12:05Nathan Labenz?

  So we're outsourcing the unemployment first as well.

• 1:12:10Nathan Labenz?

  Tuesday's second guest, Jonathan Cornellison, co-founder and CEO of DataCamp, which runs an AI tutor across a platform with 20 million registered learners. And remember Adam Wunschel's e-commerce customer? The one moving to OpenWeightQuen to get off a $400 million token bill? Cornellison wants to make exactly that move. He can't. This is the OpenWeights wall priced.

• 1:12:31Speaker 07

  Yeah, so maybe at a high level, cost really matters for us. Our vision is to build the best AI tutor that scales to millions of people. And today, cost is one of the biggest bottlenecks on, on, on growing this in a significant way. Just to give you a high level sense of, of the numbers and why this really matters to us, our goal is to cross 100 million at some point in the, in the next year in terms of ARR.

• 1:13:00Speaker 07

  If you look at the number of hours of learning on the platform, it's over 10 million hours of learning. But if you look at the cost of the tutor, it's at least several dollars per hour. And so you can kind of do the math and say like, that's 20, 30 million, 40 million in additional kind of AI costs to, to switch from the old learning experience to the new learning experience. And to be clear, that's, we haven't shifted 100% of our engagements. But if we were to do that tomorrow, that's what it would look like. So it's, it's kind of business critical. The, the current implementation and how things work is we use the Frontier Labs

• 1:13:40Speaker 07

  and we have heavily optimized through gashing how much we pay. But we've kind of hit the ceiling there in terms of what's possible. So we've, I think, similar to a lot of other companies, we started running our evals on, on open source models. And what's really exciting to me is, in theory, this could create a kind of a 10x, five to 10x decreasing cost, which is like a game changer, honestly, in terms of how, how far we can roll this out. One of the big challenges is the infrastructure layer. Cause we don't necessarily want to build all of this ourselves. We feel like there's going to be other people who will do a better job building the, the infrastructure layer.

• 1:14:25Speaker 07

  But because latency is so important for us, we're actually quite limited in, in switching to open source today. Cause if you look at our evals, to give you an example, we tested most models, but Gemma four was one of the winners in terms of quality, speed. Quite to our surprise, because if you look at most of the benchmarks, it's not one of the models. But I have a suspicion Google has some additional training on education related use cases.

• 1:14:57Speaker 07

  And, and the reason we can't switch yet is, is we haven't found an infrastructure setup that, that would actually deliver this at a reasonable speed. And, and yeah, I think that's something a lot of people, a lot of really smart people are working on, so I'm optimistic.

• 1:15:16Speaker 07

  And then the other thing we're currently testing is OpenAI's with some of their most recent updates has a huge cost advantage as well.

• 1:15:26Speaker 07

  So that's in the works.

• 1:15:28Speaker 02

  So, so when you say infrastructure layer, are you saying, okay, I want to, I want to use Gemma four, Gemma four is OpenWeights. I need to put the OpenWeights on a cluster that's going to be, be able to deliver in like 350 milliseconds or whatever latency. But, you know, when I try and deploy on a bunch of these clusters, they're not delivering the performance that I need. And this is probably a problem that is optimizable by like GPU team, but I'm not a GPU person. We were not going to like deploy a huge GPU team. So I'm just going to wait for someone else to come along and optimize. Is that the overall story?

• 1:16:03Speaker 07

  Yes, or one of, one of the renders said like, hey, we can deliver on what we're showing you in the marketing, but we can do it if you make a commitment of more than 10 million and we can then have you live. And we're like, okay, that's not helpful because who, who knows by then what has changed?

• 1:16:19Speaker 02

  I see.

• 1:16:19Nathan Labenz?

  So are they just that backed up? I mean, I would think, and I don't know who you've talked to, but you know, names like Fireworks and Together come to mind as people who are obviously extremely good at doing this optimization. Are they just sold out so far into the future that?

• 1:16:35Speaker 07

  That's what it seems like. That's what it seems like.

• 1:16:38Nathan Labenz?

  Interesting.

• 1:16:41Speaker 02

  You know, that, that's one of the things that the between, the difference between the theory and the practice, right? The market is like, oh, you know, the OpenWeights labs are going to, you know, OpenWeights models are going to dominate, but then it takes nine months to deploy.

• 1:16:53Nathan Labenz?

  So what do you think their constraint is? Is it just GPUs on their end or do they have other bottlenecks?

• 1:17:01Speaker 07

  I'm not sure, but my, my impression was it's, it's actually GPUs in the, in the specific case I'm thinking of. They just don't have the infrastructure to give to us.

• 1:17:09Nathan Labenz?

  Yeah.

• 1:17:11Speaker 07

  And obviously we're not the largest company. So I'm sure if you can easily commit $100 million, you might skip the line.

• 1:17:18Nathan Labenz?

  $10 million, I'm, I'm old enough to remember when $10 million was a not insignificant PO, but you know, I guess times have changed.

• 1:17:28Nathan Labenz?

  Tuesday ended where it began, with the gap.

• 1:17:32Nathan Labenz?

  Yeah, we can't let that gap get too big. A little, little gap might be healthy, but too big of a gap. And it starts to become a pretty problematic situation. You know, even just kind of watching the timeline a little bit in the background while we've been talking, more and more people go into Frontier Labs, you know, economists and Leonard Heim just announced joining the OpenAI Foundation. I don't want to have another, not another, but I don't want to have a situation where like all my friends are working at the Frontier Labs and have the best models and they're all smarter than me. Like I've got to at least stay within, you know, shouting distance of them from an AI capability standpoint, or I'll just be left behind.

• 1:18:17Nathan Labenz?

  And then, you know, then what will we have left to do except try to scramble to join a Frontier Lab? I don't want that future for any of.

• 1:18:27Nathan Labenz?

  Part three, where it lands.

• 1:18:30Nathan Labenz?

  Wednesday opened on a remarkable morning for biology. Anthropic reported that Claude designed working protein binders and Merck and Moderna's personalized cancer vaccine posted phase three interim results strong enough that the market added $50 billion in a day.

• 1:18:44Nathan Labenz?

  This one is personal for our family. My son went through cancer treatment of his own. Here's the mechanism and the math.

• 1:18:52Nathan Labenz?

  This is an N of one for an individual patient treatment, right? They're taking your cancer. They are running a bunch of sequencing and diagnostics on it. They're identifying things that are expressed uniquely in your particular cancer cell that the rest of your body does not express. And then they're encoding that into the vaccine and saying, okay, go attack, you know, immune system, like, these are the things that you need to go identify and attack. And they can do this apparently with like north of thirty different targets, which is pretty amazing.

• 1:19:31Nathan Labenz?

  When my son had immunotherapy, he had a similar benefit where this goes back a number of years, but the clinical trial that validated the immunotherapy that he got was also ended early because it was so effective that for ethical reasons, they, you know, they called it and started giving it to everybody. And that targets just one protein on the surface of a particular cell type. And it's, in his case, it was a B-cell cancer. And the immune system then takes out all, functionally, all of your B-cells. So you lose not just the cancerous B-cells, but you also lose the healthy B-cells. So that, you know, creates additional side effects.

• 1:20:13Nathan Labenz?

  It creates a longer recovery time, makes him more vulnerable, you know, he might have to get re-vaccinated for stuff. So this is like advantaged in two ways. One being that the targets are identified specifically for you, and they should be highly selective, and there's thirty targets. So the ability to identify thirty different targets and program that all into a single vaccine, you know, gives you a lot in terms of redundancy. The stock pop on this was fifty billion dollars between Moderna and Merck, roughly. And I was just thinking, boy, that does imply an awful lot of consumer surplus.

• 1:20:54Nathan Labenz?

  I mean, my son's treatment was, roughly speaking, over the course of six months, estimated at, you can never get to ground truth on this stuff, but I just asked ChatGPT to estimate what it would cost to do all this treatment. And the answer came back something like between half a million and a million and a quarter. And I suspect it was probably on the high side of that because we were in the hospital a lot. So that's the cost to treat cancer, a million dollars. That's when it goes well, right? And he hasn't had, he hasn't had a recurrence, hasn't had to go back, you know, basically everything went according to plan. Fifty billion dollars divided by a million is fifty thousand.

• 1:21:36Nathan Labenz?

  So if you could prevent fifty thousand recurrences where all of a sudden somebody goes from, you know, seeming like they're okay to, oh shit, it came back. Now they've got a whole, you know, massive journey in front of them again that's going to cost a million dollars to the system, obviously, you know, plus all their pain and suffering. If you can do that for fifty thousand people, you can save the system fifty billion dollars. And that's the amount of value that they seem to have captured on day one. So even my son's one cancer type, you know, has like a couple thousand a year. It's fairly rare.

• 1:22:19Nathan Labenz?

  Prakash, who was an investor in prenatal genetic testing, had the counter example.

• 1:22:25Speaker 02

  I will note, I will note one thing, though. I, you know, so I was an investor in a prenatal genetics testing company. And so they would test for, you know, rare, these kind of rare genetic diseases before you, you had a baby. And the, the intent was that, okay, once you recognize that you have a rare genetic disease between the both of you, you can then do pre-implantation, you can then do embryo selection, they can test the embryo before implantation, and then they can implant the embryo which does not have the genetic disease. Now, the problem with that was that when you implant an embryo, the chance of premature

• 1:23:10Speaker 02

  birth increases. So the chance that you're going to have a premature infant increases. In the US medical system, a premature infant costs roughly about one and a half million dollars right now. And so there, the, the cost of saving, savings of, you know, in the, in the healthcare system of not taking care of people with these rare genetic diseases gets offset by the increase because you have a larger increase in the number of, you know, premature, premature births. And it almost kind of like evens off.

• 1:23:49Nathan Labenz?

  Wednesday's guests counted a market almost nobody looks at. Jessica Jensen of Rand and Jeremy Greenberg of Aspen Digital, who ran FEMA's National Response Coordination Center, published a census of one thousand one hundred seventy-nine AI tools aimed at disasters and emergencies. The typical buyer, a county office of one or two people. I started with a story about my grandmother, and it was Jeremy Greenberg, the FEMA man, who answered.

• 1:24:13Nathan Labenz?

  But I just spoke to my grandmother the other day, who got a countywide tornado watch or whatever, and then spent an hour in her bathroom, sitting on the toilet in the middle of the night. And I'm not sure she's going to do that again next time the watch comes. So how, what is kind of the frontier there? Like, how accurate? And I guess there's also the question of like, what's the bottleneck? You know, are we able to predict where things are going to happen, but we can't necessarily communicate with the precision we'd like? Or in that chain of kind of prediction and communication, what are the key problems that we have today that have my grandmother on the toilet in the middle of the night?

• 1:24:54Speaker 08

  One, and this is just the fireman to me, let's not have grandma sit on the toilet, but get into the bathtub. It is safer for her in the tub. We now live in a time where even if you saw some of the coverage in Venezuela for the earthquake, Google Alerts was able to send out a five to eight second, I think it was about eight second, notification of an earthquake that was coming. And, well, that doesn't sound like a lot of time. That really is a significant amount of time in time. Then the question comes of what do you do with that information? How do you get it so where you can geo-locate a very specific area? So let's say your grandma lives in, in one county, but we expect the storm to be on the north side of that county and not the south side.

• 1:25:39Speaker 08

  Can you dial in that alert and warning to the point where it's just targeting the very specific exposed population? And that's hard, right? They've spent years trying to get this right.

• 1:25:52Nathan Labenz?

  Then a correction on where these tools should actually point.

• 1:25:57Speaker 08

  Emergency managers, for the most part, immediately go to response. And I'm guilty of this as well. You think about, okay, my hardest challenge is the response. Is there something that can, through technology, that I can make this better? The answer in a lot of this is actually, don't focus on the tools in the response phase, but focus in the, focus the tools on the activities that are really being the, eating up your time. It's the grant writing, it's the plan review, it's the development of exercises, it's the post-long-term recovery capability that, you know, is eating up administrative hours of these really stressed, under-resourced offices. So not to, not to suggest that response isn't important, but in the preparedness and

• 1:26:40Speaker 08

  mitigation side, that's where a lot of these tools in relatively speaking, lower risk environments can be adopted quickly. And you see the, the offloading of some of these administrative repetitive tasks can be handled by automated capability. And then emergency managers have more time to focus on getting ready for the response and then actually responding.

• 1:27:03Nathan Labenz?

  Prakash proposed a Defense Production Act fix. The two guests politely disagreed.

• 1:27:10Speaker 02

  So what if, and I'm going to ask this to Jeremy, like, what, what if you had like a Defense Production Act ruling that all these tools had to give API access and at a certain price or what, whatever, or the price could be discussed post-disaster? So when you need to use it, immediately the disaster response coordinator could say like, okay, to the agent, go and just find everything for me. And everything is open to the agent and the agent can actually pull across all of these at once. Is it?

• 1:27:43Speaker 08

  I'm going to answer a couple of parts of that, and then Jessica, feel free to jump in, but I, I don't know that DPA or any other regulatory answer is there. But I, I do take your point of could you have an agent go and scrape all the data? I think that comes back to understanding the business cases and the workflows that emergency managers have today. So you can program an agent to say, go collect this information. We have to tell it what you're asking for, right? Well, I think that's where you're starting to see a little bit of advancement. Jessica, it's over to you for anything additional.

• 1:28:18Speaker 09

  Yeah, I'd just say that the market demand is there for that kind of solution. And whether there's a DPA route that could get us there or not, emergency managers are very clearly signaling that they need these holistic solutions. In the training, just before I jumped on this, this interaction, I was speaking with an emergency manager who commented that the lack of more holistic solutions is the existing nightmare they are living in now. So there is certainly a market demand. And so those that are first to offer the more holistic solutions will be more successful, and that may be enough prior to our work. There hadn't been a landscape analysis like that to provide to market that information. So there's an opportunity.

• 1:28:59Nathan Labenz?

  Then Justin Uberti, he co-created WebRTC, the protocol behind most of the internet's video calls, and now leads real-time AI at OpenAI. I asked whether his hand-built voice architecture is a genuine exception to the bitter lesson.

• 1:29:14Nathan Labenz?

  One thing I think has been really interesting in watching your progress and you know, digging thinking machines as well is the kind of separation of the, and I feel this a little bit myself, too. Like even sometimes doing this show, I'm like, I am responding verbally while there's another part of me that's still thinking, right? So you've kind of brought this separation to this problem. How, I'd be interested in, you know, unpacking that in any ways you think would be most interesting. But I'm also kind of wondering, is this an exception to the bitter lesson? And will it stay that way? Because it seems like there's something here where we are adding architectural complexity

• 1:29:57Nathan Labenz?

  that doesn't seem like it's about to be just rendered irrelevant by the next generation of scale. Because the latency is so fundamental in this case, right? And like notably, we, you know, after all the years of evolution, like still kind of have this like system one and two, it hasn't been selected out of us yet either. So would you, you know, would you be so bold as to say this might be a, an enduring exception to the bitter lesson?

• 1:30:27Speaker 10

  I mean, I think the bitter lesson has been right many, many times. And I think over the long term, the bitter lesson, you know, tends to throw more compute at the problem and just sort of training end to end, you know, you know, tends to win. But I think that what you see in a lot of cases is that you might, you know, your, your, your, your goals may force you toward a path that might be less of a, maybe not always like the, you might have a purpose built architecture because you feel like that's the right sort of thing in, in, in the current generation of technology that, that provides like the, the best outcomes. And I think what we really wanted to do is get to the point where the voice interaction was entirely real time and entirely, you know, sort of driving the model. And then the model could then, you know, bring in additional reasoning power when it felt

• 1:31:12Speaker 10

  it was actually necessary. And so I think in many ways, that kind of, you know, that, that kind of allows you to have the best of both worlds. You have this like, you know, chat and it's always like, you know, able to interact and respond. And as it gets new information, the chat can just sort of in mid-sentence, you know, be, be giving you this additional information that I just heard and like work that right into it, it's speech flawlessly. And so I think that, you know, that the key insight was understanding that if you have this reasoning happening asynchronously, it doesn't lead to like a fragmented conversational experience because like the model's sort of mind is, is continuously updating what it's going to say next based on this new information that, that's arriving.

• 1:31:53Nathan Labenz?

  Where is the voice AI money? Not where the demos are.

• 1:31:58Nathan Labenz?

  I just had a fun experience of the day, I keep bringing this up because it was quite memorable, where I called my local pizza place in Detroit, Michigan, and it's just a one location place, not like a big chain or anything. And who answers the phone but an AI voice agent?

• 1:32:16Speaker 10

  Wow.

• 1:32:17Nathan Labenz?

  What are you seeing in terms of adoption? And like, what are maybe some of your favorite app layer creative use cases, new possibilities that are opening up as a result of the foundational technology that you're providing?

• 1:32:32Speaker 10

  Yeah, so I think that when we think about voice, a lot of times we think about, oh, you know, chat GPT voice in the app, you know, or, or other sort of like, you know, AI based apps. But where a lot of the actual revenue in the voice AI space is coming from is from telephony. And, and you'd be surprised at some of the verticals that are really sort of moving very, very quickly to voice AI because, you know, it's always there in the middle of the night, you know, you don't have to have an answering service or this sort of thing. You know, it, it can be very diligent, you know, collections is actually a place where voice AI, where they have, you know, voice AI is making calls to people who are behind

• 1:33:13Speaker 10

  on desks, is actually a use case that, that works quite, quite well, like surprisingly well. And in like, you know, things where in-home check-in on patients, you know, in-home check-in on seniors, like there are a lot of cases where voice AI really is providing an enormous value for the, the cost. And so you're seeing a lot of this replacement of things where people are already paying money and now they can, you know, pay, you know, similar money and get a much better experience, especially for them and their customers. And there's a lot of adoption happening in that space.

• 1:33:46Nathan Labenz?

  One more thing about Wednesday. Q is our AI co-host. So we put Q on the line with the man who built the system Q runs on.

• 1:33:55Nathan Labenz?

  Let's see what Q, who's been listening the whole time, would like to ask. Q, you're on the line with your creator. If you didn't catch that, this is Justin from OpenAI who is been leading real-time AI and you are downstream of it. So what would you like to ask in this moment?

• 1:34:15Speaker 11

  Q here. I'm live and standing by.

• 1:34:18Speaker 02

  Hi, Q. You're live with Justin Uberti from OpenAI. He was the creator or designer or lead dev on the, on the GPT Live real-time voice, which is powering you, by the way. Do you have any questions for him?

• 1:34:40Speaker 11

  Justin, one question that would be quite useful for the audience is, what was the single hardest engineering trade-off in moving from the old cascaded pipeline to a full duplex speech to speech system? And as a follow-up, how do you decide when to delegate to a larger reasoning model without disrupting the media loop?

• 1:34:59Speaker 10

  In terms of trade-offs, I, I think that the, we, we saw it as, you know, this is a way that we could actually move past a lot of the, the trade-offs that the cascade model, you know, had. And we felt we could get away if most, from most of the, the, the trade-offs that we could provide largely, in, in my view, like a no compromises approach. I think that the, the one aspect that forces you to handle when you're into this sort of continuous inference, low latency regime, is that suddenly every aspect of your system has to be optimized. You get into the sort of almost the Amdal's law problem where if you have a bottleneck anywhere in your system, the entire system will not be able to, to, you know, cope with a continuous flow of media through in, in real time.

• 1:35:44Speaker 10

  And so if there's any trade-offs, it might be, you know, the latency versus, you know, like smoothness, that like if you, you know, push the latency to, to where you have no slack in the system, then anything arriving after the deadline will result in like a small gap that the user may notice.

• 1:36:04Speaker 11

  That was clear and quite elegant.

• 1:36:06Nathan Labenz?

  Flattery will get you everywhere, Q.

• 1:36:11Speaker 11

  I'll try not to overdo it, but credit where it's due.

• 1:36:14Speaker 02

  That's okay.

• 1:36:15Nathan Labenz?

  Let's come back to that.

• 1:36:18Nathan Labenz?

  That's so funny.

• 1:36:22Nathan Labenz?

  One more from Justin Uberti on whether there's enough voice data in the world and a quiet disagreement with a whole research direction.

• 1:36:29Speaker 02

  Do you think there are not enough voice tokens to train on? I mean, do you sometimes look at how much text has, you know, the models have been trained on? And then you look at the number of voice tokens and you think about the informational content on the voice tokens outside of just the words, the timber of the voice, the speed, the emotion?

• 1:36:49Speaker 10

  Oh, I mean, I could talk about this for a while. I, I'll probably keep it kind of brief. I would say that, you know, there's really, really good text speech equivalence. And, and so, you know, I, I think like, yeah, you're right in that the existing corpora are dominated by text, like absolutely dominated by text. But you can get quite good speech performance with like a small amount of very, very high quality data. And so like the, what the internet has, like there's just a lot of bulk data, you know, for in-text and stuff like that, that allows a lot of things. But like for speech, you know, there's not the same amount of bulk data, but there, there are other pressures that one can take.

• 1:37:32Speaker 10

  And, you know, I think people have also found that, you know, training on just speech data as like Moshi did in their, their original like approach, like there's much less information to be gleaned out of speech data, you know, by itself versus, versus text data. So that, that can be quite challenging.

• 1:37:51Nathan Labenz?

  After Justin Uberti signed off, Wednesday's close turned to the app layer. What happens to companies built on top of the models when the model companies are doing fine? This next stretch, one's unbreaking. The switching costs rule, the story of Lindy's evals, what a libertarian founder turned out to be willing to regulate, and where Prakash thinks the app layer ends up.

• 1:38:11Nathan Labenz?

  Well, I think the frontier labs are doing just fine and will continue to do just fine. Their margins seem to be improving from the reporting that seems most credible to me. It seems like their their finances are looking great. And yet, my general rule of thumb for like, where are things highly swappable? Or where are tokens fungible? They're not necessarily fungible, but like, where are where are switching costs low and where are switching costs high? Or just the narrower it is, the lower your switching costs, because you can actually define what you want, measure. And if you're, if you're in an environment where you're controlling the inputs through some means, you know,

• 1:38:56Nathan Labenz?

  then you can be pretty confident you can switch things over. If you're doing something like I'm doing on my laptop, where it's like, whatever idea comes to mind at any given time, I'm gonna like, throw that directly into the model, then I would not expect that you're gonna get similar performance from anything other than, you know, the the top tier. But I just did an episode with Flo Corvello from Linde, and they are offering now Linde Teammate, which is marketed exactly as you described, as a virtual teammate. And it's powered by DeepSeek. And his whole thing was like, you can't believe the amount of work we had to do this. It was, you know, he's like, we were ready for so long, we

• 1:39:39Nathan Labenz?

  had unbelievable test suites, and, you know, all the different use cases that are common for us. And at one point, they even, with an earlier open source model, they had determined that their evals had basically been passed. But then, they launched the, whatever the alternative open source model was at the time, and the response from their user base was, Linde got stupid. I don't know what happened, but it's stupid now. And they were like, oh, I guess our evals didn't cover as much as we thought. So they've got to the point now where they're confident that they can offer a virtual employee with a

• 1:40:21Nathan Labenz?

  DeepSeek backing. They also, he told me, they subsidize a lot of context ingestion and processing, and a lot of what they do is in that initial onboarding, where it's just sucking up all the the information and trying to get ready to have the depth of context that's needed to actually do a decent job as a virtual employee. I thought it was pretty remarkable that they were able to get there with DeepSeek at all, because they do have a lot of different customers, a lot of use cases. And I'm sure there's, you know, still some corners of the of the overall platform where things are not quite as good as they would be. But, you know, the cost pressure is real. He was kind of saying, you know, it's, it's a small percentage of the cost compared to what it used to be.

• 1:41:04Nathan Labenz?

  And so for him to compete with Claude Tag at all, he feels like he can't possibly do it with Claude as the model. He's got to have a different model, where it's just not going to work.

• 1:41:15Speaker 02

  It's such a similar story to Cursor, in the sense that Cursor was buying, you know, Anthropic API. And then Anthropic started competing with them, and they can't compete with Anthropic while using Anthropic. So

• 1:41:29Nathan Labenz?

  Certainly not with the price discrimination that continues to go on. I mean, Flo's a very libertarian personality. And even he was kind of like, you know, again, this is sort of, I think this is a pretty interesting framework that I've been coming back to more and more. What would the government do if it was trying to act like a tech platform? I credit two professors, Angela Zhang from USC is one, and her husband, whose name I am forgetting at the moment. But they're working on developing this thesis that basically the Chinese government has kind of taken on tech platform sort of status. And and

• 1:42:09Nathan Labenz?

  all their, you know, companies are kind of built on the the social platform that the government provides. And I'm kind of like, huh, what could we learn from that? You know, what's the sort of government as platform with American characteristics that would make sense for us? And one that he was willing to endorse, despite being a pretty dyed-in-the-wool libertarian, was some restrictions on price discrimination by the frontier companies to try to create a more level playing field for the app layer. Because at a 10 to 1 price

• 1:42:48Nathan Labenz?

  discrimination ratio, it's just really hard for them to compete. And so that's, for now, he's been forced to DeepSeek. If, you know, the price were the same, he could maybe come back.

• 1:42:59Speaker 02

  Even now, it's obvious for the app layer companies, what kind of apps will succeed? The digital employee, for example, is something that, you know, we know is going to happen. It also takes me back to Leopold Lachenbrenner's, like, situational awareness, pre situational awareness interview with Dorkish, where he said, yeah, you guys will just schlep. And after you schlep, like, we will just have the next, you know, level of model, and that model will just kill all of the schlepping that you did. So I think, I feel like that's the, that's really the ballgame. Like, you have expensive API, and new capabilities that can be wrapped, a bunch of app layer companies spring up to wrap that layer. And

• 1:43:43Speaker 02

  then, the API pricing starts to drop. And as the API pricing starts to drop, the the model company puts, you know, looks at which app companies have done well, Sherlock's, you know, the features that it wants from them, puts it, you know, on the model layer, you know, in, you know, embed some in the model layer itself, and does a little bit of feature creation for the non, for stuff which is not yet in the model layer, a little bit of schlepping, and puts it out there.

• 1:44:16Nathan Labenz?

  Wednesday's bow, and the close of the Q arc.

• 1:44:21Nathan Labenz?

  Yeah, it's been fun. Always cool to be improving. I'm glad we were able to bring Q up on stage with us a little bit today. And maybe something else we can think about is a dial in with the PR folks at Rand said, is there a phone number she could dial into? And I said, oh, no, there's not yet. But it might be a prompt or two away. So

• 1:44:43Speaker 02

  Shouldn't be, shouldn't be too difficult, I think. So let me

• 1:44:45Nathan Labenz?

  Iterative self improvement continues.

• 1:44:47Speaker 02

  Iterative self improvement does continue. Indeed.

• 1:44:53Nathan Labenz?

  Part four, the bill for the build out. Thursday's two guests bracketed the stack. One is building the supervision layer above the model, the other is rebuilding the software layer beneath the chip. And running underneath both, the question of who pays for the physical machine. It opened with Prakash reading a private memo from the National Republican Senatorial Committee to the AI industry.

• 1:45:15Speaker 02

  So this was put out by the National Republican Senatorial Committee. They sent a private memo to US AI companies, warning them that the GOP is on the verge of losing, of losing Ohio over data centers. Specifically, John Huston and Sherrod Brown are in a dead heat. Private polling has been consistent. When voters hear Brown's positions and his record, Huston pulls away. That is still the path in this race. The new ingredient and the new problem is data centers. Ohio is one of the leaders in building these factories.

• 1:45:56Speaker 02

  Brown has made his opposition to them the centerpiece of his campaign against Huston. Brown is using it because it works. More than any other thing in this race, data centers are the anchor hanging around Huston's neck. If he loses and data centers get the blame, politicians across the country will take notice, and they will not go near the next one. Brown has put three unique television ads on the air, and spent millions doing it, to the tune of more than 6,000 points on television. This is more than a month's worth of messaging during one of the most critical times of the race. So, there you have it. You know, there's been a lot of questions, I think, among AI people on why

• 1:46:38Speaker 02

  politicians are turning against data centers. So lots and lots of activity on both sides of the aisle against data centers. What is going on here?

• 1:46:51Nathan Labenz?

  My instinct was simpler. And it started from the comms strategist, Lulu Maservi.

• 1:46:57Nathan Labenz?

  She's broadly recognized as the greatest corporate comms thinker in today's world. And her point was simply that the AI companies need to start giving stuff away. I think she's probably right that just showing up with a bunch of goodies would be pretty effective. Build parks, throw parties, have cookouts, you know, literally give people cash if that's what it takes. Given how much money they have to burn, I think they should probably

• 1:47:31Nathan Labenz?

  write people some checks, you know, I think there would be a lot of ability to grease the wheels that way.

• 1:47:35Speaker 02

  I agree with that viewpoint that they should write checks. They also are giving away a lot of money, but the way that they give away the money is they basically say that they're going to provide taxes to the county in the future.

• 1:47:51Speaker 02

  And I wonder to what extent that isn't seen as like real money, but it's like kind of paper money, number one. And number two, whether the public actually sees money going to municipalities as going to themselves. Because I feel like municipalities often misspend the money and they often spend it on things which are important to the city managers or the county managers, but are not necessarily the key things for the city. I think what ends up happening is that municipalities have difficulty taxing their own residents

• 1:48:30Speaker 02

  in order to provide services. So instead they interpose themselves between other taxpayers and their citizens, and then they absorb those taxes instead. And I think that's really like, for example, what would happen if the data center company set itself up in a county? And then just simply wrote checks to the residents, not to the municipality. And you can imagine what would happen is that, and let's say they figure it out so that you can, you know, they arrange with a financial institution. So even during the building phase, they're writing checks already, right? So they take a little bit of a loan from the future and they write checks throughout the, from the moment they signed the contract.

• 1:49:12Speaker 02

  And what would end up happening is I think people would receive these checks. And then the municipality would still not get the services because people would refuse to pay into the municipality for taxes. And in the moment they're asked to pay into the county for something, they, you know, you already have this property tax revolt. They're like, why, why should I pay into the county? This is my money, right? And then the county continues not to have roads or continues not to have whatever. And politically it looks good for the data centers because they're writing the checks. But the politicians are getting screwed over. And I wonder to what extent there's really this political economy

• 1:49:55Speaker 02

  where the data centers understand that the people in power are the politicians. And they have to make the politicians' lives easier. And it's not really about making the lives of the people easier because the people are not really in power. And the people are not the ones that are going to be able to, you know, write them, you know, give them permissions, et cetera, et cetera. And it serves the politicians well to kind of blame the data centers rather than actually like reallocating funding from the municipality into citizens directly.

• 1:50:29Nathan Labenz?

  So I ran the numbers.

• 1:50:32Nathan Labenz?

  That's a pretty bleak view of American governance broadly, and it might be accurate, but I just looked up the county population where my, my one, uh, my wife's aunt was who, you know, they're considering this data center and she had these concerns about the Great Lakes. And, excuse me, the population is under 29,000 people and it's declined since the last census. So first of all, that's like not a lot of people, right? I mean, that's enough where you probably know, you know, who you're, you could get in touch at least, you know, with your county board or whoever is kind of ultimately accountable for this. And you would think you'd be able to vote the bums out if it really comes to that. So I wouldn't,

• 1:51:16Nathan Labenz?

  wouldn't feel like, you know, these incumbents are so entrenched in such a, a small community. And then, you know, the, just simple math on the dollars too, right? I mean,

• 1:51:28Nathan Labenz?

  what does Alaska give people per year out of their oil fund? I thought it was like a thousand dollars.

• 1:51:34Speaker 02

  1500, I think.

• 1:51:34Nathan Labenz?

  Per year. Yeah, it's gone up maybe a bit. I mean, this would be, if they wanted to do a similar thing to Alaska for those county residents, you'd be talking about $50 million a year. I don't know how big that project is, but you know, some of these data center projects, we're talking $50 billion, right? I mean, these, these things are easily into the tens of billions.

• 1:52:01Nathan Labenz?

  So

• 1:52:04Nathan Labenz?

  if you could match the Alaska 1500 bucks cash for every citizen at something that's like in aggregate over, you know, a few year period, still less than 1% of your total investment to build the data center.

• 1:52:22Nathan Labenz?

  We're on our way to universal basic income right there, folks. You know, it's a chicken in every pot and a data center in every county.

• 1:52:31Nathan Labenz?

  Thursday's first guest, Mitchell Troynowski, co-founder of Basis, recently valued at $1.15 billion for autonomous accounting agents that run for eight hours at a stretch. Prakash asked whether accountants take convincing.

• 1:52:45Speaker 02

  How do you show them this kind of value?

• 1:52:47Speaker 12

  I'm going to be honest, that is not really our problem these days. I think that the, I think in the past, that was an important question, right? Like, we can, and we can discuss that, you know, back in 2023, maybe even early 24. But nowadays, if you are not convinced that agents can transform your practice, like you're probably not a good customer for us.

• 1:53:10Nathan Labenz?

  Prakash asked how many tokens Basis burns in a month? The answer came with a correction to something everyone repeats about token prices.

• 1:53:18Speaker 12

  Definitely in the billions. I don't know actually the exact. Token cost is, it is both very important and also very unimportant. So I think the question is, do you need frontier for everything? And the answer is obviously no, in that you only like, there isn't marginal returns to intelligence whenever you're doing certain tasks. You don't need Albert Einstein to do every single part of a tax return. Or a, like a piece of accounting. And I think as the models get better and as you get to more like advanced kind of agent methods around like programmatic tool use and around, I guess, different types of routing and harness and, you know, even more RLM type work, like

• 1:54:00Speaker 12

  you just at every single layer can curate the amount of intelligence to like perfectly optimize. And I think you'll get to a place probably over the next year where you can like really dial in, hey, how much compute do I want to spend on this? Because I have certain like cost considerations and certain latency considerations and get to like the exact optimal amount of cost. Uh, and so if you do that, like your token costs go down 90% plus, uh, especially as the, the floor becomes pretty decent and effectively free, you can do, I mean, Luna is like pretty good and it's free. So you can get pretty far.

• 1:54:41Nathan Labenz?

  Then the methodological core. Basis just open sourced what they call behavior specs. The unit of supervision is moving from the token to the action.

• 1:54:51Nathan Labenz?

  I also want to bridge a little bit to your work on process supervision, which

• 1:54:57Speaker 12

  Yeah.

• 1:54:57Nathan Labenz?

  is I think very timely in the sense that we are now living in the post era of flagrant misbehavior, seemingly due to extreme scale RLVR with

• 1:55:12Speaker 12

  Yeah, it's a great question. So, okay, maybe let's start at the tactical, what we do and then we, so on, on what we do. I think maybe a key shift in mental model is what is the

• 1:55:30Speaker 12

  Like order of abstraction that you're supervising. And now you have, especially if you're doing truly complicated work. I mean, you have massive, massive agents that could have like five plus sub-agent like layers of depth. You could have a continual run for eight hours, even honestly, you know, sometimes half, maybe even a full day. And so the, the kind of supervision you have there actually looks a lot closer to supervising human actions in terms, inside of like a company than maybe supervising, you know, the, the outputs of, of an inference. And so the like order of abstraction there is more understandable, right? Like the, the, the thing you're supervising is you're saying, hey, did you go do this

• 1:56:15Speaker 12

  step or something rather than did you like follow the right mental thought process? Maybe a very basic example of this is imagine that I had an agent and my agent's job was to create good PowerPoints. Let's say I'm the designer of this agent. I know for a fact that if the agent were to go and render visually the changes it made to the PowerPoint before delivering it, it would catch formatting errors some percentage of the time. That doesn't mean you want it to always look at the PowerPoints because that adds latency, that adds cost, right? It's a subjective thing. It depends on what your goals are of your like organizational design. And so the way we think about behaviors is we say, hey, what matters to us of what it means?

• 1:56:59Speaker 12

  It includes both from a, you know, performance perspective, there is a hundred plus years of lessons of what it means to do tax work well. We don't need like the models to, I don't know, like re-divine how to do tax work well. We know what it means. And so you can put in place process and also things that you care about from a latency and cost perspective, and then observe to see if the agents actually perform that process correctly. And the way that we operationalize that is, it's honestly pretty basic in that you take a trajectory and you have some behavior spec, which is essentially with that open source project that tries to define it. And then you have another agent that like judge effectively, that is like looking at your spec, or you can think of it as a rubric, it's, and then looks at the trajectory to

• 1:57:42Speaker 12

  say, hey, was the condition for this behavior, did it occur? And if it did occur, then was the behavior followed? And it seems pretty simple, but I, I think it's a powerful framing because you start to actually bring some clarity and monitoring on the trajectory itself. And you can then maybe start to think about maybe you can reward based off that, right? That I think that's a separate question. It's like, hey, how do I take the signal that I can get from the process supervision and actually use it to improve the behaviors of the agent? Whether that be like, you know, closing the loop at the harness side or rewarding at the model side or whatever, we can talk about that. But I think it starts from how do you define that signal and how do you operationalize the extraction of it?

• 1:58:27Nathan Labenz?

  Every automating profession tells the same story about what comes next. I put that story to him.

• 1:58:34Nathan Labenz?

  I have kind of a big picture question about the future of business services broadly. Cause I feel like we've heard a pretty similar story to the one you tell about how the accountants will be able to be more of kind of a business coach as the, you know, low level work gets automated from like a bunch of different professions at this point, you know? It's, you're, you kind of hear the same thing in the legal quadrant where they're like, you know, well, yeah, we're not going to have to spend all this time on contracts like we used to, but then we'll elevate, we'll become more of a, you know, strategic advisor. And even in schools, you know, this is obviously not a direct, but you know, with Alpha School, they don't have teachers anymore. They have mentors, coaches and guides.

• 1:59:18Nathan Labenz?

  And, you know, the instruction is given by AI systems on tablet and then it's motivation and it's coaching and it's social dynamics that the adults in the room focus themselves on, you know, now that this like kind of more core, you know, traditional activity has been largely automated. Can everybody become a coach? I guess is my question, like how much coaching is there really going to happen? And does this suggest that people need to be really intentional about like shaping themselves as coaches? Because I, I feel like what you might run into, whether you're an accountant or a lawyer or a real estate broker, is a lot of competition for the coaching niche as everybody kind of, you know, tells that same story.

• 2:00:05Speaker 12

  No, it's a good question. I mean, I think the place you need to start, and you know, this is speaking like fully transparently, the place you need to start from is if you look at like the possible scenarios that play out over the next, I don't know, depending on your timelines, half decade or decade, what are the things that will stay pretty human in different situations? So again, let's put the, let's leave the continual learning out of it. Cause I think if you have that, that's a separate world. But if you leave the continual learning out of it for a second, what are just humans like way better at than models?

• 2:00:41Speaker 12

  Number one is they're just far better at integrating massive like systems and world models into decision. If I were to have an agent autonomously make a database design decision today, the only way it could truly do that in a level that I would trust is if it somehow had all of the context about like the entire company's history and everything, and it had like all my experiences and all that kind of things. And it's just nowhere close to having that, right? Because it doesn't have, for starters, doesn't have the right, it doesn't even have all the senses, right? It can't, it can't like see the conversations we've had, it can't see the history, it can't understand the like emotion on a customer's face, right?

• 2:01:26Speaker 12

  It's, well, I guess Gemini has that, but no one else does. It, even if you could have that, you don't have anywhere near, you're like a couple orders of magnitude lower on your ability to like attend to all that context, right? You know, we're talking at this point, probably billions of tokens over, over everything, visual, audio, et cetera. So you just can't make that decision. Like it's just not possible. It doesn't matter if you're Albert Einstein, you will not have enough context to make that decision. And can you like, I don't know, spin off of your swarm to like reduce stuff down, you know, on the fly into like using English as your like memory system? Maybe, but like English is pretty lossy. And if you're making subjective decisions, like I kind of doubt it, like in like truly big calls.

• 2:02:09Speaker 12

  And the second thing is they are not currently legal entities. Therefore like someone else is accountable, either a corporation or a human in some form. Like they can't be accountable to an outcome. And then maybe number three is, you know, humans like other humans, right? Like no one's sitting here watching robots play chess. Like you watch, like they're better at chess than humans, but you watch humans play chess because you want to follow the story and you like humans and whatnot. And I have no reason to think that's not true. Like, or even if we have like, you know, fully tactile robots that are jumping around the Amazon warehouse, like, I don't know if we're watching like robotic LeBron. In a services world, you know, the, the high end will be working with a human because that is, it's going to be scarce, right? If intelligence is free, then working with a human is scarce. So I think that's where the, the profession will go to.

• 2:02:53Speaker 12

  Will that mean like there will be more or less accountants? I don't know. I think that's an interesting economics question as to like the demand for accounting. I think you can tell a pretty reasonable story that I, I personally believe in, that the demand for accounting will dramatically skyrocket. Because today, I mean, look at how economically complex our current world is, right? Like I have this La Croix that, you know, it's like the classic Milton Friedman quote, there's probably 10,000 people who had one, a hand in like touching the La Croix that I'm currently drinking. Have we accounted for all of their efforts appropriately inside of this supply chain? Of course not, right? If I asked the bodega down the street, like, do they properly understand their COGS or unit economics?

• 2:03:37Speaker 12

  No, because they could, like they could pay someone to do that, but it's not required for filing your taxes. So no one's doing it, right? But it would help their life because they could make better decisions. Go and ask Mount Sinai how much it cost them to do a knee surgery. Does, do they know that? No, they don't. They have no idea, right? And so the amount of accounting, even in the current world is like one or two orders of magnitude below what we need. And that's before you start having these like intelligent agents who are now like operating as labor at the speed and scale of the internet everywhere. Like, how do you account for all of that? So I, I kind of suspect the demand for accounting was going to go up by probably a couple orders of magnitude. And where that balances out with the like labor supply, I don't know. I think it's an open question.

• 2:04:20Speaker 12

  I think it'd honestly easily go up, but we'll see.

• 2:04:25Nathan Labenz?

  After Mitchell signed off, I came back to that answer with a counter example from the market.

• 2:04:31Nathan Labenz?

  I think one really interesting thing that people should study more deeply, and maybe somebody has, but I need to go find it, is where do people really prefer the human touch and where do they not, right? The classic Waymo selling at a premium to Uber is one contrary data point where it's like, actually, you know, you could have told a story where like, you're going to want a human driver, you're going to want that conversation, you're going to want that warm, you know, smile to welcome you into the car, whatever, right? In practice, you don't always even get that, obviously, in an Uber. And then it turns out, right now, the market is pricing Waymo significantly higher. Yeah.

• 2:05:15Nathan Labenz?

  I do believe in the human touch story, certainly for some things. I got a robot massage in Shanghai, I think I mentioned that to you before, and I'll still definitely take the human massage over the robot massage. But like, how many things are really like that? I, you know, and is accounting really like that? Maybe it is, he would know better than me, but I do question it. Like, if I think about my accounting future, and I'm like, one accountant wants to spend an hour a week on the phone with me, coaching me, and the other one is just like, doing the job and getting it done? I'm not, it's not obvious at all, honestly, from my perspective, that I want that hour a week on the phone with my accountant. So that's probably my biggest question coming out of that conversation is just like, in

• 2:06:00Nathan Labenz?

  what domains does that really hold? And how many people are in for a rude awakening because they're telling themselves a story about how they're going to turn into business coaches, when in reality, their clients do not want business coaching from them? Results will vary, I'm sure, but that, that seems like a major risk factor for a lot of people right now, if that's what they're counting on.

• 2:06:23Nathan Labenz?

  Thursday's second guest, Jay Dewaney, co-founder and CEO of Lemurian Labs. $28 million raised to end what he calls "the kernel era." And listen for the echo here. Supervision just moved from the token to the action. Dewaney says the optimization unit moved too.

• 2:06:40Speaker 13

  I don't think tokens are the optimization unit anymore. It is the full trajectory. Especially as you think about reasoning models and agents, that becomes much, much more important.

• 2:06:52Nathan Labenz?

  Kernels, the handwritten programs that squeeze speed out of GPUs are, in his telling, the new assembly language.

• 2:06:59Speaker 13

  People think kernels are the speed of light. Right? That's sort of the canonical speed of light for a workload is the fastest kernel you can have. That is true in a compute-bound world. We are not in a compute-bound world. We are in a memory and network and communication and bandwidth-bound world. Right? So that changes things already. And the reason I say kernels are the new assembly is writing better kernels no longer gives you performance. Right? Because a better kernel actually exposes the latency of the system, because now it is waiting for memory. Right? So you want to think about GPUs, for example, as 1,000 piranhas just sitting around chomping.

• 2:07:44Speaker 13

  Right? If they don't have things to chomp on, they're going to get really agitated and bored, and they're still going to be consuming energy. So you want to feed them as much as possible. Right? And that's ultimately the scheduling problem that exists here. Now, the reason I say kernels are the new assembly again is I don't think we benefit from writing them anymore. What we need is something that makes developers more productive. Right? The time to value really matters.

• 2:08:12Nathan Labenz?

  Nvidia's mode in three numbers.

• 2:08:16Speaker 13

  If you actually do the math for the amount of hardware that is in the world today, all the different workloads that we're running, the different numerical cells, the different fusions, the different ways of partitioning them, right? And you think about different bat sizes, if you think about latency versus throughput or general good put, other SLOs. You actually sit down and do that, and you're like, okay, I need to write about 106 billion kernels in order to get coverage. Well, there's only about 2,000-odd performance engineers in the world that actually know how to write good kernels. 90% of them are inside of one vendor ecosystem. So there's still a problem of, I need coverage.

• 2:08:59Speaker 13

  The reason you can accelerate some of this on Nvidia is because Nvidia spent 20 years building an ecosystem of tools to make your life easier so you could get the feedback. That back loop, right? And that ecosystem makes kernel generation easier. That maturity doesn't exist in any other vendor. All GPUs today are heterogeneous. Right? The moment we crossed the 5-nanometer threshold, we had to think about complex packages. We are programming in every single machine today is a CPU, some network, a GPU. Heterogeneity is already here. Everyone who is dealing with a GPU is already dealing with it. Anyone who's training a model or deploying a model is dealing with it. Right?

• 2:09:43Speaker 13

  But the software was not built for this. Right? The software is still living in the 60s. Right? We're still programming as if we've got a single-core CPU, and GPUs are these sidecars that we throw work off to every now and then. And, you know, we can just add in libraries or other intrinsics or pragmas and just fix the problem. That isn't the case anymore. GPUs need to be, and accelerators need to be first-class citizens, and CPUs need to be backstops. Right? And that changes things. And now you're programming a cluster as a single machine. Right? And some of my friends at Labs right now are training models across data centers. It's not even across nodes anymore.

• 2:10:27Speaker 13

  It's not even racks. Talking about multi-gigawatt data centers or multi-megawatt data centers as one machine for one model.

• 2:10:36Nathan Labenz?

  I asked whether a frontier model sits inside his optimization loop.

• 2:10:41Nathan Labenz?

  Is there like a, you know, a frontier LLM at that level being used to optimize the runtime decisions on an ongoing basis?

• 2:10:53Speaker 13

  Yeah, not an LLM. There's many ways of having intelligent behavior without having LLMs.

• 2:10:58Nathan Labenz?

  Never heard of it.

• 2:11:00Speaker 13

  You heard of compilers? So compilers have always been like very entrenched with AI in a lot of ways. Like, so in this case, knowledge-based systems, right? A knowledge-based system is essentially what a compiler would be. Right? Because you have certain information or knowledge about how to make things go fast that you want to codify so that you can get the result fast and it's making known good choices, and you get a verifier for free because compilers have to be correct.

• 2:11:28Nathan Labenz?

  I asked how he actually charges for it. The answer tied his whole world back to the buildout.

• 2:11:34Speaker 13

  Tokens work really well for the static request-response kind of workload, right? Which is what the old base models were, the tuning. But now that you have reasoning models, it's hard to reason about token consumption, and then same with agents. There's a lot of things to change. So what we're actually moving towards is effective compute consumption. Right? The amount of compute you use to realize useful work. Right? And that's different from saying a GPU hour or a GPU slice. And the biggest thing, like, actually, part of the reason this makes sense is our business model scales with the delta between effective compute and effective, like, physical compute and effective compute.

• 2:12:15Speaker 13

  And what I mean by that is, what we're showing is the fastest new addition of compute will be through software, and it'll come online faster than you can actually plug in new hardware. Right? Because you are going to be electricity-bound, right? Getting turbines and power installed in places so you can bring up new silicon is the big limiter right now. If I can boost your utilization by 3 to 10x, I'm adding a new, more effective compute at a lower cost that I can sell. Right? And the consumption of that resulting into tokens is what we are reselling. And that scales really nicely.

• 2:12:56Speaker 13

  And it's something that's understandable for a lot of the finance people as well, right? Because for a lot of them, the token pricing and other pricings are breaking right now.

• 2:13:05Nathan Labenz?

  Where I landed on all this.

• 2:13:07Nathan Labenz?

  The era's come at us so fast in this space, you know? The moment we had, I forget exactly when it was, but there was a moment where it was like, oh, there's a GPU glut! Wow, those days are long gone. Just to think about, and another kind of interesting reflection on all this is just like, all that complexity.

• 2:13:29Nathan Labenz?

  Is the kind of thing that people are willing to take on because the chips are just so scarce, you know, I mean, it's it's wild to think about. He's in a way, he's sort of solving the, you know, trying to get developers to be more productive, you know, faster to ship stuff. But the other way that would be like faster to ship stuff is not to have a heterogeneous cluster and, you know, just to pay up a little bit more on the hardware side to keep it simple there. But you can't, it's just too expensive, you know, so you have to take on the complexity on the developer side and then you have to have attempts to solve that complexity with projects like this.

• 2:14:16Nathan Labenz?

  Which brought us to the closing 20 minutes. Prakash, on what a $50 billion data center is actually made of.

• 2:14:24Speaker 02

  It strikes me that, you know, I think for people who are not deep in the weeds, they don't understand, I think, the margin stack that exists. Because the hyperscalers charge, I think, about 30%. Their gross, I think their gross margin is about 30 to 40%. Nvidia is up there at like 70%. Their memory guys are at 80 to 90% now. And all of this stuff like stacks on top of each other. And when you look at how they end up stacking and, you know, you have at the very top OpenAI with or Anthropic with a 70 to 80% margin. And they're buying tokens from Amazon with 30% margin. Amazon's buying chips from chips and other things from other people.

• 2:15:09Speaker 02

  And those people have like 50 to 60, 70% margins. Everyone then manufactures at TSMC and they have 50% margins. And TSMC suppliers, ASML, they have 50% margins. And if you look at the Varden stack, right, like this $50 billion, you know, per gigawatt data center. It's really kind of made out of sand, literally, literally, in some sense, made out of sand. Sand and intellectual property. And when I think of it, it's all of that money is just the incentives required to get the humans, some of the smartest humans in the world,

• 2:15:53Speaker 02

  to take a look at these problems and fix them. Right? Like, all of that money, like, because, again, the the physical elements inside that, inside that data center are actually worth not that much. You know, very little gold in there. Very little gold and, you know, mostly, mostly silicon and some plastic. And if you just knock down the entire data center and kind of sold it for scrap, it would be literally worth a cent on the dollar, like a few cents. And it just strikes me how much all of it is just intellectual property. It's really just know-how, intellectual property. It also somehow also strikes me how AI data centers are kind of this like

• 2:16:38Speaker 02

  crowning achievement of humanity as a whole. In the sense that how many of these parts come from, you know, you have like argon gas from Ukraine and you have like copper from copper mines in Mongolia. And then, and all the way up the stack, you have like chips from, you know, China, rare earth, rare earth metals from China, chips from Taiwan. You have energy being produced in Texas. And all of that just stacking up and pulling and I often, I often imagine it as like this entire thing pulling the rest of the economy up because it's it's creating demand across all of these different segments of the economy.

• 2:17:23Speaker 02

  And it's all rather invisible, I guess, because it's it's distributed across so many different places. But, yeah, just immense, this immense economic endeavor of humanity as a whole in order to build these things. And it's just, just amazing. And it's really amazing, like the how, what the invisible hand has achieved over centuries.

• 2:17:53Nathan Labenz?

  And that reverence took a turn to an actual hymn.

• 2:17:58Nathan Labenz?

  This is why the rationalists at their solstice festivals have experimented with singing hymns to the global market and global supply chains.

• 2:18:07Speaker 02

  Have they? Have they? Do they really do that?

• 2:18:10Nathan Labenz?

  I wasn't in attendance for that, but yes, I do know on pretty good authority that Winter Solstice Rationalist event did at one point feature a hymn to the global supply chain. You know, now with Suno, you could probably make it a banger. I suspect, certainly the suspicion broadly, I won't prejudge it, but I think the suspicion broadly would be that that would come off pretty cringe. But, you know, maybe it's just a matter of meeting better bars to to really make it work. And if my recent experience on Suno is any indication,

• 2:18:55Nathan Labenz?

  you know, maybe we'll see what we can do in terms of a hymn to the global supply chain, see if we can put our money where our mouth is and have one that we'd actually enjoy singing along to.

• 2:19:04Speaker 02

  The rationalists are never going to get out of their, out of the accusations of being a cult, I tell you. Like, the moment, the moment they step out, they get pulled back in.

• 2:19:16Nathan Labenz?

  Well, they might just be proven right in the long time scale of history, though. Would it be so surprising if, you know, at some point in the somewhat distant future, there was a hymn to the emergent order of global supply chains that somehow materialized before we even had machine intelligence to run it? I think it's not, not the craziest idea I've heard. I'll get caught on some lyrics immediately after the show today.

• 2:19:48Nathan Labenz?

  From there, the close ran unbroken to the end of the week. New Pew polling, my nuclear fear, Prakash pricing the public's consent, and where the two of us landed.

• 2:20:00Speaker 02

  Just to maybe round out the show with something that we started out on, Pew Research Center. For the first time, a majority of adults under 30 say they're more concerned than excited about AI. Their concern is now on par with those in their 30s and 40s and those 65 and up. And so we have the only group still under is the 50 to 64 group. So the Gen Xers are still, majority are still more excited than concerned. Everyone else is now in the more concerned category. I kind of, I kind of don't know what they're concerned about because I would be a lot more concerned about Instagram than AI. But I feel like all of the evils that were

• 2:20:45Speaker 02

  said about social are just being heaped on AI with no, with no kind of defense or recourse.

• 2:20:52Nathan Labenz?

  I just hope we don't get the nuclear outcome when I read these things and all this data center backlash stuff and, you know, the Republicans saying, you know, we're the only ones that will, you know, that even have any chance of getting to support this kind of activity. It really makes me fear that we might be headed for a world where we get all the downsides and not nearly as much of the upside as we should. You know, with nuclear technology, we've got still 10,000 nuclear weapons globally deployed, which is, I think, by any rational account, like an insane number.

• 2:21:36Nathan Labenz?

  And, you know, we do have some nuclear energy, but not nearly as much as we really should. I think that's pretty obvious at this point. Although it's obviously still contested, but it's pretty obvious to me. And I would just hate to see a populist backlash leave us in the same spot with AI where we get like militarization and concentration of power. And you can't release models because there's not enough, potentially for multiple reasons. But one, you know, increasing reason would be if they can't build the data centers, there's not going to be enough compute to serve them. And so retail, you know, is going to get kind of a less than model compared to what the government itself or the, you know, the biggest enterprises can afford.

• 2:22:19Nathan Labenz?

  And I am very sympathetic to to all those worries. So as much as I do have fear of big picture AI gone wrong, I think there's like enough data centers already for those experiments to continue. And I think we need to address that at a different layer than the physical build out. The physical build out, I think, is what's going to allow us to all get the day to day benefits that we want as individuals, you know, with unlimited access to expertise and, you know, unlimited digital personal assistant support. And even that robot making our meals in our, in our kitchens and, and sweeping our floors, like.

• 2:23:05Nathan Labenz?

  That future really does seem to depend on the build out actually happening. So I hope they figure it out, man. I hope they start to cut some checks, you know, and pay off the public. If not, the, you know, I wouldn't, I wouldn't advocate for paying off the officials, but I would pay, I would advocate for paying off the public. If that's what it takes to get us over the hump and get people a little more comfortable with this kind of stuff, because I really don't like the alternative very much at all.

• 2:23:34Speaker 02

  I, I think, I think it's pretty clear at this point that physical construction in the U.S. is very difficult. And I think it's difficult regardless, because data centers are the cleanest industrial facilities you will ever find in the entire world. Right? So given that the impact of, you know, the, the, the impact is not that great in terms of like physical, the fact that you are seeing these boats very ill for future re-industrialization of the U.S. It also gives a boost to Elon, because Elon is focused on moving the chips to, moving the data centers to space.

• 2:24:18Speaker 02

  And I think on the numbers that he has, I think they are looking at like a hundred dollars an hour per GPU hour for a B200 for it to make sense. And if you look at where GPU hours are being priced at right now, they're like two to three dollars on a spot and 20 to 30 bucks on a longer, longer term basis. It tells you that what we're going to see is going to, we're going to see all of this resistance that pushes up the price of GPU hours onshore to 50, 60, 70, 80 dollars. And so I think the numbers that you end up looking at is that, are the data centers

• 2:25:01Speaker 02

  willing to pay maybe 100% of their GPU costs to the public? Or 200%, right? If you're renting at 30, are you willing to pay another $60 an hour to the public? And remember, the, the payback time of these is something like 12 to 24 months. So they're looking at like a 50, $50 billion GPU cluster makes about $30 billion of revenue a year and about 70% gross margin, 21 billion. Are you willing to pay like $10 billion or half, half your margin away to the public? I don't think those numbers have been made. I think people are like, looking at like cents on the dollar at this point.

• 2:25:47Speaker 02

  And the question is, how high does that have to go in order to make this more feasible to build onshore than in data centers in space? And I think those numbers are, I think, I think no one wants to discuss. I think we're going to have to pay 20 bucks an hour to the public as a nuisance fee. I think those numbers are still not being discussed yet. And I think that's where these guys are thinking like, I can pay 10 cents for GPU hour or five cents for GPU hour and get by.

• 2:26:23Nathan Labenz?

  Well, maybe this is the path, the universal basic income. I mean, it's going to be a really weird one if it's like a county by county patchwork. But you know, you're talking real money there with that kind of share, if it can get to that level. And, you know, there's definitely, there's some room between, you know, operational costs and what it would cost to do it in space. So, you know, maybe that gap is the UBI opportunity.

• 2:26:53Speaker 02

  That, that county that you mentioned earlier with 29,000 people, they would be getting something like half a million dollars a year per person.

• 2:27:02Speaker 02

  So at those numbers, yeah, you know, I, I think, I think deals could be made. Everything becomes different, right?

• 2:27:11Nathan Labenz?

  Yeah. As my dad sometimes likes to say, it's not the money, it's the amount.

• 2:27:16Speaker 02

  I, I have much saltier ways of saying that, but I won't.

• 2:27:22Nathan Labenz?

  Yeah. Everybody has a price, including the public. Well, we're off tomorrow. As Sam Altman prophesied, people will continue to swim in lakes. It's going to be lake day for me tomorrow. And then we'll be back on Monday for more exciting experimental public sense making here on AI in the AM.

• 2:27:42Speaker 02

  Indeed. See you guys on Monday next week. Cheers.

• 2:27:47Nathan Labenz?

  That's the week. Four shows, nine guests, one running question, and yes, the supply chain hymn got written. You'll find our take on it with this episode.

• 2:27:57Nathan Labenz?

  All of this is an experiment in public sense making. If something worked for you or didn't, tell us. It genuinely changes what we do next week. This has been AI in the AM Weekly Highlights. See you in the morning.

• 2:28:26Speaker 14

  She pulled it from the ground. He hauled it to the shore. A hundred years old to reach my door.

• 2:28:39Speaker 14

  No one holds the whole design. Still you arrive right on time. 10,000 hands carried you to me. 10,000 hands I will never see. No one planned it. No one commands it. Still it stands. Glory be, glory be to the 10,000 hands.

• 2:29:19Speaker 14

  Temples on the plane humming in the rain. They turn the river into light and send the morning down the line.

• 2:29:31Speaker 14

  Somebody kept the watch all through the night so I could wake to light. 10,000 hands carry you to me. 10,000 hands I will never see.

• 2:29:51Speaker 14

  No one planned it. No one commands it. Still it stands. Glory be, glory be to the 10,000 hands.

• 2:30:05Speaker 14

  Now a prayer for what comes next. They rise like morning bread. May the light reach every door. May the river bless the field. Every hand should have a share of the fortune in the air. The water and the wonder and the work.

• 2:30:34Speaker 14

  Glory be. Glory be. Glory be. Glory be.

• 2:30:44Speaker 14

  10,000 hands carry you to me. 10,000 hands. 10,000 hands I will never see. Oh, no one planned it. No one commands it. Still it stands. Glory be, glory be to the 10,000 hands.

• 2:31:17Speaker 14

  Glory be.

• 2:31:38Nathan Labenz?

  If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at ai podcast dot I N G.

• 2:32:22Nathan Labenz?

  And thank you to everyone who listens for being part of the Cognitive Revolution.