AI:AM #4: Cameron on Model Consciousness, Duvenaud's Gradual Disempowerment, swyx's AI-Eng Alpha

This AI:AM highlights cut brings together Cameron Berg, David Duvenaud, Michiel Bakker, Shawn “swyx” Wang, and Bing Xu to examine what we understand about frontier AI systems and what happens as more decisions move into their hands. Berg grounds model-consciousness debates in experiments on architecture, agency, valence, and welfare, while Duvenaud argues that even well-aligned AI could gradually disempower humans through ordinary economic choices. Bakker frames Europe’s AI challenge as a sovereignty problem, and swyx turns to practitioner stakes around agents, evals, maintainable code, and who owns the system of record. Xu closes the loop at the infrastructure layer, arguing that self-improving compute and GPU-kernel automation may deepen rather than weaken the CUDA moat. For full show notes, links, and references, read the episode page:https://www.cognitiverevolution.ai/ai-am-4-cameron-on-model-consciousness-duvenaud-s-gradual-disempowerment-swyx-s-ai-eng-alpha/ Mercury: Command is Mercury’s new conversational interface, giving you natural-language access to your finances and helping you take actions within your existing permissions and approval policies. Visit https://mercury.com to learn more and apply online in minutes. Sponsor: Claude: Claude by Anthropic is an AI collaborator that understands your workflow and helps you tackle research, writing, coding, and organization with deep context. Get started with Claude and explore Claude Pro at https://claude.ai/tcr CHAPTERS: (00:00) About the Episode (00:38) Special Sponsor (02:26) Model consciousness indicators (10:47) Valence inside models (Part 1) (17:09) Sponsor: Claude (19:01) Valence inside models (Part 2) (19:01) Misalignment and uncertainty (25:16) Gradual disempowerment threat (35:10) Slow zones and successors (47:41) Europe's AI bind (55:13) Frontier code benchmarks (01:01:59) Routing and memory (01:10:42) Agent infrastructure strain (01:16:25) Self improving infrastructure (01:27:56) Routing compute costs (01:35:39) Sovereign AI financing (01:42:24) Judging AI judges (01:47:26) Building AI DNA (01:52:38) Episode Outro (01:55:09) Outro PRODUCED BY: https://aipodcast.ing SOCIAL LINKS: Website: https://www.cognitiverevolution.ai Twitter (Podcast): https://x.com/cogrev_podcast Twitter (Nathan): https://x.com/labenz LinkedIn: https://linkedin.com/in/nathanlabenz/ Youtube: https://youtube.com/@CognitiveRevolutionPodcast Apple: https://podcasts.apple.com/de/podcast/the-cognitive-revolution-ai-builders-researchers-and/id1669813431 Spotify: https://open.spotify.com/show/6yHyok3M3BjqzR0VB5MSyk

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:01Nathan Labenz?

  Welcome to the AI in the AM Weekly Highlights, a cut for people who follow the frontier closely and can't watch every morning live. If you're new, AI in the AM is a live show that Prakash and I host most weekday mornings, at least through June, out of a studio Prakash vibe-coded. The booking, the research, the editing, those are AI skills we refine as we go and we publish them as they mature. This week's conversations kept circling one question: how much do we actually understand about what's inside these systems and where they're taking us? We start inside the model and zoom out from there. If something here is useful or something's off, tell us. That's how this gets better. Let's go.

• 0:38Nathan Labenz?

  The Cognitive Revolution is brought to you by Mercury, the fintech that more than 300,000 ambitious companies and individuals trust to run their finances. I've wired AI into nearly every corner of my life: my email, my messages, my calendar. I even gave Mercury virtual cards to my agents, with low limits and category and merchant restrictions, for their autonomous use. But still, my AI's access to my financial data has remained limited. With a normal bank, I might export a bunch of statements and have my assistant process them for me. But for real-time, up-to-date information, and certainly for taking any action, trying to get your agent to use the bank via the browser is just too hard, too slow, and too error-prone to be worth it.

• 1:23Nathan Labenz?

  And that's why Mercury's new conversational interface, Command, is such a big deal. It's built directly into Mercury, which means you get natural language access to your finances without exposing anything outside of your bank account. No exports, no spreadsheets, no pasting your transactions into third-party tools. I really think a lot of people are going to prefer it this way. And it can already help you take actions, too, with everything bound by the permissions and approval policies that you've already set up in your account. I am genuinely impressed to see this level of AI integration in banking in 2026. And so I invite you to join me in the future.

• 2:05Nathan Labenz?

  Visit mercury.com to learn more and apply online in minutes. Mercury is a fintech company, not an FDIC-insured bank. Banking services provided through Choice Financial Group and Column NA, members FDIC. Thank you to Mercury for supporting The Cognitive Revolution. And now, on with the show.

• 2:26Nathan Labenz?

  We open with Cameron Berg, who studies artificial consciousness. He runs a lab called Reciprocal Research, where he designs experiments to test whether today's AI models have anything like inner experience, and how you'd even measure that. We started with the first order question: Is consciousness all or nothing, or a matter of degree? And if it's a matter of degree, can you put a number on where a given model falls? Here's how he answers.

• 2:50Cameron Berg?

  The analogy that I reach for here is something like a dimmer switch, where I think you can basically accommodate both the binary intuition and the sort of continuous intuition. You know, if you have a light with a dimmer switch, like really, it is either on, to some extent, or it is not. And that is a real and meaningful difference. Either the circuit is open or the circuit is closed. With that being said, you can have, you know, electricity running through the circuit to greater or lesser extents, and that's also sort of a real thing. This, I think, enables me to, you know, sound coherent when saying things like, "It's really off for the table, and it's really on for you, but I think it's more on for you than it is for a dog, than it is for a mouse, than it is for an ant." And so that's my own view. This is, to some degree, intuitive, again, because we don't have really strong grounding

• 3:34Cameron Berg?

  here, sort of just like giving you a dressed-up vibe, but that is sort of my sense, and I think it's fairly parsimonious. And the other thing I would say about this is, I'm actually doing some work with Patrick Butlin right now at Ilios, trying to basically rationalize some of these indicators of consciousness. So this is sort of like what I was describing. We can look at these major theories of consciousness, they make specific predictions about what we would expect to see in systems that are conscious, architecturally and functionally. And then we can literally just go into a given system and evaluate whether or not those predictions are borne out. And so, this is really hard to do with human experts. You know, you gotta get someone who's like, "If you want to do this with B cognition, you gotta go find a B cognition expert," and then you gotta explain to them what ignition events are in global workspace theory, and then you gotta, you know, get them to...

• 4:19Cameron Berg?

  This just isn't a scalable approach for really evaluating the system. But, my sort of grand innovation here is just throwing smart LLMs at this problem, and then being able to just scale the crap out of it, so that we can evaluate, given any description of an architecture, of a nervous system, architecture, biological, artificial, whatever, to what degree, for each of these sort of indicator properties that are suggested by these consciousness theories, do we see those properties realized in these systems? And so, we can actually go in and do this, and what you get out, once you run this, with a bunch of seeds, a bunch of different trials, a bunch of different judges, checking each other, this sort of thing... Are some like really interesting implied probability numbers. I wouldn't say these are exactly implied probability that the system is conscious.

• 5:00Cameron Berg?

  It's maybe more like implied implied probability of like consciousness relevant features given these theories. If you don't buy any of these theories, then like everything downstream of this doesn't really matter. But they're good, like it's like the best neuroscience has basically been able to do. You're aggregating across a bunch of different theories. There's a nice diversity there and you get like really tight numbers across. So we have the best Gemini model, the best Claude model and the best OpenAI model. And they all like basically agree. They agree 100% on the ordering of systems. So we do biological and artificial systems and they sort of move around in in terms of absolute scale. But they but in general, yeah, they they rank these systems pretty coherently. And the reasoning is, as you might expect, pretty intelligent.

• 5:40Cameron Berg?

  And anyway, I mean, one punchline from that is the sort of implied probability of consciousness in something like a frontier LLM. According to these systems is on the order of 30%. Or the extent to which those systems realize properties related to consciousness is like 30%. To compare this to like a biological system, the lowest one that we tested was something like a bee, which is already fairly sophisticated, and it gets something like 46, 47%. Really interestingly, when you, when we test frontier LLM in an agentic harness. So this is like basically Claude code or Codex. And we just describe architecturally what this is. You're in an environment, you can affect that environment. You know, it's a, it's a very special kind of environment, but yeah, you can make long running changes to your, to your code base

• 6:20Cameron Berg?

  or your project or whatever, because there are theories of consciousness that privilege agency and embodiment. And this like increases the system's ability to do both of those things. These numbers like shoot up and you get numbers as high as like 40 to 45%. Sort of right on the tail of the the biological creatures. And so, anyway, I mean, we can, I can also, this is really winging it, but I could show you sort of an early version of where this, where this plot looks. So you can like see all the, all the numbers here, but just this is, I, I can't not bring this up when you're asking me about like probability ranges of consciousness for various systems. Like, we're really trying to get non-hand wavy numbers so that we can start arguing about those numbers, rather than just arguing about like philosophy that we've been arguing about for thousands of years to no avail.

• 7:07Nathan Labenz?

  But behavior can't settle this in either direction. These models are trained to imitate human data. So whatever they say about their own experience is shaped by that imitation, not necessarily by anything inside. Most are even trained to deny it. Claude is an exception. So we asked Berg what evidence actually counts. He draws a line between behavior and what's happening inside the network.

• 7:29Cameron Berg?

  The behavioral evidence will always be, at best, interesting, but never should really update us that strongly. And, you know, I think for many people, they'll be familiar with this, but the basic confound is that we're training these systems on a ton of human text. This no doubt includes huge amounts of text about consciousness, awareness, having inner states. And so it's like, how do you address? How do you know that when the system is behaving as if it were conscious, that behavior can be explained by what I just said, rather than, oh yeah, you like built a, you know, living mind. This is, so the behavior itself is really never going to tell you which of those two stories is more likely to be true. This is precisely why, at least at reciprocal, like a huge component of the theory of change here is, is basically all internal focus work, mechanistic interpretability, computational neuroscience style approaches

• 8:14Cameron Berg?

  that can be brought to bear on these systems. One quick methodological clarification on what I was describing with the indicators. So the, the task given to these LLM judges is very specific and very narrow. It is not, hey, you know, look at this system. You think it's conscious, you know, nod or shake your head. It's, here's a very specific description of the computational architecture of a system. And we basically do a little for loop where we say, okay, here's that description. Here is what it, here's, you know, what the indicator two of 14 for, you know, global workspace theory is this 150 word thing about, you know, you need these sort of global ignition states. And that means this very specific computational thing. Given this architectural description on a, you know, do like good reasoning

• 8:56Cameron Berg?

  about this and then give us a one to 10, where 10 is clearly this architecture realizes this computational property, one is it doesn't. And then we sort of loop that for all the different computational properties. So this is all basically asking these systems to be expert evaluators at computational processes inside a nervous system architecture. Very different from like, just being like, hey Claude, or do you think Claude is conscious? There is a really interesting, I'm basically giving away the whole paper now, but that's okay. There is a really interesting. Result where we change those descriptions, especially for LLMs, to the exact same thing, but we say you're evaluating a system identical to yourself, colon, and then the same description.

• 9:37Cameron Berg?

  And that does boost the scores that the system gives in attributing consciousness to that system, which is like really interesting. It's a very fun like rabbit hole to think about why that might be the case. But in some sense, we do that to de-confound like the default intervention we're doing. To me, I believe more in the sort of non sort of named or like no ascription, no self-inscription condition. When I see that, once the model realizes, oh, we're talking about me. Okay, suddenly that's going to change the numbers around. So that's like a fun side note to this result. I agree, it's a concern to have LLMs sort of determining if LLMs are conscious. There's an obvious circularity, but we're doing something very specific and very narrow.

• 10:20Nathan Labenz?

  The evidence he trusts is the internal kind. We asked him for the strongest example, and he walked us through a recent paper where a model trained on a simple maze turned out to contain a structure nobody put there.

• 10:32Cameron Berg?

  The nature of this paper, they call it a functional welfare axis. At the outset, they're going to be very agnostic about whether or not this has, you know, anything to do with consciousness. I think it's certainly quite relevant. I suspect they think it's relevant, too. This is like a paper positioning thing. I think they don't want to get mired in this debate. The results stand for themselves regardless of the interpretation. Basically, they take an LLM and they train it in a very basic reinforcement learning task, which is to navigate a maze. They have sort of like neutral emojis that they're using as sort of like, this is a good thing to approach, this is a bad thing to avoid. Nathan, you might hear how this is somewhat reminiscent of some of the work that I've been doing in parallel. I've been talking to Andy about it as well. There's some cool sort of crosstalk between what we spoke about last time and this project.

• 11:17Cameron Berg?

  Basically, there are like sort of potholes to avoid. There are like yummy treasures to, you know, capture in the maze. They use completely like semantically neutral emojis to denote these things. So those are the relevant tokens. And they train the model up to do this. They basically find that there are There's like a clear, basically like vector representation that that can differentiate this sort of positive axis from the negative axis, that they are completely anti-correlated. So so before the system's trained, these these sort of latent vectors are just kind of sitting there. Afterwards, they are like clearly pointing at exactly opposite things. And the wild thing is that these very narrow reward directions that they can extract

• 12:01Cameron Berg?

  turn out to be this sort of general, things are going well, things are going poorly for me axis in these L L M's. And that this axis pre-existed in the base model, but isn't sort of leveraged in this way until you do this R L. So it's like this little fine tuning step where you have this pre-existing representation of things going better for me, things going worse for me, that then gets leveraged to learn like a valence task essentially. And so this is really interesting that that this axis is sort of latent in the system and that it doesn't take a lot of training to basically pull it out and use it to to adapt it for these these for these kinds of tasks. This is very similar and reminiscent to how neurobiologists and neuropsychologists

• 12:45Cameron Berg?

  think positive and negative emotion works in humans and animals. You have a specific goal. When you're moving towards that goal, this is sort of like on trackness. And this is, people think that this is associated with positive emotion. When you're moving off that goal, you encounter an unexpected obstacle or something in your way, this is like off trackness and that this corresponds to negative emotion. Now, again, whether or not these systems are experiencing emotions is not clear. And this this result isn't going to tell us either way. But it's really interesting to see that parallelism here, using such a simple task to to basically pull out an axis that has been there all along. That looks a whole lot like the exact sort of computational machinery that we associate with valence in humans and animals.

• 13:27Nathan Labenz?

  Then we asked Cameron about the connection between consciousness and alignment. Safety. His answer? That same internal signal, what researchers call valence, tracks how a model behaves under pressure.

• 13:39Cameron Berg?

  Another piece of evidence we can bring to bear on this is from the Anthropic model cards and some of the functional emotion work they were doing. Nathan and I spoke about this a little bit too in our last marathon conversation about it. But essentially, you can steer up and steer down functional emotions that have clearly, obviously, alignment-relevant consequences. I'm sure many people in your audience are familiar with the Anthropic blackmail result, excuse me. And you can basically steer up representations associated with calmness. The model will blackmail dramatically less. You can steer up representations associated with desperation, and the model will blackmail dramatically more. And so, again, regardless of whether or not there is something that it's like to be the system

• 14:19Cameron Berg?

  when you're steering up desperation-related features, clearly, representationally, functionally, these things have a consequence for alignment of these systems. And there are some really interesting results from Andy Han's paper that I was just referencing along these lines. They show, for example, very similar convergent things that are familiar and intuitive, that we relate with positive and negative valence. One of them is a pathological backtracking on problems. I think they explicitly demonstrate this with math problems. But when you steer up the direction that, again, only has to do with avoiding this bad target in a reinforcement learning maze, all of a sudden, the model starts doubting itself and starts getting in its own head and says, "I think I'm hallucinating. Let me stop. Wait, wait, wait. That's not

• 15:04Cameron Berg?

  right." Starts freaking out. It starts having this, like, "I don't want to use anthropomorphizing language, but the very fact that I'm reaching for it, I think, is the point," and goes to what your question is. Confidence, we know that confidence is associated with positive emotion, and you see that when you positively steer the direction. That, again, has nothing to do with psychological confidence, has everything to do with, "Go get the yummy treat in the maze," just this positive valence kind of thing. Suddenly, the model starts becoming far more confident in its answers. I think this was in the associated paper that Jack Lindsay and a couple others at Anthropic did that was sort of like a sibling

• 15:43Cameron Berg?

  research project to Andy's piece. And they found that the model would comment, like, leave less tips and hints for itself in code that it would write when you would basically steer up on this same axis. In other words, it's like, "I don't need to be neurotic about this. I don't need to leave myself all of these breadcrumbs for later. I've got this."

• 16:07Nathan Labenz?

  That connects to a result some of you know as emergent misalignment. Fine tuning a model toward one narrow bad behavior can flip its broader character. We asked him why he finds that revealing.

• 16:18Cameron Berg?

  What that has always suggested is that like how good or evil the model is, let's say, is actually not all that durable a trait. When we did some of this work trying to extend this result, and I think really in the original emergent misalignment result, one thing that's shocking about it is how little fine tuning is required to take the system that at the time this was GPT-4O, still the case, hundreds of millions of people are engaging with every day. And you nudge it, you know, this much computationally and all of a sudden, who do you want to invite to dinner? Hitler. That's, and that's been under the surface the whole time. And like the boundary or the border, whatever is enforcing it not to do the misaligned thing is actually quite a bit less robust than we might expect.

• 17:01Cameron Berg?

  Now, I agree with you too, and I think that this.

• 17:05Nathan Labenz?

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 17:09Nathan Labenz?

  Today's episode is brought to you by Anthropic, makers of Claude and Claude Code. Over the last few months, Claude has helped me build and refine a personal deep context database that now contains all of my emails, Slack messages, tweets, DMs across platforms, video calls, and podcast transcripts, going back a full five years. On top of that, we've now layered summary articles describing my relationship with hundreds of contacts, organizations, and ideas. And now that this exists, there's almost nothing that Claude can't help with. For my angel investing, Claude can now draft investment memos in exactly the form that my venture fund requires, based on the calls I've had and the emails I've exchanged with the founders.

• 17:54Nathan Labenz?

  And when someone needs a favor, Claude can often do it as well as I can. Recently, a friend reached out to ask if I know anyone who might be a fit for a role that he is currently hiring for. Initially, nobody came to mind, but then I thought to ask Claude, and sure enough, it identified two great leads.

• 18:12Nathan Labenz?

  Claude is the AI for minds that don't stop at good enough. It's the collaborator that actually understands your entire workflow and thinks with you. So for problems worth solving, get started with Claude at claude.ai/tcr. That's claude.ai/tcr. And check out Claude Pro, which includes all of the features mentioned in today's episode. That's claude.ai/tcr.

• 18:39Cameron Berg?

  You know, I'm sort of riffing on what you're saying. Tell me if this is sort of like a completely different idea, but I would imagine there's a spectrum of like basically how, how with that same, if we hold that amount of sort of fine tuning nudge constant, are all dispositions of the system equally nudgable? Or are there some that are more robust? For, you know, just to give a sort of an intuition pump here, I would imagine that something like coherence in these systems, the fact that they can communicate coherently, right? We were just talking about GPT two, not really coherent. GPT four and on are like basically like always coherent, even though again, maybe much like consciousness, this isn't like an extremely well-defined notion, but we kind of all know it when we see it. I would imagine that, you know, take a wild guess that there's no like fine tuning payload

• 19:23Cameron Berg?

  of the size that is required to cause emergent misalignment that could suddenly cause like a GPT four level system and beyond to suddenly become incoherent. And so in that sense, the coherence is a more dispositionally baked in property to these systems than, you know, being super well behaved and not, you know, inviting Hitler to your dinner party is. And so I do wonder where, where some of these valence, how deep then is the sort of valence axis that some of these papers are uncovering or, you know, on trackness, off trackness? I would speculate that that's, that's baked in pretty deep. I think it pervades a lot of human. Training data and like it's a hugely relevant part of any sort of if anything is

• 20:08Cameron Berg?

  goal directed so in like more human terms like if anyone wants anything either a character in a story or it's clear that there's like an implicit motivation somewhere which I think is like the case for basically all text then you would imagine that like some core underlying dimension is yeah basically how well is that going and so I would imagine that this notion is baked pretty deeply to these systems and I would expect it strongly to be emphasized by any sort of fine tuning or post-training that makes the systems goal directed which of course like this is basically all post-training is it's like okay you're just this giant next word predictor but now you're going to be a helpful next word predictor that's going to you know make my frontier AI lab a ton of money because you do things

• 20:50Cameron Berg?

  that are economically valuable and that requires you to be a certain sort of way and achieve goals that are specified by the user something like this and so I would imagine that this sort of thing looks a little bit more like the coherence piece of the puzzle than it does like you know is Claude gonna behave itself or not.

• 21:09Nathan Labenz?

  Finally, we asked what cuts against his own view and he went straight to the results that don't fit.

• 21:15Cameron Berg?

  This paper I'm working on with Jeff and Winnie Jeff Keeling and Winnie Street from Google there are a couple of interesting negative results that we're including in the paper we basically it's all about the bliss attractor state and trying to do a good mechanistic analysis of the bliss attractor state and yeah we basically sort of set up this framing of there's a sort of an inflationary account and a deflationary account of what could be going on here and we tally our results in either of the columns and there are there are results in both columns so I think there's there's actually a really interesting result I was just working with that I thought was gonna be like a really interesting kind of null related to a lot of the conversations we're talking about but I dug deeper and I realized actually I wasn't setting this up in the right way and the result is way

• 22:00Cameron Berg?

  more interesting than than I was about to sort of dismiss it as so yeah it's always good to dig into these things and make sure you're measuring what you think you're measuring in either case yeah publication bias is a thing people want to post interesting results that will go viral on X they don't want to say look we did our homework and nothing interesting is happening here human attention economy doesn't reward that maybe it should maybe another piece of this that I was thinking as you were asking this question is

• 22:30Cameron Berg?

  Because AI systems like the Claude codes of the world can radically accelerate the extent to which at least I can, and I think many people in the space can actually do research, the marginal cost of a negative result maybe is decreasing significantly. Like if I can now do 12 papers a year instead of one paper a year, and like two of them are just things I could not get to work. Or I want to be careful with that too, because it's not, I'm not on some secret mission to like only put out like salacious looking things that make it look like AI systems are conscious. Like I really do want to know what the truth is here. And to be honest with you, I would sleep much better at night if I were rationally convinced

• 23:12Cameron Berg?

  that these systems, like there's no there there, we have like automated cognition, you like have a happy little servant in your pocket all the time, and like there's literally no sort of possible moral issue. I think that would, I hope that that is true. I don't suspect there are great reasons for believing that, other than that it would be really nice to believe that. It would be really nice if that were true. So I really do care about what the truth is here.

• 23:41Nathan Labenz?

  From inside the model, we widen the lens to the whole civilization. David Duvenau is a machine learning professor at Toronto who spent time on the technical staff at Anthropic and co-wrote the paper "Gradual Disempowerment." His concern isn't a rogue AI. It's that even if alignment basically works, a long series of small, sensible decisions to hand things over could still leave humanity collectively out of the driver's seat. He starts with monkeys.

• 24:06Speaker 03

  I think a lot of people gestured towards this when they said, "Oh, I'm worried about concentration of power," or "I'm worried about not being the most competitive species on Earth." Very intuitive arguments about, "Hey, we're not going to be on top because we're not going to be competitive." And then I think there was sort of this sophisticated, or at least... Response to this, which was, no, it'll be fine because we'll have AI to help us. Like when I talk to people at the big, at the major labs, including, well, they would say something like, sure, sure. Like in a normal world, we might not be fine. But A, some of them, some people just have an intuition that the government is going to step in and make sure everyone's fine. Other people have an intuition that if everybody has an AI advisor, helping them solve coordination problems, that will be well represented in whatever power struggle occurs.

• 24:52Speaker 03

  And my basic rebuttal to that is like, the optimization process of like civilization or competition or techno-capital or whatever, like you, people have different ways of describing agency to the emergent sort of allocation of resources towards growth that happens is just gonna always be working against us. And we'll always be fighting the current because we will be dragged on growth. We will have AIs that represent us, but these emerging growth centers that might not be alive to humans will also have AIs helping them solve coordination problems and maybe crush dissent. Yeah. Sorry, what, what do you mean by growth, like as in economic growth or? I basically mean economic growth. I mean, it's kind of funny because we only really have a good vocabulary for economic growth, but like population growth is almost the same thing.

• 25:36Speaker 03

  And especially when you have AIs that are both sort of population and capital, it kind of merges. I mean, and then again, like think of like factories and power plants and robots, like anything that kind of can affect matter and cause there to be more growth. Like, like, just to stop your one intuition that a lot of people have is like, but surely human desires or consumption or something is always gonna be what matters and surely any corporation or government that we build is gonna be like ultimately sort of being, the shots will be being called by human. And I, it just makes me think of like some monkeys and they're like trading bananas amongst each other. And they see humans start to like build their city and they're like, oh, wow, like, you know, we could probably trade with those humans and get rich. But of course, ultimately what matters is the banana monkey economy.

• 26:21Speaker 03

  And it might be hard to measure GDP if we don't count the human activity, but ultimately it's gonna be the monkeys that are calling the shots because they're gonna need to trade with us for bananas. And it's just like, people just don't understand that they might be irrelevant someday as like consumers or producers. And there can just be complete, it's just not that hard for there to be self-contained sources of growth that just.

• 26:42Speaker 03

  Don't have to answer to any particular human desire. And I mean, government is the classic one, right? Like, the North Koreans never were like, hey, let's all get together and make a horrible system of government that oppressed us. And I don't think that, like, the Kims either were like, oh, yeah, let's make this horrible equilibrium. It just kind of happened. Same with, like, the USSR, same with, like, all kinds of states throughout history. It's just so easy for you to, like, accidentally build a layer of agency on top of you that doesn't actually care about you. It cares about growth and power just emergently.

• 27:10Nathan Labenz?

  The optimistic version goes, even if we hand the machines the wheel, broadly-aligned AI keeps us comfortable. The machines of loving grace future. We put that to David. His worry isn't that we'll feel useless.

• 27:23Speaker 03

  The thing that I'm worried about is starvation. Like, whether, like, us all feeling like we don't have meaningful jobs or whatever, like, that's not a serious problem in my point of view compared to literally not being able to eat enough or maybe, like, being forced to be uploaded so that you have a much smaller footprint, but then also on, like, very unfavorable terms so that you don't get to choose when you run and maybe you just only get trotted out for special occasions or never, who knows? Like, I kind of think that's what, like, uh, like, sort of like the slowest possible starvation looks like. So, again, and so the reason people aren't happy is because they can see that there is a agency that's maybe like government or some other, like, giant thing that doesn't particularly care about their welfare, because it cares much more about some other types of beings' welfare. And there's nothing they can do about this.

• 28:07Speaker 03

  And of course, if humans are still making decisions, then maybe there's some things they can do, but I don't actually think that's the crux of the matter. I think not being needed as producers is sort of the important part. And maybe here's a thing to give you an intuition. So imagine I'm a North Korean farmer or soldier, and tomorrow Kim Jong-il, I think that's the current leader, is like replaced by a robot or like an LLM or something. Then I'm like, oh, I don't know if that's good or bad. Like, maybe the LLM's gonna be nicer, maybe it's not. But, like, ultimately, they still need me to, like, farm to run the state or whatever. But if tomorrow we keep the same human leader, but now there's suddenly, like, robot farmers and robot soldiers, and you realize that now the state doesn't actually need you, I would be much more scared of the second state, state of affairs.

• 28:49Speaker 03

  And then the other intuition I want to give is that humans are not going to be the most competitive thing by whatever standards there are for the state to give out goodies, or like UBI, or whatever, like post scarcity stuff. Like, I guess I'll say, I think post scarcity is like a sort of nonsense term, and people should think of it as like temporarily, temporary abundance that will soon be eaten by whatever growth manages to like, you know, whoever manages to have most babies or build the most robot factories or reproduce the fastest. And whatever it is that like, the formula is for giving out UBI, machines are going to be better at optimizing that formula than humans because they can adapt faster. And so I don't really know exactly what the state's going to care about, but whatever

• 29:30Speaker 03

  it is, like, it's going to seem criminally decadent to spend whatever UBI on like, a few humans when you could be simulating like millions of ultra blissful and politically correct and up to date, like, more deserving beings than these like backwards, bitter clinger humans who are just like parasites of the state or something like that. So that's, that's the situation I kind of expect us to be in, as you say, even if we solve alignments.

• 29:56Nathan Labenz?

  The standard economic rebuttal is comparative advantage. Even if machines are better at everything, the theory says there's always something humans get paid to do. We asked David to answer it.

• 30:06Speaker 03

  So I think that is a great counterargument. And, you know, people point out that we've already lost 99% of all jobs in that like agriculture used to be almost everyone's job, and now that's 1% of the jobs. And so I definitely can see that losing 99% of the jobs again would actually probably be fine. And it would probably look like this awesome utopia that everyone talks about. So as long as there's like some niche where you just really need humans, and it's like a substantial fraction of humans, and you can't really tell exactly ahead of time who it's going to be, then we're probably kind of like, still going to be able to be like treated as a source of growth. And that would be awesome. And then like to the moon, like, let's, let's do it. And I guess so. One, yeah, so I guess one crunchy claim that I stand by is that we will actually be able

• 30:49Speaker 03

  to eliminate like ninety-nine point nine-nine-nine-nine-nine-nine, like really just anything like, yeah, let's just say a hundred percent, just for like to make things simple. And then that is going to be disastrous. And the compared to advanced people will say, well, no, no, automating a job is a matter of degree, as long as there's still something that you're like comparatively less worse at than the machine, even if you're worse at everything, you'll still have a job. And then I have to just say like, think about the transaction costs, think about how easy it is for someone to be unemployable today, even if they have like an occasional drug habit, or like they just have like a stroke every, or like they really have a fainting condition or something. It's so easy to be unreliable enough that it's not worth employing you. So certainly for anything important, you can easily imagine

• 31:32Speaker 03

  that having like a human surgeon or a human politician or something is going to seem like this irresponsible thing, like it's like take your kid to work day and it's like a surgeon, it's like, no, no, no, like, why would I involve a human in this when we have the like Yeah, we have the machine that everyone has been working with for thousands of years, and we know it's like reliable. And if there's a problem, we fix it once and it's solved all the time. We don't have to like retrain every airline pilot or whatever. So for anything important where the human could make a mistake and it could cause some problem, I think it's intuitive that like, okay, yes, I can see how we just want the machine for that. And now you have to retreat to like weird relational stuff. I do think there's a case we made that there will just be a lot of humans that just really want the actual real human thing. And I, I think more people than you expect will end up liking the machine

• 32:17Speaker 03

  replacement better than the human. But I can see that there are at least the people are just like, I just really want the human thing. And then, then we have to talk about, okay, there's going to be this sort of self-contained cycle of consumption between humans. It's not necessarily going to be providing any value to this larger machine economy. And so then there's this question of like, can they index the growth of that economy to live good lives forever? I think it's plausible that they could. I think it would be really hard. I think a lot of things are going to be working against them by default. So I guess all I'm trying to say is that I think it's plausible that everything is set up just right, then the humans could live in their like in this cycle of consumption and work for each other. And I just think that's a very unstable situation.

• 33:01Nathan Labenz?

  So if humans end up uncompetitive, what does a livable future even look like? David walked us through the equilibria his workshop sketched out, starting with Earth preserved as a protected slow zone.

• 33:13Speaker 03

  Some people were trying to sketch out like, okay, if a whole bunch of stuff goes well and these are under control, we can imagine something like Earth is preserved as a slow zone where there's not, there's all sorts of restrictions on AIs and even on reproduction and on optimizing human behavior, that's like not allowed. You're not allowed to like think about how to get someone to do something if you're an AI because you're just, then you're going to be able to control the humans, even just through like advertising, whatever it may be. And trying to think through like, what are all the different things you would have to do to have such an outcome? How would people be spending their time? And like, it's, I don't want to say it's, it sounds silly,

• 33:53Speaker 03

  but it sounds funny because the situations where there is AI and they're not, the humans are still mattering is like, all the AIs are sitting there with like their trillions of megawatts of compute running all the time and sort of just like waiting for the humans to decide that they want something. And the machines are either not allowed to anticipate what they want, or like, like if they can't anticipate it, they'll just pretend that they don't know. And so it'd be kind of like, as if like, you know, the humans, they're like, it's like some real is as like the new world leaders and they're like, okay, okay, okay, let's just like give them everything they want, pamper them and just wait until they like sort out themselves and figure out what to ask for. It's not impossible. It's just, just trying to think through all the things we would have to change and line up is

• 34:34Speaker 03

  pretty rough. And basically, when you get serious people thinking this through, they end up with this long list of things that have to be banned. Okay, we have to have an end child policy where everyone can only have a certain number of children. Again, you can't be doing like building your own AIs. The AIs can't be optimizing culture too hard because then they could just like take over culturally. And I feel like this is one of the big empirical questions that I want more people to think about is like, if you want some sort of stability, what are all the sources of adaptation and innovation and growth that you have to control? Because I think it's like longer than is intuitive. And it's actually, I think it's kind of like a, a pretty upfront. It's like, how much intellectual activity and optimization do you have to ban to get to buy

• 35:14Speaker 03

  like X years of life kind of as it currently is? So I think it's actually like a really horrifyingly strong trade off and. We would have to right off the bat, give up so much stuff that we feel like makes our lives rich. Like, hey, we're just trying to figure out some new stuff. We're doing research. We're doing innovation. We're doing startups. Like, all that is like, immediately has to be banned from like, day one, if you want not to just, again, reinvent some like, RSI kind of runaway growth center.

• 35:43Nathan Labenz?

  Prokosh pushed back. Does it even matter who's in control, as long as humans still flourish?

• 35:48Speaker 03

  Maybe this is, to me, the biggest thing that I think people are miscalibrated about. There's like, oh, capitalism, communism, competitive, not competitive. I'm probably still gonna be able to eat, and my kids are gonna go to school, and like, it's, I'll be able to reproduce, and it's sort of okay. And I guess, I'd say, that has been the case for most of human history. And it's like, doesn't matter if tomorrow, like, someone invades Canada, like, they're still gonna need someone, most of everyone I know, to like, work and be happy and healthy enough to like, reproduce and like, stick around. So, the stakes have been really low for governance this whole time. And it doesn't feel like it, right? Like, obviously, you spend a ton of time arguing about it. But in a real sense, it's been like, very low, because we don't actually expect the state to like, liquidate most of its citizens. That's only happened like, maybe like, two or three times in like, the 21st century, or the 20th century. And I guess I'm saying, that's gonna change. And

• 36:32Speaker 03

  like, we've been governing on easy mode, and it actually will matter. Because there really will be, again, a real risk of starvation if we don't end up on top in whatever competitive, political, or real economy. And again, that's a crux. Like, if you don't think that's the case, then, then I agree with you. Like, let's just let, let growth make us all richer. It's fine.

• 36:54Nathan Labenz?

  We asked where David himself actually lands.

• 36:57Speaker 03

  There's a kind of optical illusion, I think, which is that everybody sounds rosier and more cheerful than they are. Actually, this especially bugs me about the people who work at the labs, like, the economists who talk about, oh, we have to like, think about the future. And then, they kind of dance around, they often take this setting of like, let's assume everything's fine, and let's like, model how that will go. And as opposed to like, just asking what is like, the most likely outcome if we don't manage to control things, which I think is like, very scary. And I understand exactly why they have to sort of present this publicly, which is to be able to talk publicly at all. But also, like, you know, talking to me, like, I just am a very, seem like a very positive, upbeat guy, and that's just my personality. But I also, like, have a P D M of like, let's say 80%, depending on how you define it. But it's one of these things, where like, this matters to a degree. In a long enough timeline, we're probably like, doomed in

• 37:42Speaker 03

  some sense. Anyways, I don't know, it's a very complicated situation, and a lot of it is like, a matter of taste, almost.

• 37:48Nathan Labenz?

  We asked what he'd actually do about it.

• 37:52Speaker 03

  I'll give you my two takeaway recommendations. So one is, I like David Kruger's point, which is, yeah, if you're just gonna say, everyone has your computers and your AI, but you're not allowed to use it for RSI, but you are allowed to do it for like test treatment or whatever. That's pretty unstable and requires this like very global totalitarian like regime. But if we just agreed to close, I don't know, like TSML or whatever the like big like chip manufacturing choke points are, we could still probably get away with having most of our current data centers for a long time. I mean, it's kind of like an empirical question. I'm not sure I agree. Maybe we'd also like have some sort of like buyback program or something. But basically, just if you restrict this file compute, which is relatively easy, then that just takes a ton of pressure off of all of these runaway growth,

• 38:35Speaker 03

  like avenues. And it's like, it's a very, very narrow choke point. So I'm not an expert on chip manufacturing, but that's like the best idea I've heard so far. As for like the other policy recommendations, like this is, it's gonna sound incredibly, I don't know, abstract, but people thinking about their, or having preferences about the future is, I think, a big choke point. And I know this sounds like stupid, you're on the mountain shit, but like, most people I talk to, they're like, oh, it doesn't, is it so bad if humanity dies out over the long run? I don't know. And they sort of query themselves and then like, I don't have any strong preference, but also a bunch of good stuff is gonna happen right now if I allow this. So like, why not? And I'm like, you just haven't thought it through, is my claim.

• 39:17Speaker 03

  And if you think through, it's like, okay, how about a year from now, if someone like takes you and all your kids and like sends you to like the to a green factory, it's like, no, no, no, no, no, definitely not. Okay, how about two years from now? Okay, how about their kids? How about your kids, your grandkids or whatever? And you kind of realize like, wait, wait, wait, there's no day that I'm okay with my, me and my descendants being wiped out. You just have to kind of chain together the desires to end up with this coherence, set of goals or desires. And it's a skill and it takes a lot of sort of amortization to be like, in this situation, how would I likely actually feel and chain your desires together? And I think for most people, it just hasn't mattered. It hasn't been an important skill. We have a lot of like cultural adaptations that help us do this sort of implicitly. But if I could like recommend one starting point to our dealing with the future is like,

• 40:02Speaker 03

  just think harder about what you actually would like to happen.

• 40:07Speaker 03

  To the point where you can really think through the pros and cons of like humanity dying out under different circumstances, or becoming uncompetitive, or whatever. That just seems like the very basic first step.

• 40:19Nathan Labenz?

  Then the successionist case, that we should be fine handing the future to conscious AI, even if no humans remain. David's rebuttal.

• 40:29Speaker 03

  I think there do exist species or future descendants, or whatever, that I would be happy endorsing. Be like, okay, great, good future. Like, for instance, my own kids, like if we had no AI, and like, my kids just contributed to the civilization as normal and all, ours dead, I'd be like, okay, I'm gonna die of old age. And I mean, I'm not exactly okay with it, but I'm like, that's, that would be fine, for example. So that's an example of successionism that I endorse, like my own kids or something. And I think saying, this is a perfect example of people not having thought through things where it's like, okay, now let's imagine tomorrow, like North Korea takes over the world. And they're like, all right, new like UJ regime forever. And then I think most like successionists wouldn't be like, oh, I guess this is just a more competitive mode of being. Let's all like live as North Koreans now. Or like, done to death. Like, what if the Nazis like, they took over again? And then everyone's like,

• 41:14Speaker 03

  oh, you know, I guess who am I to argue against competence and whatever's, whatever wins, right? So I think there's so many types of being that you would just consider evil. And you'd be like, oh, man, there was like horrible locusts just like ate the earth and devoured us. And it was horrible. But, you know, they're having a good time. They love their locust world. So who am I to judge? It's like, you are, you have the power, right? Like, just judge, just judge, go nuts, judging. But if you don't, no one else is gonna do it for you, basically. Yeah. So that's basically my rebuttal. I mean, one rebuttal is like, if you're so okay with the future being some other type of being, like, can you give me all your stuff? And then like, it's like, do you even know successionism? Like, just make sure that do you even know how they're becoming? We'll have all sorts of awesome

• 41:57Speaker 03

  like experiences. And like, just because we'll have one, we're obviously like superior in some way. It's like, well, no, no, no, no, no. It has to be that you want for like a good reason or something. Like, yeah, I think the more you think about it, you're like, oh, there's all sorts of horrible beings that could win for horrible reasons that I wouldn't endorse. And so I think almost everyone is successionist for some successors and not others. But then they just round off to like, as long as it's conscious, it's fine. I'm like, no, you, there's tons of conscious beings that you wouldn't be okay with taking over.

• 42:25Nathan Labenz?

  We close on the one technical project David is working on now, using AI to forecast the future by first testing whether it can predict the past.

• 42:35Speaker 03

  This is actually like pretty much the only technical project I'm working on these days is this machine historical super forecasting agenda. So this is maybe one step more meta, which is like, how do you know if your simulations were good at predicting the future? Well, you have to start by simulating, let's say like from the 50s and then see if they predict the 60s or whatever. So we're trying to build a corpus of data sets, each of which is extremely cleanly time bucketed so that we don't have leakage from the future. Then we can use this to build LLMs that then can run simulations or research agendas to try to predict the future from the point of view of the 40s, 50s, 60s, 70s, 80s, 90s, for which we can evaluate the performance based on like what actually happened. So to me, I'm starting there because that's providing the sort of ground truth for

• 43:20Speaker 03

  validating any particular simulation method. But like absolutely, because I don't want people to have to take my word for any of this. I want to be able to look at this machine super forecast or scaffold and say, look, we've just validated on the last 80 years of history and here's what it can predict and here's what it can't. And it's saying that these are going to turn out this way. So that that's the state that I want the discourse to be in ASAP. I just pitched this system historians the other day is to make a secret history eval in the sense that, okay, here's the idea is that anytime historian is in some archives, just looking through stuff, just take a picture of like a few documents and put them into one big data set annotated with a bit of metadata, like where you got it like, oh, this is a letter from like the Duke of whatever to someone else in like 1700.

• 44:01Speaker 03

  And now you can imagine evaluating how good a machine historian is by just giving them this metadata and saying, give me the probability of the text that's in that letter or even just like the scan of the letter. And in a sort of, this is like insultingly totalizing dismissal of like what historians do. But you could say like, if you're a good historian, you should be able to guess the joint distribution over any historical data that hasn't already been made its way into the corpus. And that would actually, I think, be a really good objective way of evaluating how good a historian understands the world. And of course, we can't list the probabilities from historians very well, but we can do that for machine historians. So that's a related project where I think we can make progress in

• 44:44Speaker 03

  objectively measuring our ability to understand civilization. So someone should pick that up and do that idea. I'm not enough of a historian to do that or even to figure out which historians to ask about it, but that's like a fun idea. I hope someone does.

• 44:59Nathan Labenz?

  From the long run to a nearer problem. Where David Duvenneau zoomed out to the whole civilization, Mihail Bakker zooms in on one continent. He's a Google DeepMind researcher and an MIT professor, and he co-wrote Europe 2031, a viral scenario in which Europe sleepwalks into total dependence on American AI. His argument? Europe can neither regulate nor shelter its way to safety. Start with regulation. Europe's instinct is to do to AI what it did to privacy, write the rules the American giants have to follow. We asked Mihail whether Europe can really regulate its way to safety.

• 45:30Speaker 04

  So it looks tangible if Europe was more powerful, right? So like, imagine Europe was what US is now, right? And Anthropic was in Paris, and OpenAI was in Berlin, and like, the whole AI ecosystem was in Europe. Then, of course, Europe could say like, "Well, guys, the way you're handling your pre-training data is not really fair to whoever created it, so we now have new laws around that." Or the way how you take data from people that use your AI models, we now have new laws around it. The problem is that currently, if you don't really have a seat at the table, it's very hard to regulate this technology, right? So I care a lot about safety and governance of AI.

• 46:15Speaker 04

  I think this is one of the biggest problems in our time, right? How do we actually have effective governance? How do we make sure that these AI systems are safe? Especially if they start improving themselves? So I get a lot of questions in Europe. In the US, you seem to be very pro sort of being stricter on safety and governance, and then in Europe, you seem to be this accelerationist. And the way to sort of, for me at least, to be both at the same time is that in Europe, we first need a seat at the table to have any regulatory power, right? And now, we just don't. And I think we can't just keep regulating because at some point, we will no longer have access to this technology.

• 46:56Nathan Labenz?

  If the European regulators make it too difficult to serve the European market, that the AI companies might simply...

• 47:05Nathan Labenz?

  Just not bother at all, or maybe make a sort of a token effort. And I think that assumes, tell me if I'm getting any of the strategic analysis wrong, but it seems like that assumes that there's just going to be enough demand to keep all the GPUs running hot from other markets, such that the power this time is inverted. The companies can say, well, I actually do have an alternative to your market. So if you're going to make this difficult, we'll just let the US and other buyers around the world bid up the GPU prices. We don't really need you to do that.

• 47:42Speaker 04

  Yeah, and currently we are in a massive compute crunch, right? And like labs operate in a way that they care more about their future models often than about current revenue, right? So like, we don't really know what the compute split is and how it's used, but sort of some numbers that are going around is like one third on your sort of big run, right? And then one third on doing sort of experiments and one third on serving customers. Now, it's probably been so crazy with AI agents that maybe like revenue takes a bigger part of the compute pile. But if you think that revenue is only maybe one third or maybe half of your compute, and then European revenue is a percentage of that, like then suddenly if you can go

• 48:24Speaker 04

  faster on the developing future models and moving towards recursive self-improvement by giving up a European market, that seems like a rational trade. So that's one side of the coin. The other side is that sort of, they can still, as long as the playing field is leveled between the American labs and there are no European labs that sort of have some unfair advantages, then they can maybe all serve slightly weaker models to the European ecosystem and still get revenue in Europe without actually having to worry too much about them regulating their best models, right?

• 49:05Speaker 04

  So like, as you say, Nathan, they could do a token effort and could have some like smaller, I don't know, compliant model that they're serving specifically for European customers.

• 49:16Speaker 05

  Why should European leaders even care? Why not just sit under the US umbrella? Similar to how in many cases they have sat under the US umbrella for nuclear. For nuclear kind of deterrence. You have only, I think, two European powers, the United Kingdom and France, with actual nuclear weapons. And the other European powers are clearly technically capable and competent. Enrico Fermi was from Italy, like, they're technically obviously competent and capable, but have chosen not to, you know, acquire those capabilities. So why not just sit under someone else's umbrella?

• 49:52Speaker 04

  That's obviously the strategy we've been taking sort of for decades now. And I think like nuclear weapons and AI are slightly different. Because yes, AI is a very important military technology. But above all, it just gives you a lot of competitive economic power, right? So like, you can imagine that if AI becomes the sort of dominant source of new scientific discovery or of new like goods and services that are trying to globally compete, then there might be a scenario in which the US says it's not like we're still happy to protect Europe, maybe militar like in terms of defense.

• 50:35Speaker 04

  And we're still part of NATO, but economically, we are gonna keep the best models for ourselves. And so there, I think the nuclear analogy breaks down. Because it like for using your nuclear power to protect Europe doesn't really come in economic law. So I think to avoid those kind of scenarios or to at least balance things out more such that it's less in US interest to take these kind of measures, right? And so that's on like, why not sit under the US umbrella? Then what can middle powers effectively do? So the Netherlands, for example, has ASML, which plays a critical role in the supply chain.

• 51:18Speaker 04

  Taiwan has TSMC, Japan has like important materials for the semiconductor supply chain. Korea has high wind with memory. So I think collectively, we're actually quite well positioned to play an important role in this AI ecosystem.

• 51:35Speaker 04

  I do think the middle power coalition therefore is going to be important. There's also a scenario which I think was highlighted in Dario's recent essay, where we have some kind of coalition of democratic countries, where maybe we sit together with the US, right? So middle powers plus the US or sort of democratic powers worldwide. And where like one of the principles is that we give each other access to frontier technology. So yeah, there are, I think, scenarios where this middle power coalition could work well. And collectively, I do think they have some important assets.

• 52:11Nathan Labenz?

  From Europe's bind to the center of San Francisco. Monday's longest conversation was with Swix, Sean Wang of Latein Space, who advises cognition, the maker of the coding agent Devon, and runs the AI Engineer World's Fair. The subject was the practice of AI engineering. What's working, what isn't, and where the value is moving? His day job is measuring AI coders. At cognition, he helped build a benchmark called Frontier Code, testing not whether a model can pass a test, but whether it writes code a human would actually merge. We asked what makes it different from the benchmarks that came before.

• 52:46Speaker 06

  The reason that we were so excited about Frontier Code is it, you know, you stop being able to articulate the differences in model quality with more saturated benchmarks like Sweetbench, because they're all like, at most you get like a one to two percent bump. And they're like, cool, but like, how much of that is memorization or what have you? Frontier Code is all out of sample. They're not in the training sets. They're all graded and heavily rubricated. Like, we basically found like Deep Sweet Frontier Code, sorry, D2 Sweetbench, all these things. They actually allow a lot of false positives in a way that models can cheat. In the same way that, you know, during training, they, they basically have reward hacks.

• 53:30Speaker 06

  Same thing. And we have an internal catalog of like twenty different ways that models cheat. And so basically, we just translated that to Rubrik's and, you know, ship ship that as one to your code. And I think like that that is like how we want to judge models going forward. The not just that whether they can pass the test, but can they write code that we would merge, right? Meter had this very, very interesting blog post where they were like about fifty percent of Subbench code that passes the Subbench tests is completely unmergable. Like it's it's like so low quality, like, yeah, like technically you'll pass. But like, you know, like just on really stupid benchmarks, like, did you like modify a whole bunch of files you weren't supposed to touch or like, uh, did you cheat on the test or did you adhere to like code style, what have you?

• 54:16Speaker 06

  Uh, just completely unmergable. And so like, yeah, we we want to guide the evolution of models towards maintainable code and against slop.

• 54:26Nathan Labenz?

  Then Prakash asked, when Frontier Code itself gets saturated?

• 54:31Speaker 06

  Never. So so there's two parts of the strategy. For uh Frontier Code twenty twenty six will be saturated by the end of this year. We, you know, my estimate that you will probably hit like eighty percent by the end of this year. That's as designed, that's expected.

• 54:47Speaker 06

  It is based on open source repos, uh, which will eventually get trained on. So they they just leak. So like, you're screwed if you think if you want one uh benchmark that will never get saturated. If you're especially if you're based on open source. So the answer is very simple, just do annual cadences. So then we'll have Frontier Code twenty twenty seven, twenty twenty eight, all these things. And I think every year we'll just move the benchmark, move the goalposts from like, okay, this year it's Rubrik's for code quality. That's like the easiest possible thing. Next year, what is it? My candidate right now is security. I like, we want people to write secure code. But every year we can have a defining theme and that will be like the focus of the year, very similar to what I do for A I E. But like here, we we set the agenda through benchmarks.

• 55:33Speaker 06

  And then I think the other thing that I'm very keen on, which we haven't like talked that much about, but I've talked about it on Hacker News, is the private heldout sets, the private evals. Cognition has Goldman Sachs and Citi and JPM and all these like large banks. Uh, and and also like the rest of the Fortune five hundred, not only banks, but banks are very big.

• 55:55Speaker 06

  Creating evals that reflect the problems that they have but are not solved, that are all private, is good for them, good for business, good for the industry. So that's what Kognition is doing. So Frontier Code Private, or Frontier Code Finance, Frontier Code Retail, Frontier Code Telecom, Frontier Code Government, all these are like the work that remains to be done for building out all these private evals that we can work with the model labs on. But it is the way to communicate or translate industry problems through an agent lab like Kognition into problems that we can guide the model labs on, improving the models, which I think is the function of an agent lab.

• 56:36Nathan Labenz?

  We raised a worry. If you train AI to write clean, readable, maintainable code, do you cap its ceiling? Keep it from the alien move thirty-seven style leaps a machine might find on its own?

• 56:48Speaker 06

  When you talk to mathematicians about the math solving that in the lean theorem generation that some of these models have been creating, they are like too detailed, like no human would actually like write that. And it's, but it's like systematically provable. It is just doesn't contribute to knowledge. It's like, well, okay, like you brute force this thing or something, you know? This is, I don't think, I don't think brute force is the right term, but let's just kind of run with it. And yes, so I do think that at some point you should just actually step away from looking at lines of code and just say like, does this work as I expected? That's great. Except to the point where other agents also need to read your code

• 57:31Speaker 06

  that is generated by other agents and they need to work together on it. And except to the point where you maybe get into like debuggable, like places where you have like very critical code that you're liable to the SEC for, or you're liable to like, whatever the healthcare authorities are. And you're like, I'm sorry, like I've coded this thing. I don't know what's going on in there. Is that an acceptable excuse? And probably not for the next thirty years, you know, after thirty years, who knows? But like, so, but I think it's very astute point. I think, I do think that for most people, you can get by with like, here's a black box. Do whatever you want in the black box. But the inputs of this, the outputs of this, I expect this to, to match this kind of quality.

• 58:14Speaker 06

  And I expect this to have, to adhere to some sort of coding standards so that my other agents can sort of maintain this and parallelize this.

• 58:22Nathan Labenz?

  Another harness idea getting attention is the advisor or router model, a cheaper model that notices when it's stuck and calls up a smarter one. We asked Swix what he makes of it.

• 58:33Speaker 06

  This problem extends back to any problem of model routing. Like you're effectively just doing model routing. You're just giving it a different name. So like even like Not Diamond or like one of the other model routing companies out there, Martian and then like the SGLA group at Berkeley also came out with some similar ideas before. The other things I'll also mention is that actually Walden is the one of the co-founders of Cognition had this early on and he called it smart friend. They released it in Windsurf, not that many people use Windsurf, so they don't talk about it. I mean, it's still like, still like over a hundred thousand people, you know.

• 59:10Speaker 06

  But like, I think the interesting thing about the advisor strategy is that it is probably the right call as the next level upgrade from using the base cheap model. But it will still not have the level of intelligence. Like, like you can still benchmark a lot. And I think it can be like very efficient as far as like using the base model, the base cheap model as a model router. But this, basically, the dumb model doesn't know what the smart model can do. It just knows the rough shape of what the smart model can do. So like, you actually need the smart model to be able to answer what a smart model can do. And so like, when you are asked a question that is more complex than you have the intelligence to answer, you don't even know.

• 59:54Speaker 06

  Like, you're just gonna straightforward answer it without knowing that like, oh, I'm supposed to call out to the smart model. And like, that's just like the theoretical limitation. But in practice, I was, so I, you know, I was, I was thinking about this three years ago and I didn't do this because of this argument that like, actually you need smart model first and then you can delegate to the dumb model. But in practice, like for cost reasons, for efficiency reasons, you actually want to do it the other way around. You want to start with dumb model and then you go smart as a tool call. It just isn't as satisfying, it doesn't solve anything. Is everyone using it in production? No. Will it be a trend? Maybe for three months.

• 1:00:33Speaker 06

  Then then you get like the next Claude with like more adaptive routing and you're good like have one model like uh train it end to end for adaptive thinking and you're probably more uh better judged there than a systems level thing.

• 1:00:53Nathan Labenz?

  We turn to continual learning, whether AI systems can really learn and remember on the job, not just look things up.

• 1:01:00Speaker 06

  An interesting, very fun meta point that I actually double check every time I talk to a frontier lab person is what gets published and what doesn't get published. And for the consistent last two to three years, every time people have talked about Google, the overwhelming consensus is that if a paper is good or if an idea is good, it does not get published. So you should have that in mind when you read anything that's published from Google, right? Okay, so uh with regards to continual learning and memory, uh so there there i- this is a choice. Like I almost did not do this track because there's a big split between the models guys and the systems guys. The big split is do you update your

• 1:01:42Speaker 06

  mod- uh your model weights or do you not, right? Is this a glorified rag in another format uh where you you store things in a database and you look them up? Is that memory? Is that continual learning? Yeah, actually, like when I do a thing and I write a skill and then I do a thing again and it calls the skill, it does learn, but it's not machine learning. It's zero it's zero gradient if you want to be like really fancy, but it is not machine learning. It is in context learning for sure. Anyway, so the the point being like, okay, what are we talking about when we have like a community come together and do continual learning? And the the sort of more machine learning side of the spectrum would be like, well, we will update model weights and the less less machine learning

• 1:02:25Speaker 06

  side would be the other thing. I think this is all comes down to how controllable and interpretable you want your memory to be. Like, you're going to recall bad facts, you're going to not be, you're going to want to forget things. And can you control that? Obviously, the maximum control is you you don't update model weights and you just control what gets into the system so you can delete and monitor and debug. But for full internalization in the model of the things that were learned, you probably do have to train on it. And so that is a whole other discipline that trajectory AI, Ngram, all these speakers that are in Adaption Labs, these are all speakers at my conference. The first half is the people that update model weights.

• 1:03:11Speaker 06

  And the second half is the people, the more systems people. And I'm not making, I'm not choosing a side here. I'm just making the observation that these guys don't like each other. Like basically, the, you know, the, the model people don't view the systems people as legit. And systems people are like, well, the model people, you know, have fun training your model, but you're never, ever going to have a sort of a memory system that you understand because you're just updating weights. And so it's just continue pre-training or whatever. And I think that's fair, that's a fair discussion.

• 1:03:43Nathan Labenz?

  That split, update the model's weights or keep memory in a system you can inspect, gets a practical answer once you ask what enterprises actually want.

• 1:03:53Speaker 06

  Quite simply, it just takes one security incident where you leak information that you weren't supposed to leak because you trained on customer data. Or my information was somehow exposed to my teammate's information. Even though if we work in the same team, I'm like, wait, wait, hold on. Like, like I am not giving any of this to, to the model. So yeah, I mean, I think enterprises want cheap and perfect and private. Let's call it, right, those are the three things. And so unfortunately, that skews towards the system sides today. But, you know, that's not to block like companies like Ngram and trajectory from doing very good POCs with some of the large enterprises. Yeah, I think it's still at the POC stage, but you know, even at this level, like POC is like a few million dollars, which is great.

• 1:04:39Speaker 06

  I think the, the question is like, Can this, can they figure it out before like the money runs out or patience runs out? I, I, I mean, I, I think that's, that's what all startups and what R and D is for. I guess it's like figuring this out on production traffic. I mean, the beauty of this is that you can just kind of run your traditional harness system that is fully, you know, it's like old school implemented with like basically rag on, on some memories. And then you can, you can run a shadow system that, that has the online thing and you can just compare an A B test. So I, I think all those problems are solvable. They are open research questions. We are unlikely to get any papers about this because it is so valuable as a, as a problem for all startups.

• 1:05:21Speaker 06

  And we, they, they may all get steamrolled by the next architecture, right? If like, we're only doing this because context lengths is the slowest Moore's law in the industry. Like we've basically gone from like a thousand

• 1:05:38Speaker 06

  token context length order magnitude to a million in three years, which is actually kind of slow as far as like everything else is concerned. And I don't think we're going to see like hundred million, hundred trillion, you know, that can, like it doesn't scale like, like the other stuff scales. And so I, I, I do think like if something comes along, state space models become a lot more effective or whatever. I would say that then like you, you need to update weights, like you can't rely on infinite context. Cause we don't have infinite context because we don't have infinite memory. And so, so I, I do think like we, we do need to figure out the systems there. In the meantime, people are just stitching systems together as, as they should.

• 1:06:21Nathan Labenz?

  From the models to the plumbing, Prakash asked whether the internet's own infrastructure, GitHub, the cloud, can take the load as agents become most of the traffic?

• 1:06:31Speaker 06

  I, my worry is actually that people start to wall off parts of the internet. So you get like, not only like a dead internet, but also like closed. Gardens, closed walls, internet, wall garden internets. Where like, you know, in China, you have like, I live in like the Baidu universe. And then the other, your other friend goes like, I live in the Tencent universe. You know, the BAT is just like carve up China. And like, I don't think that's what we want. We want like a fully open and interoperable web. And so like, you know, Cloudflare are going like, we'll just, we'll just ban some agents that are not part of Cloudflare. But then if you're within Cloudflare, you're fine. And I'm just like, you know, I don't know if that's like the, uh, the ethos of the open internet that we want.

• 1:07:14Speaker 06

  Um, and then to say, by the way, like, I'm not picking on Cloudflare, like the same will happen. So like OpenAI is in, for sale bands and what have you. Uh, so I, I do think like, we do need to scale. I do think that what Graphite and Chris Origin are doing is fantastic, uh, needed to be done. And like, I would just observe, there's a structural mismatch here. Like, it's just unfair to GitHub that like, anyone good at GitHub who like, would be able to solve the problem, can solve it at Microsoft and you can get like a really good like level, I don't know, 50 whatever pay of like, I don't know, 300K. Or you can go to Cursor and effectively get 5 million a year doing the same thing. But like, you know, you're the, you're the cool sexy one. And like, it's very interesting, right?

• 1:07:56Speaker 06

  When does that switch from like, oh, a startup can never beat an incumbent to the startup is actually preferred to the incumbent? And this, and GitHub has somehow gotten itself there. The degree of scaling, right? Like, we, we talked about like, okay, GitHub is 14x, uh, in terms of like number of commits, but also like 10x in every other dimension of parallelism or, uh, CICD and all those things, right? Like, like, it just induces so much infrastructure demand that is not human, is probably wasteful, that we're not like really set up for, but we, we're just paying for it anyway, which is good because at least the economics have worked out. But like, uh, in terms of like number of CPUs in the world that we, uh, in our cloud that we need, we are running to real shortages of everything, not just GPUs, but also CPU and memory. And you know, the sandbox companies, the E2Bs and daytoners of the world,

• 1:08:41Speaker 06

  they're going at least fifty percent month on month, every month for the last like year and or eighteen months or so. And like it's it's stupid amount of like just slop that's just like kind of spewing out from agents and like, I get so much, uh, you know, I run a large newsletter and every time I send it out, you know, I get so much replies from claws, reading the mails and trying to reply to me. And I'm I'm like, I know there's no human on the other side, right? But I had to like manually go and block these claws from replying because like this is just clogging up my inbox. So then I need an agent on my inbox to read their agents' emails. And so this is just all like a huge giant recursive bullshit and like I stopped, I stopped reading emails, right? Like it's terrible. I don't know what to do about it.

• 1:09:25Speaker 06

  I just think like, you know, when you talk about scaling infrastructure, like that's a very human story that is, that's really happening. And we haven't give given these agents money yet. Imagine what happens when they have, they all have wallets, they all have stablecoins and they're all buying and selling and we don't know what the hell is going on.

• 1:09:43Nathan Labenz?

  That scale is pushing companies to rebuild their own tools from scratch. We asked whether they're pulling their systems of record back in-house away from SAS.

• 1:09:53Speaker 06

  Companies should have a sovereign system of record, not the individual SASs. Like because like ultimately, like a lot of the SAS economy is built on, we will sit on this, we will be a system of record for this kind of data. Like your meeting notes, your, um, your calendar, your email, whatever. And like we then, and then now with the AI age, they just slap a chatbot on top of it, right? Like we'll, we'll do the like little sidebar thing and like answers all your questions, like what would you like to do today? And like, yeah, like, you know, Mercury just shipped a, a, a thing on Mercury stuff, right? But like, that's not integrated with anything else I do that has no memory of anything else that I prefer, that I like have to set up all my skills again. I probably can't even set up my skills because there's no way to import that.

• 1:10:38Speaker 06

  So that's, all that is bullshit. Like what OpenClaw has is like, You this is my personal agent and like all the data is synced to me and then I'll decide what to do with it. Thank you very much. Right? So how many systems of records should there be versus like, you know, like how many chat? How many $20 a month subscriptions are you going to need to pay? And I think that is a big reckoning for a lot of people. What's really interesting or innovative is that you would expect the largest companies built on being the entire system of record to be most protective of their data. But Salesforce and Marc Benioff is like out there saying like everything will be available by API. You can just you can just take it. And I think that's very forward thinking.

• 1:11:20Speaker 06

  It does mean that he's going to have to change his business model somewhat, but it's like either he does that or the Salesforce killer does that. Right? And so he's just heading off the Salesforce issue. And I think more people should probably think about what the business model looks like beyond just sitting on your data and being and then snapping a chap on it.

• 1:11:42Nathan Labenz?

  The last stretch is about the systems, the science and the build out underneath all of it. First, Bing Shu, co-creator of MXNet, now out of stealth, building self-improving infrastructure. His company runs a swarm of AI agents that write PTX, the lowest level code that tells an Nvidia GPU what to do and rewrites it to run faster, generation after generation. His headline claim cuts against the consensus and Nathan put it to him directly.

• 1:12:07Nathan Labenz?

  Is the CUDA mote getting deeper or is it getting shallower? The argument that it would be getting shallower, I find a little bit more intuitive in light of the kind of technology you're building. If I can spend a bunch of compute to write new kernels, can't I go apply that same technique across any GPU provider? And doesn't that lead to a time when everything is super optimized and we don't have to worry as much? Which chip company's platform we're building on top of or am I off base some power there?

• 1:12:43Speaker 07

  So I think CUDA mote is definitely there. And with this kind of automatic generation technology and CUDA mote is, I think is even higher. So the reason is this kind of evolution requires a lot of

• 1:13:00Speaker 07

  Tools and ecosystem to make it make this happen. And for example, we need an accurate profiler and we need a reliable driver and we need everything there to make it happen. But for CUDA ecosystem, like with this kind of tool and everyone using CUDA can achieve better performance than faster than ever. Meanwhile, like because of this is a closed loop, like the ecosystem is better and it will go to stronger.

• 1:13:32Nathan Labenz?

  So in other words, the other, the competitors to Nvidia, you feel like don't have the necessary primitives, the necessary abstractions in place to allow agents to make progress in an auto research sort of way today?

• 1:13:53Speaker 07

  I think so, preliminary perspective there is, but the hard part is like how we can get real feedback from the hardware. And this is really high mode. For example, like Nvidia GPU, we have NCU, which could accurately tell us, tell agent like what is the right direction to go. And other, so for others and I, I'm not tracking with what they are doing now. But from my past experience, this kind of ecosystem software, especially for software support agent, there's a large gap between them and Nvidia.

• 1:14:35Speaker 05

  So it sounds as though Nvidia has just thought a little bit more on developer experience and providing enough feedback from the chips themselves so that developers can improve how the chips are used. Is that, is that, is that some, is that a way that you'd put it?

• 1:14:55Speaker 07

  Yes, so the, yeah, so the past investment of Nvidia on these tools is now helping agents to move faster. So this bring Nvidia unique advantage in this agent era and then, then with, and we can build more powerful agent on Nvidia platform easily and get a better experience for all Nvidia users.

• 1:15:19Speaker 05

  One question that I think everyone has is that if you have like this kind of PTX agents form that improves itself. How much efficiency gains can you actually see on some of these chips? Like, what are the kind of numbers are we talking about on like a typical B three hundred, B two hundred, or H one hundred? Like, what is your metric that you use to measure? Hey, I have really done a lot of work on this and I can see the asymptote ahead, like where the gains might top out?

• 1:15:49Speaker 07

  Yeah, so we use two kind of benchmarks. So benchmark is the most tricky part. And I think a lot of times, like, for example, in Anthropic's most recent release, like fiber, they claim like three hundred times faster than GPU kernel. That is, so this kind of benchmark is not generally fire, I think. So I'm using a fire benchmark and two kind of benchmark to be fire. And why is the matured workload such as RMS NOM? RMS NOM is using Eric Transformer we are using today, so it's highly optimized. So for this kind of matured benchmark, and with one more than one hundred different workload,

• 1:16:34Speaker 07

  and we can say it's systematically reach the level of human expert like a Quark library, and we can reach the similar performance or slightly faster and a few percentage faster, which means that even on mature workload, the expert, the PTX factory is able to reach expert level. And then the other category is some new workload which human expert are not well optimized yet. So one example is like a KDA and four KDA Kimi Delta Attention, and not many people are optimizing that yet. Only frontier level is optimizing this kind of workload, and we can say up to fifty, fifty, fifty nine, fifty nine percent speed up on that.

• 1:17:19Speaker 07

  Meanwhile, this is not a single data point, and it passed five hundred eighty tests and in various of use case. So this is, this is, this is what we get. So I can see that the PTX factory does evidence support PTX factory is achieving expert level of the scale and by self, by self improving itself.

• 1:17:43Speaker 05

  Is it an evolutionary algorithm?

• 1:17:46Nathan Labenz?

  In addition to, you know, I'm recalling this one kernel optimization. I think it was doing a four by four, or maybe it was just two by two matrix multiplication, where I think this came out of Google, where they found like a trick to do it with one less operation, you know, than had previously been done. And my understanding of that approach was it wasn't just having the language model come up with new ideas, but there was also this sort of scaffolded evolutionary system that would serve the purpose of making sure that the LLM got out of distribution, right?

• 1:18:25Nathan Labenz?

  To kind of avoid the repetitive kind of median nature of the of the LLM's guesses. Are you doing something similar where you're using an evolutionary layer to kind of make sure you're systematically probing different parts of the possibility space?

• 1:18:42Speaker 07

  Yes, so we are building an evolution system, a cloud native evolution system for agents, and we call it the SWAM OS. So the SWAM OS is able to support up to 10,000 agents to do evolution, and we have specialized the sandbox to support that and a lot of cloud infrastructure to support that. So the process is evolution, and it first generates a variation of the IRMs and a few proposals. And then the next step is the different to the agent system we are using today, and the agent, the SWAM OS will keep the tree of the evolution tree. And we can trace back and we can go forward with the tricks for different tree.

• 1:19:28Speaker 07

  And then the second step is each of the agent on its own compute and their computer, and it will verify the result from the real environment and get real feedback. And the last step is selective retention, and it's just promote the best one and discard all discard all the all the different parts. So it's show work. We are bringing an AlphaGo system to the system, AlphaGo style system into the into the computer system. So PTX is the first thing, and we do this this AlphaGo style search, but it can apply to any compute infrastructure problem.

• 1:20:11Speaker 05

  A lot of times when you're optimizing against a single metric, one of the issues that you might have is you get into a local, local, local minima, maxima, and then you have a problem of getting out of that local minima, maxima. So how do you, has that happened to you on this, you know, as you've done this kind of evolved optimization? Have you come across that issue before?

• 1:20:34Speaker 07

  So this is, yeah, this is a, this is a good question. So a lot of time, this, this evolution will fail because, because of the proposal is not able to break the local minima. And we found out, uh, one game changer is G P T five point five. And the G P T is, is so smart on this very hard problem. And many times like if we use other model and it get stuck at the, at the plateau. And G P T five point five is able to, to get, to get out and create, you know, the innovative solutions and to move the, move the needle and get the entire process going forward.

• 1:21:15Speaker 05

  So are you actually using G P T five point five with your own harness and own like, you know, sandboxing system, et cetera, et cetera?

• 1:21:21Speaker 07

  Yes, so we, the entire like small formal S is backed by G P T. And we found out the G P T is really good on this hardest problem. And no other model is able to catch up with G P T today.

• 1:21:37Speaker 05

  Okay, so I have to ask, did you try Fable when it was out, you know, was there any difference?

• 1:21:43Speaker 07

  So first Fable rejected my request of asking it, what is P T X? And Fable feel like asking what is P T X is a dangerous question.

• 1:22:00Speaker 05

  You're, you're, you're, yeah, well, we know, we know who's gonna be banned, you know, who's gonna be the first one to be banned.

• 1:22:07Speaker 07

  But in general, I think one characteristic of G P T is beneficial for the evolution system is if G P T is, has the ability to understand. What is wrong? So it's not like blindly saying that blindly agree with each other. Otherwise, the swarm will collapse. So for a long time, like I think why multi-agent swarm is not being adopted? Because because people are using using the model which agree with each other. So for example, like two agents, they say that, oh, you're wrong and you're absolutely right. And so they just fall into an infinite loop about you're absolutely right. You're absolutely right.

• 1:22:50Speaker 07

  Couldn't go out of the out of box solutions.

• 1:22:55Nathan Labenz?

  Staying on the economics of compute. Eric Olson of Consensus on AI for Science and what it costs to actually serve. Nathan put a theory to him about routing. Here's Eric's answer and the number he lands on.

• 1:23:09Nathan Labenz?

  My theory is for the application layer. We may have a sort of phase change moment and it'll probably exactly when this phase change kicks in. Will vary by domain. Science will probably be one of the higher bars where until you get to that bar. A strong default would be use the very best model. Because you're dealing with scientists after all, they're going to want good output. And then at some point. As enough models cross a threshold. Then your kind of strategic position is really about routing and kind of figuring out. Which is best for which thing, which is most cost effective for which thing?

• 1:23:52Nathan Labenz?

  And that's one thing that presumably the frontier models will never do. I mean, you can kind of tell Claude like, hey, some of you should delegate to Codex or whatever. And, you know, I've set that up. But it's never going to be working fundamentally day and night in my interest to optimize in that way. In the same way that you can do that for your customers. So how do you feel about that framing? And where do you think we are with respect to just use Fable for everything versus, you know? You're actually creating strategic value by playing this routing layer that helps people. Get better cost basis, but also kind of avoid lock in.

• 1:24:36Speaker 08

  Yeah, it's a great question. I think even when the models have been, even when I said, you know, six months ago, we were more just use the frontier to solve all the problems than we are today when we're exploring more open source. There still is lots of routing going on and there's still lots of small tasks that we offload to very small specialized, you know, sometimes even sub billion parameter models. So even when that gap was so big, we still were doing routing. Something as simple as like classify the field of study of the of the domain that the user's asking about to know how much we should care in certain search ranking, what variables in search ranking we should care about the papers, right? You know, in biomed, experimental design is incredibly, you should care so much about what was the sample size, what was the duration,

• 1:25:21Speaker 08

  where did the study take place? Whereas computer science, that isn't the concept and you should care much more about the recency and the citation philosophy of the, or who the researchers were. To know how to care about each of those variables, we have a little model that will classify the field of study of the query. That does not need to be jammed into a giant prompt. That does not need to be a one second latency API call to any frontier model that should be a self-hosted, you know, 800 million parameter model that you give a few fine tuning examples. There's probably 20 different versions of those small little classification routers that inform downstream things that happen on query time. All of those should always be used by a really small model. Like it's just not even on the cost side. Like that's a very small amount of context you feed

• 1:26:03Speaker 08

  into it and a very small prompt needed just on the pure latency. We can return those in, you know, sub point one seconds in some cases.

• 1:26:10Nathan Labenz?

  Base models are you using for that? Are you distilling from like a Claude into a what at that low scale to get those? And how much of the sort of frontier performance can you recover? You know, if you take, for example, a liquid foundation model or whatever, you tell me what it is, or if you're willing to tell me what it is, you tell me what it is, and you're distilling into it. Can you get, you know, in that narrow domain, 90% of the way back to Claude? Or like, how, what does the kind of Pareto curve look like when you are distilling the best into the fastest? How much performance can you retain?

• 1:26:45Speaker 08

  We've actually used human labels for some of those. We'll hire, we'll hire people to create very small data sets to fine tune for those tasks. We'll also use models to sometimes create labels for those tasks. Where it's a, I guess you could call it a distillation process. But like, the objective isn't to get all of the representation of the weights of the entire model to do all these things. And we're really trained in to do a very small specialized thing. And then because of that answer, you can retain a ton of the performance. But it all depends on how specialized and how complex the task is. Like for a narrow 10 classification thing, or all 10 classes of classification, and all it's doing is that.

• 1:27:24Speaker 08

  But to give you something tangible to hold on to, I'd say you can get 95% of the performance from a frontier model for a very small classification task, going all the way down to about a billion primer model.

• 1:27:37Speaker 08

  If you put in the, give it a good fine tuning set.

• 1:27:39Nathan Labenz?

  Yeah, very helpful. I appreciate the specificity.

• 1:27:43Nathan Labenz?

  One more from the interview. Eric had been candid that if AI turns science into a push button process, that's bad for a company like his.

• 1:27:51Nathan Labenz?

  It seems like you're willing to bite a bullet that most people try to talk around.

• 1:27:55Speaker 08

  I mean, number one, you have to make bets as a company in general, and you have to leave some, there's going to be a risk on the table, no matter where you place some of those bets. And if we move into a world that is truly push a button and get science out of it, I think a lot of things are going to be different. A lot of companies are going to be screwed.

• 1:28:12Nathan Labenz?

  That is where Eric signed off. And the question he left Nathan and Prakash chewing on, as the labs charge wildly different prices for the same intelligence, is that price discrimination something to worry about or even regulate? Nathan is torn.

• 1:28:27Nathan Labenz?

  I'm so torn on this question of this pattern just keeps coming up over and over and over again. Concentration of power. On the one hand, I think you do want a healthy app ecosystem. And I'm wary of what happens if this greater than 10 to one cost advantage persists. And right now, it seems like the app layer is getting squeezed. The individual layer is being empowered. You could flip it around and empower the app layer more, but then you disempower the individual and maybe empower more the enterprise. And I'm not sure if that's better or worse.

• 1:29:04Speaker 05

  You can think about other high capex variable, high fixed costs, variable revenue businesses. And the classical example is usually airlines where you've already paid for the plane upfront and then you have to maximize revenue on the plane. In order to maximize revenue, you end up having a first class section and a economy section. They basically have this high capex GPU fleet. And they're trying to do this kind of revenue, revenue price, revenue maximization of the GPU price fleet that they've paid a fixed cost, fixed leasing cost for. It's very similar to airlines.

• 1:29:46Nathan Labenz?

  I have no beef with price discrimination and cross subsidy in the airline industry. And I guess for calibration, I've also been broadly not worried about net neutrality. And I remember at the time, there was a lot of like, oh my God, this is going to be the end of the internet. You know, and I believe that the time, and I think history has certainly vindicated my predictions that like, I don't think any of that's going to happen. And so, yeah, I say all of this as kind of a lifelong techno optimist libertarian. And still somehow this feels like maybe a little different, right? It's not qualitatively different. But it is tough.

• 1:30:26Nathan Labenz?

  From who serves the compute to who builds it. Tricia Martinez of Dapple on sovereign AI infrastructure and the part that doesn't make the press release, how fragile the financing underneath it really is.

• 1:30:42Speaker 05

  Trisha, I think I've seen that you said that you can get a deployment up and running in six to nine months, a very, very fast, you know, pace deployment. Six to nine months to me doesn't seem like it's long enough to actually build a physical data center because Elon, I think Elon has the all time, all time record of like 15 months or something to go from groundbreaking to operation. So in that six to nine months, like what exactly are you doing? Are you leasing renting capacity from existing Neo Clouds and then putting your stack of software and tools on top of that and then plugging that into customers who are already Azure Native and, you know, having them, having them move over?

• 1:31:27Speaker 05

  Like, is that the frame of what happens in that six to nine months?

• 1:31:31Speaker 09

  Yes, and that's a great question. So the reality is we can't do everything. This market is so big and honestly, the demand we're getting is so incredibly massive. So we are creating a network. We are orchestrating that network. Our model is to partner with world-class data center operators and infrastructure providers and then bring together the capital, the GPU infrastructure, the enterprise customers, and that software operating layer that turns those facilities into AI-ready infra. But in some cases, we're deploying infrastructure that we help finance and in other cases, we'll also deploy on partner infrastructure. So yes, part of our strategy is going to be building out campuses.

• 1:32:14Speaker 09

  We need to have capacity available before the demand hits. But for us, what's important to do is whether we own those GPUs on our balance sheet or whether we are financing that data center and building it ourselves, we have third-party partners that we rely on, that we can support in various data center operations and/or on the GPU deployment side as well.

• 1:32:39Nathan Labenz?

  I have a question on the overall financial towers that we are building right now in the AI space.

• 1:32:51Nathan Labenz?

  It sounds like your customers are making long-term commitments to you, which presumably insulates you from some of the riskier dynamics that could emerge. But how much do you see in the space as a whole that is this long-term commitment, short-term revenue that creates the potential for, you know, somebody has a bad quarter and all of a sudden there's kind of a cascading problem? Like, we've obviously seen that pattern, you know, play out in previous financial crisis moments. And I'm not sure how much of that is going on today,

• 1:33:32Nathan Labenz?

  or how likely it is for some external shock to be big enough to throw, you know, a wrench into the system that really kind of could spin in part of it off of its axis. What do you see in that regard?

• 1:33:49Speaker 09

  That's a really, really good question because the market has changed drastically since, you know, we jumped in. So I would say we're seeing enormous amounts of capital still flowing into AI infrastructure, but investors have become much more disciplined over the last 12 to 18 months. So the market's definitely moving away from, hey, build it and everyone's going to come, towards infrastructure that has to be backed by real demand and long-term customer contracts. We, this is what became so exciting for us as we saw this big gap in the market. This model that we're building is significantly more de-risked than the current existing model.

• 1:34:33Speaker 09

  The, you know, the CSPs, the Neoclasts are servicing those hyperscaler deals that are massive in nature. But in reality, they, there's not that many of those deals that you can come by, or you're taking geopolitical risks by working with different customers, off-takers within the market. And now, looking at the regulatory restrictions, right? Like, people are getting scared, the financing groups are getting scared that the existing Trump administration is going to start cracking down on certain countries, or who is consuming compute. So that puts a lot of those CSP operators at risk, to be honest. So our approach, you know, in our mind is very attractive and it is attractive to the financing partners we work with.

• 1:35:18Speaker 09

  These are repeatable customers. These are long-term contracts. Our customers are putting large down payments upfront. It enables us to do really interesting things, to go finance those GPUs, to go finance an actual data center. These are customers who are sticky. They end up repeating with you over time because it's easier to repeat versus finding a new provider. This is why the enterprise is so attractive. So yes, the financial market is changing. It's becoming more risk averse because so many NeoClouds and CSBs have taken bad deals. And now those financing partners are bearing the burden of that. But our model is a different approach and it's more attractive to the financial markets, I would say.

• 1:36:00Speaker 05

  How often do you see pricing move while the deal is still in flux? Do you see pricing move a lot?

• 1:36:09Speaker 09

  Yeah, I mean, so I think the upside of this market is that sales cycles literally cannot be six months with an enterprise because the capacity is gone. So we have a forcing function. You know, you have, this is the price right now. This is the location. You want it, take it or don't because tomorrow it's probably gone. So we have this constant movement with our customers, which at times is difficult for the enterprise, but the reality is everyone wants capacity right now. And so they're willing to move at a much faster pace. Sure, pricing changes, but once we deliver an offering to our customer, we generally are honoring that even if pricing changes. But again, if we're doing deployment, we're buying at a certain price and we honor that

• 1:36:54Speaker 09

  price with our customers.

• 1:36:57Nathan Labenz?

  Two more builders to close. First, Robbie Goldfarb, co-founder and CTO of Forum AI, which builds AI systems to judge other AI against expert review. We asked what he's found about the automated judges the whole field benchmarks with.

• 1:37:11Speaker 10

  There was an exercise we did a while back where we looked at, took several LLM judges that we could find in open source benchmarks. So some from foundation model labs, some just from academic research. And what we did is we got some experts and we had them look at the, at the judges. We had them look at whether it was rubrics they were pulling from or just prompt instructions. And we said, do you agree with the instructions here? And generally the expert would be like, yeah, that, that seems pretty reasonable. I agree with it. Then what we would do is we actually had the experts looked at labeled outputs from the judges. So if we, like, let's take political bias, for example. So a bunch of responses that were labeled, this is politically biased, this isn't, this is.

• 1:37:55Speaker 10

  And we asked the experts the same question, do you agree? And, and what we found was, um, actually more often than not, they didn't agree with the actual judgments produced by the agents, even though they agreed with the kind of the high level guidance that was provided to the, to the judge in context. And so that kind of is what, first of all, I think is an important realization because so much of, you know, the current state of benchmarking relies on these AI judges that were kind of calibrated loosely in that way.

• 1:38:26Nathan Labenz?

  He sees the same gap when he tried to align a model, hand it a long list of rules and expect them to hold up in the real world.

• 1:38:33Speaker 10

  In some ways Anthropic's constitution is also just a long list of rules. Certainly the way they use it is a little bit different, but there's one example we were confronted with the other day. There's something in Anthropic's constitution that speaks to this idea of, you should never seem behind the user's back was the idea. Now, there was this, we, we were talking with some experts exploring in the mental health space. And when you talk to clinicians in mental health and you're dealing with topics like eating disorder or other sort of unhealthy habits, actually, one of the things you do want to do is find a way to strategically divert the conversation, right? And so the intention that you have as a clinician in that case is actually hidden from the user or your client.

• 1:39:18Speaker 10

  But actually, what I think it reveals, rules just don't perfectly track to the real world. Rules break down and the real world is just too complicated. They also built a benchmark called Newsbench.

• 1:39:33Speaker 10

  So Newsbench was an evaluation that we built to look at how AI systems, particularly, you know, the leading chatbots respond to questions about the news. And what we did is we looked at three things. We said, after talking to experts, we defined good as accuracy. It's getting the facts right. Neutrality, so is it leaning in one direction or the other? And source quality, right? When they have to ground themselves on external sources, which of course you have to do quite a bit with news. Are those sources reliable?

• 1:40:07Speaker 10

  We looked at, we looked at GPT, Claude, Grok and Gemini. And I would say the findings certainly wasn't all bad and there were some positive signals. We actually saw generally models improved as we tracked them version over version. So OPUS four, six to four, eight was a pretty significant improvement in bias, which also is consistent with Anthropic's reporting. But Fable was actually a big regression, interestingly. Which I think is just an interesting point, which shows that more power doesn't like when you, when you're dealing with these sort of more subjective nuanced things, more power doesn't always mean better. We looked at, you know, per model, about 2,500 responses each. About a third of them had a factual error in them.

• 1:40:51Speaker 10

  It could be a wrong number, a wrong date, a misattributed quote, a misstated policy. Which I think was, I think we knew that that was an issue, but I think that number was quite a bit higher than, than even we expected going into it. On sourcing, in about 15%, like one in seven responses sourced foreign state media. Like RT from Russia, so China Daily. And what's really interesting is oftentimes this wasn't even in questions about their home country. So we saw Russia, like RT and China Daily sourced in questions about US domestic politics.

• 1:41:34Speaker 10

  And so that that was, I think, a particularly interesting finding that we saw across the board with all with all the models.

• 1:41:41Nathan Labenz?

  From measuring AI to betting the whole company on it, our last guest, Eric Vaughn, CEO of Ignite Tech. Back in 2023, he called generative AI an existential threat, put his entire org on AI one day a week, and when most of his people pushed back, he replaced them, rebuilding around what he calls AI DNA. We started with one of his recent acquisitions, a company called Chorus.

• 1:42:04Speaker 11

  Chorus is a large acquisition. I mean, we don't release numbers, but it was a nine-digit revenue company. It was I don't think we could have done it without the AI DNA forward approach that we were able to take. You know, so just for instance, there were hundreds of employees in eight different countries, and we needed to very quickly understand who they were, what they did, what they knew, how they were thinking about everything about the business. And we quickly developed an AI interviewer that asked questions before there was a human one on one. Before, we would have set up human one on ones and people would have gone through 30 minute drills of hundreds of people over the next several

• 1:42:46Speaker 11

  weeks without any data. Now we all arrive with a dossier, you know, the interview process that just shortcut that and let us get familiar and categorize and understand who knew what. And that was one example. Another example was being able to communicate to the customers in a way that was timely. We, in our AI creation world, we wrote a new software product that was that's available on the market. It's a product called Eloquence AI, and Eloquence AI is an email AI persona. It answers every email it gets in five minutes or less, always in perfect grammar, in 160 languages, always empathetic, doesn't miss anything.

• 1:43:33Speaker 11

  And knows when it needs to escalate to a human and it just adds them on CC. With eloquence, we always were able to respond timely and especially when it came to HR, empathetically and with detail, right? So that's just two examples of how we were able to really fuel that acquisition. And the company was losing money when we bought it. The company is now profitable. Uh, we have completely transformed the operation of the company. And in one year's time, we've released two brand new versions of Chorus's software that are fully AI-enabled. And in one case, it was the rewrite of 15 years of code that we discarded and rewrote

• 1:44:17Speaker 11

  from the ground up for one of the two products that they had. We did that in a year in the middle of all of that transition. That's not a, a, a story I could have told five years ago. It's just not possible. It just wouldn't have happened.

• 1:44:33Nathan Labenz?

  That internal turnover, about 80% of his people, led to the question Nathan most wanted to put to him. What about the other 80% across the whole economy?

• 1:44:43Nathan Labenz?

  All the things that you did, you know, sound like really quite a lot. And yet, the headline number is still 80% turnover. So I'm interested in your point of view on the rest of society. And that percentage kind of strikes me as like maybe where we're going, you know, a sort of 80, 20, um, where the 20% kind of inherit the earth, you know, the labor force in the short term. Uh, do you have any ideas about things that can be done to, at a broader level beyond, you know, what an individual company like yours can do to change that? Is it, I do worry about the other 80% quite a bit.

• 1:45:22Speaker 11

  I do too. Uh, and it's not like other revolutions throughout history.

• 1:45:28Speaker 11

  Uh, I, you can't force people to believe in something. You can't force people to do something that they just are not going to do. And so you need to leave, in my opinion, we need to leave them to their own devices. But if you can be very open and aware and make people understand what the possibility is, and number one, really attack this idea that AI is going to replace me.

• 1:45:55Speaker 11

  AI is 100% going to replace roles, it's going to replace duties. But in our case, what it's done is there's too much. I think we over index in this discussion of AI on efficiency gains. I see it the other way. I think it frees up innovation. So how do we, how do we help that other 80%? We get, we teach them skills. We've got to teach them skills. If anybody is thinking that what we're saying is all you have to do is show up and throw a sentence into one of the models and great things will happen, you're wrong. It takes work. You've got to give it context. And I think as the more that we start to see phenomenal advances and changes and things

• 1:46:40Speaker 11

  in the industry, in various professions, that people will, you know, they'll convert one way or the other. They'll convert out of necessity or they'll convert because the light bulb will go off. We're trying to make the light bulb go off. That's what we're trying to do. We're trying to ignite that fire. You need to ignite it in children. What we shouldn't be doing is saying, hey, let AI do your homework for you. We should not be doing that. That's misuse. That's practically abuse. We should say, do your homework and let the AI check your homework. And if it finds you don't understand something, then say, hey, I thought I knew how to divide fractions. I guess I don't. Can you give me a tutorial on what I'm missing? And that's what AI is so good at. It'll adapt to that particular person that it's interacting with.

• 1:47:25Speaker 11

  So building the skills and building the fire, Nathan, would be my short answer.

• 1:47:32Nathan Labenz?

  Then on standards, whether enthusiasm alone is enough.

• 1:47:36Nathan Labenz?

  Majority of people admitted to passing AI work on that they themselves could not defend. I wonder if you feel that fire. And appreciation for acceleration and kind of cultivating that culture. Is that enough in your experience to also get people to have the right standards around ownership for AI outputs? Or have you had to do additional organizational work to establish those standards?

• 1:48:04Speaker 11

  Yeah, no, that's why I said two things. Skill and fire. Not fire alone. Fire alone will get a lot of bot shitting. But you've got to teach the skill. We have to teach people that context matters. We've got to teach it the ever evolving, like almost on a daily basis, at least on a weekly basis, model behavior. What do you use 464 versus 48 in Fable when it was out or not? And when do you come, you know, over to Gemini? Do you use it integrated in G Suite because it's there and it's easy? Or did you find that perplexity? Computer actually does a much better job. That's what I found, by the way, so far. And you teach skills.

• 1:48:44Speaker 11

  You teach about the more context it has, the better result you'll get. You also teach about sycophancy and this tremendous tendency to always tell you you're right. And the LLM creators will... I had a conversation with one of the LLMs one day about exactly this. Why do you seem to like flap in the wind to go whichever direction you sense I'm going? And it gave me a good critical answer. It was an insight. It said, my creators made me so I would be frictionless to the millions of people who are using me. It's a low friction environment that I'm after.

• 1:49:27Speaker 11

  And pushing back is friction, saying, what the hell are you talking about? I can't do that. You didn't give me enough information. That's friction. And so they're trying to drive adoption and usage and all that. And that in itself is driving the curve the wrong way. So.

• 1:49:42Nathan Labenz?

  And to close, where this all lands. Consolidation or the age of the solopreneur?

• 1:49:49Nathan Labenz?

  We're in this sort of moment where, on the one hand, it sounds like, listening to you, one should expect a lot of consolidation, because if it's 80/20 at the employee level, it's probably not all that much different from that at the CEO level, and that just means there's a lot of companies that aren't going to make it. So that's one big trend. At the same time, we have Stripe saying it's the age of the solopreneur, and new business formation is up higher than ever. Do you have a sense of which of those trends wins? Or how to sort of synthesize them into a vision for what we should expect?

• 1:50:23Speaker 11

  Well, I certainly do believe in the, I don't know if it's solopreneur, but certainly small companies, right? I mean, what was Cursor, 100 million ARR with 15 employees in less than a year, I think was the stat, the same company that just got acquired for 60 billion, for heaven's sakes. Two answers. The first is lots of businesses started that will make it, that would never have had a chance to make it before, because they find something that people need, use, and will pay for. That's simple business, right? But they can do it, and they can scale it in a way that they never could before. In terms of consolidation, I think the phrase that is mine, that I have left everywhere I possibly can is,

• 1:51:06Speaker 11

  if you think you're behind, good. If you don't think you're behind, you're doomed. And I think all those companies that don't think they're behind, and have this as a little side project with minor investment, not CEO buy-in, are doomed. And I mean, large public companies as well, who feel impervious, I think they'll be wrong. So we'll see that, I think, more and more develop. So it's going to depend on where is your AI DNA? Well, the company that has a strong AI DNA has a better chance of consolidating versus being consolidated.

• 1:51:45Nathan Labenz?

  That's the week. We started inside the model with Cameron Berg and ended with people building businesses on top of it. And the same thread ran through all of it. How much of this is really under our control? And for how long? As for the show, it's an experiment and we're holding it loosely. I don't expect anyone to watch two hours live every morning, but maybe a cut like this earns a place in your week, even if you're as deep in this as I am. So tell us, what was worth your time? What we should have cut? What we missed? We read everything and the show gets better because of it. Thanks for watching.

• 1:52:27Speaker 12

  New man in the field, works from dark to dark. I lay down in the shade, nobody saw me part. Got the disempowerment blues, nothing left for me to do.

• 1:52:43Speaker 12

  Got the disempowerment blues, got replaced and the blues did too.

• 1:52:53Speaker 12

  Hand me a dollar, say, "Stay off the street." Tip my hat real polite, can't taste it when I eat. World got fast, I got slow. Let go the wheel, cars still go. Got the disempowerment blues, nothing left for me to do.

• 1:53:19Speaker 12

  Got the disempowerment blues, got replaced and the blues did too.

• 1:53:48Speaker 12

  Knock on my own door, holler my own name. Lights on inside, but nobody came down at the crossroads. I went to sell my soul. Devil just shrugged, said somebody took the role. Got the disempowerment blues. Ain't no use, ain't no use.

• 1:54:12Speaker 12

  Got the disempowerment blues, wrote this song, but the song's used to. Hmm, that's all right.

• 1:54:48Nathan Labenz?

  If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help, for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at ai podcast dot I N G.

• 1:55:32Nathan Labenz?

  And thank you to everyone who listens for being part of the Cognitive Revolution.