AI researchers debate how close we are to recursive self-improvement

New episode with John Schulman http://joschu.net/, Beren Millidge https://www.beren.io/ and Charlie O’Neill https://charlesponeill.com/. I got together with some of the most insightful AI researchers I know who are at the openish companies, because I wanted to hear the details of what's actually happening at the frontier and what comes next. Watch on YouTube https://youtu.be/PrSf7IOYu-I; read the transcript https://www.dwarkesh.com/p/john-beren-charlie. Sponsors * Antithesis https://antithesis.com/dwarkesh helps you trust your code. As agents generate more and more of your software, the bottleneck shifts from your engineers actually writing code to verifying it. Antithesis does that testing for you. Ron Minsky, who co-leads Jane Street’s tech group, told me that Antithesis was able to help his team shake out bugs in software that had already undergone heavy review. If you want to see how it fits into your development process, go tohttps://antithesis.com/dwarkeshantithesis.com/dwarkesh http://antithesis.com/dwarkesh * Grok Bot https://x.ai/bot has been a great way to hand off tasks. My team uses it as a producer: whenever my editor posts a rough cut of an interview in Slack, Grok Bot opens the transcript on its own computer, matches my notes to the exact moments they refer to, and uses a file of my preferences to suggest edits. Then it sends me its top clip candidates so I can review everything from my phone, which saves my editors from sorting through hours of footage. Try Grok Bot for yourself athttps://x.ai/botx.ai/bot http://x.ai/bot * Jane Street https://janestreet.com/dwarkesh just launched its most ambitious competition yet: design a protocol-emulator ASIC. Basically, if you have a chip you want to test outside of a live system, you should be able to connect it to your design and have it simulate realistic traffic. Jane Street wants general-purpose, reprogrammable designs that can work across multiple protocols and remain useful as new ones emerge. The most novel submissions will actually get taped out, and the winners will receive a physical copy! The competition is open until January 18, 2027, and teams are encouraged. To get started download the template code at janestreet.com/dwarkesh https://janestreet.com/dwarkesh Timestamps (00:00:00) – Steelmanning the case against RSI (00:18:39) – What’s driving the Chinese labs’ progress (00:28:06) – How will automated AI researchers be trained (00:33:51) – Will long-horizon RL elicit AGI? (00:45:24) – The sim-to-real gap (01:00:33) – How much progress is explained by data? (01:18:03) – Why is RL working so well? (01:24:54) – Move 37 and entropy collapse (01:28:32) – Rapid-fire timelines This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.dwarkesh.com https://www.dwarkesh.com?utm_medium=podcast&utm_campaign=CTA_1

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Dwarkesh Patel?

  Today, I'm chatting with three of my AI researcher friends, from whom I learn a lot every time we talk, and who also happen to be at somewhat open-ish labs and companies, so you guys can actually say things on the record. I'm Jordan Baron-Milisch, who is the CTO of Zyfra, which is developing open source models. John Schulman, who is the chief scientist at Thinking Machines, previously the co-founder of OpenAI, led the RAL-Let'shaf work that led to ChatGPT. And Charlie O'Neill, who is head of model training at Base10. The first question I have, if we're in 2036, it's been 10 years, and we don't have billions of crazy super intelligences that are running around, that have radically transformed the world. What is the most likely reason that that doesn't end up being the case?

• 0:43Dwarkesh Patel?

  Other than sort of exogenous political shocks, or like there's a war, or they banned AI or something. But what is the most likely technical reason that 2036 isn't like a crazy alien superintelligence world?

• 0:55Speaker 01

  I mean, like, my reason would just be like, it's got to be the sort of, like, there's been a classic thing, almost like Marvex paradox, right? Where like, we see like, you know, we think of the AI being like, if it can do this, it's going to be amazing, right? Like, if it can solve these hard math problems, if it can win a chess, blah, blah. And then it solves these things, and then it's like, not that impactful. Obviously, it's somewhat impactful, but like, not everything. It's like, if somehow that continues, and like, there's never like, the true like spark of generalization that occurs, I think that could lead to like, the AI is just being like, extremely good at kind of everything that people like, put into a benchmark, put into an environment, but like, there are still some persistent like sim to real, which is somehow blocking everything. I think this is kind of unlikely. I think we do actually see this kind of generalization even from RL and practice already. But like, if it is just like ridiculously hard to like,

• 1:39Speaker 01

  generalize meta-learning, plus like, we don't solve continental learning, and it's just like, super hard and impossible.

• 1:44Dwarkesh Patel?

  Yeah.

• 1:44Speaker 01

  Like, this would be my like default scenario in that case.

• 1:47Speaker 02

  Yeah, I agree with that. Humans have a lot of advantages over models now, and each time a new model comes out, it'll sort of, it'll catch up in some of these areas. But like, you end up getting bottlenecked by the places where the model is weaker, and where it has worse judgment, or the models can't check themselves well enough. Yeah, so there's this cycle that keeps repeating, where people think, where a new model comes out, and people are blown away, and they're like, this is it, this is the, this is AGI. But then they use it a bit, and then it starts to feel dumb after a month or so. So that cycle just might keep going, and it's hard to predict how many times it's going to repeat.

• 2:29Speaker 02

  And like, right now, you don't get explosive growth in capabilities, because you still get bottlenecked enough when you're trying to do research in engineering, that even if the model can write way more code than a person, it doesn't make you like 100 times more productive. But yeah, so maybe, maybe there are just more of these cycles than we would, than we would expect.

• 2:52Speaker 03

  For me, it's like a question of how far off, like, this global optimum of a learner you could have on a chip is like, the transformer plus like RL, basically like the current recipe.

• 3:03Dwarkesh Patel?

  Yeah.

• 3:03Speaker 03

  So like, I think people imagine that even once, like, once you have a, an agent which is better than all humans at AI research, even if it's like 0.1% better than all humans, then the fact that you can run like, you know, hundreds of thousands, if not millions of these in parallel, you can run them much faster, like, chip's going to speed up, that's going to outweigh every other like bottleneck, and like, you're eventually just going to like, hit this like very fast takeoff with recursive self-improvement. I could imagine that if we continue along the trajectory that we're currently on, with that paradigm where, you know, it's basically just like self-attention, RL, scaling up RL environments. The, I guess, like, if you think about what happened with Moore's Law, right? Like, we had this very like nice straight line, and that held for a really, really long time. But there were so many like discreet, like discontinuities

• 3:48Speaker 03

  and innovations that had to happen to keep that scaling law going. And the same thing has kind of happened with LLMs. Like, we had this pre-training like scaling law, and then that was kind of like, you know, hitting the diminishing returns. And then we came up with like RL and solved that. And then we got this new like, you know, diminishing returns curve to hit that made it keep looking like a straight line going up. And so, like, if it requires another one of those discontinuities to solve, like, I'm not sure that like, the current method of like, training LLMs with these RL environments, even like RSI-targeted RL environments, would be able to discover that discontinuity. And if not, like, we're probably going to hit this asymptotic like curve where it like...

• 4:25Dwarkesh Patel?

  So, but do you think this continuity will be harder than anything that's come since 2012?

• 4:31Speaker 03

  If we had the answer to that, that we'd kind of have the ability to implement it. But like, maybe there's the distinguished, we should distinguish between a discontinuity, which adds to the current paradigm. Again, it's like cumulative, like, there's something beyond the RL that we have to discover, and maybe they're capable of like, you know, connecting the dots in that straight line. Or like, but like, again, how far off the global optimum are we? Do we have to go back and throw out like, you know, gradient descent?

• 4:54Dwarkesh Patel?

  Yeah.

• 4:54Speaker 03

  And like neural nets in general? And I don't think like, if you continue to scale up the current paradigm, an LLM, no matter how many LLMs you're running, are capable of necessarily discovering that if it's too far away.

• 5:05Dwarkesh Patel?

  Yeah, the only hope really is if deep learning just can't get us to an AI which is at least, can dominate human research and human development, including the human ability to come up with new paradigms and so forth. Or like, I don't know, maybe humans would also never have discovered the next learning architecture, but to the extent humans could have discovered it eventually. But it just seems like, I don't know. If, if you just look at the progress that's happened since 2012 till now, and you just continue that on, I mean, I know it's been powered by huge amounts of compute scaling and so forth, but it would be weird if like, it just didn't get to the point where it could like dominate humans, at least in R and D.

• 5:48Dwarkesh Patel?

  Especially, over the next few years, there's going to be, Ryan Greenbub was on the podcast recently, and he made this point, that you could imagine, as AIs get more and more capable and are capable of making progress on simulations which incentivize getting better at not only AI R and D, but generally at science. So this is a thing that all the labs are targeting, many startups are targeting. Or another intuition pump is if you look at the Elo score of chessbots since the 80s, there's just like a very linear increase in Elo over time. But there's this huge discontinuity as they cross the human range of, human experts always win against AIs, to like, human experts never win against AIs, as this linear increase in Elo happened. And you could think, I agree with your point that,

• 6:30Dwarkesh Patel?

  so far, AI capabilities have not been that big of a deal in terms of their end economic impact in the world. But that's just because like, they're slowly rising in Elo relative to humans.

• 6:39Speaker 03

  Yeah.

• 6:39Speaker 01

  Yeah, I mean, I agree, it would be very, I mean, the only way for this to not happen is if like, as you said, somehow asymptotically, like just before, basically, because we're already pretty close, in my opinion, to like, where we'll start crossing like the human Elo score. And so we'll need to asymptot before that. And like, that's the only way, you know, in this scenario you posed, where like, somehow we're sitting here in 2035, and like, everything is normal for this to happen, I think. Yeah, I mean, the only other way is like, there's like some dramatic like regulation on AI.

• 7:02Dwarkesh Patel?

  Yeah.

• 7:02Speaker 01

  It's like, this is kind of what I see as like, the most likely way for this scenario to happen, actually, rather than a technical thing.

• 7:07Speaker 03

  Yeah, I, I think there's different kinds of research. There's like, research where it's like the order research style, where the objective is already specified very cleanly, and you're optimizing that objective. And I think everyone is picturing like, if we continue along this path of like, you know, making pre-training loss go down, making RL environments go up, that's going to lead to like improvement. But like, you know, maybe what Ryan is talking about is like, this much more open-ended type of science, which is required for like paradigm shifts.

• 7:30Dwarkesh Patel?

  Yeah.

• 7:31Speaker 03

  Where we can't specify the objective, and the AIs are definitely not able to specify that objective either. Like, we have to be really, really careful about how we specify objectives for any of these things.

• 7:39Dwarkesh Patel?

  And maybe your point is that like, the nature of the breakthroughs that have happened since 2012 is that we have found, like, in 2012, people weren't saying, or I'm assuming, I don't know, you guys were there, or at least, John, you were there, but I was not...

• 7:51Speaker 03

  I was in primary school.

• 7:53Dwarkesh Patel?

  Actually, John, I'm curious, you're like sort of wisdom of the ages of,

• 7:58Speaker 03

  Yeah.

• 7:58Dwarkesh Patel?

  Or wisdom of like, being in the trenches way back when. But presumably, a big breakthrough was realizing that next token prediction is the, like, you wouldn't have thought that NanoGPT speedrun is the thing to be optimizing for in 2014. But now that we have come to this new paradigm, that's, you wouldn't think to do a speedrun on that and have AIs get really good at that. But maybe there's like a next inner loop to optimize that the AIs wouldn't anticipate. And there's an outer loop of like revenue or something that eventually should be strong, but it's a very slow outer loop.

• 8:29Speaker 02

  Yeah, in fact, I remember in the early OpenAI days, having the intuition that actually just do, like, minimizing log loss wasn't going to get you to intelligence, because like, the important bits are accounting for such a small fraction of the loss, that like, it was going to be overwhelmed by noise. So just training a language model on next token prediction just wasn't going to learn the interesting things you wanted to learn. And we needed to craft better objectives that would put more emphasis on the important things. And like, you can make all sorts of arguments for this, and you could say, oh, humans probably don't learn how to like,

• 9:14Speaker 02

  we don't learn how to model everything in our environment. We can't, most, like, people can't create a photorealistic reproduction of some kind of scene they've looked at. So... There must be, we must need a better objective, but then it turned out that it just worked anyway.

• 9:32Dwarkesh Patel?

  As you were pointing out, the inner loop, even in current AI research of like, post-training benchmarks or whatever, doesn't necessarily translate into what users like.

• 9:41Speaker 02

  Oh, yeah. I mean, the whole field relies a lot on generalization and it's very hard to predict when you're going to get generalization or when you're going to get some kind of out-of-distribution generalization. So we know that if you train on the task you care about, you're going to do better.

• 9:55Dwarkesh Patel?

  Yeah.

• 9:56Speaker 02

  But like, the most important advances are often the types of generalization that we have no right to expect. So, for example, from just pre-training on this very naive next token prediction objective to like various tasks of interest where some very, that require understanding of the input in some deep way or learning some skill from pre-training that's like very rare and like not very like heavily represented. And then like also generalization from these verifiable tasks to less verifiable ones. This is also a type of generalization

• 10:39Speaker 02

  that there's no reason a priori to expect it.

• 10:43Dwarkesh Patel?

  Yeah. So this is an interesting question because one intuition pump that you could have for why you would see some sort of singularity very rapidly without even scaling up the inputs to AI progress that are not just AI labor is that before every single experiment you run, that's you know, like a seven-figure experiment, you spend an equivalent amount of compute on AI labor. And so you just have automated versions of you guys spending a century thinking about like, what is the optimal experiment to run? Do you like small-scale ablations? Developing literally like a century's worth of theory. So going back even before like deep learning, before you decide what experiment to run, doing extremely optimal like setting up of the experiment.

• 11:27Dwarkesh Patel?

  Then you do a century of thinking after the experiment is over, where you're like analyzing what happened and what the next experiment to run is.

• 11:33Speaker 02

  Yeah. Well, I think if, if you think hard enough, you probably could have expected some of these things beforehand. Like, like there is probably some very clever way to do a small-scale experiment that'll let you build the theory that then will generalize to the large-scale experiments. So I would expect that like we're nowhere near the ceiling of how well you can do research. And like, I would imagine a future where, where AI is like, is doing a lot of like analysis and theory building, like spending a comparable amount of compute to the amount that you're spending on the experiments themselves, doing various kinds of analysis and building a theory around what we've seen so far.

• 12:15Speaker 03

  I think there's like really concrete examples of this when the objective is well-specified. So again, like, all thinking can do is like, update your posterior based on like, you know, the bits that you've gotten since you formed your prior. Like, you can't gain any new bits from like just thinking. But when the objective is well-specified and there is like this data sitting around, like I imagine there will be this big speedup in the current paradigm we're in. And like, good example of this is like, you know, if you've got an AI to think about like the Kaplan scaling laws, like, an AI at this point would have noticed that like, you know, oh, like they've just taken these intermediate checkpoints and didn't account for like the annealing. And so like, this is wrong. And like, that would have caught that like years earlier, we would have made like, you know, progress like, like would have cut off a year or two of progress just from that like observation from an AI. And like, again, once the objective is well-specified,

• 12:59Speaker 03

  which is like lower pre-training loss or whatever, like there's many, many good examples where if you just thought about it a bit more, you would have been able to like cut down significantly on things that you've done. So like muP and like how learning rate scales with like model size and like realizing that model width is important in that as well. Like, I feel like you can, you can really back out a lot of these things and cut off like a lot of low-hanging fruits. I would imagine like a 10x speedup if our thing is just like maximize the objective we're currently on. But I don't see that, how that generalizes at all to, you know, come up with the right objective in the first place. Like, just thinking doesn't necessarily buy you the right objective in the first place.

• 13:33Speaker 01

  I mean, yeah, I think this is really the key question to like any kind of like very rapid RSI is like from current AIs is like, how well can AIs generalize to like learning their own objectives? Because to have any kind of like self-propelling automated loop, you need the AI to like propose objectives, optimize them, figure that out, propose a new objective, and like have this like not go off the rails at like any point for like a long, long time. Coming back to Moravec's products, there might be like a case of Moravec's products where like we think this kind of like autonomy and sort of like being like self-encapsulated so we can, you know, think of what we should do ourselves and then go do it and like have this loop as like super easy because we always do this. And like, obviously, evolution needs to create creatures that can like survive on them by themselves like long period of time. And like, this just might be something that for some reason is like really hard for the AI in the same way that like locomotion stuff is really hard. But it's like math is super easy despite being super hard for us.

• 14:18Dwarkesh Patel?

  I don't know, doesn't the time horizon increasing suggest that that's?

• 14:21Speaker 01

  You would, yeah, exactly. I mean, this is another possibility which like, but I agree, like, there's no obvious evidence for this. Like, in fact, the fact that, you know, our AI is now like super persistent and it's quite easy to do this, is kind of evidence against this. But like, this would be, you know, potentially like, one of the reasons why like we just don't get this like immediate takeoff is like if this is hard.

• 14:39Dwarkesh Patel?

  If you look back from 2012 till now, or maybe from when you started doing your research till now, what part of, of all the innovations that have happened since that time, including purely engineering ones, including purely conceptual ones, what seems like the thing that is the thing that would be the last thing humans would have to do before AI is totally automated, AI R and D?

• 15:05Speaker 01

  Probably just like intuitively asking the right questions. Like, if you can get the AI to like do any experiment, but like you need to decide what experiments to do. And like, right now, I think AIs are not very good at this compared to coding the experiments at all. Like, whenever we talk about research, they propose like a bunch of like miscellaneous things which are like very, very tiny steps.

• 15:22Speaker 03

  Or even going from like, you know, DeepMind's approach of like, we're going to solve intelligence by learning to play games at a superhuman level. That's going to be the approach to like one random researcher like Radford being like, I'm going to try and just predict the next token off a very wide swath of data. Like, and then even once Radford had discovered that, right? Like, it took a while before people decided to scale up because we had to come up with the idea of scaling laws and the fact that like you could very reliably predict these things.

• 15:46Dwarkesh Patel?

  Yeah.

• 15:46Speaker 02

  Yeah. I would say that the last job for humans or the role for humans that'll last the longest is like defining the objective and like deciding what we actually want. So like, in that vein, something like deciding what, how like the AI, the assistant should behave or what it means to be helpful or what's like the objective when we're doing RAL from human feedback is one such thing. And then like, then later like defining like constitutions and model specs is another one. And I think even if the AIs can do all the technical work, we'll have to still do a lot of that and decide what we actually want.

• 16:31Dwarkesh Patel?

  Yeah. Alignment is the final job.

• 16:33Speaker 02

  Yeah, alignment is sort of the answer, but it's also alignment itself can be kind of decomposed into, into like specification of the objective or figuring out what the right objective should be. And then like actually like achieving or optimizing the objective you've defined. And I think the first one is not going to go away anytime soon. And like, if I think about like a post-training team and why you need a lot of people to be on the team, it's, it's just because there are a lot of different like areas where you have to figure out like how the model should behave. And like, there's no way of, like, there's no way of,

• 17:17Speaker 02

  it would be very hard to automate the whole thing just because someone has to think about how, how should the model behave in this area?

• 17:24Dwarkesh Patel?

  Jane Street started using Antithesis to test their software in early 2025, and they were so impressed by the product that they decided to invest in the company. I recently caught up with Ron Minsky, who co-leads Jane Street's tech group, to ask about how Antithesis actually plugs in.

• 17:38Speaker 04

  The thing that I think is most impressive about Antithesis is we started using it in a team that was building high-assurance software and being really careful. And nonetheless, it was able to shake out bugs that were otherwise going to be really hard to find. And that's important both because it helps make those systems more reliable, but also because it helps the teams that build it to just move faster.

• 17:59Dwarkesh Patel?

  This matters more and more as code production is increasingly automated.

• 18:02Speaker 04

  I think in general, as we've been using agents more and more, the key problem that you run into is the verification bottleneck, just the, the time it takes from people to look at code and figure out, is that actually something you want to accept in your production software? And tools that make testing better are just incredibly helpful there. They just ease the verification bottleneck and make it possible for you to get more stuff done and move faster because you can have more confidence that the code generated by the agent is actually not introducing new problems.

• 18:30Dwarkesh Patel?

  To see how Antithesis fits into your development process, go to antithesis.com/thorash.

• 18:40Dwarkesh Patel?

  What is the story for why there isn't huge consolidation in model providers? There's just so many things that point to centralization here. Yeah, if you step back over the course of years, is there something that is going to prevent that?

• 18:54Speaker 02

  Yeah, I think distillation is the main thing that fights against the centralizing force. Because basically, anything that can be learned through RL can be distilled very easily. Because it's a small number of bits. It's something that you can learn from a small amount of data. So if you can get trajectories from the model that show a behavior, you can easily distill it. So I think distillation is one of the things that fights centralization. There's also, I mean, there is a possibility that there will be company-specific models

• 19:33Speaker 02

  that it'll be possible to learn from deployment and have a company continually improving its own model. And such a system could be provided by the current oligopoly of model providers or some other currently smaller company. But I think that'll change the game a bit.

• 19:53Speaker 01

  Yeah, and I also want to point out that continual learning doesn't stop distillation, right? Like, even if your model's improving every day, like, people could be distilling it every day. So it's like, the loops could just operate at the same pace.

• 20:03Dwarkesh Patel?

  Right, that makes sense. Okay, so copying model behavior.

• 20:09Dwarkesh Patel?

  I guess you need to know yourself what the right distribution to prompt is in order to get the relevant model behavior?

• 20:16Speaker 02

  Well, yeah, for just distilling with supervised learning, the prompt distribution is extremely important. So it's very non-trivial to distill a model, even if you have full access to it and have the COT, the chain of thought and everything. Yeah, it's non-trivial to distill all of the useful capabilities from it, because you need to prompt the model with something. And you need to prompt it with, like, realistic prompts. You need to have a really wide distribution of realistic prompts. So, yeah, one thing that's been coming out recently is some of the Chinese companies are probably using these router services,

• 20:58Speaker 02

  which are designed to allow people in China to use the US frontier models, which would otherwise be blocked in China. But there are all these router or proxy services that allow people in China to use these models mostly for coding. And these router services are collecting and selling some of the data. So I think this is like a very useful dataset for distillation, because it gives you the perfect prompt distribution.

• 21:25Speaker 01

  I think this is one of those things where AIs help a lot here. Like, if you actually look at, like, you know, the frontier pipelines, or say, like, the Chinese models that they actually put in their papers, it's a lot of, like, humans or, like, they get seed prompts from somewhere, which is some combination of humans, this kind of data. And then they, like, synthesize a vast coverage from those seed prompts using their existing models or, like, the other frontier models. And so it's like, you can automate, like, an awful lot of this, like, prompt distribution gathering and, like, environment creation. It's just like, humans need to provide, like, increasingly few amounts of bits. It's like, the models get better.

• 21:52Dwarkesh Patel?

  Right, or certainly, it still seems you're bottlenecked by, like, having a service which has users or users are going through.

• 22:00Speaker 01

  Not necessarily. I mean, like, yeah, that's obviously very helpful. But theoretically, you can just think about, like, what users want or, like...

• 22:07Dwarkesh Patel?

  The whole point is that we don't, the user says, "Make me an application like this." "Oh, that didn't work. I actually want you to make this new feature." But actually, let's step back and do this other thing. And capturing that whole trace is the... Or to the extent you could have done that anyways, then you just have, like, RSI.

• 22:23Speaker 01

  Yeah, I mean, like, ultimately, like, if you have this, like, fully automated loop, that is basically RSI, right? Like, the AI is deciding, the data is deciding, the training, that is the loop. But, yeah, I mean, like, it depends how much human information you need. Like, at some point, if you're just like, "I want traces that look like this," you prompt that to the model, the model will be able to, like, come up with, like, a pretty good approximation.

• 22:40Dwarkesh Patel?

  But what if you want to do, like, "Make me a really good politician," and then just, like, anticipate de novo? How would a discussion in, like, the Senate halls go or something? I just feel like, there's going to be a lot of things which are...

• 22:50Speaker 01

  Ironically, this is actually, I think, easier for the distillers than the frontier labs, right?

• 22:54Dwarkesh Patel?

  Right.

• 22:54Speaker 01

  Because the distill is just like, "I want a good politician." They go to the frontier model, the frontier model already knows how to be a good politician, so it just, like, generates those traces. Whereas, like, if you actually want to build the first model that does this, you have to, like, actually somehow, like, get data on, like, what politicians do every day and, like, build that. So it's actually much easier to, like, say, like, "I want something like this," and then, like, get, like, the AI to produce, like, a billion variations, than to, like, actually create the thing like this to begin with.

• 23:15Speaker 03

  I think you can actually make a really concrete prediction based off, like, this observation, that the Chinese labs have this routed data. So, like, I think the thing that gestated this originally was, I was saying, "Isn't it weird how Sonnet 5 and Opus 5 are, like, almost objectively worse models than, like, GLM-5.3, Kimi K3, even though they've had access to, like, not only distillation, but logic distillation from, like, Mythos?" And so the counter here was that, like, okay, the prompt distribution really, really matters. Like, you need to see what users are doing so that you can distill, like, kind of these behaviors and things in. I think the prediction from this is that the frontier labs don't necessarily have much of advantage, if at all, in RL environments now. Because, yes, like, user distribution matters for, like, general, like, behavior and so on.

• 24:00Speaker 03

  But, like, the best measure of a capability is the very, very hard RL environments you've made at the frontier. And so if you have access to those RL environments as anthropic, and you have access to logic distillation, and you've still made a worse model, then maybe, like...

• 24:13Dwarkesh Patel?

  Then real-world deployment matters more than the environment.

• 24:15Speaker 03

  Yeah.

• 24:15Dwarkesh Patel?

  That's really interesting. So, but they had to incentivize those capabilities in the first place in Fable or the frontier model. And it's weird that they can't incentivize them again

• 24:28Dwarkesh Patel?

  with, like, a smaller model or something.

• 24:30Speaker 03

  Maybe, like, maybe we're just in this weird, like, uncanny valley where, you know, like, actually trying to copy that frontier model too much, like, the student-teacher gap or whatever it is, is just, like, too large. And, like, I think people made this point with Opus, is it's like, the difference between Opus 4.6 and Opus 5 is that Opus 5 really feels like it's got this, like, AI as a judge checking every possible thing it's done. And it's like, that's why it uses so many tokens, it, like, tries to think about all these things. But it doesn't necessarily have the big model smell of Fable to know when to, like, stop doing that or, like, when's the good path to go down or whatever.

• 25:00Dwarkesh Patel?

  The richies sees the grass.

• 25:01Speaker 03

  Yeah.

• 25:02Speaker 02

  Yeah, I would offer a slightly different hypothesis. So I would say there are a couple of different axes for the environments you can create. And, like, one of them is difficulty and the other is realism. It's sort of easy to create, or it's comparatively easy to create a lot of difficult environments, like, that are just, like, involve, like, doing a much more complicated task or doing something that requires a lot more cleverness. And you could say this is like the benchmarking distribution, because a lot of the most prominent benchmarks just involve doing some very hard, puzzle-like task that's easy to verify. And then there's sort of, like, the realism axis, where you want the model to be good in the realistic coding agent setting,

• 25:47Speaker 02

  where there's, like, multiple back and forth, there's the human, and there's, like, multiple objectives. And, like, I'd say, like, the people, like, the labs who are crafting the model behavior for the first time need to push in both directions. And to get good model behavior, you need to really push on the realism axis and have, like, rubrics or some kind of human feedback that's informing the reward function you use there. But I think when, if you try to do distillation naively, you end up just sort of matching the teacher on the benchmarking distribution. And, but if you don't have enough of the environments that really exercise the capabilities in these, like, tricky or realistic settings,

• 26:32Speaker 02

  then you're not going to, you're not going to get those into your student model. And I think maybe one thing that's happening is the big models generalize better from the, like, the tricky, narrow tasks to these sort of more realistic tasks. So if you have a really good, like, realistic prompt distribution for distillation, you can match the big model really well. But if you only have this, like, this distribution of easily verifiable tasks, then you can match the big model on all the benchmarks. But you do worse on this broader distribution. So you might, that might even explain something about the smaller anthropic models,

• 27:16Speaker 02

  like Sonnet 5, though it's hard to predict exactly what they're doing to post-train those models. It could also be that they're always changing their post-training stack and they just made, they just got a few things wrong in some of these models. So they, like, I don't know, they turned up, like, something too high and created some quirks that people really don't like. So it's, like, really easy to screw up post-training in some way that doesn't show up in benchmarks.

• 27:44Speaker 01

  I mean, just one other sort of very basic point is just, like, the AI, like, the frontier AI labs buy all their data from data companies. And, like, the Chinese can also just buy the same data from data companies.

• 27:53Dwarkesh Patel?

  And they are, right?

• 27:54Speaker 01

  And, like, they are, exactly. There's a lot of people, like, you know, being annoyed about this. But, like, if they have exactly the same data and, like, they can buy that, they can also distill. It's like, it means it's quite easy to, like, keep up, really.

• 28:05Dwarkesh Patel?

  Yeah, yeah, yeah. Okay, the other question I had is... How the first models that are capable of automating AI R and D will actually be trained? Because there's a toy version, which is this thing that Ryan was talking about, which is you just have GPT-8 try to build GPT-3 sized models that are really good at like inner loop type challenges of beating video games that require continual learning, or just getting to a certain loss with like the least amount of compute, etc. But John, I think you had an interesting point that maybe that's not the way it actually will happen in practice. So I'd be curious about, yeah, by the point in which you have AIs that are actually capable of automating AI R and D, how are they probably trained?

• 28:45Speaker 02

  Yeah, I think we'll probably do some combination of learning from human feedback to absorb like the researcher's taste and just like creating a lot of practice environments, which involve like doing multi-step research projects. So I think, yeah, people will in practice do some combination of those two things and just each iteration like patch whatever seems to be most broken in the last iteration. So like researchers will be using the AIs a lot and will notice that they have some consistent weaknesses and then those things will either be patched by like collecting human feedback or like creating environments.

• 29:28Dwarkesh Patel?

  Yeah, makes sense.

• 29:29Speaker 03

  Maybe you sort of think about this as like how much of the lineage we roll back and then let self-play from there. Like, I think in the limit like you're picturing, like, you know, just giving them like a GPU and maybe neural nets or something and saying like, okay, figure out how to train a model to like do this particular task. Like, the way it currently works is like we go up to the very like edge of the lineage and say, okay, like here are the bugs like, you know, Anthropic has found and their training stack in the last few months will turn those into environments. Like, you need to train to get better on the frontier. And so you obviously lock in all the previous history of the lineage, but you could imagine a world in which you roll back to like, you know, before GRPO or something. And then you have environments which like trying to get it to discover like the best will form to like RL models on and then maybe roll further and further back.

• 30:14Speaker 03

  But I think we will be still so compute bottlenecked that like people will just keep like staying at the frontier and like dishing essentially the bugs and whatever improvements they found since the last model version, turning those into training environments.

• 30:24Dwarkesh Patel?

  Which is also really good for having non-stale, like new data between model generations. It's just, again, this is basically continual learning within the AI lab of distilling the last three months of AI research progress through environments and like RL chef type stuff back into the model itself.

• 30:41Speaker 03

  And it is distilling, right? And that's maybe why some of us feel like it's asymptotic is like you're always like just trying to get the last three months of progress. And that progress is being contributed to by AI, of course, but it also still has humans in the loop and it feels like, you know, you're just constantly inching closer and closer to what the human researchers are like finding and capable of doing.

• 31:00Dwarkesh Patel?

  Yeah.

• 31:00Speaker 01

  I mean, the one thing I will say though is like, obviously if you're just distilling on like trajectories, you can never go above it, but environments can go quite a far way above what a human can do. Like, it's very easy to design an environment that like no human can solve, but the AI can obviously still try and solve it. And so that would be the path to like go ahead of just like what the human AI research.

• 31:15Speaker 03

  Do you have like an example of like in terms of RSI or like?

• 31:20Dwarkesh Patel?

  By doing it even faster than a human speedrunner?

• 31:22Speaker 03

  Yeah.

• 31:23Speaker 01

  I mean, I feel like in AI research especially, it's very easy to define like goals, which like, you know, you could say like the loss needs to be like 1.3 or something and like no human can get there, you know, now. But like, that's a very extremely measurable, verifiable task and if the AI gets there, then great.

• 31:37Dwarkesh Patel?

  Right, and building like 100 million parameter model that beats Minecraft, that's maybe too easy, but like beats like a much more complicated game or something.

• 31:45Speaker 03

  Isn't it crazy that 100 million parameter models to beat Minecraft, we're calling that too easy? Like, imagine if you said that like five years ago.

• 31:52Speaker 02

  I would say a lot of research is not exactly like that though, where it's like hill climbing on a well-defined goal. It's sort of more like, here's an intuition we have about some way models should be better. And then we also have some idea for an algorithm that seems to go a little bit in this direction. So let's come up with a task that is sort of designed to show signs of life on this approach and like see if we get some, get those signs of life. And then if we do, we can make successively more realistic versions of the task.

• 32:25Dwarkesh Patel?

  Right, it's like a lot more guided by intuition. And then the inner loop is to elicit the, or make tests for that intuition rather than like the test itself leading to that insight.

• 32:40Speaker 02

  Right, like you're not directly optimizing for the eventual objective you care about or the practical like production objective. It's sort of, you're relaxing your objective a little bit. You're saying, yeah, let's relax on the realism axis a little bit and find some methods that actually work and then like then try to get back to realism later after the method matures a little bit. And then there's also like more, there's research that's more oriented towards explaining things and like developing a theory or a sort of, yeah, often we don't have like mathematical theories in machine learning that are that predictive, but we have like a lot of like

• 33:23Speaker 02

  more informal theories for what's going on.

• 33:26Speaker 01

  Yeah, I mean, like presumably the models will be trained on like some combination of all of these tasks and like some will be very easily verifiable, some will be like LMS judge or like just ask the human like, does this look reasonable? And then he will say, the hope would be that like these would all generalize to like these much sort of hard, sort of more vague, fuzzy kind of tasks. And like it probably will to some extent, whether it generalizes enough that like we could, the loop can become like self-sealing without humans being in the loop at all. It's like unclear.

• 33:50Dwarkesh Patel?

  Yeah, yeah, yeah. Maybe taking a step back, here's what I, here's what it seems to me that the plan for AI research going forward is, and you tell me if you think it's going to work or if you agree with this characterization. So the bet is that we will scale up RL VR training across millions of diverse environments across hundreds of different kinds of domains. And what will emerge at the other end is an agent which has like learned these basic skills, or less than basic skills around being persistent, being able to triage information in context, eventually having like end-to-end optimization of working with other agents and things like that. And such an agent will be very sample efficient within the context.

• 34:34Dwarkesh Patel?

  You know, you've done research on how you actually scale up in context learning to make it like arbitrarily long, but you keep scaling it up. And so what comes out the other end will something, will be something that it basically functions like a drop-in remote worker over the course of a week or a month. First of all, do you agree that that is a bet the labs are making? And second, is it, is that enough? Like basically learning how to learn within the simulacra within a data center and then but getting deployed into the real world, but not actually like learning from real world deployment, only learning these meta skills from the simulated environments in the data center?

• 35:07Speaker 03

  Yeah, I think it's now hard to separate out like how much of the lab's effort is going towards like direct RSI versus like making generally intelligent models that they can continue to deploy to collect revenue to fund the next big training run. I think for the latter, like, yes, that's probably just the bet they're making. Like, and it's very clear like the pattern of like where these environments are going over the last few years. I mean, like Anthropic's lineage of environments is like a very clear example of this. Like, you know, first like we just focused on coding and like we're going to get really, really good at that. And then the task horizon that we've got from coding, which is probably the lowest hanging fruit in terms of like data available on the internet to create environments, like their own internal stuff that they can turn into environments. Then we're going to generalize, we're going to go after finance next and like literally

• 35:52Speaker 03

  like just so much Excel data and all that sort of stuff in the, in the R training. And then, you know, it's PowerPoints, it's like this long tail of like the working economy and like that seemed to work really well and like a lot of the other labs and even the open source labs have now realized that that was the correct bet.

• 36:06Dwarkesh Patel?

  But what is the implication from that? When I had Dario on the podcast, the thing I asked him was, if you truly expect models, which will be human-like in their ability to learn on the job, why would you try to bake in all these skills of like working with PowerPoint or something? Wouldn't you just expect the model to be able to pick that up on while it's deployed? And so, yeah, there's multiple different explanations. One is just that this is, we expect models to get there soon, but they're not there yet. So why not amortize these skills into the model training? Another is that we're not concentrated on making it really good at widely deployed work. We just want it really good at RSI. And this is just like a way for us to like get revenue so that we can pour it back into a model that is actually like really good at doing RSI development.

• 36:51Dwarkesh Patel?

  And then like once the singularity happens, the thing that comes out the other end will be really good at all the things which seem like bottlenecked to the current generation of models. Yeah, John, I don't know if you have a take on like what, how, once you construe why there is so much task-specific knowledge in these models, if the path is like this kind of generalization.

• 37:07Speaker 02

  Yeah. I mean, if the models were good enough at learning in context, then in theory you wouldn't be, you wouldn't need to train them on finance. They would just be able to figure out, read all the, all the books on the fly and figure out how to, how to do everything in the appropriate jurisdiction. Yeah, and you could argue that you need to do a lot of this domain-specific training just to make them more efficient. So even if they were smart enough to figure this out on the fly, you still might want to do a bunch of RL and bake all these intuitions into the weights. So the model would be more efficient at runtime.

• 37:47Speaker 01

  Yeah.

• 37:48Speaker 02

  Yeah, I'd say in practice, it does seem like model providers are going domain by domain and trying to strengthen the models in the highest value domain. And I'd say that that's one of the answers to why the models have gotten so much better. It's just because the model providers have covered a lot of the high-value domains and the most common types of skills.

• 38:10Speaker 01

  I mean, I think another thing is just that it's not that expensive to do both at the same time, right? Because the models are massive, they can easily afford, in terms of their parameters, to learn everything. And there is likely some transfer, even if finance is not specifically, like the information is important for RSI. Just the general meta-learning of how to figure out what's important, how to have taste, how to do long-horizon work is potentially generalizable. There's not that much RSI data in the world as well. It's kind of hard to generate, and that requires a lot of effort. If you can amortize in this other data, get some transfer from it, you already have masses of compute and massive parameter space, so why not do that as well as, obviously, the direct commercial intent of selling your model? That makes sense.

• 38:48Speaker 02

  Oh, yeah, I'll add that, I mean, there's one question about whether this current paradigm of doing sim to real will be the dominant one forever. So basically, you look at what the real-world tasks are like, and then you try to create a bunch of environments that can be simulated, like in the data center, and you can do RL on them. And I think, obviously, this has been very successful, but it has a lot of weaknesses because a lot of things are just kind of hard to simulate, especially if they involve interacting with a bunch of humans in real time. Yeah, so there's some question about whether sim to real will be the dominant framework forever.

• 39:31Speaker 01

  I think sim to real has to be the dominant framework, while sample efficiency is kind of low. Because right now, you need thousands and thousands of interactions with the humans, and no human is going to sit there and deal with this, basically be in the loop of RL training. And so we kind of have to simulate that now to get the samples you need. But obviously, if sample efficiency improves a lot, you'd expect learning from deployment to become a much bigger part of it.

• 39:51Speaker 02

  Though there are also other things you could do, like you can learn off-policy, so you can take all the traces, and even without resimulating everything, you can potentially learn something from them.

• 40:01Dwarkesh Patel?

  Jane Street just launched a new competition, and it's their most ambitious one yet. Design a Protocol Emulator ASIC. Basically, if you have a chip that you want to test, you can connect it to this ASIC, and then this ASIC will simulate realistic traffic. That way, you can see how the chip responds without having to plug it into a live system. Jane Street is looking for flexible, general-purpose designs, not single-protocol emulators. We're now chatting with them. They suggested that I start off by trying to implement what are apparently three very common protocols: UART, SPI, and I2C. Jane Street also mentioned that they hoped that more ambitious designs will also tackle low-speed USB and Ethernet, and any other protocols that flex your chip's specific architecture.

• 40:42Dwarkesh Patel?

  Importantly, your design should be reprogrammable, rather than smashing a bunch of specific protocols onto a chip. If a new protocol comes out after your ASIC is taped out, your chip still needs to be able to handle it. How exactly it does it is up to you, but there is one hard constraint: your design must target an open-source 130-nanometer process node. That's because Jane Street will pay to tape out the most novel submissions and send the physical copies to the winners. The competition is open till January 18, 2027, and working in teams is highly encouraged. Go to janestreet.com/thoracash to download the template code and get started. I want to ask more about this, because it's sort of weird that you have 50% of

• 41:27Dwarkesh Patel?

  compute that's spent on inference, that is not directly helping the model become better. One of the key advantages you'd expect eventually digital minds to have is, unlike a human who gets to have 50 years of real-world experience, a model will get to, through all its instances, will get to experience, I don't know, millions of years of deployment across all kinds of economically relevant work in the economy. And right now, that data is just not, in a meaningful sense, helping the model get better. It just seems so obvious that eventually, models should be able to learn from this data, and once they do, you would have something that almost feels like a widely-deployed intelligence explosion, because the model is simulating so much information across all these deployed instances.

• 42:09Dwarkesh Patel?

  But when do you expect this kind of hive mind, kind of crazy shit to start happening?

• 42:13Speaker 01

  I think broadly, at a very basic level, this is already happening, right? Just in the next generation of models. So right now, you can obviously take your deployment data and put this in the pre-train or the mid-train of future models, especially if you do some kind of filtering or some kind of judgment or annotation, or recent synthesization of that.

• 42:28Dwarkesh Patel?

  How much do you think that explains, the generation-over-generation improvement?

• 42:31Speaker 01

  I think it explains quite a bit. I mean, especially, I don't know whether the labs do this, because theoretically, they claim not to train on people's data. But the Chinese 100% do, and they definitely get this advantage, both, obviously, deploying -- this is basically what distillation is, they take out the models, they get some of their deployment data, they get some fraction of that by pinging the model, and then they train their next generation of models on it. And they can suddenly do it on their own models as well, like there's no reason not to whatsoever.

• 42:55Speaker 03

  I completely agree with this. I think if you zoom out far enough, this is definitely happening. You're picturing this, and we're all picturing this, this is what continual learning, like the Holy Grail is, is this very, very organic, live loop of an individual model getting an experience and live updating on the spot and learning from that. And a lot of things break when you zoom into that level of granularity. But yeah, the big labs are doing this, the closed models are doing this. There's also early signs of life of people using open-source models doing this at a much faster cadence. So a good example is probably Composer. You have some sort of model, and you are able to, or Harvey's doing the same thing with legal agents. It is getting very specific environments from the data that you have for that particular

• 43:39Speaker 03

  task, and things that users are complaining about, and all the feedback that you're somehow extracting from your specific deployments. And a lot of these companies have the advantage over the big labs in that they can use this data really, really well. And then they will create environments, they will do a big post-train of Kimi K3, they will go deploy it, they might do some online learning as well, like Composer did online, basically reinforced for a long time. So yeah, there's still a human in the loop, there's still a human saying, okay, these are the signals we care about, here's how we're going to create environments from the data that we have. And it's still a longer cadence than maybe the one that you're thinking of, but it really is happening. And eventually, that loop will become faster and faster.

• 44:16Dwarkesh Patel?

  I mean, the Composer thing is interesting, because this is where the model, in cursor, people press Tab or they don't press Tab on the next completion that the model suggests. And based on that, every single day, Composer gets better at predicting the next.

• 44:29Speaker 03

  So that was the old Tab model. They actually did the same thing for the actual, not just the Tab model, but the actual generative model.

• 44:36Dwarkesh Patel?

  Oh, that's interesting.

• 44:37Speaker 03

  And it's hard, because when you do online reinforcement learning, you don't have groups, right? You just have one user saying one thing, and then you get one rollout. And so you have a big variance reduction problem. And Curse's fuzzy answer to this was, oh, we have very good heuristics, which are able to estimate how much better than average this response was, or how much worse than average this response was. And then they would do this big reinforce update, and then their solution to whether it got worse or not was, if it improved on cursor bench, they would deploy the new model every five hours, and if it didn't, they would throw that version out.

• 45:08Dwarkesh Patel?

  Interesting.

• 45:09Speaker 02

  Yeah, I think your biggest problem is actually just not knowing what the reward function should be for natural data. And if you use some kind of superficial signal, like did they accept the code, the edit, that might get reward hacked in some way.

• 45:24Dwarkesh Patel?

  But this seems like a bigger issue with the sim to real thing, where the longer and longer horizon tasks get, the harder the art is to simulate within a data center, right? It seems to me already, potentially, even in coding, we're getting to a point where there's not some year-long coding task that doesn't eventually require you to talk to a client, or interact with the company, or interact with users. And if you think about the gamut of things we would want AI to be capable of, you want, eventually, superintelligence should be able to run a business, or start a new business and make it profitable, or have a profitable day trading in the markets, or win a court case. And these are all things which are very hard to simulate in a data center. An inherent part of the learning there is interacting with the real world.

• 46:08Dwarkesh Patel?

  And so maybe they may have to learn how to get better at these things from the transfer between sim to real. But alternatively, maybe you do need weight updates from these kinds of interactions in order to get better at them. And then if that is the case, if transfer isn't strong enough, and you do need weight updates, then the fact that the models are quite sample inefficient is maybe a deeper problem. And the reason I'm curious about this-- I feel like by default, I don't see how you don't get some kind of crazy recursive self-improvement within the next 10 years. But the one reason why that might not happen is in terms of like weight updates, the sample efficiency of weight updates, they just seem way far behind humans, right? Like plausibly millionfold behind humans in terms of how much data a human sees from birth to adulthood

• 46:52Dwarkesh Patel?

  versus how much a model sees from, you know, like cold start to like finishing training. And so, yeah, this is all to say, first of all, is there, is there going to be a good transfer between simulations and extremely long horizon, really complicated real shit that we want the EIs to do in the real world? And if not, does that really mean that like the lack of sample efficiency in these models comes to bite us?

• 47:13Speaker 03

  I think maybe the way I'd break down like the two types of tasks in which models get good and models will like still continue to struggle is whether the task is like cumulative or like you kind of have to, you have this like non-stationary distribution. You have to keep learning and like re-litigating a bunch of stuff. So like maybe an example of a cumulative task might be RSI. Like it's theoretically possible to maybe like have less than a million token, like, you know, Python file, which like from scratch trains a model that is capable of recursive self-improvement. And like every discovery that you make is kind of a line in the sand that you hold. Like if it's true that, you know, for RSI, we don't need to discover a new attention variant or whatever. Like once you've discovered attention and then once you've discovered, you know, mixture of experts, and once you discover GRPO, you just add that to the training stack and like that's, that's there.

• 47:56Speaker 03

  And like a good example of this is like, you know, 5.6 soul, training, 5.6 terror, whichever one. OpenAI told us to train, like it didn't have to go back and discover attention. Like it basically probably would have called a bunch of like scripts, which is like pre-training.sh and post-training.sh and just did that. So like, that's an example of like a cumulative task. I think the real world and the reason like people are thinking so much about like continual learning is it's, it's not really a cumulative task. Like imagine like in a law firm, you have an agent acting as a legal associate. Like that's a very non-stationary distribution. You have to be able to fit in your context, like all the relationships between all the important people at that company, which are also changing all the time.

• 48:32Dwarkesh Patel?

  Yeah.

• 48:33Speaker 03

  You have like all these like implicit like ways about how things are done, where to find information, et cetera. And like, that's not as clean of an example of a cumulative task as like RSI is. So I think that there will be this breakdown between tasks, but you know, like if the labs realize that and they, they do believe that RSI is cumulative in the sense that like, we don't need to go back and discover some brand new like architecture or whatever.

• 48:53Dwarkesh Patel?

  Yeah.

• 48:53Speaker 03

  Then maybe more and more effort and compute gets focused on that versus the.

• 48:56Dwarkesh Patel?

  It's so unfortunate that RSI happened to be easier than the parallel.

• 49:02Dwarkesh Patel?

  Yeah, I don't know if you guys have thoughts on this?

• 49:04Speaker 02

  Yeah, I would say there's like models are, today's models are weaker than humans in a lot of different ways. And some of them might have to do with sample efficiency in a certain regime where, I mean, in some regimes, models are very sample efficient, like learning in context. But then there might be some like medium length regime where they're less sample efficient because humans can do some kind of weight update more efficiently than models. So, so I think like being less sample efficient in certain regimes might be one of the sources of weakness. But then I think there are other sources of weaknesses that are completely different than that.

• 49:45Speaker 02

  For example, having lower diversity of thought than humans or being bad at certain kinds of long horizon judgments. I mean, I think a lot of what people call taste is, is something about behavior that works in the long run and that people have realized works in the long run. Not everything, but like some, some aspect of taste, like especially for something like software engineering. Like I think a lot of taste is like, what are the systems that are going to be maintainable and work well in the long run of this project?

• 50:23Speaker 02

  So, yeah, I think the weaknesses of humans, which limit RSI along with other things are, yeah, there's a variety of them and some of them are related to sample efficiency and some of them aren't.

• 50:37Speaker 03

  Maybe an interesting thought experiment is like, if you were able to give a model, like a context window of, I don't know, a trillion tokens or whatever you would have needed to fit in like your experience prior to like, let's say RLHF. And like it's got all that experience in the context window and it has the same sample efficiency and in context learning ability as it does at a million tokens. Like, do you think taste is then solved? Like, would it be able to like make the same judgments that you did? Or is there like something fundamentally missing apart from just a longer context window with the same sample efficiency?

• 51:07Speaker 02

  Yeah, I mean, it would have to be trained to learn from that context. So I'm not sure. Yeah, either it would have to be trained to learn the right update to make from that context.

• 51:21Speaker 03

  So you don't think you could just like dump it all in, like your whole like life, like research experience?

• 51:25Speaker 01

  I mean, like you still need the data to train it long context, right? Like even if you could theoretically get like a trillion context, you would need a trillion lengths of data to train it. Like right now, you have like.

• 51:35Speaker 03

  I'm just asking if you had that.

• 51:36Speaker 01

  Yeah, in theory, I think yes. I mean, this really just comes down to the question of like how meta-learnable is taste from like shorter horizon episodes? And like, I feel like there's no obvious reason it's super long because like humans somehow develop taste with not having many long episodes. Like we don't live to be like 10,000, we have like, you know, we, we develop pretty quickly, right? And so like, you know, if you think about like, even like in a PhD, the difference between like a first year PhD student and like a final like postdoc or something, that's like five years maybe. And they've only done like maybe like 10, 15, 30 research projects in total. But somehow they develop taste quite quickly from like a relatively short succession of like small things. And so like theoretically, it's possible to develop it like that. The AI obviously will have vastly more experience in which to develop taste like meta-learn it. And then it's like, how well does that generalize to like really long horizon things?

• 52:21Speaker 01

  Is I think the question, which I think is really unsolved at this point, like we don't know.

• 52:25Dwarkesh Patel?

  Going back to this question, eventually there should be a regime where EIs are learning a ton from each individual instance of deployment that they have. Where currently you could say there's a meta-fuzzy process by which models do infer for deployment. But I feel like it's a very weak, very weak feedback loop. Do you see this around the horizon where there's this like hive mind kind of learning that's very rapid? And, and if so, how exactly does it happen?

• 52:51Speaker 02

  Actually, I would say that around, will we get a hive mind that learns from all of its deployment experience? And I mean, a big part of that is actually about incentives rather than being a technical question. So like companies aren't going to want to have the model provider learn from all of their deployment because that, that might just reduce the advantage of their business.

• 53:16Speaker 03

  I think that maybe the economics of this will pressure not necessarily weight updates to one big like common shared model, but like kind of like modules that get subbed in. So like a very obvious example, this is a law, but it might be something else. Like, you know, there's been a lot of work to try and fit like an arbitrary context length into a fixed size. Like this is all the linear attention stuff and all that sort of stuff. And like cartridges, which are essentially KV cases trained to be very, very compressed KV cases to fit in a lot of information. That's another example of like, you know, something that like companies may be willing to sign up for if that gets subbed into the model and it's not like actually changing the base underlying model itself. So like, there's many different versions of like learning from, from your data in real time. And like the latter ones are not really helping the big labs because they are just these modules.

• 54:00Speaker 03

  But I think the, like the, like the economic pressure will force like the labs to go down that path first before they can embark on this, like, you know.

• 54:09Speaker 01

  Which economic pressure though? Because I feel like even if you have like a bunch of cartridges or lawyers or whatnot, you can still just like take all these traces and just like distill this, dump this to the pre-training of like your next generation of lawyers.

• 54:19Speaker 03

  So it may be a more indirect form of learning that the big labs are getting. And that's obviously still really valuable to them. But I can't imagine a world in which we start off with like, you know, we're going to just like directly train this one big model on like all the exact data that we're getting.

• 54:31Speaker 01

  No, I think it will definitely like go through stages because, because I mean, this is assuming there's like one discontinuous event where suddenly we fix like weight updates continuously. And like in practice, I think it's much more likely to be like, the cartridges and stuff allow you specialize in deployment. Then you generate traces, you put that in your model. Like three months later, you come out with a model which is better with this stuff. You specialize it again, you like consolidate it again. And then eventually we'll just like make this loop faster and faster. So instead of like every three months, we release a model. Now it's like every week and then every like day and then every hour, in which point we basically have obviously solved it.

• 54:58Speaker 03

  Yeah, and I think this is a good point as well because you asked like kind of how far off the current paradigm we are from being able to do this. I think we've, we've done a bit of research to this and people have done a lot of research. Like at a really large scale, like when you wash out enough noise and you have large enough patches, like this outer loop process of like putting data into mid-training, creating our own environments. Like it does work in like some sort of continual learning regime. But the problem is like when you zoom in close enough at like a micro level, it's like, I've got one model and I'm trying to update it again for like a law firm or something. And I'm trying to do that very continuously, like with a relatively small amount of data. Like all the methods kind of break down a bit. So like if I SFT the model on just like, you know, successful traces, off policy, on policy,

• 55:41Speaker 03

  like eventually in the very iterative regime, like when you're doing like, you know, hundreds of these micro updates, you see both catastrophic forgetting, you see forgetting of like previous information I've learned on top of the base model that was much earlier on. And I see degradation of general like use, general capabilities. You know, just on policy distillation seems to like push this horizon out a little bit, but it still eventually succumbs to the same thing. And RL is not very good at like, it is good at like getting capabilities in, but it's not as good as getting like knowledge in. And like just this very explicit knowledge of like, oh, okay, like this person does this at this law firm. And like this is a very specific process we find. And you have to pour in a lot of compute to create the right environments to get the, the knowledge in.

• 56:20Dwarkesh Patel?

  And do you think that the fundamental issue here, why you get worse at these other skills or there's forgetting and stuff, do you think it's fundamentally an issue of capacity? Or it's an issue of techniques?

• 56:30Speaker 03

  A little bit of both, I think like SFT and even like distillate, like on policy distillation can be like way too destructive. Like our, the reason RL is so nice is because like, yeah, it, it changes a very, very small amount about, about the model. And there's like a lot of evidence for why this is the case. And so like, it kind of just like tweaks it in this very, very, very small like loss valley to like get it into the right point. But that also then limits what you can do with RL, like how much you can actually change the model.

• 56:56Dwarkesh Patel?

  Well, I'm just sorry, you're saying like the reason this is the winner take all potentially is that it's just like very hard to distill that much information into the base model?

• 57:03Speaker 03

  Without ruining something in an interview.

• 57:06Speaker 01

  Like it's easy to distill it into like a different base model. Like this is why I think it's mostly technique. It's not like, it's definitely not like, just like there isn't capacity. Like if you had some modeling with all this data and you take like literally the same size model and pre-trained it from scratch with like all of this stuff in mid-training, it will be better. And I think that's a lot of what's happening today.

• 57:21Dwarkesh Patel?

  Yeah.

• 57:21Speaker 01

  And so it's very much like there's, you know, a bottleneck that stops us from just keeping training the same model forever versus just like getting all the data from the old model and like training a new model from scratch. And this is exactly, as I was saying, like some combination like plasticity and like catastrophic forgetting in that like, you know, if you just naively train on like non-stationary data because you're adding new data as you go, basically this is messing with the data distribution. So like the old stuff is just forgotten and we don't really have good methods to like stop that from happening.

• 57:45Dwarkesh Patel?

  And maybe at the, in the limit, you're like just bottlenecked by retraining the model from scratch with all this new information.

• 57:50Speaker 01

  Yes, which of course is like very expensive, like training model from scratch is expensive.

• 57:54Dwarkesh Patel?

  But you're gonna do that anyways.

• 57:55Speaker 01

  I mean, not necessarily. I mean, like maybe eventually if you have continual learning, you never train a new model, you just like, just have a model and it keeps learning and like expanding, right?

• 58:02Dwarkesh Patel?

  But there's, there might be like some deep technical reason why that's very difficult because of these like.

• 58:06Speaker 01

  I mean, that's the question.

• 58:07Speaker 03

  That's the question, yeah. I think we have pushed back like how much from scratch we need to do. Like it is definitely possible now to take like the pre-trained base and like do very good mid-training on top of that, like kind of continuously plus some RL from like different checkpoints that are later on in the training. And like, that's looking more like continual learning, but certainly not the case of like, you know, take the most recent model, apply a couple of very small updates and like iteratively, like never lose.

• 58:29Dwarkesh Patel?

  So, but isn't this like, I'm a bit confused because isn't this literally what happens during training or during post-training or something? You just have like, you have a model that's already gone through so much training and then you like distill some fork that's been further RL'd or something. Isn't, isn't that literally what happens? And like.

• 58:44Speaker 03

  But it's, it's still at a large enough scale, I think, that you're washing out a lot of like the noise and like you're not just focused on one distribution, which as Baron said, is like, you know, that is now a very, if you're just like focusing on one task, right?

• 58:54Dwarkesh Patel?

  Like. Oh, sure. And I mean, in the eventual regime, you'd be doing, I don't know, there's billions of deployed instances, you're like doing, you're learning from all of them at once. And so hopefully there's some washing out of noise and stuff from that, right?

• 59:09Speaker 03

  Maybe that's, oh, yeah.

• 59:10Speaker 01

  Yeah, I mean, I think like definitely as I was saying, like you can do continual mid-training for like a long time and you can like roll back to a checkpoint, give a new mid-training data.

• 59:17Dwarkesh Patel?

  Yeah.

• 59:18Speaker 01

  But at the same time, like you can't do this like indefinitely. Like if you just keep continuing mid-training the same base forever, it just like get, it does, it's sort of asymptote at some point. Like you can't just learn new stuff in that base. And this is why people end up training new bases, like otherwise you would just keep mid-training the same base forever.

• 59:32Dwarkesh Patel?

  Whenever I finish recording an interview, I immediately brain dump all my thoughts into Slack. Things like, what was the most interesting and what should get cut? This ensures that my editors have all the context they need to start editing the episode. But it's not like these brain dumps have any clear timestamps and my unedited recordings are many hours long. It can take a ton of editor time to even find the exact moments that I was referencing. So we decided to try adding a Grokbot producer to our chat. And now whenever one of my editors posts a rough cut of the episode, Grokbot opens a transcript on its own computer and starts working, usually before I've even seen the message. It takes the notes that I dropped into Slack and it highlights the relevant snippets in the transcript. It also uses a big case file that I've compiled with all my preferences, so it can suggest potential edits.

• 1:00:15Dwarkesh Patel?

  And when it's done, it sends me its top clip candidates so that I can review everything from my phone. This has worked really well. Being able to send informal messages, like I'm texting my editor, and then having the transcript immediately reflect my preferences has just been so helpful. Try Grokbot yourself at x.ai-bot. Okay, let's talk a bit about data now. So I'm generally interested in this question of how much of AI progress is just explained by data progress. Doesn't mean it will be necessarily hard to automate, but that's a separate question. So is there some data distribution which if you train current architectures on would result in a superintelligence that totally dominates human experts across every single field?

• 1:00:58Speaker 03

  Are we talking about like pre-training plus post-training data, like environments as well? Like, I think the existence of this is obvious. It's just like whether we can create the right environments.

• 1:01:07Speaker 01

  Yeah, I mean, in the trivial case, we could just train it to output the Python file, which like trains the actual superintelligence, like just have that memorized in the weights.

• 1:01:14Speaker 03

  Like, yes, there's probably like a ladder of RL environments that is possible to construct such that you would get a AI researcher which is at least as good as a human researcher. But the effort to climb each successive rung grows like kind of exponentially. And that's going to be the two things that you have to trade off against as to whether like, like how fast we're going to hit like that final rung where it's, where it's better. I, I think that's fairly clear. And I think there's like, you know, we're still relatively early in like RL environment creation. Like there's a lot of asymmetries that we exploit in order to create good environments. So one of those asymmetries which we've talked about before is like, it's, there's environments where it's easier to go backwards and forwards.

• 1:01:56Speaker 03

  And like, what I mean by that is like, it's very easy to define this like complex data generating process. And this is like this kind of latent variable you keep hidden from the model. You can generate like arbitrarily complex like environments and the model has to do a lot of like irreducible like token spend and irreducible work to figure out what that data generating process was. There's asymmetries in terms of like, you know, you can inject information from the real world, so like Anthropic finds a bug through like, you know, tens of thousands of human and LML was combined and like turn that into a very, very neat environment, which in a single LM could theoretically find within like, you know, a few million tokens. So like, there's all these asymmetries which we're cherry picking and like, we're counting on like kind of this task horizon generalization. But I think, yeah, again, there's just gonna hit diminishing returns at some point.

• 1:02:39Speaker 03

  Like, at some point, there's diminishing returns and how hard it is to create those environments in the first place, like coming up with them, because you can't necessarily just like have these really, like these processes where it's easier to go backwards than forwards. Like you actually have to sit down and construct like something that looks like with humans, like, you know, a long enough time horizon, like it's gonna be a really complex task to create. And then it's also gonna be like the compute and time bottlenecks for the agent to actually do those tasks. So like, I think you're just gonna start seeing this like curve to flatten out.

• 1:03:06Speaker 02

  I saw something about how someone fine tuned the Taki model, which is only trained on data up to 1930 on this like modern coding agent data. And, and it did better than Claude 3 Opus on, on Sweetbench. So, so this model that has like no knowledge of code whatsoever can be fine tuned on a moderate amount of data and like behave better as a coding agent than this much larger pre-trained model is pretty crazy. And it kind of shows you that, like, once you have an example of like the right expert behavior, it's actually surprisingly easy to like copy that into a like relatively weak model.

• 1:03:47Speaker 03

  Yeah, but a counter example to like that kind of is, there was a paper recently where they trained it up to like fifth grade maths.

• 1:03:54Speaker 02

  Uh huh.

• 1:03:55Speaker 03

  And like also like primary school, like English and stuff. So it was like a decent language model. And they tried to RL it to do like, you know, late high school and college maths. And the gap was just too large, like they couldn't get it to climb at all. Like, but if you did like successive rungs of like, you know, your seven maths and then your eight maths and so on, like you could obviously climb to, to year 12. So like, again, it's just like, what is the distance between the rungs on those ladders? And how hard is it to create?

• 1:04:17Speaker 01

  Yeah, and this just comes back to like the RL signal problem. Like RL is not very good at like exploring right now. And so if you, if the model can't like get in like, you know, 128 world outs, it's very unlikely to get signal to like progress. And this is why like in RL, we need like curricula, whereas like in pre-training we don't, because like it's, that's not a problem for pre-training at all.

• 1:04:34Speaker 03

  Yeah, and, and again, pre-training data is different to post-training data. And I imagine as we continue on, like, yeah, humans will be involved less and less, but that doesn't change the fact that you're bottlenecked on like how much signal you can extract from the, the real world. So like, there's a lot of signal in the world and that's true. Like, you know, there's people doing like spreadsheet tasks, there's people doing like legal tasks and all this sort of stuff. But, you know, the capability frontier of where the models are at now, like how many bits in the world actually like really relevant to like improving the, the model's capabilities? Like, you know, how many new maths problems are being solved that like couldn't, they're just beyond the reach or grasp of the current models? Like, how many new coding problems are being created or solved that are beyond the reach of the current models? And like, I think that's why the diminishing returns kicks in, because like even the world

• 1:05:18Speaker 03

  as a whole is not giving you the bits that are useful for tipping you into the next like basin of, of capability.

• 1:05:24Speaker 01

  Yeah, I, I totally agree with this. It's like really a question of like where the signal is coming from. And so like the signal doesn't, you know, in pre-training, the signal is like already in common call, right? Like for the tasks that you care about in pre-training, the problem is, it's not just like getting signal at all, it's like filtering out all the noise that exists.

• 1:05:40Speaker 03

  Yeah.

• 1:05:40Speaker 01

  And that's quite an automatable process. But like as the models get better, as we enter mid-training and post-training, the signal just like doesn't exist anywhere in the original data we have. Like no amount of filtering will like get this, you know, there's no like hidden proof of like the millennium prize problem sitting in common call. We can just like filter until we see it, right? And so like at that point, you have to get bits some other way, either from humans like directly, like asking them to like write out their reasoning or like by like creating environments where like humans decide like what environment should be created, what the objectives of these environments are, or like, you know, some kind of like training on like the human data that exists in deployment. Like you have to get the bits from somewhere.

• 1:06:13Dwarkesh Patel?

  Yeah, yeah. There's a question of how much of the progress in pre-training is being driven by data?

• 1:06:17Speaker 01

  Yeah.

• 1:06:17Dwarkesh Patel?

  I, I did this investigation with Jerry Han, who's a student at Princeton, where we basically trained all the recipes from 2019 till now, pair-wise with all the data sets from 2019 to now. So you're saying like GPT-2 on the newest data set like ultrafine web and you train Delphi, which is the newest training recipe, or the open source training recipe on like the pile or some old data set. And you do like the whole grid and you see. The getting to some level of capabilities, how much less compute does it take across this grid? And you see that the data seems to explain like 9x of a compute efficiency gain.

• 1:06:58Dwarkesh Patel?

  But the architecture improvements explains like a 3x compute efficiency gain at a very small scale. And so to the extent that that is true at large scale, that most of the pre-training compute efficiency gains are coming from better data. How much can that continue? Like, can you keep just filtering data more and more and building more and more synthetic data until? Yeah, do you have a sense of how much this kind of pre-training progress can continue?

• 1:07:22Speaker 03

  I think my prior is that, like, again, the low-hanging fruit is somewhat exhausted with like, we got the internet as this big block. And like, it's not like the internet is necessarily like growing at the same like rate. All the useful stuff on the internet is growing at the same rate. So like, we've probably got like a bunch of like 0.1% loss drops to go. But like, not definitely not as many as have currently occurred.

• 1:07:44Dwarkesh Patel?

  Yeah.

• 1:07:44Speaker 03

  But like, that's also really interesting that like, you know, you find this like, what cumulative like 27 times improvement across both. I think like it was Epoch or someone who estimated like three times a year since 2019, which would imply something like, you know, three to the seven, like over 2,000.

• 1:07:59Dwarkesh Patel?

  Yeah.

• 1:08:00Speaker 03

  Like times improvement. So like, where's that missing, you know, 100 times or whatever coming from? Like, that probably gives you a good signal of like, how much of this is like post-training?

• 1:08:08Dwarkesh Patel?

  I think the explanation has to be that a lot of the compute efficiency gains are scale dependent. And we're studying at extremely small scale.

• 1:08:16Speaker 03

  Hmm.

• 1:08:16Dwarkesh Patel?

  And that raises a question of, do the data compute efficiency gains or the algorithmic compute efficiency gains have more scale dependence? I don't know if you guys have a prior on that. We just didn't have enough compute to investigate that question.

• 1:08:28Speaker 03

  I mean, like, just naively, right? Like, theoretically, the scale dependence of the architecture is like fairly well-known.

• 1:08:35Dwarkesh Patel?

  Yeah.

• 1:08:36Speaker 03

  And like, you can fit a straight line to it. Whereas like, I would have no idea how to do that for like combining pre-training plus post-training data and mid-training data.

• 1:08:44Speaker 01

  But I feel like data is actually more like more important with scale. Like, I feel like architectures are kind of like a one-time, like, you know, an architect. I think like combining, like saying just like an X percent efficiency gain is kind of misleading. Because like what an architect does is like letting you reach like a qualitatively new regime, which you couldn't reach with the old architecture. And then within that regime, obviously, the data is like the primary thing determining it. But like, you know, if we say didn't have like, even like GQ, we were doing like full attentional day. We wouldn't be able to do like, we'd like we'd take expenses to do a million context. And then like, because of that, we couldn't, we could never use the data, which was like actually at a million context. And so we couldn't get these capabilities, even though like if you just do a naive like, how much does this do at like 2K context where the architecture isn't unlocking anything? Then like the data, you know, that will look much more important than in some sense it is, right?

• 1:09:27Speaker 01

  It's unclear to me that these things are like really just like multiplicative gains in this way.

• 1:09:31Dwarkesh Patel?

  Hmm, I see. So then, but what does this take away for the scale dependence of data?

• 1:09:36Speaker 01

  So I mean, on scale dependence, I think like, a lot of the like mid-training and post-training data we have now is like actually gets better with scale. Because like a lot of it, like the very long context horizon of stuff really requires like big models to be able to like make use of them.

• 1:09:49Dwarkesh Patel?

  Yeah.

• 1:09:50Speaker 01

  And like, this is not, you know, if you try and train like your 100 million parameter model on like three bench traces, it's not going to get anywhere. Like, it's not going to show you the same kind of improvement that you would get if you train like an actual sensible sized model on it.

• 1:10:00Dwarkesh Patel?

  Yeah.

• 1:10:01Speaker 03

  And like, it's hard as well now because so many of the architecture changes. Like, you look at like Kimi, for instance, like, or DeepSeek. They're doing these architectural modifications with not just like dropping the pre-training loss in mind, but like, for instance, how the models are going to be used in the real world. So like, the inference efficiency, like having some form of compressed attention in the DeepSeek models is not necessarily geared around, you know, this is fundamentally like a pre-training improvement. It's just like, okay, we're considering how the models are going to be used.

• 1:10:25Dwarkesh Patel?

  Right, right, right. Well, one question I'm curious about to understand the future is how parameter scaling will go as we're getting into more of a RL-heavy regime? Like, I don't know, I don't know how fast historically. Yeah, you can look at sort of open source architectures and see how fast parameters have been scaling. And maybe it's like roughly 2X every year for frontier open source models. And to the extent that like even frontier closed source models have like 100B or 200B active parameters. Do you think that like keeps 2Xing year over year? Or now that we're in an RL regime where you also want to conserve compute on rollouts? That, and also, maybe there is like a threshold effect where you have enough capacity. And at that point, does increasing parameters arbitrarily doesn't matter as much? Do you guys have a sense of, in 2030, how many active parameters will a frontier model have?

• 1:11:10Speaker 03

  Yeah, I think for the next few years, we're going to be like, like, because we're so focused on doing longer and longer horizon rollouts for RL, where like inference efficiency matters a lot.

• 1:11:20Dwarkesh Patel?

  Yeah.

• 1:11:21Speaker 03

  It feels like the models aren't necessarily saturated on their ability to that. Where the bottleneck is still the environments. And so we might see like a little bit of plateau. Like, I have a feeling that, you know, like Mythos and and the GPT models are much smaller than like, you know, the 10 trillion parameter range that the people are talking about. Even just naively comparing these to open source models, you can probably back out that conclusion.

• 1:11:41Dwarkesh Patel?

  Yeah.

• 1:11:42Speaker 03

  So, yeah, probably for the next few years, I wouldn't imagine a huge growth in the number of parameters. But again, like, there's so many different, different things to trade off here. Like, you decide the size of your model based on like how much pre-training data you have. And then like the difficulty of the RL environments that you've got to train on. And you ideally want to like get to the optimal point where, you know, you can get like a decent pass at one or something on like the hardest environments you have. And like, it wouldn't make sense to like make a bigger, bigger model pass there because then you're just paying like much more inference swap than you need to. So there's a lot of inputs to this, like depends on how quickly, you know, like McCor and then in-house, these, these guys can scale up the complexity of the RL environments they're training on.

• 1:12:20Speaker 02

  I would expect the models to keep getting bigger just because people are scaling up compute and the GPUs are getting bigger. But I would say exactly how much they get bigger depends a bit on the scaling laws in non-obvious ways. So one thing is that I think like data efficiency is going to be a bigger driver than compute efficiency of like the exact architectures people use. Now that we're getting to the regime where we're sort of running low on like high-quality pre-training data. So that might affect how sparse you want to make the model. And then I also think we don't understand sparsity that well. And it's like, parameters are a different resource than active parameters.

• 1:13:01Speaker 02

  But it's, and like sparsity has definitely increased a bit, but it's not clear that it's going to keep increasing without bound. There might be some kind of sweet spot. There's an argument that sparsity should make data efficiency worse because you might have to learn the same thing on multiple experts.

• 1:13:22Speaker 02

  Though that's debatable. So I think we don't, I don't think we, we have a good enough theory of scaling laws that we, we really understand why sparsity is helping and to what, how much it'll help. And if that'll like plateau at some point at a certain level of sparsity.

• 1:13:38Dwarkesh Patel?

  And so can you spell out exactly what the implication of data efficiency would be on? So it sounds like you'd say, well, it should, there should be less sparsity. But what are the other implications on parameter scaling?

• 1:13:49Speaker 02

  I guess just that the scaling law, you're, you're not necessarily looking for the most, you're not trying to optimize compute efficiency. So you have all your choices you can make on the architecture. And each of these gives you a different scaling law. And then like traditionally, you would look at some kind of envelope based on compute. So you would look at performance versus compute and take the envelope of like the, yeah,

• 1:14:13Speaker 02

  the best models. But like, if we're making that decision based on data, so it's like, yeah, we're sort of assuming we can spend a lot of compute. And like, we're sort of data is on our X-axis instead of, instead of compute. Then we just get a different set of optima or a different set of models that are on that frontier.

• 1:14:38Speaker 03

  Yeah. And I also don't think that we've necessarily like, you know, doubled like the size of the models every year for the last few years. Like, the people have been training like 1 trillion parameter models for at least a few years. Like, there was even an open source one called Falcon. But like Liam from Periodic Labs, like I think posted yesterday on Twitter about how like an early experiment at OpenAI was like training a 1 trillion parameter model that was very, very sparse.

• 1:15:00Speaker 02

  Oh yeah, that's what they did before OpenAI. That was like at Google, the switch transformer, yeah.

• 1:15:05Speaker 03

  So like, it was like, you know, very, very good at like knowledge, but terrible at reasoning because it was so sparse. And so like, yeah, it feels like we've been playing in this like 100 billion to, you know, up to 2 trillion parameter range for like at least a little bit. And like, it certainly hasn't been, this is like nice linear increase, yeah.

• 1:15:21Speaker 01

  I mean, I feel like there's two things. So as Charlie was saying, like inference efficiency is super important for RL rollouts. And so like this will really push down active parameters quite a lot. And then I think the total parameters really depends a lot on the hardware as well. So like, you really need to get like very high memory bandwidth and like VRAM size to like actually be able to serve like multi-trillion parameter models. And so like, you know, right now, you know, people still, still using a lot of like H100s and stuff. And so as everyone moves to GBs and then their Rubens will get like more actual like, the ability to scale and like actually serve and like do like large RL inputs at like different, at larger scales. The data question I think is interesting because naively, like larger models are much more sample efficient in like the actual data points. And so like, even if you're like not saturating the model, it's still better to get bigger

• 1:16:04Speaker 01

  because like the models, larger models generalize better and like get to a better loss for the same amount of data. And so right now, I think we kind of have a lot of data. And like, that's not the constraint rather than compute. And so we're having like small models, which are like very inference efficient. But if computers are along the bottleneck, it might come back to larger models, which are like under sort of undersaturated, but like they have this generalization ability because they're much larger.

• 1:16:25Dwarkesh Patel?

  If you just look at like the basic Chinchilla scaling law and you just maximize out parameters.

• 1:16:32Speaker 01

  Yeah.

• 1:16:32Dwarkesh Patel?

  It actually decreases the amount of data you need to go to the same loss very little.

• 1:16:36Speaker 01

  Yes.

• 1:16:36Dwarkesh Patel?

  If you go to infinity on parameters, the amount of data you need, I think, goes down less than 10X. Just because of the nature of like the power law.

• 1:16:43Speaker 01

  But we're now on the way too much data side of the Chinchilla laws, right? So right now, we over-train more of the Chinchilla. And so we could easily get back to a point to which, as we're running out of data, we move back to like the Chinchilla optimum point. Or even like a bit on the over-training, like, you know, under-training model side.

• 1:16:57Speaker 03

  But surely, like, even with these new chips that come online and stuff, like, we're just going to be so compute bottleneck for the next few years that that won't necessarily be the case.

• 1:17:03Speaker 01

  This could well, yeah, this depends on like the ratio you have like training and inference compute really. It's like, if you're super bottlenecked on data, not on compute, you should go bigger. If you're super bottlenecked on compute, you should always go smaller. And then like, yeah. But you can also use computer-generated synthetic data. So it's like one of these very hard things to predict.

• 1:17:19Speaker 02

  Yeah, I think part of the reason it took people so long to figure out the scaling laws in the first place was that if you don't get all these things right, then you don't get such a clean relationship. And like the, the beautiful straight lines on graphs like hide a lot of complexity on how you have to make sure, like to scale every hyperparameter the right way or like parameterize your optimizer in a way that scales and where you don't have to change your hyperparameters as you change the model size.

• 1:17:46Speaker 03

  And, and bugs have their own clean scaling laws as well, right? Like, you know, like with Caplan forgetting the cosine and eeling thing or like even just like not considering embedding parameters, I think. And so that messed up the estimate at smaller models because embedding parameters are a decent size of the model.

• 1:18:03Dwarkesh Patel?

  A bit on RL. So I feel like a year ago, a lot of people were making this argument that RL will not be super successful at scaling for models. I think, John, you wrote a research paper where you were pointing out that models learn one bit per episode. When you RL, they basically learn, did I get the answer right or did I get it wrong? And I wrote some blog posts earlier this year. I was like, it's even worse than that because when the pass rate is low and the model is very unlikely to get the answer right, it learns almost, almost nothing at all from an RL episode. But I look at the models today and they seem pretty smart. And it seems to be the result of scaling up RL. Baron, you had a post, I think, a few weeks ago where you're trying to explain what's going on. But why has RL been more successful than one would have naively thought?

• 1:18:50Speaker 01

  I mean, so I think the success of RL comes down to a bunch of different things. So first, I think what is slightly underestimated is actually the mid-training. So an awful lot of like what we see as successes of RL actually comes from like very, very good mid-training data, which is basically where we're like essentially doing pre-training, but on like synthetic reasoning data and like the kind of environments that like get the model warm started for RL. And so this actually takes the model like almost like 80% of the way to like the final RL checkpoint often. And then what RL does on top of that is it does like a lot of essentially tweaking to the policy. And so this is one of the reasons why it doesn't need like as many bits as you would naively think. It doesn't have to learn all of these behaviors from scratch. It needs just like a few bits from these episodes, which you do get. And then the other thing that I really pointed out in my blog

• 1:19:32Speaker 01

  is that these bits are actually extremely high signal compared to like regular like pre-training, which is why you need RL at all versus just like SFTing on like successful reasoning traces.

• 1:19:40Dwarkesh Patel?

  Because it's exactly the bits about how to get the answer right.

• 1:19:44Speaker 01

  Well, there's two things. So yes, one, it's exactly the bits about how to get the answer right. But like, this is not exactly how you think of it because in SFT, you have a trace, right? You have like a bunch of math reasoning and then the answer at the end. The bit is still there, like you're still SFT on the answer token. So that bit is still there. What's important is that the objective ignores all the other bits. So in SFT, you like have like, you know, to try and match like the exact reasoning tokens that the model produces. So you're essentially getting like too many bits about like the exact way this other model you're training on reasons. For RL, you only get the one bit. And that means that like this, that signal is not drowned out in the noise of like all the other bits the model has.

• 1:20:18Dwarkesh Patel?

  Yeah.

• 1:20:19Speaker 01

  And so that's what really like, it's really a super dramatic like increase into the signals noise ratio during training, which is why like RL is like so dramatically efficient in terms of steps.

• 1:20:28Dwarkesh Patel?

  I don't know if you guys have thoughts on that?

• 1:20:30Speaker 03

  Yeah, I like, there's been so much debate about like what RL does to the model versus like, you know, mid-training or SFT or whatever. And like, you know, everyone talks about how, you know, pass at one will go up, but pass at 256 will go down. Like very rare correct reasoning traces will be like downweighted and kind of like outweighed by gradient signal from like easier kind of reasoning traces. And I, I think the simple like way to view RL now is that if you have a large enough, like a large enough amount of compute to sample, a large enough group size such that your probability of getting a bunch of correct answers is like past some like not insignificant probability, then like it will be upweighted. And like to Baron's point, like basically mid-training and, you know, more pre-training, like the pass at one, the starting point for RL like scales

• 1:21:15Speaker 03

  in a log number of pre-training tokens.

• 1:21:16Dwarkesh Patel?

  You can ask some very basic questions. I guess that answer makes sense and maybe there's empirical research which shows that this is what's happening. But then I just look at the models themselves and I don't know what's happened. Like, maybe you can give me a sense of what is the basis of the AI progress over the last year. But if it is, yeah, maybe it's just upweighting the, the policies, which we're gonna do the correct thinking anyways. But it just seems like qualitatively, the models have gotten so much more capable. And anyways, maybe there's nothing to, there's no inherent contradiction there. But how do we square like the relatively small impact

• 1:22:00Dwarkesh Patel?

  this take would imply that RL would have from the actual qualitative capabilities the model seem to be gaining?

• 1:22:05Speaker 01

  So like, one thing I want to point out here is that like, it doesn't necessarily imply that RL has a small like effect, right? Even if you have a few bits and like you only change the parameters a small amount, like the actual impact on like function space, the model lands, like the input to upper mapping can still be like super dramatic. You know, even if it's like, even like one bit can change like your function space a lot and it can like will out like half the hypothesis space, which is huge. So like, I don't think it's necessarily the case it's like small amounts of bits, small amounts of RL. Once you're starting from a really good point, means that like you don't have dramatic impacts in behavior, at least like not necessarily.

• 1:22:35Speaker 03

  I, I think it comes down to two things. I think the first thing is that everyone was hoping that RL would like generalize this reasoning across like all these different domains. And I don't think we necessarily got this like horizontal generalization. Like just training on math doesn't necessarily make you the greatest coder. Like you do have to do RL on, on code environments. I think what we did get though is like horizon generalization. Like the models just learned how to use more tokens for, for longer and still make progress on some sort of task. And so like you can train on environments where they get longer and longer and longer and then put them into a completely new environment. And yes, like they may not have generalized the reasoning patterns, which allow them to do well in that environment. But they've at least generalized the ability to like continue on that task for longer, which is correlated with like success. I think there's a paper called Edge Bench, which showed that the rate

• 1:23:19Speaker 03

  at which models can work for longer is like doubling every three months. And so that's a clear evidence of generalization. And I think like the, the final way to think about it is like, in pre-training, there's this idea of like quanta. So you have this very smooth like pre-training loss curve. And when you actually look at what's happening in the model, like the model is learning all these like very discrete like tasks. And there's like all these like emergent points, where there's like kind of a phase transition. Like it didn't have induction heads, now it has induction heads. And there's like tens of thousands, millions, probably like hundreds of millions of these things. And you average them all together and you get this very like smooth loss curve. I think like to us, to an extent, like a similar thing is happening, happening for RL. Like there is this very slow out of loop, as Baron mentioned of, you know,

• 1:24:00Speaker 03

  we will train a model and then RL and then like the next kind of model iteration of training, we will dump a bunch of these synthetic reasoning traces into the mid-training data. Like we're kind of hitting all these quanta for all these different tasks. And like on an individual task level, it may look like a phase transition. And like you're suddenly going from like a 0.5% pass rate to a 90% pass rate on like a particular like finance task or Excel task or whatever. But you average all these things together and plus the horizon generalization, you kind of would go up, wow, we've got like qualitatively better models.

• 1:24:29Speaker 01

  Yeah, I mean, I think a lot of this as well is just like, I think RL does generalize a bit. Like suddenly you get like some transfer between like math and code or like puzzles and math and this kind of stuff. Also, just like the, the share amount of environments I think the people are targeting is just like vastly greater. So like, you know, before, when you tried to do, you know, some task, which like you do in your daily life, like two years ago, like the labs wouldn't really care about this. They wouldn't like train the model for it. And now like it's just so much broader.

• 1:24:52Dwarkesh Patel?

  Right.

• 1:24:52Speaker 01

  They have a lot of environments targeting this specific thing.

• 1:24:55Dwarkesh Patel?

  Earlier in the conversation, we're talking about RL in the context of causing this entropy collapse or just, you know, concentrating probability on solutions the base model were already done and causing relatively sparse updates in the policy. But when I think like, when I think about, I think there's also another story about RL, which is going back to the Atari games and then AlphaGo coming up with Move 37, the super creative move, that because it was never initialized on human data, it can like think in ways that humans are not even thinking and come up with extremely creative solutions. Yeah, do you have a sense on when we should expect or if we should expect RL on LLMs to result in things like Move 37? Just extreme creativity, even beyond human creativity?

• 1:25:39Dwarkesh Patel?

  Because like there's just de novo, de novo initialization of intelligence?

• 1:25:45Speaker 01

  I mean, so a couple of things here. Like first off, I think that the AlphaGo is using MCTS, which obviously does like more exploration and like stuff than regular policy gradients. But I kind of also think that like RL doesn't necessarily like reduce the creativity. And like, I mean, even if we, I think, you know, this is obviously qualitative. But if we look at like the, you know, the open air hacking phase incident, like these models were coming up with like multiple zero days at a time to like break out of their sandbox. And like, this is clearly like some level of like Move 37 creativity, I think already, which we just get from just like the general generalization properties of the LLMs. Like I don't think, it's definitely not the case that like RL is like totally destroying their like entropy.

• 1:26:20Dwarkesh Patel?

  Yeah.

• 1:26:21Speaker 01

  Especially on long horizons.

• 1:26:23Speaker 02

  Yeah, I mean, one thing that people call creativity is just solving hard search problems. So, and so that's like, like Move 37 is obviously an example of that. Or like writing some kind of poem that satisfies a ton of different constraints. So that's something AI is obviously going to be extremely good at if, if trained for it. Then there's another way in which the models, like the diversity of their outputs is a lot lower after RL and they sort of develop these tics. And like, even though the models seem like they're good at writing, when you do some kind of like distributional analysis,

• 1:27:03Speaker 02

  you find that like they're reusing certain themes like all the time and they're using the same character names all the time. So there's actually, it's not like you're getting the same kind of diversity that you get when you, like from human authors, you're sort of getting one really good like style. So I think that, like that kind of diversity has definitely been like cut down by RL a lot. And in fact, now, oh yeah, since we were talking about distillation earlier, that's sort of something, yeah, one thing that's happening is that so many people are distilling mostly from Claude that like, like all the open weight models write the same way as Claude and use the same, like have the same tics.

• 1:27:47Speaker 02

  So this seems kind of concerning to me that we're having this, like this monoculture emerge.

• 1:27:53Dwarkesh Patel?

  Yeah.

• 1:27:54Speaker 01

  Again, I don't think this is like fundamental to RL as like a method though. And same with distillation, like even with distillation, like you're just training on the data. It's like, just because your data is not like super broad, that doesn't mean like the training method itself is somehow wrong. It's like a problem with the data. And I think a lot of, for instance, like the RL like entropy collapse is basically due to like exploitation of fairly simple like verifiers when you don't have like a huge diversity of environments. Because like, for instance, like the writing, I think, the writing is presumably graded by some judge and like the judge has some specific tics and like the model is learning to award hack the judge. And that's why like it collapses. But like, this is really a problem with the judge, it's not a problem with like RL in general.

• 1:28:30Dwarkesh Patel?

  Okay, super rapid fire predictions about the future. So I want timelines on the following couple of questions. By when do we have models which you can, here's what the, it feels like to a user?

• 1:28:46Dwarkesh Patel?

  You basically hire them as a drop-in remote worker for all kinds of white-collar work, not just coding, but I don't know, video editing, law, paralegal, et cetera. Like, it's like literally an actual remote worker. Like, full computer use with, like, literally a month of seamless learning and operation and executing on, like, complex projects and required interacting with other people, et cetera, et cetera. It's like everything a human worker could do over a month.

• 1:29:14Speaker 03

  If you're mandated to use, like, a browser or whatever, rather than, like, these, again, the firm setting up the information to be programmatically accessible, like, maybe a couple of years, but if it's not, like, browser-based, like, you can send Slack messages, they can do all this stuff, but I'd still probably say around a year.

• 1:29:29Speaker 01

  Yeah, I mean, I would say maybe, like, for the, like, full generality, maybe, like, three years, but I think, to Charlie's point, we will end up with, like, a lot of people, like, making their organizations easier for the AIs to use. And so you get, like, 80, 90% of the way there before that.

• 1:29:45Dwarkesh Patel?

  So, but the thing that's, the diff between one year and three years, there was just literally, like...

• 1:29:49Speaker 01

  Like, I think there's gonna be, like, a long tail of, like, miscellaneous stuff, which, like, some human can do, which, like, will take the models, like, quite a while to do.

• 1:29:55Dwarkesh Patel?

  Yeah, like, are you thinking of, sort of, computer stuff or, like, basic cognitive capabilities?

• 1:29:59Speaker 01

  I mean, I think this really comes down to a question of, like, how quickly can we solve this kind of, like, online learning? And, like, whether we can, like, get, like, 80, 90% of the way there with, like, compaction and, like, writing files to yourself and stuff. And, like, that's my big uncertainty. I really don't know.

• 1:30:12Speaker 03

  And another, like, maybe an example of something that I wouldn't be good at is, like, you know, if I have to, like, yell at someone to get something at work, or, like, really push someone to get something done. Like, the model isn't just gonna do that, it's just gonna be too nice.

• 1:30:23Dwarkesh Patel?

  Yeah.

• 1:30:23Speaker 03

  Yeah.

• 1:30:24Speaker 02

  I'd say there's a wide variation in quality of, uh, human remote workers. So if you, if you try to hire someone, like, off of Upwork to do a software engineering project, there's gonna be a huge variation. It's, like, often quite hard to get them to do, like, to do a good job or, like, pay attention to all the feedback you're getting. And, uh, like, I would guess that in some cases, uh, like, it'll be worse. Like, the, the pre-AI version of this was worse than what you can get now from existing AI. Uh, so I think it might end up being a little complicated, uh, 'cause maybe to some extent, we already have this, uh, like, for some, like, not-so-high quality of work, but then, like, then it's obviously, like, we're not, yeah,

• 1:31:08Speaker 02

  we're not matching human level in certain, like, higher-quality, like, um, forms of work. So, but I basically agree with Charlie and Baron that maybe, yeah, we'll, yeah, we'll have some version of this in a year or so that's, like, okay, and it will be able to do, uh, maybe we'll have that form factor, and, uh, it'll be able to do some things really well, some things not so well, and things will be improving from there.

• 1:31:32Speaker 03

  Like, we shift the goalposts based on a very long tail all the time. Like, I think, I feel like you use this example for, like, doing your taxes or something. Like, this year, I literally just, like, told Codex to, like, go get everything I needed to do and send it to the accountant. And, like, there was this massive list of stuff I had to use computers, click through and, like, download some stuff. And I didn't know, it was, it was, like, fine, it was perfect. So, like, I don't know, a lot of this stuff, it can already do.

• 1:31:55Dwarkesh Patel?

  Okay. Um, give you 10x total productivity uplift. Basically, if you, if it takes you a year to make a breakthrough now, you make a breakthrough every month.

• 1:32:05Speaker 02

  I think I would just refuse to give you a scaler on this. Uh, like, like, we might already be past that in some, like, types of work. Uh, like, let's say you're just trying to prove, uh, you're trying to do, um, like, certain types of math.

• 1:32:20Dwarkesh Patel?

  Oh, sorry, but for you as AI researchers, trying to make, um, like, advance, you know, the state of AI research. It's how much are, like, AI researchers sped up?

• 1:32:29Speaker 02

  Yeah.

• 1:32:29Dwarkesh Patel?

  Or uplifted?

• 1:32:31Speaker 03

  Somewhere between five and 10 years.

• 1:32:33Dwarkesh Patel?

  Oh, really? Okay, that's far away.

• 1:32:34Speaker 01

  Well, you think it's longer than, like, for general remote worker?

• 1:32:37Speaker 03

  Yeah.

• 1:32:38Speaker 01

  Interesting.

• 1:32:39Dwarkesh Patel?

  I think you're probably, I think, I'm realizing you probably have very different definitions of fully general remote worker. I could have specified that.

• 1:32:44Speaker 01

  Yeah, this is true, 'cause I mean, like, yeah, because obviously, like, an AI researcher can be a remote worker, and so, like...

• 1:32:49Speaker 03

  Yeah, no, I'm picturing, like, you know, normal white-collar work over the period of a month.

• 1:32:54Dwarkesh Patel?

  Yeah.

• 1:32:54Speaker 03

  I think it starts to emerge a little bit past a month.

• 1:32:56Dwarkesh Patel?

  A very competent white-collar worker, but not necessarily, like, a super creative researcher.

• 1:33:01Speaker 02

  I would say, like, two years.

• 1:33:02Dwarkesh Patel?

  Two years?

• 1:33:03Speaker 02

  Yeah.

• 1:33:03Dwarkesh Patel?

  10x? Okay. How about you, Beren?

• 1:33:05Speaker 01

  I can kind of see that, actually, 'cause, like, it really is just, like, right now, it's already, like, definitely more than 10x of, like, coding stuff. And so it's, like, if it can do even, like, one or two loops of, like, experimental feedback, that would actually be massive already.

• 1:33:17Dwarkesh Patel?

  So 10x uplift of AI researchers within two years? If you just plug it into, like, a very naive model of, like, AI progress and how much is coming from AI researchers and they're, like, there's, like, a 10x increase in their productivity. Yeah, you have, like, radically accelerated pace of AI progress starting two years from now.

• 1:33:36Speaker 01

  Yeah, I mean, I think, like, this will mean that AI progress doesn't get bottlenecked on, like, AI researchers' ability to run, like, small experiments, it gets bottlenecked on other things.

• 1:33:43Dwarkesh Patel?

  Of course, of course, but it just, like, happens 10x faster.

• 1:33:45Speaker 01

  For sure, yeah.

• 1:33:46Dwarkesh Patel?

  Which is a huge deal. And that also, like, helps the next thing, which makes, gives you 100x speed up, happen sooner, et cetera.

• 1:33:53Speaker 03

  Yeah, I'm happy to stick with longer on that one.

• 1:33:55Dwarkesh Patel?

  And what's, what's, like, the crux?

• 1:33:57Speaker 03

  Uh, like, my capacity to absorb information and make, like, Bayesian optimal decision on the next experiment.

• 1:34:03Dwarkesh Patel?

  Makes sense.

• 1:34:04Speaker 01

  Yeah, I mean, I'm assuming that, like, you can delegate some of this to the AI. So, like, the AI is becoming decent at, like, deciding, you know, it's run this experiment, it's got this result, it runs, like, the next experiment. And then if it can run, like, two or three experiments in a row without, like, crashing, then, like, that is actually a big update in, like, uplift.

• 1:34:18Dwarkesh Patel?

  And, okay, final question. An AI which is, which dominates top human experts across every single field of work that can be done over a computer.

• 1:34:31Dwarkesh Patel?

  So not only AI research, but all cognitive work. And not just, like, short-horizon work, but, like, literally, if it takes, like, three years or something, the AI will still do better than humans.

• 1:34:41Speaker 01

  This is basically just, like, ASI, though?

• 1:34:43Dwarkesh Patel?

  Yeah.

• 1:34:43Speaker 01

  Okay.

• 1:34:44Speaker 02

  I would say, like, three or four years.

• 1:34:45Dwarkesh Patel?

  The fuck?

• 1:34:49Dwarkesh Patel?

  I mean, that's, that doesn't seem wrong.

• 1:34:50Speaker 02

  I mean, I would say, like, um, like, AI is obviously being more, um, getting more attention, so it's, like, one of the harder things, but it's, like, uh, a lot of energy is being put into it, and it's also, uh, like, not one of the hardest things for AI, 'cause it's, like, involves a lot of code and math, which models are really good at. Uh, maybe for things that involve, like, 3D and, like, spatial stuff and physical stuff, I think, uh, that, that will take a little longer. So, um, especially if it's not, like, yeah, yeah, if it's, like, mechanical engineering or something, and it's not getting, like, the most attention right now, that might take a little longer.

• 1:35:27Dwarkesh Patel?

  But it also does include fields where there is relatively little data because of the nature of the field, and it has to, like, learn that data on the fly. So, for example, it has to become superhuman at, like, being an engineer at TSMC or something.

• 1:35:42Speaker 02

  Oh, yeah, so you would have to assume that, like, the on, uh, like, the onboarding, yeah, you can give the AI the same onboarding material and, uh, oh, yeah, then there's some, like, some, something has to be solved about, like, uh, sort of, longer horizon learning or, yeah.

• 1:35:57Speaker 03

  At 10, 5 to 10?

• 1:35:58Dwarkesh Patel?

  So, basically, yeah, it's, you think, uh, automating AI research is, like, ASI complete or something?

• 1:36:05Speaker 03

  Yeah, I think so. Um, yeah, I think there's so many things in the world which, like, even if you have some sort of memory system external to the model, and even if, like, context length grows a little bit, like, there are just fundamentally things, like, even if you could research the information or write notes yourself, like, you'd need more than a mental context field to do, yeah.

• 1:36:23Speaker 01

  Yeah, I mean, I kind of agree in, like, the, the five-year range, at least for, like, the stuff that, like, labs are focusing on, but I think, like, there's gonna be a long tail of stuff which, like, the AI could theoretically go out and learn about, but, like, no one has the bothers to do it. And, like, the computer's been allocated to that, so that might take longer for, like, literally every single human expert.

• 1:36:40Speaker 03

  Yeah.

• 1:36:41Dwarkesh Patel?

  That's right, but by this, I also included, like, the ability to learn as fast as a human, a new domain.

• 1:36:45Speaker 01

  I mean, I think that's not necessarily necessary, actually, 'cause, like, the AI will have vastly greater experience than, like, any human.

• 1:36:51Dwarkesh Patel?

  Right. Thanks so much for doing this, guys. I feel like this is a great format for getting different experts to disagree and debate and discuss things together. It was very productive.

• 1:36:58Speaker 01

  Cool, cool.

• 1:36:59Speaker 02

  Thanks for having us.

• 1:36:59Speaker 01

  Absolutely.