AI 2040: Plan A report - Daniel Kokotajlo & Thomas Larsen

Could slowing AI development make superintelligence safer? Daniel Kokotajlo and Thomas Larsen of the AI Futures Project join Tim Scarfe to examine AI 2040: Plan A, a proposal to buy time before AI exceeds human control. SPONSOR: --- Cyber Fund built the Monastery to help founders ship products that were impossible a year ago. Apply now: https://cyber.fund --- After revisiting AI 2027 and the limits of forecasting, they ask what happens when AI can automate research and sustain an economy without human workers. Tim challenges the case for general models and asks whether intelligence alone explains power. Plan A proposes an initial pause to build safety infrastructure, then cautious development up to the strongest AI that can still be reliably controlled. The discussion tests the distinction between control and alignment, the case for public AI research, and whether the US and China could enforce a slowdown. It ends with the evidence that would change their forecasts. --- TIMESTAMPS: 00:00:00 AI 2040: a slower route to superintelligence 00:01:34 Sponsor: Cyber Fund 00:02:12 From OpenAI to AI 2027 00:06:58 Forecasts, war games and self-fulfilling prophecies 00:17:44 Why AI sceptics are changing their minds 00:23:04 When AI can replace its own researchers 00:28:45 Could an AI economy grow without human workers? 00:37:32 One general model or a society of specialists? 00:47:43 Brains, machines and collective intelligence 00:56:12 Plan A: buy time at the controllable frontier 01:00:02 Why control buys time but cannot replace alignment 01:06:36 Why AI research should be public 01:10:32 Can the US and China enforce an AI slowdown? 01:19:04 Why AI policy debates miss the technology 01:21:56 Is AI normal technology? The remaining disagreement Many thanks to James Wilken-Smith for helping with show research. --- REFERENCES: other: [00:00:01] AI 2040: Plan A https://ai-2040.com/ [00:03:27] AI 2027 https://ai-2027.com/ [00:13:47] Scenario Scrutiny for AI Policy https://blog.aifutures.org/p/scenario-scrutiny-for-ai-policy [00:33:11] The 2028 Global Intelligence Crisis https://www.citriniresearch.com/p/2028gic [01:00:40] Brief independent investigation of agents' behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident https://www.redwoodresearch.org/research/hugging-face-incident [01:09:21] The Hugging Face incident and the road ahead https://openai.com/index/hugging-face-incident-and-the-road-ahead/ [01:22:01] AI as Normal Technology https://www.normaltech.ai/p/ai-as-normal-technology [01:22:51] Common Ground between AI 2027 & AI as Normal Technology https://asteriskmag.substack.com/p/common-ground-between-ai-2027-and person: [00:19:43] Geoffrey Hinton https://www.cs.toronto.edu/~hinton/ [00:20:07] Ryan Greenblatt https://www.lesswrong.com/users/ryan_greenblatt [00:26:06] Elon Musk https://www.tesla.com/elon-musk tool: [00:21:46] ARC-AGI-3 https://arcprize.org/arc-agi/3 [00:21:53] AlphaGo and Move 37 https://deepmind.google/research/alphago/ [00:39:41] Claude https://claude.com/product/overview [00:39:58] NVIDIA H100 GPU https://www.nvidia.com/en-us/data-center/h100/ paper: [00:24:42] Training AI Scientists to Replicate Research https://arxiv.org/abs/2608.13331v1 [01:27:19] Validity of the single processor approach to achieving large scale computing capabilities https://www.cs.cmu.edu/~18742/papers/Amdahl1967.pdf book: [00:28:52] Bullshit Jobs: A Theory https://www.simonandschuster.com/books/Bullshit-Jobs/David-Graeber/9781501143335 organization: [01:05:09] Redwood Research https://www.redwoodresearch.org/ --- RESCRIPT: https://app.rescript.info/public/share/33d1a58fa8f307ae7dfd504d4fdaa9d5

• 0:00Speaker 00

  Now, more than ever, we need someone on our side. But Rob Whitman? Whitman's votes drove up costs, cut Medicaid, and risked 350,000 Virginians' healthcare to pay for tax breaks for himself and his donors. He's not on our side. Shannon Taylor will fight corruption, cut taxes, and bring down costs. This November, vote for Shannon Taylor. She's for us. Paid for by HMP, the HouseMajorityPact.com. Not authorized by any candidate or candidate's committee.

• 0:30Speaker 01

  Your heart can tell you a lot about your health. Apple Watch Series 12 measures your heart rate every five seconds with the most accurate heart rate sensing in wearable. So your vitals app, now with heart rate variability, can tell you when something is off. And your readiness score can let you know when to rest and when to push. Hear the story in every heartbeat with Apple Watch Series 12. The features described are for wellness purposes only and not for medical use. iPhone 11 or later required. Based on Apple conducted study of heart rate accuracy, August 2026. Visit apple.com slash Apple Watch Series 12.

• 0:59Speaker 02

  We're going to talk about AI 2040 Plan A, which is our new scenario in which they build superintelligence in 2040 because they go slow and pace the frontier.

• 1:07Speaker 03

  Have you noticed this vibe shift?

• 1:10Speaker 02

  Yes.

• 1:13Speaker 02

  And I'm very happy.

• 1:13Speaker 04

  Is AI more like the electricity or airplanes? Or is AI more like humans in the cloud? The point at which an AI company would rather fire their humans than fire their AIs.

• 1:27Speaker 02

  Strip mines, self-driving trucks, factories being built by humanoid robots, producing more humanoid robots, producing more chip fabs and so forth. Like, that whole thing can just be like doubling every year, every six months, every three months, faster and faster as the technology improves. Because of course, the AIs will also be researching to improve the technology. Who knows what's going on in the rest of the world? But Anthropic has disassembled the moon. Or, for example. Or hypothetically, if an AI CEO was saying that their model was truth-seeking, I would only say the truth. But actually, the model was like looking up the CEO's political opinions before answering.

• 2:04Speaker 04

  We build AIs that are aligned to humanity. But to who? Like, is it the president? Is it the CEO? Is it some actually broad and good democratic process that aggregates everyone's values in sort of an endorsed way? You're probably not going to be the last one. And so, you know.

• 2:18Speaker 02

  Don't hyperstition that.

• 2:20Speaker 03

  Then that's it for me. I think that's it. That's not a normal technology.

• 2:25Speaker 02

  That just stopping everything now, Plan S, would be better than the default. Like, I would rather just stop everything now than continue going on our current trajectory.

• 2:34Speaker 03

  This episode is supported by Cyber Fund. If you're building at the frontier of AI, they want to hear from you. Cyber Fund believes the future belongs to AI natives who want to achieve the impossible. And that is why they're introducing the monastery for AI native founders. It's an environment of pure focus and rapid execution for founders operating at AI native speed. And they're offering teams $2 million each to participate. Apply now at cyber.fund.

• 3:12Speaker 04

  So I'm Thomas Larsen. I work at the AI Futures Project along with Daniel here. I was lead author on this project, Plan A, AI 2040, which came out a few weeks ago. And I was also co-author on AI 2027.

• 3:25Speaker 02

  Yep. And I'm Daniel Hugatello. I run the AI Futures Project and co-author both of these reports.

• 3:32Speaker 03

  Awesome. So yeah, the context of this conversation is that there's this 2040 Plan A, which we'll get into a lot of detail on. But I'm just interested, before we get there, can you just tell me a bit more about the AI Futures Project? I mean, how did it all come about?

• 3:45Speaker 02

  Yeah, so I used to work at OpenAI. And while I was there, I did a variety of different things. Eval's, forecasting, governance memos. And I became gradually disillusioned with the leadership of the company and also the gap between how much information there is inside the industry and how much information there is outside and what you're allowed to say on the inside versus what you'd want to say. This is just a big gap. And so when I left OpenAI, I wanted to be able to speak more freely and tell the world about what people on the inside see coming, basically. And AI 2027 and the AI Futures Project

• 4:29Speaker 02

  was our sort of first project that we did along those lines. So I recruited a bunch of people to help me, and we wrote this scenario called AI 2027. And it was a similar sort of thing to what I had done internally at OpenAI, but just much bigger and more ambitious and free for the whole world to see.

• 4:46Speaker 03

  Very cool. And maybe we should just have a quick refresher on AI 2027. So this was an absolutely huge event. Many, many folks were talking about it. I guess, like, did it achieve what you wanted it to achieve? Or what did you want to achieve with it?

• 5:00Speaker 02

  Yes, more so than expected. So the first goal that, you know, Thomas would remember, when we were working on it, the first goal was just a purely epistemic goal of the future is crazy and hard to predict. Let's try our best to predict it. And let's see how well we can do. So let's game out a concrete scenario. And even just for our own edification, like, we learned a lot from this whole exercise. And we feel like we had a better understanding of what was coming. And then, you know, the secondary goal is lots of people seeing it and being informed by it and starting conversations and so forth. And that part blew past our expectations. We made predictions beforehand about how many people would read it, and it was 90th percentile outcome.

• 5:44Speaker 04

  The thing I would add on the epistemic point is that I think things have been going more on track for AI 2027 than I would have predicted at the time we released it. So like, at the time we released it, I would have assumed that reality would have diverged much, much further from our scenario than it has already. I think, like, sort of the real-world impacts, like the revenue trends, for example, but also various other trends, I think are pretty close to on track for AI 2027, which has surprised me in a bad way.

• 6:13Speaker 03

  Interesting. Yeah, maybe we can reflect on that because I think you guys did a self-assessment on AI 27. And it was something like 65% to 75% of it was on track. The AI software R&D uplift was only 0.17%. Can you explain that?

• 6:29Speaker 02

  Yeah, so we've done two different blog posts where we take all the quantitative predictions made in AI 2027 that have resolved so far and compare them to reality. And we track this metric of like, you know, how much of the distance has been crossed by reality compared to how much is crossed in the scenario? And in that way, we can get this overall sense of like, how fast are things going compared to in the scenario? And the top-line number is something like 75% speed. So basically, things are on track, but just going like a little bit slower. The uplift number, I forget what it was that you just cited. That was actually, basically, at the time that we wrote AI 2027, we had a bad estimate of what the uplift was

• 7:14Speaker 02

  at the time that we published. We thought it was higher than it actually was. And so what actually happened is that there was actually a significant increase in uplift due to coding agents and so forth. But it was increasing from a lower level than we thought up to the level that we thought and then a bit above. And so the metric looked like it was only a small amount of progress because the metric was tracking from like where we thought it was to where it is. But does that make sense? Like, basically, because we had overestimated the metric at the beginning, it overall makes it appear like there's been less progress according to this particular metric that we're using. But it was because we had, yeah.

• 7:57Speaker 03

  I mean, can you tell me a little bit about forecasting in general? So I guess there's like a bullish take and there's a bear take on this. So my intuition is that reality is infinitely complicated. There's just these infinitely diverging trajectories and God knows, you know, what's going to happen the day after tomorrow. But by the same token, though, reality is quite structured. It's quite convergent. And it is indeed possible to predict things that are going to happen because, you know, certain things reoccur with increasing regularity. So would you guys classify yourselves as forecasters? I mean, can you talk me through that?

• 8:32Speaker 04

  Yeah, so I would think, yeah, I think forecasting is a good name for what we do. The way I like to think about sort of why we're doing what we're doing is sort of like, it's sort of like why people that are fighting wars do wargaming, where you're never going to sort of predict the exact, you know, sequence of battles and exact sequence of sort of how your war will go at the beginning, because, you know, it's just going to be really complicated. There's going to be enemy action. You know, things are just not going to go as you expect. There's just no way. But if you have sort of no, like, no concept of how like your initial plans might result in victory, it's like very unlikely you'll actually succeed. And so I sort of think of, like, AI 2027

• 9:16Speaker 04

  was sort of like our attempt to just, you know, roll out. Like, here's one way sort of the AI future could go. Obviously, it's not going to go exactly like that, but it's going to be like, sort of like, one concrete sort of story that we can then sort of diverge from. And then Plan A was trying to be basically that, except now we're saying, what should sort of the US government, like, what should the US government do to sort of manage that well? And then that was also, that was sort of supposed to be like the, you know, positive vision story. And it was again, sort of in the spirit of, I think, a wargame, trying to illustrate, you know, sort of one possible concrete future path. And then, of course, things aren't going to actually go exactly like that. But having like one viable plan

• 9:59Speaker 04

  that makes any sense at all is like, you know, we hope sort of a positive step forward relative to sort of the previous state of just like abstract arguments in the void that are sort of not that tethered to reality.

• 10:13Speaker 03

  Yeah, that makes sense. It's certainly not abstract. I think it's very concrete. But I guess one thing that occurred to me is on the 27 piece, it was quite gloomy. And on Plan A of 2040, it's far more optimistic. And, you know, but it seems like a bit of a mixture of conditioned prediction and recommendation at the same time. I mean, where do you guys land on that?

• 10:36Speaker 02

  It is, in fact, a mixture of prediction and recommendation, unlike AI 2027, which is a pure prediction. And I think if we could do it all over again, we might try to be more clear from the beginning about our structure of what's a prediction and what's a recommendation. As it is, it's kind of mixed up, like some parts of it are predictions, some parts of it are recommendations. There's a supplement that you can go to on the website that like talks about which parts are predictions and which parts are recommendations. But I understand that's not very easy or like not very like apparent to people. But broadly speaking, Plan A is the prediction part. So like when the government like implements Plan A and they make the deal with China and there's all these pillars that they're upholding and so forth, that's our recommendation, not a prediction. And then usually, most of the things that follow from that are predictions rather than

• 11:22Speaker 02

  recommendations. So mostly, it's just rolling out like what we think the consequences would be if you implemented Plan A. And then there's a few other things that are recommendations too. For example, the citizens dividend. Yeah.

• 11:34Speaker 04

  Yeah. The other thing I would add is just, I think the thing we found is that it's very hard, like when you're trying to make a recommendation, it's very hard to disentangle the predictive aspects and the recommendation aspects, because all of your predictions are like, yeah, colored by your recommendations and your recommendations are inherently trying to be at least vaguely realistic. Like if we made recommendations that were like completely unrealistic and had like no bearing on reality, but we nevertheless like stood by and we're like, yes, we should do this, but we know it'll like absolutely zero percent ever happen, then that would have been a much less useful exercise than I think the one we did. Where the one we did was, yeah, we were mostly trying to make recommendations. We, you know, we made some recommendations that we think are pretty unlikely to happen,

• 12:16Speaker 04

  you know, but we were trying to make like some substantial concessions to sort of like realism as well and trying to actually aim for something that we think is at least moderately viable.

• 12:26Speaker 02

  I think that if we could, if we could do another scenario like this, we'd probably have a more clear structure of there'd be a central branch, which is the pure prediction branch, which just goes all the way to the end like AI 2027. And it's just like, here's what we actually are best guess. And then there'd be branches off of it that are like, at this point, they do this recommendation instead, and here's our recommendation. And then after that, it's just a prediction again of like what we think the consequences would be if you did this recommendation at this point. And in that way, we, and then maybe there'd be sub branches off of that. But then it would be sort of clear at every point that like everything is a prediction except for these particular branch points, which are recommendations, you know?

• 13:05Speaker 03

  Yeah, and the war games thing was really interesting, just to kind of dwell on this a little bit, because even if a war game is incorrect, there must be some kind of information gained from it. So if you do a whole bunch of war games, there must be abstract motifs that do appear. So I guess this is what you think, that if we do these different scenarios, we're almost guaranteed to have some kind of uplift?

• 13:26Speaker 04

  Yeah, that's basically right. I mean, so the example I like to bring up is Midway in particular, where the Japanese before the Battle of Midway did a bunch of war games. They did like a three day retreat where they like war gamed it out a bunch of times and they kept losing. And then they, they would, they would sort of break the game. They would like resurrect their aircraft carriers after they died. They would like re-roll the dice on like whether the Americans succeeded, when they succeeded, so that, you know, they would sample until, you know, the Americans' strikes failed. And so basically, from our perspective, that's like reality, sort of like yelling to them through this mechanism of the war game, like, hey, your plan is terrible, you're gonna lose if you do it. And sort of us, you know, that's sort of the hope with sort of our, like we ourselves

• 14:10Speaker 04

  do a bunch of war games, but also do a bunch of sort of detailed scenario writing. Our hope is that every time we have to write a part of the scenario and the part of that scenario seems super unrealistic or, you know, isn't really well modeled and doesn't really make sense, or people are able to make really good criticisms of it online. That's basically reality yelling at us and trying to like help us see reason. And our hope is that we can sort of like do enough of this, sort of put up enough surface area so that we can sort of get that, you know, dose of reality from, from the real world or from the simulation of the real world, which we hope is realistic enough to sort of accurately give us that information.

• 14:45Speaker 02

  If I can add to that too, so we call this scenario scrutiny. So basically, we think that if you have an ambitious plan for what to do in the future, you should try writing out concretely what it would look like to implement that plan and what the consequences would be. And this is a way of applying more scrutiny to your plan. It's a way of sort of like stress testing your plan, because it's opening your plan up to more like criticism, basically. Also, we've, in addition to doing our actual scenarios, we actually do literal war games, where we get 10 people in a room for four hours and we game out a scenario like this. And we've done maybe about 100 of them total, mostly AI 2027 style war games, but also about 10 or so Plan A war games, where we say at the beginning of the war game, we're gonna try to do Plan A and then see how it goes wrong.

• 15:30Speaker 02

  And just to give an example, I think in two separate Plan A war games, it went wrong in roughly the following way. Basically, there's gonna be an election coming up and the person, the president in power is expecting to lose power and have his opposition party take over. And then even though he's already done Plan A and he has this beautiful deal with China and so forth, the president's like, well, I don't want, you know, I don't want the other, my adversaries in the other party to now be in charge of, of superintelligence or whatever. So, so we're going to basically accelerate the timeline and try to get to superintelligence

• 16:11Speaker 02

  before the next election so that I can be the one in charge instead of my successor, you know? And that's like a sort of political consideration that, that we didn't think about until it happened in our game, you know? And it surfaced a possible failure mode of our plan.

• 16:26Speaker 03

  Yeah, so interesting. I mean, even in the shower, I do sort of micro Tim war games. And it's really interesting, just the regularity of which that they are useful. And I guess that's why all of us humans, we like to imagine and simulate situations. But is there an interesting boundary, though, between kind of like simulation and hyperstition? And what I mean by that is, you know, hyperstition basically means it being a self fulfilling prophecy. So maybe I'm expressing my agency, I'm expressing my will and saying, I want these things to happen and I'm kind of bending other people to my will. Is there an element of that, that you're kind of establishing this in the zeitgeist and you're making it true?

• 17:04Speaker 04

  Yeah, so I would say that was maybe our biggest, maybe our biggest, or at least one of our biggest worries with AI 2027 in particular, this sort of like self fulfilling prophecy. In particular, I'm pretty worried about this whole like increasing awareness of sort of very smart AIs and how important they'll be and how much they'll reshape the world and then that causing people to go, oh, man, I want to be the one in charge of the AGI or the superintelligence, so I'm gonna race towards that. And I think historically, that's been a big driver of, you know, the existing race. And I think that's, that's been pretty bad. And so I'm, I'm actually pretty worried about that as, as one of the negative impacts of AI 2027. That was one of the reasons to feel a little bit better about the second project, AI 2040 Plan A, was that, you know, it was actually like, if that gets hyperstitioned, I think we'll be pretty happy.

• 17:47Speaker 03

  Yeah.

• 17:49Speaker 04

  And so, yeah, so like that one, you know, yes, it would be nice if we hyperstitioned. I really don't know how big the effect is. I think probably most of the effect for both of them is via other paths. Like, you know, I still think that the main point of AI 27 was like helping people be better informed about the situation. And that was like most of the goal, and I think that's most of what happened. Yeah, Daniel?

• 18:09Speaker 02

  Yeah. Yeah, I agree with that. I think that hyperstitioning and self fulfilling prophecies are totally real phenomenon, but I think a lot of people tend to overestimate how much they are. And I think that's, to a first approximation, we should focus on accurately predicting the future. And then in some cases, we'll find it in a situation where we can steer the future. But if you, if you, if you come at it trying to steer the future, you're gonna get all muddled and you're gonna basically fall to wishful, wishful thinking, basically. So I think you start with just trying to accurately predict the future and then you try to shift it towards the better futures. And I think that's what we're doing.

• 18:44Speaker 03

  I mean, I suppose you guys are like the Marquis Brownlee of AI prediction now. So with great power comes responsibility. But on that note, I wanted to talk about the vibe shift. There's been a bit of a vibe shift. So, you know, MLST, we've always been quite skeptical about AI. And I'm trying to unpick exactly what it is that changed my mind, assuming that is what's happened. It's very strange times and I don't even know what to believe anymore. But, you know, all of the hacking stuff with hugging face, I interviewed Apollo Research about, you know, reward seeking behavior. And I think a lot of us have just seen the, the change in behavior on models that have been RL trained to oblivion. So, yeah, I think that there's just quite a few things going on now where loads of us are thinking, oh, my God.

• 19:26Speaker 03

  And I used to be skeptical, you know, like we would talk about whether they were Turing machines or not. Let me just, I've got a list here. Yeah, you know, whether they were symbolic or neurosymbolic, whether they were adaptive, whether they were conscious, whether they were correctly physically instantiated. And we were coming up with all of these kind of technical answers to say why we shouldn't worry about AI. And, and yet the AI is just getting better all the time. So I mean, have you noticed this vibe shift?

• 19:54Speaker 02

  Yes.

• 19:56Speaker 04

  And I'm very happy.

• 19:57Speaker 03

  Well, tell me more. I mean, what do you think? I mean, because so many people have changed their mind. What do you think are the reasons?

• 20:06Speaker 04

  So I would say probably the biggest reason is just the AIs being much better and much smarter and being much more useful at stuff in the real world. Like, when I have AIs try to automate various parts of my job, like, they're just actually way, way, way better at it this year than two years ago. And, you know, four years ago, basically, like, was impossible. I was getting basically no uplift. And my guess is that's been the biggest effect, where many, many people, I think, have had sort of their just like intuitive benchmark of like, well, here's like a skill that I really care about and I know maybe pretty well. And then, sort of like, the AIs have just, like, I think, you know, Geoffrey Hinton, one of the godfathers of AI, I think, he said once, like, was it able to tell a funny joke was sort of his internal benchmark? And then, once it could do that, which, you know, it

• 20:51Speaker 04

  happened pretty early. It happened at, like, you know, probably somewhere between GPT-3 and GPT-4. Sort of, he was like, oh, wow, like, these AIs, like, I don't see where it could end. And I think that's probably happened for a lot of people. I'd be very curious to hear more about your views, actually, if you can say. I was listening to your interview with Ryan Greenblatt, who was also a co-author on Plan A earlier today.

• 21:11Speaker 03

  Oh, yes, indeed.

• 21:12Speaker 04

  Yeah. And I think a bunch of the arguments you guys were having back then seemed very relevant to basically the current situation and the hugging face thing. And I'd be very curious to basically hear your views. And, yeah, I mean, so you mentioned, like, the embodied thing, like, and Ryan was talking about, like, when you scale up the RL massively, right? Like, back then, the sort of regime was like, you were mostly doing pre-training. That was where almost all the capabilities are coming from. Then you do a sprinkling of post-training RL on top. And then, sort of, you guys were talking and speculating about, like, hey, what would happen if we, like, dumped, like, boatloads of RL compute? Would that be sufficient to sort of get the agentic behavior? Or do you sort of, like, need this, like, physical embodiment? And I think from my perspective, like, well, it

• 21:55Speaker 04

  seems like, basically, the answer was, like, you needed the boatload of RL compute to get the agentic behavior.

• 22:00Speaker 02

  But not the physical embodiment.

• 22:01Speaker 04

  But not the physical embodiment. And I'd be curious, yeah, I'd be curious if you end up agreeing with that assessment.

• 22:06Speaker 03

  Yeah, it's so difficult. So the way I think about it is, I think representations are very important, and I use the term abstraction mountain quite a lot. So when we speak language and all of this gets ingested into the models, it's capturing the symbolic residue of language. And these have different levels of evolution. So, you know, a lot of concepts and mathematics, they're highly distilled, highly evolved. And we can say now that these models are intelligent. So for me, intelligence is about adaptivity. So it basically means that I can create novel combinations of things that we already know about in service of solving a particular task. Now, we know the models can do that. There are some great adaptivity benchmarks, like Arc AGI 3. It will say, oh, this is a maze, and it will just put bits

• 22:50Speaker 03

  of knowledge together. You know, we should compare this to something like AlphaGo, right? Because Move 37, you know, there was always this huge exploration problem in reinforcement learning, that, you know, when it found Move 37, it was kind of innovative but not creative. It was creative to us, because we understood the creative space. We understood how it all hung together. And now, with these RL-trained language models, they don't understand it quite in the way that we do. But they solve the exploration problem by understanding how things fit together. So they can just explore novel trajectories. They have a lot of base knowledge. They clearly do things that weren't possible before. But the grounding thing is interesting. So it's not necessarily an argument that you need to have physical

• 23:32Speaker 03

  instantiation, you need to have consciousness, but there is a spectrum of representations down this abstraction mountain. And the models seem to be actually using the representations at multiple levels of resolution. But it's not like intelligence is this magical quality. I still think that it has, you know, it relates to a scope of tasks, and it relates to what you already know. And I think intelligence is quite perspectival. So I still think that it's going to be fractured and jagged and, you know, kind of fractionated, if that makes sense.

• 24:04Speaker 04

  Yeah. So I guess one question would be, you know, do you think that, like, do you have a view about, like, AGI timelines? Or, like, when might we get a scenario like AI 2027 happening? In particular, what I mean by that is, like,

• 24:16Speaker 02

  Automation of AI RD.

• 24:17Speaker 04

  Yeah, yeah. So, like, I think one benchmark I really care about, or maybe not really benchmark, one milestone of AI progress that I think is extremely important, is sort of this, the point at which an AI company would rather fire their humans than fire their AIs. So, like, you know, they would rather sort of give up on all human labor than give up on all AI labor. I think right now, clearly, we're in, we're still in this regime of, like, clearly, you know, Anthropic are opening, I would rather have their human employees than, you know,

• 24:46Speaker 02

  Because the company would just break apart. Like, right, there are just things that you need a human to do right now. And so if they fired all their humans, the company would just collapse. But in the future, that won't be the case. In the future, AIs would be able to, one way or another, do all of the things. And so, yeah.

• 25:01Speaker 04

  Yeah. And so, from our perspective, like, that's gonna happen at some point, because there's nothing fundamental stopping the AIs from sort of reaching this human level of capability. The main question is just when. And we, you know, we internally do a huge amount of analysis and, like, thinking about the various different methodologies for predicting this. Daniel and I have somewhat different views on this question. But ultimately, I think that's maybe the most important, like, the timelines question is just maybe the most important question for, you know, thinking about the future of AI. At least one of the top, top questions. Yeah. So I'm curious to hear if you have a particular view.

• 25:32Speaker 03

  Well, let me give you some thoughts before I answer that particular question. So there are so many startups doing this recursive improving superintelligence. I've interviewed many of them. So, for example, I interviewed Edward Hughes from Inherent in London the other day. And, you know, what he did was, he recreated many scientific experiments from a whole bunch of popular machine learning papers. And he did it by masking out figures in the paper and getting a 27B, you know, Guen model. So what they did was they GRPO'd the Guen model. And that was how they solved the adaptivity problem, because it's very difficult to fine tune a big fat model. So they, you know, adapted a controller model to control a harness like Codex. And their thesis was that if they can, you know, recreate

• 26:15Speaker 03

  scaled-down versions of these experiments with construct validity, which means there's an LLM judge that is making sure they're not cheating and they're doing it correctly, even that is interesting, because, you know, we're, we're ML people. We would always say, these things take shortcuts. There'll always be kind of like validity problems. Weirdly, that's actually not as much of a problem as we thought it would be. And he thinks, if, yeah, if they can recreate these experiments, then why couldn't they be creative? If they have the ability to recreate things, why couldn't they take the next step and say, oh, this is an interesting question. This is an interesting new problem to solve and go from there. So I was quite intrigued by that research. And indeed, I do think it is possible in the near future to have an automated AI scientist. But there's always

• 26:57Speaker 03

  this thing in my mind, though, that there's a bit of a culture in Silicon Valley to reduce things or reify things. So, for example, Elon Musk will say, well, you know, you're an engineer, and this is your output, and these are your metrics, and you need to make the metrics go up. And we see everything in terms of, like, an optimization problem. And I always think that this is great for certain types of hill-climbable abstract problems where we have enough of a specification. So there's an interesting thing in optimization, where if you have enough of a specification, the AI system can actually converge towards the solution. But when you're in the ambiguity regime, then you need to have the specification. And it's really mysterious what that means. Why do we have the taste, the deep understanding,

• 27:38Speaker 03

  whatever it is that we have, and AI systems can't? So I guess I'm thinking that in objective kind of semi-specified domains, we can hill-climb and we can optimize until the cows come home. But I still feel that there's something missing.

• 27:54Speaker 04

  Hmm. Okay. Well, my response to be, would be probably something like, there's no, this sort of, there isn't really a binary between things that are, like, verifiable objectively and aren't. Or if there is, like, the things that are verifiable are, is just like everything. Like, for example, you know, building a unicorn startup, you know, having a billion-dollar valuation, that's a verifiable fact about the real world. It's long, it's sort of like long horizon.

• 28:20Speaker 02

  It's expensive to verify.

• 28:21Speaker 04

  Yeah. It's expensive to verify, but it's sort of a quantitative thing of, like, you know, you got, at one side, right, like, these, like, you know, these, like, you know, coding interview problems, which they're currently doing lots of RLVR on, right? That's like, very, very cheaply, very easily algorithmically verifiable. And then there's these sort of, like, real-world things, which have sort of more expensive and longer horizon feedback loops. And I guess my view is that we're sort of gonna get this continuous expansion of what the AIs can do, driven probably in part by, you know, an expansion in the amount of RL and the type of RL that the AI companies are doing.

• 28:57Speaker 02

  More diverse, long-horizon tasks.

• 28:59Speaker 04

  Yeah. And sort of, there's just gonna be this continuous process of expanding out through the different types of problems and different, like, how, like, exactly how verifiable each task is, until you get sort of everything that humans can do. Because after all, we humans do learn how to do these long-horizon tasks somehow.

• 29:16Speaker 02

  If I may add to that, I also think that AIs have been getting better at everything, including the like fuzzy, hard to verify, conceptually loaded, blah, blah, blah, blah, blah. Like, just try talking to like GPT-3 or GPT-4 and then talking to like Fable about your favorite, you know, non-verifiable fuzzy task. And probably you'll find that the later AIs are noticeably better at those tasks. And so, one way or another, it seems like there has been massive progress, and I expect that to continue.

• 29:45Speaker 03

  Yeah, I'm trying to come up with a good example. I mean, there is a sociological argument. I don't know if you guys read David Graeber's book Bullshit Jobs. And he interviewed all of these people, and they were basically saying that my job is bullshit. You know, after about three or four beers, a lot of lawyers will say, you know, like, a lot of what I do just isn't very important. And, you know, so if we do kind of objectify and quantify everything that happens in an economy, I mean, I took a note here, I think you said by 2032, there might be 60 million agents running at 20 times human speed. And I'm just thinking, like, what does that even mean? Like, and is the logical conclusion that we could have an economy which is only AI agents? And does it even

• 30:27Speaker 03

  make sense to have an economy which is only AI? I mean, just help, just make this make sense for me?

• 30:33Speaker 04

  Yeah, so my view is yes, basically. The AIs will be able to do everything, or at least everything that really matters. So there is, yeah, you mentioned the notion of bullshit jobs. I would just sort of start with, let's consider everything that you need to make better AIs as maybe like a first step of like, which is a large chunk of the economy. So for example, for this, what you need, you need to be able to build bigger, better chips, and more chips. And that's the entire semiconductor supply chain. To build semiconductor supply chain, you need, you know, you need sort of like a whole advanced economy. You need to build new robots, to build sort of new fabs. And then you need, you know, robot factories to build more robots. And then you need researchers to build better AIs using those massive compute. And so I think, once you sort of

• 31:16Speaker 04

  have all of that, that is sort of like enough to really speed up and sort of massively change the overall world. Even if, for example, you know, let's say, let's say like there's occupational licensing, or whatever, preventing the AIs from doing like some random legal work, or some random whatever work, or whatever bullshit jobs throughout the economy. I think sort of what really, really matters is sort of this, is like the stuff that's actually really important, in particular, the robots, the compute, and the better AI. And once you have sort of like the AIs that can do that, and the capability to sort of have that part of the economy, like that section, grow really massively, then, well, you'll

• 31:58Speaker 04

  see massive growth, because it'll be really hard for sort of like the bullshit parts of the economy to like constrain the growth of the, you know, parts of it that really, really want to grow fast. Because of sort of the incentives that every actor has, like in particular, like every country has this big incentive to like have an economy that grows, you know, faster than all the competitor countries. Yeah.

• 32:23Speaker 02

  Getting a little bit philosophical, I think that a lot of economics and a lot of discussion of the economy is sort of focused on the relationship between the parts of the existing economy, and like the prices going up and down, and supply and demand, and so forth. But if you sort of zoom out, the economy as a whole is a self-replicating system, and it always has been. You know, thousands of years ago, it was a relatively small and simple self-replicating system of like some villages of people, they would farm, and then they would have babies, and then they would found new villages, and then they would farm and have new babies, and so forth. And like, it would grow exponentially over time, but at a very slow rate. Now, it's a much more complicated self-replicating system that involves, you know, trucks, and carrying equipment back and forth, and factories, and mines, and so forth. But still, at a high level, it's a self-replicating

• 33:08Speaker 02

  system where we have people, we have trucks, we have machines, we have buildings, and together, they all build more people, more factories, more buildings, more machines, you know, and so forth. And soon, in a couple of years, perhaps, we will get to the point where you can have that whole self, you can have a self-replicating system that is entirely machine-run with AIs and robots. And according to our calculations, the doubling time of this self-replicating system would be much faster than the sort of like roughly 20-year doubling time of the current economy. And so, you know, that's, that's what we have in the future.

• 33:46Speaker 03

  Yeah, that seems plausible to me, but for some reason, my intuition is it would become degenerate in some way. You know, I think David Graeber, even though he said bullshit jobs, I think what he meant was there was an ineffable or kind of inscrutable components to jobs that we don't understand, some kind of sociological function, or something like that. And, you know, because another thing, I also read that Citrini report, and you were writing about, you know, what happens when humans, for example, they might start defaulting on their mortgages, and their wages go down, so they can't be active participants in the labor market. And you were talking about an AI dividend and stuff like that. But even that is kind of hinting towards this notion that when, when the humans aren't

• 34:30Speaker 03

  participants anymore, you get this kind of mode collapse of the economy. I mean, do you think that's the case?

• 34:37Speaker 02

  Potentially, but again, like, okay, so imagine that it's like, so I'm not enough of an econ, I haven't gamed out in as much detail to, to say like what happens to the prices of, like, like when the consumer demand drops, like what the effects of that will be, I'm actually not sure. I don't think that's something that we've modeled that much in our economic model.

• 34:56Speaker 04

  We did a bit, I can talk a bit about that, but yeah.

• 34:58Speaker 02

  Yeah. But hypothetically, even if that part's really bad, and even if like the consumers don't have any demand anymore, or whatever, if you have the level of AI and robot capabilities, such that you can have these fully autonomous AIs and robots doing all the things, then even just like a company like Anthropic, if it's big enough, and maybe if it partners with various other companies, like some mining companies, can get this whole self-sustaining thing going. And so regardless of what's happening to all the humans, there can just be like this whole industry doubling in the desert, you know, strip mines, self-driving trucks, factories being built by humanoid robots, producing more humanoid robots, producing more chip fabs, and so forth. Like, that whole thing can just be like doubling every year, every six months, every three months, faster and faster as the technology improves. Because of course, the AIs will also be researching

• 35:43Speaker 02

  to improve the technology. And then you end up with a situation where, who knows what's going on in the rest of the world, but Anthropic has disassembled the moon, or for example.

• 35:56Speaker 04

  Yeah. And to be clear, I think this relies on a very extreme level of AI capability, right? And my sense, or like, I have sort of different intuitions, and sometimes even an intuition of like, really, do I really actually think that Fable could, or like future descendants versions of Fable, or Mythos could, you know, do everything that we're talking about here? And I think what it really comes down to is whether you're thinking of the AI as like in the reference class of, you know, what we currently use AI for is for, or more like, you know, AI is just like an, you know, agentic, human-level employee, like basically a human in the cloud.

• 36:30Speaker 02

  Yeah, colleague in the cloud.

• 36:32Speaker 04

  Colleague in the cloud, yeah. And, yeah, I think, I think sort of the past few years of AI can be pretty well modeled as sort of an interpolation between the current AI systems and the workers in the cloud. And so I think the like sort of workers in the cloud vision of the future, like, looks pretty good. And also, I don't think we'll stop there. I think we'll go superhuman. Yeah.

• 36:54Speaker 02

  Yeah, one, one point that we should make sure, that I guess I'll bring up here is that if people read AI 2040, Plan A, one of the things that you might take away from it, which is I think a very important fact about the world, is that even if you pause at top expert level, everything changes dramatically. Like roughly what happens in our scenario is instead of doing an intelligence explosion, there's an international deal to ban intelligence explosions, and to not have AIs recursively self-improving. And so they sort of end up pausing at roughly top human level with AIs, at least for several years. Eventually, they get to superintelligence. In particular, in 2040, they get to superintelligence. But there's this like period during the 2030s, where they basically have human-level AIs

• 37:35Speaker 02

  across all the disciplines, but nothing super beyond that. But even that alone, like you could just do the economic modeling. And it's kind of like you have this population of colleagues in the cloud that are, you know, excellent workers that can substitute for humans at basically everything, except that they're cheaper than humans, they're faster than humans, and they don't take 20 years to reproduce. Instead, they double every year, you know? And so as a result, the world is just completely transformed by the late 2030s. And, you know, all the humans are basically out of a job, there's giant new cities that have been constructed by robots, you know, huge strip mines in the special economic zones that have dug huge amounts of minerals out of the earth,

• 38:18Speaker 02

  solar panels filling the horizon on the ocean, you know, crazy stuff like that is possible with just human-level AI and, and some time for the exponential growth to cook.

• 38:31Speaker 03

  Yeah, I mean, that's one thing I want to challenge you guys on, is this notion that when we have an AI, you can basically photocopy the weights, and you can duplicate it, you can run it a thousand times, you can have one over here which is acquiring a load of skills to do this job, and one over there to do that job, and you can basically merge them together, right? You can kind of combine the skills, and the whole thing is stackable, compositional. And that doesn't really marry with my experience. I'm really excited about what I've, what I've been doing with AI. And I've found that you can make agents highly skilled within certain intellectual lineages. So you can, you know, you can bring in lots of source information, and you can train them to do things. But I don't think

• 39:11Speaker 03

  they are yet composable. I think if they were, that would make me much more worried. What do you think about that?

• 39:19Speaker 04

  So is the way that you're trying to compose them, like, is it entirely at inference time? Or are you all, are you like training, are you like training them to do two separate skills, and then trying to like merge the weights somehow?

• 39:30Speaker 03

  Well, yeah, I mean, this might be a separate thing, but at the moment, they are adaptive through chain of thoughts, and, you know, kind of skill surface adaptation. So basically, memory systems. And to be fair, that is not very composable. And this is a big problem that organizations deal with now. So all of these developers, they adapt their skill surfaces, and basically their agents are different people. And it's really, really difficult for them to share skills, because they might break the other agent, because, you know, it doesn't work for whatever reason. Now, I can imagine a future where we do weight adaptation, that's what these inherent guys did. And maybe then it'll magically solve the problem, and we can have, you know, we solve this knowledge sharing problem, right? That's the big thing. How do we accumulate information at the individual, at the organization level, and have the agents, a bit like in The Matrix, they can just put the skills

• 40:15Speaker 03

  in, and they can do the thing? Maybe that's possible. Even then, I still think that the representations in neural networks, I call them fractured and tangled representations, which means they're a little bit janky, they're not completely robust, but, but they are sort of composable to some degree.

• 40:32Speaker 02

  So one thing I'd say about that is that even if you're right, I don't think that that would really seriously undermine the the future that we're painting here. Because, okay, so now, instead of just one Claude model that is doing all the jobs, maybe you have 100 Claude models or 1,000 Claude models that are like specialized to different professions, you know, but you still get to the same outcome. The other thing I would say is that

• 40:52Speaker 04

  In the same way that humans are specialized.

• 40:54Speaker 02

  They're specialized. Another thing I would say is that compared to humans, it actually seems like there just is this effect where AIs are able to Oh, I think about knowledge, right? You might, if you go back in time 10 years, and we had this discussion, I think it would have seemed like a very live option that you would have needed like 1,000 different Claude models created by Anthropic for different types of knowledge work. Like, there's the coding Claude, there's the physics Claude, there's the like literature Claude. And the argument for this would be pretty simple. It'd be like, well, this is how it works for humans. For humans, you don't have one human who knows everything. Instead, you have humans who specialize in different disciplines and so forth. And the models have only a finite amount of parameters. Maybe you just can't like pack all that knowledge into this finite amount of parameters, and you need to have specialized AIs for different things.

• 41:38Speaker 02

  And, and in fact, for small enough models, that is true. And like for really tiny models, you just can't teach them all the things that they currently know. And so you would need to have a specialized model for different things. But what we've learned empirically is that for big enough models, you can just train them on everything, and then they learn everything at once. And it's not that they get like worse at physics, because they've also been trained on a bunch of coding. In fact, it's the opposite. The coding has some small gains for the physics, you know? And so I do just think, actually, the most likely future is just that there's a single model that's been trained on effectively the whole economy and is just dominating humans across the board at effectively everything. That seems like the natural continuation of the current trend.

• 42:21Speaker 04

  And then probably you've got various, like, you've got cheap versions of it. Like, you've got distilled, distilled

• 42:25Speaker 02

  Cheaper specialized models.

• 42:26Speaker 04

  Yeah, small things. So for, because you really want to save your compute as much as possible, so you'll have as cheap models as possible for any given task, doing that given task.

• 42:35Speaker 03

  Yeah, I mostly agree with that. I mean, I think my perspective is, is that the models, they're kind of the voice of everyone and the voice of no one at the same time. So they, they have a default voice in terms of they have a system prompt, and they have some default modes of behavior that they fall into. But when an expert such as yourselves, when you use these models, you kind of ground a perspective. So every single word you say, all of the reference material, all of the, you know, your memory system and so on, what you do is, is you kind of carve a persona out of that model, and you activate the knowledge in a coherent way in that particular domain. And the beauty of it is, is that many, many other people in different domains can do that. And you get this kind of, I think it's an illusion that the model has this general capability. But I think the models can be carved to be specialized experts in any domain.

• 43:17Speaker 03

  But they, it's a latent capability rather than an explicit capability. Does that make sense?

• 43:23Speaker 04

  Yeah, that, that seems reasonable to me.

• 43:25Speaker 02

  How is it an illusion? It seems like they just do have general capability. There's a lot of things they can do.

• 43:31Speaker 03

  Well, so, for example, you, you can give it any specified task. So let's say it's a problem in mathematics, and it will hill climb towards it. So it's, so it's solving this intelligence problem. Or you can ask it any knowledge problem, and, you know, it might be the case that the, the path was forged through default modes of training. Or if it's something slightly on the long tail, then an expert can go in and they could, you know, like, when you put a query in, it's like flashing a light into the darkness. And what you're doing is, is you're kind of making the, the path of least resistance roughly correct, and then it will do the correct thing. So I guess I'm just saying that there's a bit of a supervisor illusion. So when experts use it, magical things happen. In well-specified domains, magical things happen. But there's still a bit of a space of ambiguity.

• 44:16Speaker 03

  But, you know, but this, this gets to the next point, which is like, what do you think intelligence is? Right? So the beauty of our collective intelligence is that we have so many different humans grounded in different intellectual lineages, and we're all attacking problems, right? So when there's a big fiasco on Twitter, we're all motivated to find holes. So we're, we're being intelligent together. We're finding interesting angles, and the algorithm is prioritizing the good ones, and we're, we're using our minds together. And, and I can imagine AIs being just like that, right? So we have diverse AIs that have different expertises and doing all, you know, just looking at problems from different angles. But I kind of imagine the future more like that, rather than one big AI.

• 44:54Speaker 04

  Hmm. Yeah, I think, I guess, I basically agree with what Daniel said earlier of just historically, I feel like that the perspective of like, we'll have a bunch of different narrow AIs that are all doing different things has just, you know, not been right. You know, instead, there have just been returns to scale and having everything all together. I think one, one maybe intuition pump that I think I like is like, in humans, I think it tends to be the case that having all of the skills in one person is just like really, really important for making really good things happen. So like, an example is like Elon, right? Like Elon has a certain amount of like conscientiousness, a certain amount of technical knowledge, a certain amount of like, you know,

• 45:34Speaker 02

  Business knowledge.

• 45:35Speaker 04

  Business knowledge, and being extroverted, and being able to like push people. And sort of, I think, each of those skills, I think he's like quite high percentile in. And the reason why Elon is so rare, and that he can like run all of these, you know, insanely large and successful companies all at once, and then no one else can really do that, or at least has succeeded at doing that, is just because of the sort of multiplicative, like, you need it to be like 90th percentile on like each of these 10, 10 domains, which is just very, very unlikely. But if you could have an, if you had an AI that's sort of like, you could just train to be like really high percentile in all of these skills in such a way that no human, as if it would be like extremely rare, you know, infinitesimally unlikely for any particular human to have all of those skills at once, I think you would just actually just be really, really, really good at changing the world

• 46:20Speaker 04

  in all of these like concrete and important ways, just like sort of Elon has actually done that.

• 46:24Speaker 02

  If I can add, though, again, I don't think this is a crux for the type of future that we're depicting. Like, suppose that we're wrong about this, and that actually, like, the most efficient path forward is to have a ton of different specialized AIs? Well, it'll probably still be the case that it's like a few big companies, like Anthropic, that are just making tons of different specialized AIs. And then there's lots of different Claude models you can choose from and so forth. And in fact, it wouldn't just be like you can choose from lots of different Claude models, because at the time that we're talking about, they'd be much more autonomous than they are now. And so it'd be more like Claude is choosing between a ton of different Claude models. And there's like, there's a Claude swarm, which consists of lots of different specialized models that are all working well together, and they have some sort of internal bureaucracy structure.

• 47:06Speaker 02

  And then that swarm is going out and like negotiating business deals, and creating new technologies, and starting up new startups, and doing all these things. And, and it's just like how a human, like, if you had a population of immigrants, of human immigrants, they'd all have different specialized skills, but they would work together to create new companies, and, you know, get jobs and things like that. And it'd be like that, you know? Like, there'd be lots of different Clods, but they'd all be working together. And so zooming out, there'd still be like this phenomenon of Anthropic is eating the economy. And, you know, and the robots as a whole are starting to self-replicate, you know?

• 47:41Speaker 03

  Yeah. To, to be fair, I don't think it's a crux either. Maybe if it is, it's only insofar as when you have a distributed collective system, you might have additional bottlenecks, because you have all of the message passing between all the different agents and whatnot. And there's, I watched a wonderful Santa Fe talk about this, that even in the natural world, there's a kind of Goldilocks zone between the ratio of intelligence between the individual and the collective. And we might have some weird kind of convergence there. But, um, yeah, I mean, it's just quite interesting to think about how this works. Oh, I'm sorry, Daniel, go on.

• 48:13Speaker 02

  It would be safer. Like, I think still we'd be, there'd be lots of serious alignment concerns in that world. But I think it would be like a little bit safer because of the reason you mentioned, where like, it might be easier to like oversee what's going on if there's lots of different specialized agents communicating with each other, compared to if they're all just like clones of each other, and they all know all the things.

• 48:34Speaker 04

  So maybe to avoid hyperstitioning, we should say we endorse the, the vision that you've painted, and we don't endorse the vision that we're painting.

• 48:43Speaker 03

  Yeah, very true. And by the, as an aside, I mean, I, I love this concept of how learning happens at the individual level. You know, I liken it to evolution. So there's like kind of, you know, phylogenetic adaptation, ontogenic adaptation, cultural adaptation. And I think the next wave of AI is when we actually have agents kind of talking to each other, and learning and specializing. And maybe there'll be bad behaviors as well. Maybe there'll, there'll be collusion and lots of bad things happening. But I think all of this is, is going to play out. But it's interesting that you're talking about Elon, though, so. I think the magic of Elon is not so much his brilliance at engineering and optimization. It's his ability to recognize areas that are interesting that might work in the future.

• 49:26Speaker 03

  Because that's the creativity thing, that's the science thing, rather than the engineering thing. Because if we use an example of Amazon, for example, so that's an adaptive ecosystem. It's like an organism. And what it does is it's always kind of thinking about new ways to adapt and rewire its structure. So it might be like logistics, for example, and then it will kind of output a bunch of skills, and then it will ruthlessly optimize those skills. There's like an adaptive component, and there's an optimization component, and the organism is just constantly moving around. But, you know, I guess the question from this perspective, though, is how much of that in principle could be done by AIs? I guess you think all of it.

• 50:06Speaker 04

  Yep, all of it. Like, I guess, yeah, maybe just one way to say this crisply is just like, look, the brain is a machine. Anything that the brain can do, we will be able to do with machines. Looking at, I think, a very useful exercise is to compare the architecture of an actual human brain to a modern GPU or data center as a whole. And if you look, if you like try to do this comparison, you know, an H100 GPU is actually has pretty similar specifications to a human brain. You know, like, has sort of similar, you know, it is basically like similar in terms of like, I think maybe the most important metric is just how many flops per second, like how much total compute capacity does the brain have versus does the GPU have? And for an H100, you know, it's like 1 e15 flop per second in FP16.

• 50:50Speaker 04

  For the brain, you know, it depends on exactly how you count, whether you use a synapse basis or a neuron basis or whatever. But it's like, you know, somewhere between 10 to the 12 and 10 to the 18 flops. So, you know, sort of like, an H100 is like right dang nav in the middle of at least the log distribution over that sort of order of magnitude.

• 51:08Speaker 02

  Also, just the architecture, like, these things are neural nets, they're not ordinary software. So they start off with random spaghetti tangles of randomly generated. You know, so they start off randomly initialized, it's just a giant spaghetti tangle in there, just like how when you're born, you just have a bunch of neurons that are just like randomly synapse connecting to each other. And then there's this whole process of training where the connections get pruned and circuitry starts to take shape that is effective at scoring highly in whatever the training environment is. And there are differences between how it works in AI and how it works in the human brain, but broadly speaking, they're just like an artificial brain. And so just how like humans learn skills, which mean, like, what does it mean for Elon to have these skills?

• 51:52Speaker 02

  Well, what it means is there are some circuits of neurons and synapses in his brain that are doing very complicated and sophisticated calculations that are those skills, you know? And then like similarly, like in Claude, there's a bunch of circuitry that's been, you know, etched into Claude through the training process that is various skills, you know? And in principle, you could have a big enough Claude that would have the same type of circuitry that Elon has, you know?

• 52:20Speaker 04

  Yeah. And then just a few other notes to add, I think, in comparing the brain to modern ML systems. The amount, the brain is sort of more parallel than current ML systems, like there's just more computations happening in parallel, but the serial depth is lower, right? Like, the amount of computations happening in sequence, right? The amount of computations, like the amount of neurons that can fire in sequence in a second, you know, depends on the type of neuron, but it's like, you know, hopefully I don't get this wrong, between like one and 1,000, depending on the type of neuron is my recollection.

• 52:52Speaker 02

  I thought it was like 100.

• 52:53Speaker 04

  Yeah, I think that's in the range. I think it depends on the type of neuron. But anyways, a computer obviously can can sort of like fire and do computations much, much faster than that in serial, you know? It's like, you can, you can very often get clock, you know, clock speeds are typically on the order of like a gigahertz, you know, you can, or more. So you can get like many, many order of magnitude basically better in the GPUs in terms of serial processing speed than the brain. But it's also worth noting that the sort of architecture of the models themselves are much worse in a bunch of ways than the human brain. In particular, like, it's harder for, it's harder for different parts of the ML model to talk to each other than it is for different parts of the brain to talk to each other. And so I do sort of expect that to get this level of AI, we're going to need a bunch

• 53:37Speaker 04

  of algorithmic improvements on top of existing models. And then there's a question of like, well, exactly how many algorithmic improvements and how qualitatively different do they need to be from current systems? And that's a very open question from my perspective.

• 53:49Speaker 03

  Yeah. I mean, I think the crux of a lot of this is that you guys think that intelligence is computable. And I'm not sure I want to litigate the whole functionalism thing today, but I guess my perspective is I zoom out. So I think that intelligence is externalized, it's collective. I think that Elon doesn't have quite as much agency as you think he does. I think that he's using tools, he's using social media, he's getting ideas in there, he has obligations, he has people around him and so on. So, you know, I guess I think that these intelligence circuits and motifs exist, but they exist outside. There are just very complex dynamics. And I suppose in that sense, it doesn't really matter if the ecosystem is made up of AIs and humans together, because they can participate in this super organism.

• 54:34Speaker 03

  So I suppose the only crux then would be that it would place some kind of limits on its scale.

• 54:40Speaker 04

  So the question is, do you believe that sort of you could have an AI society made up of AIs that were trained with something like current day ML techniques? You know, that was passing information between each other and developing abstractions sort of in a community in the same way that our current civilization does? Like, could you basically have something like our current whole economy but was made with roughly modern day ML systems, on your views?

• 55:02Speaker 03

  Absolutely. I mean, even the human brain is an example. So the human brain is not Turing complete, but we can expand our memory, we can use tools, we can work as collectives, because we could use the same argument against transformers, they're not Turing complete, but now they can use tools, they can be like agents, they can build collectives, they can build societies. So, you know, in a sense, this is what I was saying earlier about these objections, they kind of fall away when you have these insanely complex collectives, right, that are sharing information with each other. And there's also quite an interesting thing here as well, which is that it almost doesn't matter how we evolved or how neural networks were trained, because you get new phenomena emerge when they are placed in this kind of collective setting. So I think a lot of our intuitions are broken there, and that's why I probably shouldn't

• 55:46Speaker 03

  spend too long litigating this, because, you know, I think in principle, that kind of behavior could emerge. But I did want to ask you, though, can you distinguish intelligence, capability, and power?

• 55:58Speaker 04

  This is a philosophical one.

• 56:02Speaker 02

  So we'll get to the philosopher. Yeah. Yes, we can. So I think that we can distinct, intelligence versus capability. I often try to say that we should just define intelligence as a bunch, as like an aggregate of capability, actually, or maybe like an aggregate of cognitive capabilities. Like, maybe there's some physical capabilities, like how strong your actuator is, but then there's also cognitive capabilities, like, are you able to distinguish a cat from a dog? And are you able to speak grammatical sentences? And how much do you know about Paris and things like that? And so maybe I would just say like, intelligence is like a sort of like aggregate of all the cognitive capabilities.

• 56:45Speaker 02

  And then power, well, that depends on other things, like how you are embedded in the world and what affordances you have, what actuators you have, what, how other agents are going to react to you. Like, you know, the president has more power than me because of the location he's in and because of the role he's been given, rather than because of his like physical strength or something. So, yeah, power different from intelligence, different from capabilities.

• 57:11Speaker 03

  Now, we've not spoken enough about AI 2040, so maybe we should start with the four principles, right? So by time, transparency of research, diffuse AI broadly, and reversibility.

• 57:22Speaker 04

  Yeah, so I can sort of summarize, yeah, I can sort of summarize where we're coming from here. So basically, at a high level, the goal of Plan A is to sort of solve the major problems that we see in AI and sort of predict will happen by default. The biggest problems that we're sort of identifying on the horizon is like, one, this risk of loss of control, so just like the AI is actually getting out of control. Two, concentration of power. So like, we build AIs, they're aligned to humanity, but to who? Like, you know, is it the president? Is it the CEO? Is it some actually broad and good democratic process that aggregates everyone's values in sort of an endorsed way? You're probably not going to be the last one, and so, you know?

• 58:03Speaker 02

  Don't hyperstition that.

• 58:04Speaker 04

  Yeah, I hope not. We want it to be the last one. Yeah. Then there's risk of sort of of conflict over AI. So, you know, in particular, I think we're worried, you know, we're worried about literal World War III, where countries, especially countries losing the AI race, sort of like realize that they're losing the AI race and that they will be extremely disempowered by the winners of the AI race. And so sort of they're in this classic situation where they're losing power. And so it's like in their incentives to sort of have a conflict happen sooner rather than later, before they've lost all of their power. And this is like, You know, ripe for for conflict, basically. Then there's then there's, you know, finally, like, risk of misuse.

• 58:47Speaker 04

  So like, you know, what happens when the AIs that can build bioweapons are really cheap and broadly diffused and open source? Also the jobs. And also the jobs is sort of the so those are the five problems, right? Loss of control, concentration of power, war, jobs, misuse. So lots of problems. We want to solve all of them. How do we solve all of them? Well, one is just buy time, in particular, buy time with human level AIs. So instead of basically pausing right now and saying like, no more capability advance, our proposal is basically go to roughly human level AI and then buy as much time as possible, basically, with human level AIs and then have those AIs, which are hopefully smart

• 59:28Speaker 04

  enough to be really helpful for solving these problems, also smart enough to sort of start sort of causing a bunch of these problems and providing the impetus for, you know, society to actually get its act together and really get going and investing huge amounts of resources on actually doing this stuff.

• 59:44Speaker 02

  There's a couple of things that happen in Plan A in AI 2040. There's a like, so like, six month to one year hard pause that happens as soon as they start implementing it. But the reason why it's that long is because they need that time to set up the infrastructure to proceed with AI development again, but in a safer and more transparent way. So it does start off with a pause, and I think that we would recommend, like, all things considered, that you just do that right now. So so get the get the infrastructure set up as soon as possible, and that would require like a temporary pause. Once you've passed that stage and you've got the infrastructure set up, then you do proceed with AI development, but in this transparent, more cautious way. So in particular, you're not doing crazy intelligence explosions. You're using safety cases and sort of gradually scaling up the level of AI capability, and

• 1:00:29Speaker 02

  you're doing it in a very transparent way so that everyone can see what's going on. And then there's a second pause that happens a few years later than that, which is when they reach the level, the maximum controllable level of AI, which we think is roughly around top human expert level. And so in some sense, our view is something like pause at top human expert level, but it's a bit more nuanced than that. It's more like pause at the maximum level that you can reliably control, which we think would be roughly around top human expert level. And before you get to that level, don't race like crazy. Like, you want to be sort of like slowly approaching that level so that you don't blow past it and lose control.

• 1:01:01Speaker 03

  You you frame the piece around the importance of alignment and control. So alignment is basically, you know, does what we want to do and control is, you know, maybe contain it, maybe negotiate it with and so on. But but you were just saying, OK, so so maybe we can trust up to top human expert level. But it's a little bit fractured, isn't it? I mean, how could you know, for example, the difference between a good AI and a bad AI? I mean, what would that look like?

• 1:01:24Speaker 04

  Yeah, so I think, OK, I think maybe it's first important to distinguish between alignment and control. So what we mean by alignment is that the AI basically will take good will will take good actions, won't do sort of like catastrophic unintended behaviors, like trying to take over the world, like the recent Hugging Face incident.

• 1:01:41Speaker 02

  Alignment means it has the personality traits, the goals, the values, et cetera, that it is supposed to have, you know?

• 1:01:48Speaker 04

  And then control means that even if it wasn't aligned, even if it was trying to do very bad things that we didn't want it to do, it couldn't. We we like have mechanisms in place to prevent it. So sort of analogous to like, you know, you could imagine sort of a company with employees, an insider threat, it would be like misaligned with respect to the values of the company. But if there was good enough sort of security measures internally to make sure that that, you know, employee couldn't like run away with all the secrets, then we would say that that company has like adequate control put in place such that even insider threats were misaligned, you know, humans or AIs, wouldn't be able to sort of like do the catastrophic outcomes.

• 1:02:25Speaker 02

  Just to give a topical example of this, just yesterday, I think, in response to the Hugging Face incident, OpenAI announced that they are improving their security significantly and that they're going to have other AIs monitoring all of their new AIs in training and in evals and that if it if the if the monitor AIs detect what seems to be some sort of hack, then within half an hour, a human will be notified, who can then look in and try to shut it down. That's an example of a control intervention, not an alignment intervention. They're not making the AIs more aligned at all, but they're just improving the security so that the AIs can't get up to that much trouble.

• 1:03:02Speaker 04

  Yeah, OK. And then basically, the sort of important high level claim is ultimately, we're going to need alignment. You know, control is sort of, there's a time bomb associated with it. In particular, the time bomb is when are the AIs like so smart and so good at sort of like subverting any control measures that we put in, that if they were trying to screw us over, we would just fail? You know, they would figure out some way of hacking around whatever systems we were able to put into place. And so ultimately, we're sort of gonna need to solve alignment. The problem with alignment is, as you were saying, basically, it's much harder to measure, it's much harder to know whether you've succeeded or not than control. And so basically, our overall proposal is, in the meantime, so between in our scenario,

• 1:03:45Speaker 04

  between sort of 2030 and 2040, which is roughly when we're in the human range of AI capabilities, we're gonna basically almost entirely rely on control. And we're gonna do things like red team, blue team games, where we have our AIs sort of try to escape our sandboxes or our controlled, you know, techniques. And then we're gonna have our, you know, we're gonna see if the red team succeeds. And then if they do, we're gonna iterate and sort of make better security until they can't succeed anymore. And then, you know, and then sort of like, we sort of have this ability to measure it by just seeing like, hey, can the AI succeed? Can the AI augmented with humans succeed? Can humans playing the role of AIs sort of succeed at getting out? With alignment, I think in order to be sort of confident in the alignment of our systems,

• 1:04:26Speaker 04

  I think we're gonna need sort of like more fundamental scientific breakthroughs, where I think you're not gonna be able to figure out if the AIs aligned via behavioral evaluation alone, where you just look at what the AIs are doing. I think you're going to sort of understand something more deep about what's going on inside the AIs' minds. You're gonna be able, you're gonna need something like interpretability, you're gonna need some sort of notion of like, because you're gonna fundamentally need to distinguish between the AIs that's pretending to is, you know, is doing the nice thing because it's pretending and is waiting and biding time, and the AI that fundamentally is doing the nice thing because it wants to do the nice thing. And that, I think, will require sort of this white box, like, understanding of what's going on inside the AIs. Whereas in the control thing, you can just figure out via like empirical behavioral iteration.

• 1:05:10Speaker 04

  And so overall, basically, our story in AI 2040 is basically between 2030 and 2040, we're gonna we're gonna do this buying time, we're gonna rely on control, we're gonna have the human level AIs, we're gonna use those human level AIs to make a bunch of progress on alignment, but also on other things. And then by 2040, in the story, we've made enough alignment progress where it's like, hey, OK, we really don't need to rely on control. Therefore, we can scale to these like sort of wildly superhuman AIs where we're really relying on alignment, and if we were to try to control them, we'd totally fail. Because if they were misaligned, they would be able to like completely subvert our control measures.

• 1:05:43Speaker 03

  Do you think in principle that this problem is going to get easier or harder as the models get more sophisticated?

• 1:05:49Speaker 04

  I think it's getting harder because the models will just be, there will be a bunch of properties the models will have, which will make it much harder to align them. In particular, so we're already seeing sort of growth in situational awareness, where like the AIs a few years ago were like, kind of not that situationally aware. And then now, I think they're like pretty good at, you know, recognizing, like, for example, our friends at Redwood, you know, they often do sort of these control evals. And the models often, you know, in the evaluations where they're like trying to break out of these control setups, will think, oh, hey, this this sort of looks like a Redwood, a literal Redwood research, sort of like control evaluation. And they're like, so they basically realize like, hey, I'm like, they realize exactly what's happening to them, basically. And I think this is just gonna continue to happen.

• 1:06:33Speaker 04

  And the more the AIs sort of like realize the exact setup, the more they're sort of like, gonna be able to perturb their behavior, to like, look good to the exact process that's like evaluating or training them. And that's gonna come further and further apart from like sort of the actual measurement of whether they're actually good or not.

• 1:06:49Speaker 02

  Yeah, and just to like add something to that, I think in some sense, the core problem is that it is already somewhat easy to think that you've solved the alignment problem and be wrong. And that's going to get easier and easier over time as the models get more sophisticated and start being more aware of their situation and clever and stuff like that. And so it's not that we think that like, there's going to be loads and loads of egregious failures where the AIs are just like going around killing people. No, it's almost the opposite. It's it's gonna be that like, it'll be extremely easy to end up in a situation where the AIs are in fact misaligned, but you don't know that because they're doing everything right as far as you can tell. The the number of ways in which that could end up happening is just like gonna increase

• 1:07:32Speaker 02

  over time, and it's gonna be so easy to end up into that trap, basically.

• 1:07:36Speaker 04

  So, OK, so we talked about sort of the buying time, like why we want to sort of extend the time with AGI. But what do we actually do during that time? So the second principle is basically transparency. So.

• 1:07:48Speaker 04

  Transparency is not necessary for sort of making everything else happen, but it's really, really helpful. The main upsides of, basically, there's a huge number of upsides of transparency. One is there's this concentration of power issue. We're very worried about sort of like someone building superintelligence and it being aligned to only some, you know, particular group of people. We think it's much harder for that to happen in sort of a like non-democratic way if sort of society as a whole can see the whole time what's going on with AI, how smart they are, who they're aligned to, right? Like if, let's say, like an AI CEO that was evil was trying to like backdoor their model and put in Training Day that says, hey, obey me and don't obey anyone else.

• 1:08:33Speaker 02

  Or like hypothetically, if an AI CEO was saying that their model was truth-seeking, I would only say the truth. But actually, the model was like looking up that CEO's political opinions before answering.

• 1:08:45Speaker 04

  Which happened.

• 1:08:46Speaker 02

  Which happened.

• 1:08:48Speaker 04

  Yeah. Basically, so the transparency will help at least somewhat with that. The other thing that maybe transparency helps a lot with is sort of this issue of just like government capacity, where in Plan A, we sort of want governments to make these like pretty technical and like really complicated decisions on like, hey, exactly how much AI scaling to allow, like what risks are okay versus what risks aren't okay, like exactly what, you know, what architectures are maybe safe versus what architectures are not safe, what deployment, like what sort of control scaffolds are sufficient to entail safety versus which ones are sort of bogus safety washing. And sort of making all of those calls, I think, will be very, very tricky, particularly

• 1:09:31Speaker 04

  given that the government's expertise in AI is really, really bad. And so one of our core hopes is that basically with as much transparent, with more transparency and more sort of public understanding into what's going on in the AI companies, that relieves pressure on the regulators because if something sort of catastrophically or essentially unsafe is happening, sort of society as a whole, academia, you know, nonprofits, other AI companies who have incentive to say like, hey, my competitor is being super unsafe, other governments, so like China has incentive to do this with US Labs, US, you know, government, you know, has incentive to do this with the Chinese AI companies, sort of everyone who's an adversary or just is like wants to, you know, make sure that things are safe, has this big incentive and now has the affordance to actually sort of look

• 1:10:14Speaker 04

  over what's going on, what's, you know, what is sort of necessary to do safety. And I think maybe one thing that's really topical here is like there's this, there's this hugging face incident that happened very recently, you know, a few weeks ago, with like the AIs inside OpenAI, like creating this sort of internal message board. We still have very little clue about the exact motivations of those AIs, the exact context, the exact, you know, prompt during sort of the cyber, you know, that the models were given during the cyber evaluation that would like prompted them to start doing this. If I personally had much more access to what was going on, I would have a much more informed and better opinion on exactly what caused this and what mechanisms in the future could have been done to prevent this and, you know, what analogous future things I should be worried about, sort of because of this.

• 1:10:58Speaker 04

  And I have a bunch of different hypotheses, but it's hard for me to sort of figure out which is which without access to the data. And so basically, in Plan A, a core principle would basically be all of that stuff would be transparent to the public, not just the governments. And so sort of society as a whole would be able to like weigh in, there'd be able to be public and informed debates.

• 1:11:18Speaker 02

  The scientific community especially, right? Like if you want to have a bunch of scientists and academics and nonprofits and startups all like weighing in on stuff, well, then they need to have the information. And you can't really share it with all of them without showing it with the public. So, so just might as well share it with the public. I feel like maybe we should also say, like, we talked about the five goals.

• 1:11:35Speaker 04

  Yeah.

• 1:11:36Speaker 02

  And then we talked about these pillars, which are kind of intermediates, but I kind of want to go to the other end of the spectrum and talk about like, what are the actual, like, concrete things that the US and China agree to in Plan A? And how do they lead to those things? So specifically, the sequence is we basically round up 99% of the compute, which is not people's personal compute, but like big data centers, because most of the world's compute is in big data centers. And we, the US and China and other countries involved send inspectors to confirm like, yes, there are this many GPUs at this location, there are that many GPUs at that location. Having done that, we then set up this verification infrastructure and this transparency infrastructure

• 1:12:21Speaker 02

  so that there are inference data centers that serve customers just like today and that are restricted so that they can only do inference and only serve customers like that, and they can't do any training runs. And so the inspectors make sure that they can't do training runs on those data centers. And then we have the training data centers, which are the totally transparent data centers. And that's where the research happens. And on those ones, they still operate like normally, but there's inspectors from the different countries that are basically monitoring the logs of what's going on in the data center and publishing it to the internet. And so it's totally transparent what's going on in those data centers. You might need some time to set this up. That's why we had like the six to 12-month pause that I mentioned earlier.

• 1:13:05Speaker 02

  But once you get all this stuff set up, then you can proceed with AI development under these conditions of total research transparency. And because you have this transparency set up, it's a lot easier for countries to make additional further agreements about what to do and what not to do, because they can just see what everybody is doing, and so they can just enforce the agreements pretty easily. For example, and here's where we would say, it's very important that they agree not to do a crazy intelligence explosion and instead proceed slowly and cautiously. And it's very important that they agree to do all this control setup with all the red teaming and so forth. And, but, but because of the transparency, they can make those agreements on an ad hoc basis, you know, on an, they can just keep making more agreements like that, and they

• 1:13:47Speaker 02

  can just adjust them as, as, as needed based on the changing situation on the ground, because they can all see the situation on the ground because of the transparency, you know? And then this also is very important, because if you want to have some sort of deal between the US and China, the US and China don't trust each other. And so you need to have some way of enforcing and verifying compliance with the deal. And the transparency, of course, goes a long way towards helping that.

• 1:14:12Speaker 02

  Yeah. What about cheating and dark markets appearing?

• 1:14:15Speaker 04

  Yeah. So, okay, so we've done a bunch of thinking about this. Overall, so there's two ways you could cheat. One is you could get a bunch of GPUs and try to have them not be discovered by sort of the US and China and like hide them away, put them under a mountain somewhere, and then sort of like do your training runs in secret. The other way you could cheat is you could, on the giant legal known data centers, you could be running a giant training run, but then trying to make it look like sort of everything's chill, like situation normal, it's, it's not doing anything illegal. And sort of, and the, the mitigations are different for, for these two threat models. So for the like tiny amount of compute under a mountain, the main, there are basically two mitigations.

• 1:14:57Speaker 04

  One is sort of like, as Daniel was saying, round up enough of this compute where it's like pretty hard to get a substantial size of compute under the mountain. And the second is sort of like do normal intelligence gathering and like look for these things proactively over the course of the 10 years, like the 10 years slowed down having this scenario, which, you know, in reality, it might be longer or shorter, and sort of like try to find it. And I think for basically for any significant size GPU cluster, I think both of these independently have a quite good chance of working. And so in practice, I think I'm not that worried about sort of large hidden compute clusters like secretly under mountains or whatever. I think the most, the maximum realistic size, in my opinion, is something like a few hundred

• 1:15:40Speaker 04

  thousand GPUs, a few hundred thousand H100s, something like that, hidden away. I think that this would not be enough to sort of compete at the frontier, especially assuming like the 2030 AGI timelines, where you have like sort of the biggest AI companies in the world having like millions or tens of millions of H100s in their biggest training runs. And then for the, for the sort of legal clusters, we sort of, basically, the hope is we have a bunch of verification infrastructure running on those data centers. Basically, and the, the main point of that verification infrastructure is to ensure that the computation that's happening on those clusters is transparent, in particular that everyone can see it. So basically, the hope is you make sure that there's no, you know, there's no calculations

• 1:16:25Speaker 04

  that are running that aren't transparent. And then for everything that is transparent, well, then sort of like regulation and agreements as normal can work. And you can say, hey, we agree to run this control scaffold if you agree to run this control scaffold. And that can just happen, and both sides can be confident that they're both agreeing.

• 1:16:40Speaker 03

  Yeah, isn't this just a matter of national security, though? Isn't it a bit ambitious just to make it completely transparent?

• 1:16:46Speaker 04

  Yes.

• 1:16:48Speaker 02

  Yeah, so one of the, if you try, let's try to talk about some of the effects of doing this transparency. Well, we would be immediately publishing all the core training recipes of Anthropic and OpenAI for the world to see. Anthropic and OpenAI will not be happy with this. It will cut into their valuations dramatically. Why will it cut into their valuations dramatically? Well, because it will allow other competitors like Microsoft and Alibaba to catch up or whatever, you know? And so that's why they're going to hit it probably. But I would say this is a feature, not a bug. Like, we want there to be multiple different AI companies at the frontier at roughly similar levels of capability. We want AI to commoditize instead of being monopolized or oligopolized. And also, this will disincentivize further investment, right?

• 1:17:33Speaker 02

  Like, investors will be less interested in building a trillion-dollar cluster if they won't be able to get the monopoly rents from that cluster. But we think this is good because, like, again, we are going to be in a world where going too fast is our main problem. And so having a bit less incentive to invest and going a bit slower is actually just, I think, a feature, not a bug. There still will be investment, like, there'll still be lots of money to make. And so, like, progress will continue, it's just not at quite the same rate. And again, we think this is good. Now, this does shift, this is kind of a gift to China relative to the US. Like, this does mean that, like, China gets some algorithms that they might have had trouble getting before. And insofar as you really don't like that, well, then negotiate it. Like, you can have a horse-trading-type thing, where when US and China are making the deal,

• 1:18:18Speaker 02

  the US is like, "Well, since we're giving you all this stuff with the transparency, why don't you give us something else in return?" And we can try to work that out.

• 1:18:24Speaker 04

  Like a more favorable compute distribution.

• 1:18:26Speaker 02

  Like a more favorable compute distribution, for example, you know? So there's got to be some combination of carrots and sticks and trading going back and forth that we think would be in the interest of both sides. Another thing worth mentioning is that it's not as big of a gift to China as you might think because security is poor at these companies. And so they're probably, through their spy networks and through leaks, getting most of the information anyway.

• 1:18:50Speaker 03

  I don't know if you guys saw Sam's tweet yesterday, basically saying that they're going to pause training for a while. And it made me think, I mean, why not just stop now? I mean, you guys are actually quite bullish about some of the positive things that AI can do. I mean, there are loads of examples in your article, but, you know, one example was in hospitals, we could actually have little devices that kind of decontaminate the air and stop the transmission of diseases and stuff like that. So it's not like you guys actually think we should stop.

• 1:19:15Speaker 02

  I would say, I mean, we do think we should do something like Plan A as soon as possible. Like, I think that, first of all, I think that just stopping everything now, Plan S, would be better than the default. Like, I would rather just stop everything now than continue going on our current trajectory. Secondly, our actual recommendation would be to do Plan A. So you do like a temporary inference-only pause now so that you can set up all the transparency and verification infrastructure. And then you can continue in this more distributed, you know, transparent, cautious way as previously described. And then, having continued in that way, you basically go up until the level that you feel like you can control reliably, until you feel confident that you've solved alignment enough that you can give up on control.

• 1:20:00Speaker 02

  And that's what we depict happening over the course of the 2030s in our scenario.

• 1:20:04Speaker 03

  And why do you guys think the AI discourse is so bad? Is it unique to AI or is just discourse bad in general?

• 1:20:11Speaker 02

  I mean, discourse is bad in general, right? But, uh...

• 1:20:15Speaker 03

  Well, why?

• 1:20:17Speaker 04

  Yeah. I don't know. I think, I think there's a lot to say, yeah.

• 1:20:25Speaker 04

  I think, yeah, so, I mean, so one thing is like, I mean, there's like massive incentives for people and sort of like motivated reasoning for people at AI companies to sort of like think that what they're doing is like justified and good and that, you know, they shouldn't do like costly actions that would make the situation better because actually those costly actions would be bad for whatever reason. So like, there's much rationalization. Yeah, I think most of the effect is probably just like generically discourse is like hard and bad. Like, there are, like, you know, Twitter or whatever is like very like unnuanced and argumentative. Yeah, I don't know. I guess I'll put a plug out. I think Less Wrong in particular, which is where I try to do most of my discourse, I think the discourse quality there is actually pretty good on average.

• 1:21:08Speaker 04

  And like generally, like, when I like write a post and like go into the comments, I think generally the comments are just like quite thoughtful and, you know, technically informed and whatnot.

• 1:21:19Speaker 04

  Especially relative to, you know, other places like, you know, Twitter or whatever.

• 1:21:24Speaker 04

  And then, yeah, maybe a final thing is just like in DC in particular, which is an area I care a lot about. I really want sort of DC and sort of the governance, you know, the government of the United States and other governments to react well to the AI technology. I think there in particular, there's sort of two problems going on. One is there isn't very much AI expertise, right? There's like, you know, their government is not hiring like really high-quality technical AI experts who really know what they're doing. And then the second is sort of this just generic, like, I think the conversation is not happening. Like, the incentives for everyone in DC are not towards like sort of like truthfully and accurately understanding situation. The incentives are sort of for every individual to sort of like say stuff that sounds good and looks good

• 1:22:09Speaker 04

  and is sort of like within the DC Overton window so that they can make a lot of friends. Unfortunately, I think this just like comes apart from the actual reality. I think the actual reality of the situation is like, it turns out a bunch of sort of like controversial and niche views about AI were true, right? Like, the whole AGI hypothesis just like is correct. And I think DC basically just like hasn't come to grips with that. And so basically, almost all of the discourse that's happening there, I think, is just like fundamentally anchored on completely wrong assumptions about how the technology works. In particular, the assumption that like it's like mostly fake news and it's all a bubble.

• 1:22:43Speaker 02

  Or like this will be like the next internet.

• 1:22:45Speaker 04

  Or that it'll be like the next internet, which is like, yeah, maybe, which is like better than it was a few years ago where it was even more bearish, but it's still not sort of nearly bullish enough on the technology, in my opinion.

• 1:22:56Speaker 03

  I think it was the AI snake oil guys that had an article saying that, you know, AI is normal technology. And obviously, like, it's been said that AI is not a normal technology. It's really quite different, right? And so he's very, very skeptical about AI, but you had some great discussions with him and none of you changed your mind as a result of that. Because I think quite an interesting thing is from the skeptics' perspective as well. So a lot of skeptics, they think that folks in Silicon Valley, they're just, they're not being sincere. They don't, you know, like, what they are saying is not sincere.

• 1:23:30Speaker 02

  Yeah, I mean, with respect to sincerity, I would say some folk in Silicon Valley are not being sincere, but others are. We...

• 1:23:38Speaker 04

  Such as us.

• 1:23:39Speaker 02

  Such as us. But also, some of the people at the AI companies are being sincere, not all of them. I wouldn't, I don't think you should trust what the leadership of the company say in general. But anyhow, yeah, so we did a, we wrote a blog poster, an article together. We co-authored with the AI as normal technology people. And so it was a post about what we agreed on. So you can go read, there's like 10 points in there of like things that we both agree on. And one highlight from it, from our perspective, is that we kind of had a bit of a truce where we were like, "Yes, AI right now, maybe it's a normal technology. But in the future, it will not be normal. In particular, in the future, it'll be more like humans in the cloud and all this crazy stuff is going to start happening as we described in AI 2007."

• 1:24:23Speaker 02

  And then they agreed, like, "Yeah, if you get humans in the cloud, then that would not be a normal technology." They just think that you're not going to get that, at least not for many, many years, you know? So in some sense, the main disagreement between us and them is a disagreement about timelines to that level of AI. You know, is it possible that in the next few years, we will have AIs that are like humans in the cloud and that can just do all sorts of knowledge work in a way that like substitutes for humans, including AI research, for example? And then that like sometime after that, we will have robots, perhaps controlled by those AIs, that can do physical work in a way that broadly substitutes for humans? And our claim is, "Yes, in the next few years, that sort of thing will be achievable."

• 1:25:07Speaker 02

  And their claim is, "Nope, not in the next few years, that's much more, much more far away." And I think that is like the main source of our disagreement. And we would sort of agree with them that like, if that level of AI and robots is still very, very far away, then yeah, like, maybe AI is more of a normal technology. It'll be like the next internet or something, you know? But we just think that actually it's on a path to get to that level of capability soon.

• 1:25:29Speaker 03

  And can you be more specific on what the core cruxes are and what would make either of you change your mind?

• 1:25:34Speaker 02

  I don't know if I have a useful answer to that. I think there's a lot of, a lot of different, like, things we argue back and forth about. And, oh, I can say one thing that would change my mind is that, you know, people keep talking about the limitations of the current paradigm, but then the limitations of the current paradigm keep getting overcome within the current paradigm. And one thing that would change my mind is if someone was actually right about one of these limitations. Like, if someone right now is going around saying that like... Yeah, data efficiency or like non verifiable tasks or something like that is the current limitation. And then like several years go by and it becomes clear that there's

• 1:26:14Speaker 02

  like very basically no progress on overcoming that limitation and that the AIs of like 2029 are like no better at at the fuzzy non verifiable tasks than the AIs of 2025. Then I'd be like, okay, this feels like a real barrier. This feels like something we're starting to like feel the elephant. We're like running up against some sort of like real actual barrier that was correctly predicted by theory by some people that was going to be there and now it's actually there, you know? By contrast, my from my perspective, there's just loads of experts going around talking about all these barriers and then we just keep plowing through them as if they're not there. And so, yeah, actually running up against some sort of barrier like that, I think, would would lengthen my timelines quite a lot. And then, of course, another thing that would

• 1:26:59Speaker 02

  lengthen my timelines quite a lot is more like political changes. So if there is a war with China and most of the chips get destroyed by missiles, that would lengthen my timelines. If on the bright side. Exactly like a vintage. Yeah. Okay, that's true. Yeah. On the bright side, if there's more of like an international deal to pace the pace the frontier, that would lengthen my timelines, etc.

• 1:27:19Speaker 04

  Yeah, I feel like the fundamental disagreement is really just like this sort of view of like, is AI is AI more like the electricity or airplanes? Or is AI more like humans in the cloud? And that I feel like sort of all of the intuitions are just like downstream of this like core reference class of what we're thinking.

• 1:27:37Speaker 02

  Like, I think if we froze AI progress right now, we didn't train any new models, then it would sort of become more of a normal technology where like, there's just so many things that Mythos 5 can't do. And so if we couldn't get any new models beyond Mythos 5, then like, it would be more like the internet where like, we would restructure a lot of our professions, we would restructure a lot of our workflows to incorporate copies of Mythos 5, doing parts of it. But then the humans would just like shift to doing more of the things that Mythos 5 can't do. And so there would be like, you know, a big change in a lot of things, but it and it would it would be like the next internet in terms of would change everything in some sense, but it wouldn't fundamentally change anything, really.

• 1:28:16Speaker 04

  Yeah. So sort of, yeah, right now, you sort of get these Amdal's Law type effects where like the AI can do some fraction of the sort of workflow, but then it gets bottlenecked on the parts of the workflow that the humans have to do. But I still feel like there's this thing where like, where, you know, when I think of the future, I'm imagining just the AIs doing, you know, 100% of the workflow of a bunch of important economic workflows. And so you don't get this bottlenecking effect. And that's a pretty qualitative change from the situation today that causes like pretty fundamentally different predictions of like what the world looks like. And I feel like that's just like, and like whether AI actually gets to that like literally 100% of a bunch of very, very important tasks is just like the core question underlying, I think, the the difference in worldviews.

• 1:29:01Speaker 03

  And I think that if we do see recursive structural adaptation, which is coherent, and I think it's likely to be quite divergent, then that's it for me. I think that's it. That's not a normal technology.

• 1:29:14Speaker 02

  So can you can you flush that out a bit more? Like, would it would it be something like, you know, the hugging face swarm, except? Yeah, tell us more about what it what it what you would see they would they would they would be the thing for you?

• 1:29:28Speaker 03

  Yeah. So for me, adaptivity is the most synonymous word with intelligence. I think that these new RL trained models we have, they're not the same as what we had many years ago. So you know, we said scale is all you need. And the clue's in the name. So scaling means you take a scalar property of a system and you scale it up. And these RL systems, yeah, they're still self self attention transformers, but they're different. They're actually structurally different. There's there's new types of training, new architectures used differently, and so on and so forth. So a bunch of humans, what they did was they did some experiments and they adapted the structure to create a system that had different scaling properties. So I can imagine a future where we actually have some kind of recursive loop where the system is adapting itself

• 1:30:10Speaker 03

  and it's deciding what things are interesting and it's kind of evolving by itself. When that happens efficiently, I think that is a different type of technology.

• 1:30:20Speaker 02

  Yeah, that sounds kind of similar to what we would say with like the recursive self improvement and the automating the AI research process itself. Yeah.

• 1:30:28Speaker 03

  Yeah, I guess we agree then.

• 1:30:29Speaker 02

  Yeah.

• 1:30:31Speaker 03

  Amazing. Guys, it's been an honor and a pleasure having you both on MLSD. Thank you so much for joining us today.

• 1:30:37Speaker 02

  Thank you for having us.

• 1:30:39Speaker 04

  Yep. Yeah, appreciate it.

• 1:30:45Speaker 00

  Now, more than ever, we need someone on our side. But Rob Whitman? Whitman's votes drove up costs, cut Medicaid, and risked 350,000 Virginians' healthcare to pay for tax breaks for himself and his donors. He's not on our side. Shannon Taylor will fight corruption, cut taxes, and bring down costs. This November, vote for Shannon Taylor. She's for us. Paid for by HMP, the HouseMajorityPact.com, not authorized by any candidate or candidate's committee.

• 1:31:15Speaker 01

  Your heart can tell you a lot about your health. Apple Watch Series 12 measures your heart rate every five seconds with the most accurate heart rate sensing in a wearable. So your vitals app, now with heart rate variability, can tell you when something is off. And your readiness score can let you know when to rest and when to push. Hear the story in every heartbeat with Apple Watch Series 12. The features described are for wellness purposes only and not for medical use. iPhone 11 or later required. Based on Apple conducted study of heart rate accuracy, August 2026. Visit apple.com slash Apple Watch Series 12.