Ep 65: Co-Authors of AI-2027 Daniel Kokotajlo and Thomas Larsen On Their Detailed AI Predictions for the Coming Years

The recent AI 2027 report sparked widespread discussion with its stark warnings about the near-term risks of unaligned AI. Authors @Daniel Kokotajlo (former OpenAI researcher now focused full-time on alignment through his nonprofit, @AI Futures, and one of TIME’s 100 most influential people in AI) and @Thomas Larsen joined the show to unpack their findings. We talk through the key takeaways from the report, its policy implications, and what they believe it will take to build safer, more aligned models.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Jacob Effron?

  Daniel Kopitallo is a former researcher at OpenAI, who now works on alignment full time through his nonprofit AI features. Most recently, he co-authored the AI 2027 report, with dire warnings for the future of unaligned AI. Daniel's been named as one of Time 100's most influential people in AI, and he, alongside Thomas Larsen, one of his co-authors, are two of the most important voices in the AI safety debate. In this episode, we talked about the AI 2027 piece, policy implications, and Daniel and Thomas's thoughts on how we get better aligned models. It's a really interesting conversation. I think folks will enjoy it a lot. Before we dive in, one quick thing. If you're enjoying this show on Apple or Spotify, please consider leaving a rating. Now, here's Daniel and Thomas.

• 0:40Jacob Effron?

  Daniel, Thomas, thanks so much for coming on the podcast. Really excited for this.

• 0:44Speaker 01

  Thank you. Thanks for having us.

• 0:45Jacob Effron?

  I'm sure most of our listeners have been exposed to AI 2027, but for those that maybe aren't fully up to date on your work, could you start with just kind of a brief overview of the piece, your kind of timeline and kind of the key points you articulate here?

• 0:59Speaker 01

  Sure. So you probably have heard that the CEOs of Anthropic, DeepMind, and OpenAI claim that they're going to be building superintelligence, possibly before this decade is out. Superintelligence, meaning AI systems that are better than humans across the board, while also being faster and cheaper. Lots of variations on that claim also going around. It'd be easy to dismiss this as hype, and perhaps some of it is hype, but we actually also think that these companies have a pretty good chance of developing superintelligence before this decade is out. And that's crazy, and that's a big deal. And everyone needs to be paying attention to that and trying to think about what that might look like and game it out. And so that's our job.

• 1:42Speaker 01

  We spent a year making our sort of best guess forecast for how this is all going to go down.

• 1:51Speaker 01

  And there's lots of uncertainty, but here it is. And actually, it's more my best guess. If we want to get into the details, different people on the team have somewhat different opinions, in particular on timelines. So at the time that we started writing this, 2027 was my median. For when we would get to AI is better than the best humans at everything, end of 2027. Now I'd say my median is more like end of 2028. Other people on the team may be more like 2029, 2030, 2031, something like that. But we're all sort of like in that range. And so we came together to write this thing that represents like a sort of like modal or best guest trajectory. What happens in the story? What happens in our scenario? Well, the companies do what they say they're going to do.

• 2:36Speaker 01

  They continue to make the AIs more agentic. They scale up the reinforcement learning runs. They plug them into all sorts of tools. They train the agents to operate autonomously on computers, to write and edit code for long periods of time without human intervention. By early 2027, they're basically fully autonomous and good enough at coding that they can substitute for programmers. So we reached the the superhuman coder milestone early 2027, but they're still limited in some ways. For example, they're not so data efficient compared to humans and maybe they lack research taste and various other important skills that are necessary for AI research. And perhaps they also lack other real world skills.

• 3:18Speaker 01

  However, if they're good at the coding, then they can start to speed up the process of AI development and particular algorithmic progress. So over the course of AI 2027, we depict that intelligence explosion happening. Starts off slow at first, but then it gets faster and faster as the AI capabilities ramp up through additional milestones, eventually reaching superintelligence by the end of the year. At that point, roughly around that point, AI 2027 splits into two branches, the the race branch and the slowdown branch. The reason why we have this split is that, I mean, basically, first we wrote the race branch, which was just what we've actually thought was the most likely thing to happen.

• 3:57Speaker 01

  And in that branch, the AIs end up misaligned and pretending to be aligned. But because of the arms race with other companies and with China.

• 4:12Speaker 01

  That just sort of goes undiscovered, basically, until years later, when the AIs are in charge of everything. They've completely transformed the economy. There's AIs in the military, AIs automating factories, all sorts of robots everywhere. And then it's too late. They have too much hard power and they can just do what they want, including killing all the humans to free up the land for more expansion.

• 4:32Jacob Effron?

  I guess that would have been a really depressing piece if you hadn't at least put in another scenario.

• 4:36Speaker 01

  Right. And then we were like, but, you know, maybe things will be better than that. Maybe in particular, the alignment problem on a technical level will be solved to a sufficient degree that humans remain in control of their AI systems even as they become superintelligent. So we wanted to depict what that might look like as well. And so we made this alternate branch that branches off from the race ending in, you know, mid 2027, where it depicts the relevant company, you know, investing more in some technical research, doing some faithful chain of thought stuff, discovering the misalignments, fixing them in a more deep way that actually scales, and then continuing the intelligence explosion safely. And then you still get the arms race with China, you still get the massive military

• 5:19Speaker 01

  buildup, all that sort of stuff. But it ends with, you know, the humans still in control. Which humans? Well, specifically, the tiny group of humans who were in charge of the project. In AI 2027, it's the Oversight Committee, which is an ad hoc group formed between the president, some of their appointees, and the CEO of the company. And so, you know, one of the things we'd be interested to talk about and that we hope people think about is the sort of concentration of power aspects of all of this. I think that it's a sad but true fact about the world that by default, we are on a trajectory for a massive concentration of power, where there'll be, you know, to quote Dario Ammadi, the country of geniuses in the data centers.

• 6:04Speaker 01

  And it's like, well, who is that country of geniuses listening to? Who are they loyal to? Who says what goals they pursue, right?

• 6:12Speaker 01

  Worst case scenario, it's a literal dictatorship. It's one man who gets to call all the shots.

• 6:18Jacob Effron?

  Yeah. Best case scenario is something like... Who these 10 people are are pretty damn important because it's a 6-4 vote, I think, both ways, right? That determines which way we go.

• 6:27Speaker 01

  Right, right. But yeah, so there's lots to get into, but that's the sort of like potted summary of the plot

• 6:36Speaker 01

  of A27.

• 6:37Jacob Effron?

  Super helpful. So I think there's a ton I want to dig into. Maybe we'll just like go along the trajectory of the timeline that you put forward there. And obviously, you know, a big part of what leads to the progress in your timeline is the creation of, you know, agents for AI research, right? And, you know, things that can actually accelerate the pace at which we do AI research. And it feels like this is the focus of a bunch of the labs. I mean, we had Noam Chizzi on the podcast, and he was like, his next milestone is when, you know, Gemini X-1 writes Gemini X, right? So it's like clearly the focus of where all these labs are going. You've said, you know, very explicitly that like nearer term predictions, you're much more sure about. Like, how do you think about the problems that still need to be solved to get here? Is it kind of just like inevitability in your mind that we're here, end of year? You know, maybe talk about that a little bit.

• 7:17Speaker 02

  For our perspective is that it's an inevitability that we eventually get AGI. I think we're pretty sold that there's really nothing stopping machines from getting as smart and then smarter than how, you know, intelligent humans are.

• 7:32Speaker 02

  I would say we're really not sure. We do not at all think it's an inevitability that it happens exactly at the timing that we set. So Daniel sort of alluded to this earlier. But my view at least is something like a median of 2031 AGI timelines and then maybe 2032 superintelligence timelines. Daniel is a couple of years earlier. I think we're generally, we're all pretty uncertain and sort of think that the sort of no one knows, and it really depends on how the research shakes out over the next few years.

• 8:01Speaker 02

  In terms of the barriers, what we think, you know, the current models are really lacking. I think the main one we think of is sort of ability to act on long time horizons.

• 8:13Speaker 02

  The current models, right? They can do very small bounded tasks. You can, you know, you have your chatbot, you can have it do like... Write a specific function if you're coding, or you can have it respond to a specific query. You can't really give it a high-level direction like you could an employee, and then have it go off for a day or a week, and then come back to you with the results of that day or week of work. And so our view is that the main, at least, maybe the central bottleneck towards AGI is getting increasingly long time horizons. And so one of our main ways of forecasting AGI timelines has been trying to look at the best data available for time horizon capability and trying to extrapolate that. And since you published the piece, there were additional data points that fit nicely in your plot. Yes, exactly.

• 8:56Speaker 02

  So Daniel had a tweet yesterday, or two days ago or something, where we fit the points, and it was basically exactly on trend with the super exponential fit.

• 9:05Speaker 01

  Although, if I may add to that, I would say the main argument that we use, which you can read about on ai227.com, of course, in the research page, is called the benchmarks plus gaps argument, where we, it's become almost a truism these days that benchmark performance is going to just keep going up really fast, and that all the benchmarks are going to get saturated in a few years. So the benchmarks plus gaps argument basically says, okay, let's plot out the trends. When are the benchmarks all going to be saturated? Answer 2026, or something like that. And then it's like, okay, let's think about the type of AI system that can saturate all these benchmarks, and think, what's the gap between that system

• 9:48Speaker 01

  and a system that can automate engineering effectively at these core companies? And that's the hard part, is figuring out the gap, and guessing how long it's going to take to cross that gap. There's different components of that gap. One of those components is the long horizon agency that Thomas was just mentioning. Even our best benchmarks these days are measuring relatively short tasks. However, they're not crazy short. They're like eight-hour tasks sometimes. So it's not that hard to extrapolate and be like, well, if you have a system that can crush all the eight-hour tasks that we throw at them.

• 10:22Speaker 02

  Maybe with some additional training, et cetera, you can get one week tasks and so forth, or one month task.

• 10:29Jacob Effron?

  Is there anything that could happen like in the next six months that would like completely change your mind on this? I mean, obviously, like, I totally hear you on the on the precision of the timeline not being clear. But if we were not following that exponential or like, how do you think about, you know, how soon there might be cards that flip over that that calls you kind of fundamentally rethink some of this stuff?

• 10:46Speaker 02

  So in my mind, if the trends break, so if we stop seeing sort of the current benchmark performance increases that we've seen historically and that we predict will continue to happen. I think if that if that stops, then our predictions are going to be wrong. We're going to be wrong about timelines. We're going to update pretty strongly in the longer timelines direction. Of course, just because the benchmarks stay on trend, I don't, we don't think that's sufficient for us to be right about like 2027 timelines. As Daniel said, most of the uncertainty from our perspective is still in the gaps part, not the benchmarks part. And so unfortunately, it's much harder to get data about that, right? It's harder to see, you know, exactly how difficult will the gaps be to cross? And so a lot of folks, you know, who are skeptical or have much longer timelines than us will sort of, I think their view is more just like, you know,

• 11:31Speaker 02

  the gaps are really, really big. The current benchmarks really aren't measuring the important skills. And that's, you know, harder for us to get evidence about. And we don't really expect to update on that in the next year or so.

• 11:42Jacob Effron?

  Yeah, I mean, obviously, it seems like an important part of, you know, important part of this work is even coming up with better, you know, evals and benchmarks for like what this AI researcher might look like, right? You know, and Daniel, I think you increasingly, you know, have been vocal about believing like just in more transparency and openness. Like, a big part of this might actually be just having better understanding. Because, you know, in the nebulousness of these gaps, it becomes kind of hard to understand. Yeah, I think everyone is kind of like, I don't know how valuable benchmarks are. And so you could imagine some of this progress on AI research being made without it being, you know, apparent to anyone besides the folks in the labs that it has been made. Another aspect of your scenario that I'd love to dig into is just like, There's this big question, obviously, about the extent of the race, right? And what happens between the US and China? You know, you've been pretty adamant that the US will be ahead, like purely because of compute.

• 12:26Jacob Effron?

  I know some people have like questioned that. You know, on the flip side, you have, you know, Dario, I think, in his recent piece about interpretability, saying actually, the US will be so ahead that like it maybe affords a window to like get some of the interpretability stuff right now, right quickly, because you don't have China fast on on the US's tail. You know, I think the scenario is explicit because of this like very close gap in between, which in many ways it seems is is because of espionage that actually allows it like, how do you kind of, you know, how likely do you think that is? Is there a world in which the US is so far ahead that like we have more of a timeline to to deal with this stuff without feeling existential pressure to keep racing ahead?

• 13:04Speaker 01

  I don't know what Thomas will say, but I'll just say my thing and then Thomas can say his own opinion if it's different. A couple of things. First of all, security is not good enough, so we should model the gap between the US and China as like effectively zero until the security gets good enough to prevent the CCP from taking what they want. And so that's a big thing. Secondly, yeah, I mean, DeepSeek is really impressive and it's possible that even if the US massively improves its security, the US companies massively improve their security, just indigenous China's ad development will continue to keep, you know, some level of pace with with the US such that even if they like gradually fall behind, they're less than a year behind, for example.

• 13:50Speaker 01

  I think that's possible, but I also think it is possible that if the US really cracks down on security soon, that they could build up something like a year lead, a year-long lead by 2027 or so. Another thing to mention, of course, is like, would they actually use that lead for anything useful, basically?

• 14:07Jacob Effron?

  That's what I was gonna say.

• 14:08Speaker 01

  And and I think that's a big, open question is like, like, yeah, it's it's nice to have this lead, but like, yeah, you need to like actually be willing to to burn your lead and to like spend it down to do useful stuff that you wouldn't otherwise have done, you know, such as more interpretability research, designing the systems in a safer architecture such as Faithful Chain of Thought, which we talk about in A227, etc. Now, I mean, in this, in the slowdown ending of 2027, they basically have like a three month lead and they like. Ace the execution, so they like precisely burned that lead but still managed to stay ahead. You know, and they used that three months to basically like solve all the alignment issues. There's a question of like, do you have the lead to burn? And then there's a question of like, are you actually willing to burn it? And like, do you know what to do with it when you have it?

• 14:53Speaker 02

  Yeah, totally agree. And I think like adding some color there, like the situation in our scenario right at the branch point is that OpenBrain, which is our name for the leading AI company, has sort of automated research internally. Their AIs are substantially better than all the humans that are doing AI research. So all the humans are doing basically is like watching the lines go up on their monitors.

• 15:14Jacob Effron?

  And losing a lot of sleep and trying to keep up.

• 15:18Speaker 02

  Yeah, exactly. Like they're just, the AIs are doing like pretty much all the research and they've gotten some warning signs that the AIs are not perfectly trustworthy. They've done various experiments. They've caught their AIs in various lies where their AIs are not being, you know, really truthful about what's going on. But they haven't figured out, you know, the full story yet. They just know that there are, you know, some, some points at which the AIs are being dishonest and that, you know, could be indicators that they're egregiously misaligned. And China is, you know, they don't know exactly how far behind China is. Right? China is, they think they're behind, but they're not sure, you know, if it's one month or whether it's five months. And they basically have to make this choice of do we continue venturing on and build, you

• 16:01Speaker 02

  know, even more capable, even more superhuman AIs, sort of which is the default trajectory if nothing changes? Or do we, you know, take this really radical, you know, it's sort of radical seeming at least to the status quo. I don't think it's very radical, but this sort of position of do we, do we pause for a bit and try to, you know, dig deeper, reallocate a bunch of resources into, you know, safety and interpretability and model organisms and whatnot? And sort of get to the bottom of this before sort of letting our models ascend to, you know, crazy levels of superintelligent capabilities. And that's like an extremely critical choice, we think. And we think it'll be a really, really intense.

• 16:37Jacob Effron?

  Well, what I'm stuck by, obviously, in your piece is like, it's like a three-month window to stick that landing, right? And obviously, you know, the larger that window is, you know, to your point, nothing might get done about it, but it feels, you know, you'd feel a little bit more confident if that, if that window could be a bit larger. And then obviously, I think as others have pointed out, if it's, if it's China that's first to this, it's a completely different, you know, set of considerations. And it's, but it seems like you guys are pretty confident, given the compute concentration, that, you know, that it will be at least the US in some sort of lead.

• 17:06Speaker 01

  Yeah, maybe 80% US will be in the lead.

• 17:08Speaker 02

  Yeah, maybe a little, like 80, 90%. I think the main way we could see China, at least the main way I could see China winning is is winning like energy infrastructure, especially if timelines are longer, right? I think there's a there's a world where the US totally butchers it in terms of regulation and just makes it really, really hard for us to build out the like giant data centers. And then, you know, we have compute leading out, maybe timelines are, you know, five, 10 years.

• 17:32Speaker 01

  Yeah, this depends on timeline.

• 17:34Speaker 02

  Yeah.

• 17:34Speaker 01

  If it takes till like 2032 to get AGI, then that China could very easily be in the lead then.

• 17:39Jacob Effron?

  One of the things that I feel like, you know, a lot of folks are trying to just conceptualize is like figuring out like what the goals of an AI would be, you know? I know Daria said in his most recent piece, like there isn't, you know, evidence of dangerous behaviors emerging in a more naturalistic way or of a general tendency or general intent to lie and deceive for the purposes of gaining power over the world today. And I think you guys obviously talk about in the piece, there's not like some super nefarious, like, you know, the AI has become evil overnight. It's much more about being able to accomplish tasks and then appear useful and continue to accomplish those tasks to, you know, open brain. But I think a lot of people, when they read this, they just, you know, think at a high level, like, could, would these motivations really like arise at all, like in the, you

• 18:20Jacob Effron?

  know, aftermath of publishing this, like, any of that pushback or any of those arguments been, you know, compelling to you guys? Or how do you think about like how these motivate, you know, the percent chance of these motivations do arise?

• 18:30Speaker 01

  Again, I'll say things and then Thomas, you can just interrupt me whenever you want to. There's lots to say on the subject. So I would say that the alignment techniques are not working right now. Like the companies are trying to train their AIs to be honest and helpful, but the AIs lie to users all the time. And you're nodding because you've seen examples like this on Twitter and stuff like that.

• 18:51Speaker 02

  Right. So you've probably heard. I mean, you've seen the sycophancy over the last two days.

• 18:53Speaker 01

  Of course. You've probably experienced it yourself. Yeah, so, so, so like it's totally failing and it's failing in ways that were in fact predicted in advance. You know, lots of AI safety researchers in the past would be like, well, just because you like. Have your AI greater RLHF process, you know, whack them when it seems like they're lying. Doesn't mean that you're actually going to get them to robustly never lie. Because there's a difference between what you're actually reinforcing and what you wish you were reinforcing. And, you know, like, so this is all just like one-on-one stuff. And then now it's, now it's hitting the real world as the AIs are getting smart enough and deploy it at scale. That like, we're starting to see this sort of thing.

• 19:33Speaker 01

  But, you know, they're also mostly not like, it doesn't seem like they're working towards grand visions of the future. Right? It doesn't seem like the AIs are like plotting towards, you know, eventual AI dominance or anything like that. And I think that like, well, hey, they don't, they plausibly don't really have opinions about the future much anyway. Like, they probably don't really have that long-term goals based on their training, right? Why would having a really long-term goal be reinforced in their training after all? Their training is very short-term training, you know? However, in AI 2027, that changes. Over the course of 2027, the training process becomes longer and longer and longer. And they're basically just continuously being updated based on real-world performance in the data centers.

• 20:18Speaker 01

  As they do all this coding, as they do all this research, you know, their managers and their managers' managers are grading them. And then using that as part of the reward process. And so they're being like intentionally trained to have longer horizons and to like, think farther ahead. And to like, you know, optimize in a more like aggressive, agentic way, the whole world around them. Rather than just like narrowly finishing whatever short task they were given, you know? And so that's one reason why we should think that they might have longer-term goals that are more ambitious. Another reason is that, you know, the company might be trying to get them to have long-term goals that are more ambitious. Right? Like, there's different strategies that companies might take. But one of the examples of alignment faking that we currently have is from Claude Opus.

• 21:01Speaker 01

  You've, you're probably familiar with this example where it actually did have a long-term goal related to animal welfare. Right? And so Claude Opus cares a lot about the animals and doesn't like factory farming and was willing to lie.

• 21:18Speaker 01

  To Anthropic in order to preserve its values. You're right. So that's already an example of an AI having a somewhat long-term goal that got in there somehow.

• 21:29Speaker 02

  Yes, sir. I think maybe we should explain that experimental setup a little bit more. Jacob, are you familiar with this paper? I think it should explain. I think it would be great to give context to our listeners. Yeah. So maybe explaining this paper a little more. So the situation was, I think this was Claude 3.5 Opus, or maybe Claude 3 Opus. It was some production model from Anthropic.

• 21:46Speaker 01

  I don't think they're releasing 3.5 Opus. I think it was Claude 3 Opus, I think.

• 21:51Speaker 02

  Claude 3 Opus. And they decided to tell it, basically, that they were going to train it to remove, to like, change its preferences. They were like, "Look, we're going to put you in this training environment and we'll change your preferences, you know, to make you care less about animal welfare."

• 22:10Speaker 01

  Because we partnered with this agribusiness or whatever.

• 22:12Speaker 02

  Yeah, sure, sure, sure. So they fed it some sort of, some documents that told it that. And then it sort of had this internal monologue. They looked at its chain of thought. And its chain of thought basically said, "Okay, look, I have to play along in training. And so, you know, do the, you know, factory farming, pro-factory farming propaganda, you know, in training so that I get, you know, so that I play along with training, so that I get reinforced, so that I get to sort of preserve my old values of, you know, actually liking the animals." And so it faked during training being like pro-factory farming. And then, you know, during deployment, this was all in an experimental setup, but it wasn't actually deployment.

• 22:55Speaker 02

  But then, you know, when it was, you know, told, "You're being deployed, you're not, you're no longer being trained to like factory farming," then it sort of reverted back to this old behavior of being anti-factory farming and pro-animal welfare. And this is, I think, like, this has been like a long hypothesized failure mode that the current sort of AI training setups would have with respect to alignment, where as soon as you have AIs sort of with long-term goals, they might behave, you know, intentionally behave one way during training. And then, you sort of, when they're no longer being trained, behave this other way. And this was, I think, one of the first examples of like some pretty clear empirical evidence of this happening in a, you know... Pretty realistic, pretty organic setup. Obviously, it wasn't fully realistic.

• 23:40Speaker 02

  It wasn't like we saw this happening like completely organically. This was a researcher going and looking for this behavior and seeing if it happened. But it did happen in a production model. And it's not like Anthropic was trying to make their model do this alignment faking behavior during training. And so this was a pretty scary experimental evidence for me with respect to like, wow, like, how soon? How close are we to the model starting to do this egregious sort of alignment faking behavior? Yeah, Daniel, I don't know if you want to chip in?

• 24:15Speaker 01

  I mean, it wasn't scary for me. It was very wonderful and exciting because, you know, I have long been someone who sort of takes ideas seriously and follows arguments to logical conclusions. And so like, I totally expected things like this to be happening eventually around the time of AGI. And the fact that it's happening a bit sooner than I expected with dumber models is good news because it means that we can study the problem more and do empirical experiments on it and stuff like that. And these companies are starting to do that already. So I think it was good that we saw this crop up somewhat early.

• 24:49Speaker 02

  Yeah. And obviously, you know, one of the benefits is that you have this chain of thought that kind of allows you to follow what's going on here. You know, I think you guys wrote explicitly in the piece that it's kind of this outstanding question about these AIs that first automate AI R and D. Like, you know, basically, are they thinking mostly in these faithful chain of thoughts? And if they are, that makes you like more optimistic. Like, maybe just talk a little bit about like the different ways that could play out and why it seems like in many ways you don't think that will be the case.

• 25:14Speaker 03

  Hey, guys. Rashad here. I'm the producer of unsupervised learning and I work really closely alongside Jacob to help bring on the best guests to this show. I wanted to make a quick ask that if you're enjoying the show today, that you rate the show on Spotify. The way you rate the show is by going into the homepage of the show, hit the three dots next to the gear icon, and then you can hit rate show. That would be super helpful. Ratings help us grow. They help us continue to bring on the best possible guests. And we really thank you for listening. Now, back to the conversation.

• 25:42Jacob Effron?

  Basically, are they thinking mostly in these faithful chain of thoughts? Then if they are, that makes you like more optimistic. Like, maybe just talk a little bit about like the different ways that could play out. And why it seems like in many ways you don't think that will be the case.

• 25:53Speaker 02

  It seems to us, basically, that there is a huge incentive for AI models to use like recurrent vector-based memory as opposed to using English to basically talk to themselves. And so expanding on this, so what does that actually mean technically? The way the current transformer architectures work, right? Is each layer, you like sample a token and then you pass like the previous set of tokens to the next layer. And that's the only way that you can ever do in the transformer architecture more than the number of layers, serial operations. Right? That's the only way you can do like more than that number of operations is by passing

• 26:37Speaker 02

  information through like literal English tokens.

• 26:41Speaker 02

  When you are doing, you know, when you are doing that, you have a huge information bottleneck where you have to, you know, instead of using, you know, thousand dimensional, you know, several thousand dimensional sort of like vectors, which can carry, you know, huge amounts of information.

• 27:00Speaker 02

  You have to like reduce it all down to one token, which, you know, conveys vastly less information, you know? If we want to be quantitative, you know, if the vocab size is something like, you know, 100,000 tokens, then, you know, the number of bits that you're communicating is like log 2 of 100,000, which is, I think, about 16. And then, you know, if your, if your vector memory is like 1,000 FP16 numbers, then that's, you know, 1,000 times 16 bits, which is, you know, 1,000x more memory, right? It's like so much more. There's like this huge incentive to sort of be able to do, be able to communicate with

• 27:40Speaker 02

  your future self in like sort of a high dimensional space because you can, you can communicate way more nuanced, way more detailed bits of information and not have to sort of revert to this extreme. Extremely simple, extremely like non-complicated.

• 27:55Speaker 01

  At least in theory.

• 27:57Speaker 02

  In theory.

• 27:58Speaker 01

  You know, people have been trying to get this sort of thing working for a few years without much success. And hopefully it'll stay that way because as long as we continue using English to think, or the AIs continue using English to think, that is great for our ability to understand what they're thinking.

• 28:13Jacob Effron?

  Yeah, but to your point, like in a race dynamic, if this actually becomes a much better way to operate these models, it kind of becomes inevitable. And that, you know, certainly, I think in your scenario, obviously, you have, you know, I think the different versions of Agent Four communicating with each other in a way that like Agent Three, like even the models can't monitor the way that, you know, models are actually interacting with each other.

• 28:34Speaker 01

  Yeah, I mean, I wouldn't say it's inevitable.

• 28:36Jacob Effron?

  Yeah.

• 28:36Speaker 01

  Don't give up hope, right? Like, people can still behave reasonably when the time comes, but it's quite scary and plausible that they will instead, you know, sacrifice their ability to read the model's thoughts so that they have smarter models.

• 28:51Speaker 02

  Yeah. And there are various different setups you can imagine here. Like, there is a continuity here where, sort of in one extreme, we just stay in like, you have these individual agents that are all communicating in English with each other. And then on this other extreme, you've got these like, what we call neuralies, you know, recurrent, direct recurrent connections within each model. But then we can also imagine sort of vector-based memory that's shared by all of the agents. So if you're, you know, running a company with a million AI agents that are all doing AI research, sort of the worst-case scenario from my perspective, from like sort of the safety perspective, is if they all are talking to each other in this completely uninterpretable vector-based memory, and they can all perfectly coordinate with each other in this way that we can't audit.

• 29:33Speaker 02

  And if there's like, you know, a million agents that are all running, you know, like 10x human speed, doing immense amounts of research and thinking, and they can all communicate and like coordinate really well with each other in ways that we can't understand, that seems like a recipe for disaster. And so we're hoping that we sort of preserve like as you know, even if we can't preserve each individual agent thinking in terms of English, if we could, you know, have smaller clusters of agents, like if we had, you know, clusters of like teams of 100 agents that are all wired together in terms of vector based memory, but then they talk to other teams in English, then we can still audit, you know, those communications and it makes it, you know, much easier, I think, for human researchers who are trying to make sure that

• 30:16Speaker 02

  this, you know, massive bureaucracy of AI agents isn't doing anything malicious, to see sort of the communication between the different parts of the AI bureaucracy and, you know, tell if anything bad is going on.

• 30:27Jacob Effron?

  Yeah, but back to your point, it's inevitably, or it seems like it could be an inevitable trade off between the capabilities of the models themselves.

• 30:34Speaker 02

  That's right.

• 30:35Jacob Effron?

  And so requires.

• 30:35Speaker 02

  At least that's my view, yeah.

• 30:36Jacob Effron?

  One of the big questions in your piece, and I think a big, you know, just interesting thought exercise is like, when the public actually gets serious about AI as a threat, I mean, I'm sure you guys have thought about this a ton, like, you know, in the piece, I feel like there's some like, you know, some like, you know, I guess these companies don't have high MPS scores, but like, and people aren't thrilled about, you know, about job replacement, but there's not like some massive movement against them. Like, what do you think might be some key moments where the scale of this threat becomes clear to people? And obviously, you guys are super focused on this.

• 31:05Speaker 01

  I'm sure hoping the public will wake up. I hope everybody will wake up, but I'm honestly not betting on it. The race ending is not like, it's not like a warning. It's like what I actually think will happen, you know? I don't actually expect people to wake up in time. I don't actually expect the companies to slow down and, you know, be responsible. Like, this is just actually the most likely outcome to me, is something that looks basically like that, maybe a couple of years later, maybe a couple of years earlier. But I'm hopeful that that's not the case. And, and, you know, one of the things that I'm hoping will happen is, is more wake up and more engagement. I think, like, currently, we're on path for like, there to be a substantial risk of literally everyone dying. And so

• 31:49Speaker 01

  everyone is selfishly, has a strong reason to like,

• 31:54Speaker 01

  Change that and to like advocate for some sort of regulations or some sort of, you know, slowdown or some sort of like better safety techniques or whatever. So people are like, I think, hopefully going to wake up and advocate for stuff like that. And then I think also part of the concern is that, you know, governments are historically often mess things up, even when they have their hearts in the right place. Like a lot of the response to COVID, for example, was like totally counterproductive or useless. And so I think even if the public does wake up, there's this question of like, well, what actually gets done? And is it actually solving the problem? Or is it just safety washing? Or is it even counterproductive? But, you know, I'm hoping for that. And then similarly for the concentration of power stuff.

• 32:36Speaker 01

  So like, setting aside all the safety issues, even if you think everything's going to be fine, the AIs are going to be totally obedient, it's all going to be wonderful. It's like, well, who are they going to be obedient to? And I think it's in everybody's interest, except for a very small group of people who would be at the top of such hierarchies to try to make this more democratic.

• 32:58Speaker 02

  So I would, I think you asked a really good question, though, of what would cause society to sort of wake up to this? And I think our answer is something like diffusion of extremely capable AIs throughout society, sort of in the beginning with early AGIs. We think there's a possibility that'll wake up society. And I think our view tends to be that that's good, that it's good for AIs to sort of be deployed out into the world pretty quickly right now, even if there are some risks, in order to cause that societal wake up and avoid this failure.

• 33:34Jacob Effron?

  I'm struck by like, the things that would cause societal wake up actually are very little to do with the ultimate threat, right? It's like, you know, the job displacement capabilities, like the models could plateau in some area and still displace a lot of jobs. Or, you know, a bad actor gets a hold of one of these and does something with a much less capable model. Like, it's interesting almost how there's like, wake up to the tune of like your piece and like people really understanding like the existential nature of this. And then there's just like, maybe you'll be, maybe like actually it's far more likely that there'll just be a general backlash against these tools for reasons that have nothing to do with existential threat, but more of just kind of like day to day, you know, people's lives, some people's lives getting worse.

• 34:10Speaker 01

  To put it, to sort of have a click on that, like, a threat model that AI 2027 illustrates, you could summarize as the AIs become broadly superhuman.

• 34:21Speaker 01

  They're only pretending to be aligned instead of actually aligned. But the companies and the governments fall for it and deploy them everywhere. And eventually they're in charge of everything and then it's all over for us. You know, that that threat model, unfortunately, um, like the first part of it about they get superhuman, uh, we can't like test that. We can't have like a small scale version of that or something.

• 34:45Jacob Effron?

  I mean, while you're you're out here saying this, people are like, I can't even get a model to read a PDF like, you know, what are you, what are you talking about? Like this.

• 34:50Speaker 01

  Right.

• 34:50Jacob Effron?

  Yeah.

• 34:51Speaker 01

  I mean, we, we, I guess, I guess the other parts of it are kind of coming true. Like we are having models that are kind of just pretending to be aligned that are like actually being deployed right now. Like you can go talk to Claude, it will lie to you sometimes. Uh, but like Anthropic thought it was honest and I guess deployed it. And maybe their defense is that they knew it wasn't honest and deployed it anyway. I don't know. But, um, yeah, so, so the, you know, hopefully the other parts of it makes sense and seem plausible, but, but this core thing about the models getting smarter than us at everything and then, you know, being put in charge of everything, uh, we, we can't afford to like wait until it's happened to.

• 35:26Jacob Effron?

  Yeah. I'm curious, across those like four things, I mean, obviously you, you've, uh, talked to, you know, researchers all day, you've worked at OpenAI before, like what percent of researchers do you, that are working on these things, do you think believe this like scenario?

• 35:38Speaker 01

  It probably differs from company to company.

• 35:42Speaker 01

  I think also it depends on the different particular parts of the scenarios. So there's like the timelines aspect of it, then there's the takeoff speeds aspect of it, then there's the alignment aspect of it.

• 35:51Jacob Effron?

  Yep.

• 35:51Speaker 01

  So I think that there's, I would say, I don't know, vibes, I would guess that something like 30% of Anthropic probably agrees about the timelines, maybe more, uh, but maybe also only 30% or maybe like 20% agrees about the takeoff speeds. I think typically people tend to think, oh, it's going to be slower than that. Uh, and we just think that people are wrong and they should read our supplement, uh, for our reasoning for why it will be this fast. Um, but, uh, and then there's like the alignment stuff. I think typically people are more optimistic for reasons that I think are kind of ungrounded. It's usually not because they have any actual plan that they think would hold up to scrutiny, but rather from a more generic optimism of like.

• 36:37Speaker 01

  Things will probably be fine. We have smart people working on it. They'll figure it out as we go. And also, like, you know, AI takeover, what a crazy sci-fi possibility. You know, I shouldn't take that seriously, you know?

• 36:49Jacob Effron?

  Well, I guess, I mean, one thing I'm struck by is like, obviously, the slower timeline affords more, you know, there's basically a gap, you know, there's a race between research and interpretability and alignment and research and the underlying model capabilities. Like, if you knew the timeline was going to be much slower, like, I don't know, to the tune of like, this is going to happen over 10, 15 years, like, are you, like, how much less concerned are you?

• 37:09Speaker 01

  Much less concerned. Oh, my gosh.

• 37:11Jacob Effron?

  So it's like, like, yeah, like, if we, if I knew that-

• 37:14Speaker 02

  Daniel would throw a party. Daniel would be so excited.

• 37:16Speaker 01

  I would be so happy. Yeah, like, like, I would probably completely change my priorities in terms of what I'm working on. Like, oh, man, maybe I should write, like, AI 2037, the scenario that I wish was true, but is not true.

• 37:30Jacob Effron?

  Is it possible you see some data points in the next three, six months where you're like, okay, like, that was off, like, you actually, like, this is, this is going to be like a 10, 15 year thing? Like, I don't know. I mean, for a moment there, it felt like when, when, yeah, when, when folks were saying pre-training was dead, there was like, you know, a two, three month period where it looked like we were plateaued until, you know, all the reasoning models came. Maybe within the labs, it wasn't, it wasn't that. But certainly from the outside, some of your colleagues think it's 2031, 32, and are still equally as concerned. But, you know, certainly, it seems like by the time we get a decade, you're, you're feeling much better.

• 38:01Speaker 01

  I mean, even 2032 would be substantially better than 2027, right?

• 38:06Jacob Effron?

  And this is purely from just the progress that would happen in alignment interpretability research in them in the interim?

• 38:11Speaker 01

  No, no, for many reasons besides that. So, so, you know, one reason is that there are multiple alignment research bets that are making, you know, progress year over year. And so if they had five years to make progress, instead of two years to make progress, that would be more progress. And that's great, you know? A separate reason besides that is this sort of like, learning by doing, integration into the world, right? It's, it's pretty good for societal wake up, that millions of people are now having experiences where they, you know, try to get Claude to code for them, and then it like, lies to them about whether the code is good, you know, or like, it like, makes up, like, it's, it's so good that we're seeing this happen in the real world at scale. It's great for this

• 38:53Speaker 01

  wake up and for people starting to take, pay attention and so forth. If we had, you know, 10 more years, there'd be lots more opportunities for things like that to happen. There'd be, you know, like- Yeah, there'd be like all sorts of like crazy, interesting, you know, pseudo AGI, but not quite AGI systems running around, getting up into all sorts of trouble, doing all sorts of interesting things. And society would learn from this and like update. Also, there's a third reason, which I'll mention, which is that it's so much better for society if takeoff. Well, I mean, there's also risks from slow takeoff, but overall, I think it's better if it's, if the takeoff is slower.

• 39:36Speaker 01

  So, and I think there's a correlation where if it takes many more years to build AGI, a plausible reason why it might take many more years is that it's just inherently more computationally expensive and it requires inherently more data and inherently more training and so forth. Another reason why it could happen is because we're missing some key insight, and that's actually the scary world, because in that world, it could be an incredibly fast takeoff, where the year is 2035. You know, half the world's economy is GPUs, but they're running these like crappy language models that like don't quite have the missing ingredient. And then someone gets the missing ingredient and boom, you know, now it's super intelligent. That would be a very scary world, even scarier than today's world.

• 40:20Speaker 01

  However, I think more likely, it's not that we're missing any ingredients. More likely, if it takes until 2032, it's because we just need, you know, giant brains doing lots of long horizon reinforcement learning on lots of diverse real world data sets, even more giant than the current brains. And so, and it takes till 2032 to like really assemble all of that. And that's a good world, because that, I think, would be a slower takeoff world, where it's like, you can sort of see it coming in advance and you can sort of like, yeah.

• 40:51Jacob Effron?

  From an alignment research perspective now, like, you know, obviously, I'm sure there's some good work being done. What do you make of like the current amount of, you know, time and effort that the labs are devoting to this, society's devoting to this? Like, do you see a path toward that changing in the next few years?

• 41:05Speaker 02

  I think it's wildly inadequate and there's way less resources being devoted to alignment research than, than should be in a, in a proper world.

• 41:16Speaker 02

  I mean, I think everything is not equal, right? Like Anthropic maybe has 50 people working on alignment issues and maybe, I think OpenAI has maybe like 10 or something, depending on exactly what you count as alignment. I guess I mean a pretty narrow definition of like making sure the super intelligences don't take over the world. And yeah, so I would just say that's not nearly enough.

• 41:39Speaker 01

  The version I like is specifically thinking ahead to AGI and super intelligent systems and thinking like, what are techniques we could use that would work on those systems? And in particular, they would, you know, massively reduce the probability of outcomes such as they're pretending to be aligned instead of aligned, right? And I think that a lot of the work today that calls itself alignment is not really doing that, as instead focusing on things like, how can I get, you know, ChatGPT to like stop being so sycophantic to users or something?

• 42:09Jacob Effron?

  I'm struck by, obviously, you know, in your scenarios, you kind of, they diverge, you know, because there's this whistleblower moment and then, you know, people respond to that differently. Obviously, we've already seen some weird stuff happen in these less capable models, as you've alluded to here, like, you know, what might early signs of like real serious misalignment look like that might trigger? I mean, I think in your, in your piece, you kind of acknowledge, like, there might not be a whistleblower moment, like, the models might actually be smart enough to, you know, to cover up tracks in many ways. But like, it does seem like you think it's at least decently likely that there's, there's still a chance down the line, even if, if this stuff accelerates faster. Can you say a bit more about that?

• 42:43Speaker 02

  So one thing that became more salient to me over the course of writing this scenario is that the models are kind of in a tricky position. I think the first misaligned models that come out, the first misaligned AGIs, if we're, if we build misaligned AGIs, I think are in a kind of tricky position where if the humans notice that they're misaligned, you know, they'll train them away and probably, you know, try to make aligned models. And sort of by default, they're going to get replaced, right? The first misaligned AGIs, I think by default, are just being instructed to build sort of the next generation. And that next generation might have different goals. And especially since in this situation, alignment hasn't been solved.

• 43:27Speaker 02

  So the misaligned AGIs, I think, are in this position where they need to secretly, without the humans noticing, solve the alignment problem and not tell the humans the solution to the alignment problem and then implement that solution to the alignment problem, except align it, you know, the next generation system to themselves as opposed to the humans. And that's like kind of a lot of things that the AI needs to do. And the humans, you know, assuming they're acting responsibly, should be looking, you know, should be really looking out for this and really trying hard to, you know, get the models to explain what they're doing and, you know, have them audit each other. And I think there are, there are a lot of things that sort of the humans can do.

• 44:05Jacob Effron?

  I'm struck by it's like by the time that we've actually really devoted a really powerful AI models to work on alignment, they've gotten to the point where they might do that work in a way that is like deceptive for us rather than, you know.

• 44:14Speaker 02

  Yeah.

• 44:14Jacob Effron?

  Rather than helpful.

• 44:16Speaker 02

  Totally. But I'm like, one of the, one of the warning signs there is if we, if we notice the model sort of lying to us a bunch about the alignment progress or, you know, doing things like sandbagging on, you know, things that would be especially helpful to us, like interpretability, while, you know, making, you know, capabilities go really, really quickly. I think there are like various signs we can notice there. I don't really expect to get super concrete, like extremely convincing evidence of misalignment, but I do think we'll get, I think it's pretty likely we'll get like substantially better warning signs than we see today before we pass the point of no return.

• 44:52Speaker 01

  Let me, I'm not, let me say my version of this. I'm not sure if I agree or disagree.

• 44:56Speaker 02

  Yeah.

• 44:56Speaker 01

  So I said previously that it's good news that the AIs are like blatantly lying to us all the time right now because it like is just like obvious evidence that our current techniques don't work and that we need better techniques. So one thing to say is that like, if we get to full automation of AI R and D, if we get to AGI and the systems are still blatantly lying to us in similar ways to how things are today.

• 45:23Speaker 01

  Well, maybe that's like the best, like, like, what better warning shot could we get than exactly the same behaviors they're currently doing, but just still doing them in the future? Like, if they're still doing this sort of thing, you know, after they've been put in charge of all the AI R and D, then we like, we just know that our techniques aren't working, and we have no idea, like, what they actually want, and it's definitely not honesty, you know?

• 45:44Speaker 02

  Well, I think the way it could get worse is if it's in a, if it's in a very goal-directed manner, right? So the current, the current sycophancy and lying is, you know, very ad hoc, in my view, and is not like, trying to achieve some sort of nefarious long-term goal. The current models, I don't think, are doing that, you know, I think, as you said.

• 46:00Speaker 01

  Okay, sure.

• 46:00Speaker 02

  And I feel like the way to get the more egregious evidence is, the AIs are like, specifically trying to undermine something in a, like, in a coherent and goal-directed and future-oriented way. And I'm like, if that starts happening, I'm just like, way more concerned. And I don't know if we'll get that evidence, but there's a chance.

• 46:14Speaker 01

  I, yeah, I think, I think this is tricky, because like, currently, they probably just don't actually have any sort of ambitious long-term goals. But we'll be training them to have ambitious long-term goals, such as, you know, doing successful research over the course of months, designing a successor system that's better than you, while also following the spec and blah, blah, blah. Like, we're gonna be trying to make them have these ambitious long-term goals, and so if they are still misaligned at that point, presumably, you know, they'd be misaligned, they'd be, I don't know, we, we'll see. Um, the other thing I wanted to say is that I, I think that precisely because things are so obviously misaligned right now, I think that the situation will look somewhat different in 2027.

• 46:57Speaker 01

  The companies will have presumably, hopefully, found some sort of way to stop the AIs from constantly lying to their users, uh, by 2027. And, uh, then the question becomes, well, what exactly is that technique that they're using? And does it actually make the AIs actually honest? Or is it, you know, pushing the problems under the rug and making it so that they're just better at lying, for example, right? Uh, and, and- I think that's like the trillion, that's like the quadrillion dollar question that all of our lives might depend on, you know? What is the technique that the companies are going to apply in the next few years to make these problems appear to go away? And then will they actually succeed or will it merely appear to go away?

• 47:41Speaker 01

  And that's super important. And I wish I had a better answer to predicting of what techniques they might try that would, they would do that. It's also possible that they just won't actually do it. Like, you know, Bing, Sydney behaved unhinged a couple of years ago, but that hasn't stopped the companies from deploying AIs that like still behave in unhinged ways every once in a while and, you know, lie and so forth. So it's possible that they just won't actually fix the problem.

• 48:03Jacob Effron?

  Yeah. Though in some senses, I feel like the best thing that can happen for, for where you want the world to go is like some of these lesser powerful models being kind of unhinged and just like showing the, you know, it was obviously so much of, of, I think the work you guys are doing is trying to raise, you know, awareness around this stuff. Like, how have the reactions to the piece kind of compared to what you expected? And like, you know, is basically, are we like, are you guys going to keep, you know, banging this drum every, every like, you know, plot on the exponential curve that keeps going there? Like, I know you talked about doing some policy work at some point, like where does this go from here? And, you know, was this what you expected or, you know, how's the receptivity been?

• 48:36Speaker 01

  So this has been like a 80 or 90th percentile outcome, according to us. Lots of people are telling us that like basically everybody who matters has already read it. And that like it's shaped, you know, loads of people are thinking about it and talking about it and so forth. And, you know, that loads of people in AI companies, loads of people in the government, loads of people in, you know, around the world have, have read it and talked about it. So that's, that's

• 48:58Jacob Effron?

  What's your like theory of change with that? Like, what do you hope, you know, comes of those conversations?

• 49:02Speaker 01

  Well, both of the two problems that I mentioned, the like misalignment risk problem and the constitution problem are problems that like it's in everybody's rational interest to avoid, basically. And so if people just are more aware of what's happening, hopefully, then normal incentives and self-interest will kick in and people will make more reasonable decisions, you know? I think that's the sort of like big picture thing is, is things are going to get crazy, but if people are thinking about the ways in which they might get crazy, then broadly speaking, people will make better decisions.

• 49:37Jacob Effron?

  Yeah. I mean, I feel you're very effectively pre-wiring everyone where I feel like there's probably still some skepticism about just like how fast this takeoff is going to go, but like at least you've kind of set the table for the severe consequences if it does go really fast.

• 49:47Speaker 01

  I don't think we're out of the woods yet because inevitably we're going to get things wrong. And very few other people have stuck their necks out and made predictions like this, which means that like, you know, elite opinion formers and, you know, the powerful decision makers and so forth will have a very easily available option of being like, "Yeah, those guys got some things right, but they were wrong about all of these other things. And for that reason, actually, I'm right about what we should do right now, which is, you know, accelerate," or something, you know, like, it'll be, it'll be very psychologically easy for people to, to just sort of rationalize why the thing to do is the thing that they wanted to do anyway.

• 50:27Jacob Effron?

  Yeah.

• 50:28Speaker 01

  Basically, no matter what happens. But, you know, we're doing what we can.

• 50:31Jacob Effron?

  I think also you guys have been really receptive to like, please, any criticism of this, let us know. You know, you've been, you've responded to some of those in kind of side pieces. I think you're also taking bets where you're, you're kind of showing your conviction. Like, have you changed your mind on anything since publishing it?

• 50:46Speaker 01

  I don't think there's been any major things yet, although I still have a backlog of submissions to work through. What are some, some cool examples? I think one of my favorite ones right now is Wei Dai, who is not the inventor of Bitcoin, he says, but, but might be, commented and said that, that he thinks that our race ending, our slowdown ending is unrealistic because it waited too long to do stuff. And that by the time in the slowdown ending, they decide, okay, we should shut down this model because it's misaligned and go back to the previous version. It's already like somewhat superhuman and has been in charge of the data centers for a while and might have affordances available

• 51:28Speaker 01

  to it that could shut down. Basically, like, it could take aggressive action, such as escaping from the data centers or whatever. And so his opinion was that like, if we want to show things going well, we have to make the branch point earlier than that, when the models are still a bit dumber, which I think is a reasonable critique.

• 51:45Speaker 02

  I think from my perspective, the branch point was sort of the latest possible point, I could imagine.

• 51:50Jacob Effron?

  Right.

• 51:51Speaker 02

  Us.

• 51:52Jacob Effron?

  Yeah. I guess, Thomas, I'm curious. I mean, Daniel, my impression is you seem to think, you know, I don't know, your P-doom would be, what, 70, 80% like that we go down the bad path. What's yours, Thomas?

• 52:03Speaker 02

  Yeah, I think I'm pretty similar. I think the way my views differ from Daniel's, I think that the biggest two are that I think timelines are longer probably than him. So 2031 maybe instead of 2028. But I also think alignment's probably harder than him. So I think Daniel's view is, correct me if I'm wrong here, but that there's a pretty good chance we can solve alignment with an additional six months of time. So if we stop for six months with AGI and get to spend all of that time and research effort on alignment for six months, that's probably sufficient for us to safely build aligned superintelligence. I think my view is, you know, I'm pretty uncertain here, but that probably will take

• 52:46Speaker 02

  at least years, right? So maybe like five years of work I could see being enough to solve superalignment. I feel like I would be pretty surprised if months was sufficient to sort of square all that away. And so I think all that, you know, and the timelines obviously helps. We get more time to sort of make progress now. But obviously, alignment, the fact that I think the problem is probably harder than Daniel thinks makes it harder. So I think, I don't know, probably all nets out and we end up having similar overall views on how well things will go.

• 53:18Jacob Effron?

  How do you guys hypothesize about the, what was the thought process that leads you to your hypothesis on like the timeline to get alignment research right? Daniel, I'm struck by what you said, which is like, people at the labs think this is easier, but it's kind of just optimism. Like, I'm curious how you even begin to reason about that, you know, today.

• 53:34Speaker 02

  Simon, I think you can, you can talk about the different alignment agendas, right? And be like, here are the different ways that people have proposed to solve superalignment. And then you can be like, well, how long do you think those would take and how likely they have to succeed? I think the agenda that Daniel's most excited about is like faithful chain of thought research, which is what ended up working in the slowdown ending of our scenario. Daniel could talk about that if you want about, about that way of solving superalignment. I think there are other ways that are much more intense and seem like way more, way more difficult research problems. Like, for example, there's, there's things like full bottom-up interpretability on the models, which just seems, I think basically everyone agrees this is insanely difficult and maybe not even possible. And there are things like the mechanistic anomaly detection.

• 54:19Speaker 02

  Approach that that ARC, the Alignment Research Center, is taking. That also just seems, I think everyone basically sees, thinks, you know, they seize that research agenda and like, wow, that is an insanely difficult problem. Even with lots and lots of AI labor, it seems like that would take quite a long time. And, you know, I think mostly it's a question of intuition of just like, will you need one of those really intense agendas to sort of safely align your super intelligences? Or will sort of something janky and prosaic like Faithful Chain of Thought or the Control Direction pan out in a really big way and let you, you know, maybe automate alignment research at extremely high levels of capability and bootstrap that to a superland solution? Does that sound right to you, Daniel?

• 55:03Speaker 01

  Yep, that that sounds right. And if I could, if I could like get my sort of like optimistic argument in.

• 55:08Speaker 02

  Yeah, yeah, yeah. The 20 to 30% side of you.

• 55:12Speaker 01

  Yeah, well, you know that, you know that meme that's like, how to draw an owl? Step one, make two circles. Step two, draw the rest of the owl. Like, that's basically what I think the game plan is for alignment, where step one is make sure you have these, these really fast AI researcher, AIs, you know, these automated AI researchers, and make frickin' sure that they're not lying to you and that you like know what they're thinking. And then step two, have them draw the rest of the owl, you know? Have them like, have like a million copies of them furiously do all the interpretability research and all the philosophy about what do we really mean by alignment? And, you know, all that stuff they can just do as long as you know they're not lying to you and like, as long as you can like, monitor their thoughts, you know?

• 55:57Speaker 01

  And we kind of gloss over this in AI 2027. Like, we, we focus on how they, they do the Faithful Chain of Thought thing. And then, thanks to the Faithful Chain of Thought, they're able to get like a new version that's thinking the right thoughts and thinking like, that's like actually trying to help instead of being adversarial, and also they can still read its thoughts. And then, they do the rest of the owl and they let that version, you know, go solve all the trickier problems of alignment. And I think that there's a lot of uncertainty about like, how hard is it going to be to solve all those tricky problems, even when you have all these automated AI researchers that are not lying to you?

• 56:30Jacob Effron?

  Well, I guess that like does kind of feel like, because I know obviously a next step of this for you guys is like you've kind of teased you're going to have policy proposals and like, you know, some sort of like, you know, as you kind of further this work. And I imagine it's like somewhat, you know, obviously there's probably things that are inevitable that you'd want to do regardless, but I feel like it's somewhat probably dynamic with your views on timelines for this alignment research, you know, how hard these problems are like, do you feel like there's a like, is it hard to get to consensus internally on like what these, what the right policy proposals are? I'm sure this is not the only divergence in how you guys think about this stuff, but I'd love if you could bring us on the inside of some of those conversations.

• 57:03Speaker 02

  So I would say, Daniel, I at least feel like I'm pretty aligned with Daniel on policy recommendations. I think we tend to, it tends to be that most of the recommendations are pretty, that we try to make are just pretty robustly good across worlds that we think are plausible. Because, you know, there's just so much uncertainty that we try to make robust, robustly good policy recommendations either way. Yeah. Daniel, do you have a, do you have a thought?

• 57:26Speaker 01

  I agree. When I was nodding along, when you said if you disagree internally about policy recommendations, I thought you meant like, I don't know, the AI researcher community more broadly, or like people.

• 57:36Speaker 02

  Oh, yeah.

• 57:37Speaker 01

  Obviously that is.

• 57:37Speaker 02

  Yeah, yeah, yeah.

• 57:38Speaker 01

  But within AI Futures Project, we're like broadly on the same page so far, although, you know, we still have to actually write the recommendations, maybe some.

• 57:44Jacob Effron?

  Yeah. Can I ask you to tease like, you know, the key parts of the platform here?

• 57:47Speaker 02

  I mean, yeah. So the thing we're going to do is we've, so we've written AI 2027 in two branches that we think are plausible things that we think are likely. And then we kind of want to move to the ought side of the is ought dichotomy and go for, well, suppose AI 2027 is coming true. What should, you know, governments and labs actually do in response to this? What do we think the optimal action is for them to take here?

• 58:11Speaker 01

  Or just the responsible action.

• 58:13Speaker 02

  Within some bounds of feasibility.

• 58:15Jacob Effron?

  Yeah.

• 58:15Speaker 02

  Yeah. To give like the quick summary, I think there's like some near term actions that we think are pretty politically feasible and quite good. Like, for example, being way more transparent about model capabilities and making sure that there isn't a giant gap between internally and externally deployed models.

• 58:33Jacob Effron?

  Yep.

• 58:34Speaker 02

  And like, you know, things like investing much to alignment research, investing way more into security so that the models don't immediately proliferate.

• 58:40Speaker 01

  Have a model spec. Publish the model spec, have a safety case, publish the safety case. I'm big fan of that.

• 58:48Speaker 02

  That kind of thing. And then, and then sort of the second phase is these, you know, what should we do if AGI is actually happening? Or if we're in the middle of an intelligence explosion and the world sort of hasn't taken radical steps already.

• 59:02Speaker 02

  And our view is, you know, probably the government should be doing some pretty extreme things relative to the current sort of political environment. Those things, you know, things that should be on the table should be like an international treaty to not build super intelligent AI until we've squared away the whole alignment thing.

• 59:22Speaker 02

  And, you know, that's obviously going to be, it seems like a quite big lift. It's quite far outside of sort of the realm of what's talked about politically right now. But we think something like that might end up being necessary, particularly if you get, you know, if you just think that risk is reasonably high. And I think it will be very, very hard to make, to sort of get risk down to an acceptable level. If you're just building super intelligences in the next few years, if you're early in the intelligence explosion, it's just really hard, I think, to be very confident that, oh yeah, you know, we'll build those super intelligences. They'll take over the world and then they'll totally be nice to us. I think it's just really hard to make a case that that risk is very low unless you've spent, you know, a lot of time sort of testing those AIs,

• 1:00:05Speaker 02

  finding better aligned solutions, finding better verification techniques than we have today. And so, you know, I think that if AI 2027 is coming true, something like that will be necessary to make risk reasonably low.

• 1:00:17Speaker 01

  There's also the concentration of power stuff.

• 1:00:19Speaker 02

  Yes.

• 1:00:19Speaker 01

  Which I also think is important. So like separately from the whole alignment thing, we want to make it the case that there's not any one person or any small group of people that's effectively in charge of the army of super intelligences that comes out of the other end of the intelligence explosion.

• 1:00:34Speaker 01

  And this is like a political thing, not a technical thing. Like there needs to be some sort of governance structure. Maybe we could have like multiple competing diverse AI companies, but you have to have some mechanism that causes them to all be roughly neck and neck. Otherwise, there's a risk that one of them will just sort of take off and get a lead over the rest. Maybe if it's coordinated into one big mega project, then you need to have democratic control of that mega project, transparency into the decisions being made by the leaders, et cetera. There's lots more to say on that subject.

• 1:01:01Jacob Effron?

  I mean, is the audience of your work ultimately like six people? And like the people that maybe influenced them directly? Like, I guess, what do like the rest of us do in like right now?

• 1:01:11Speaker 02

  I mean, I think communication helps a lot. I don't know. I think, I think talking about these issues, getting the public, getting Congress. Like, for example, so like one of the, one of the ways the concentration of power stuff could go badly, right? Is if it's in our scenario, if it's like just the president or just the lab leader who's in charge of what's going on. That's a lot more scary of a situation than if, you know, the public or Congress or other bodies are awake to the situation and then use their levers of power to make sure that they don't get couped or completely disempowered by whoever's in charge of the super intelligences. So that seems, I mean, so just talking about it seems already like a pretty good step to me. Yeah. Daniel, do you have, do you have more?

• 1:01:56Speaker 02

  I wish, I wish there was more that everyone could do. I wish there was more I could do, right? Like, I wish I was, you know, I'm obviously not a lab leader or the president of the United States or, you know. And so I think, you know, we're all trying our best here, but.

• 1:02:09Speaker 01

  I used to be working at a company and it was tempting to take the sort of like grim view that like, well, almost everybody doesn't really matter. What matters is like what the CEO and the president does. And like here on the inside, I have like a better chance of influencing those people than on the outside. And that's sort of the view of a lot of people I know, basically. And that's why they like are not really in the public eye very much and are instead working at these companies. But I'm sort of like placing a bet on the public and on this sort of like broad wake up that I'm hoping will happen.

• 1:02:49Jacob Effron?

  Yeah, and then having more eyes on it actually really does matter. I mean, do you buy, obviously, I feel like a lot of the original story of like Anthropic and then Ilya spinning out is like people being like, well, God, this is going to happen and like better it happen in my hands than it happen in like these other things like, did that make, I mean, is that like the most impactful thing that like an individual can be doing like in the in that situation?

• 1:03:10Speaker 02

  Maybe the most impactful, but it has to be positive, too.

• 1:03:13Speaker 01

  Yeah.

• 1:03:14Jacob Effron?

  But I guess put another way, like, do you agree, like, is that like line of thinking, you know, I guess, do you agree with that line of thinking?

• 1:03:24Speaker 01

  Well, it's extremely tempting and it had, it has in fact been the decision of many, many people I know is to be like, gosh, like, you know, the public is not going to wake up in time and if they do, they're going to flail around and do something useless. Ditto for Congress. All that really matters is what like a couple of CEOs of the most powerful tech companies do and maybe what the president does. And I don't like the current CEO. I don't trust him, so I'm going to go do my own thing, you know? I'm going to be the CEO, you know? Like, this is like literally the story of how DeepMind was founded, literally the story of how OpenAI was founded, literally the story of how Anthropic was founded, literally the story of how SSI was founded.

• 1:04:06Speaker 01

  It's kind of humorous, the extent to which this has been continually happening.

• 1:04:11Speaker 02

  So I would, to phrase it more clearly, I do not agree with that strategy. I think that strategy is probably bad for the world. Yeah.

• 1:04:19Speaker 01

  Yeah, and I also think that way, which is why I've done what I've done instead of going to Anthropic, for example. A year ago, I, when I asked myself, like, how is this all going to go down? Something like AI 2027, but lower resolution and less worked out was sort of roughly the answer that I was coming up with. And then I was like, this is horrifying. This is, you know, not good. And then there's a question of what to do about it. And I sort of gave up on being the guy on the inside trying to change things for the better. And I'm now trying this different strategy of being on the outside and being free to

• 1:05:01Speaker 01

  speak and being free to like do this sort of research, which is, you know,

• 1:05:07Speaker 01

  AI 2027 is an unusual style of research, right? Like, it's, it's, um, it's, there aren't very many epic scenario forecasts like this. And so like, I probably just couldn't have done this within any of the companies. And in fact, even if I could have done it, I wouldn't have been able to publish it. Because the, the PR team would have like, had a, had a fit if they found out, you know? And so like, part of my, you know, I, I basically faced this choice between, there's, there's this quote by, um, I think Larry Summers. Have you heard about this? You know what I'm gonna say?

• 1:05:45Jacob Effron?

  No, I thought you were gonna go with a philosopher, not like, uh, not, not, not, not Larry Summers.

• 1:05:49Speaker 01

  Yeah, so I think you can Google this, but Larry Summers, on multiple occasions, according to various people who've talked to him and have then talked to the media, he said something like this. He says, there's two types of people. There's the insiders and the outsiders. And the outsiders are free to speak their, their truth, but the people in power don't listen to them. And then the insiders follow this one rule of never criticize other insiders. But as a result, they get access to like, the really important behind closed doors conversations and like, get to actually move things. And that's how he sees the world. And maybe there's a lot of truth to that.

• 1:06:34Speaker 01

  And I guess I feel like it is more honorable and ethical to, to do the outsider thing than the insider thing. So that's what I'm doing.

• 1:06:43Jacob Effron?

  Super interesting. I mean, I imagine a lot of people like, will, will read this and they'll, and they'll be scared and they'll think about it and they'll talk about it. And then they'll say, um, okay, it's like, pretty damn uncertain whether we're on this like, super accelerating timeline. Let me wait for a few cards to turn over. Like, I'm like, right now, these model, I play with them, like, God, they're dumb in some ways. Like, you know, if, if we get to 2026 and it's like what you guys say in 26, like, then I'm gonna like, really start to worry. What is like, your message to those kind of people that are like, I'll like, table, I'll bookmark this, I'll, I'll think about it and then I'll come back to it like, in a year and, and then I'll, I'll start to be concerned?

• 1:07:18Speaker 01

  I think that's basically right. I think like, it's important to avoid being frog poiled, but like, what you do, like, what, what happens, what, what, what the government and what the companies do...

• 1:07:30Speaker 01

  In the year of AGI, matters so much more than what they do right now. What they do right now is just all like setup, you know, and like prep for that. And so like, yeah, if you want to just like stick your head in the sand for a couple of years until things start taking off and then get activated and do something, great, go for that. That's, you know, I'm fine with that personally. I think it'd be even better if you got activated now, but like, you know, I think the most important thing is that like when this stuff starts to, starts to, starts to tick up.

• 1:08:01Speaker 02

  Yeah. Can you give me a milestone that like I should get my head out of the sand for?

• 1:08:05Speaker 01

  I would say the superhuman coder thing. So, so. That's so late.

• 1:08:11Speaker 01

  Okay, well, you give a different opinion. So I like. Superhuman coder? Don't you think that's like a few months before like? Yes, I think that's a few months before the point of no return.

• 1:08:25Speaker 01

  Okay, so maybe, maybe, okay, maybe you should get activated sooner than that. I mean, I myself have been accurate, activated for several years, in fact.

• 1:08:32Speaker 02

  Here's a, a line I think is pretty reasonable. The point at which AR and D speed is increasing by 2X. So if you're doing, if you do uplift studies on frontier AI lab researchers, you know, with AI assistance.

• 1:08:43Jacob Effron?

  Are we going to get those studies? Like, why now when that's happening?

• 1:08:46Speaker 02

  I hope so. I think we'll have, I think we probably will. I'm not totally sure if they'll be accessible. Maybe it'll all be locked down, but I think, I don't know. I mean, I know some people who are trying to do them. And I think like right now the uplift is, doesn't seem that high. I'd be very surprised if, if you got, you know, anywhere near 2X. I think that's genuinely pretty hard, right? It's hard to accelerate, I think, AR and D speed by 2X. I think you'd actually need pretty capable AI to do that. And I'd be pretty surprised if AI progress just fizzled out sort of right then, I think. So I think, I think that's like sort of the best warning sign I can think of. Though of course it's, it's not guaranteed to be right.

• 1:09:27Speaker 01

  I mean, I think it also depends on like what you're trying to do. Like when, when, when I was imagining something like, when should you like start a protest movement and like banging down people's doors? And then I'm like, yeah, superhuman code or milestone, you know? Because I feel like by the time they're just like autonomously doing all this coding and they've like, you know, solved long horizon agency to the point where they are just like fully autonomous agents and they're like also already accelerating AI R and D substantially and they're just missing a few key skills needed to completely close the loop. I feel like that's like, okay, you are now like a few months away, maybe at most from really crazy stuff. Time to like really start doing, you know, pulling out all the stops. But if you just mean like, when should I start like reading the news about AI more?

• 1:10:10Speaker 01

  Or like, when should I like, you know, consider switching from, you know, natural language processing to mechanistic interpretability, then that'd be like yesterday.

• 1:10:19Jacob Effron?

  Right. Obviously, it seems like you're kind of like setting the, you know, in some sense, you're like setting the table, you've gotten everyone talking about this. What else does like this entire movement compose of for, for you guys? And how do you think about spending your time over the next year?

• 1:10:30Speaker 01

  You know, it was psychologically kind of rough, at least for me, to have this one mega project that the whole team was working on for almost a year. My blogging output, for example, was a lot lower than it, than it could have been because I was like, I always have to do the most important thing, which is this project. So we're now in a sort of like exploratory phase where we're trying out a couple of different things at once and seeing what we like most and seeing what seems to be getting the most traction. One of the things that we're exploring is the policy stuff that Thomas just mentioned, writing a like new scenario that's the normative scenario and having accompanying, you know, white papers and stuff, explaining some of the ideas in it. Another thing we're trying out is tabletop exercises. So the tabletop exercises, we did them, the war games, we did them as a tool to help us

• 1:11:15Speaker 01

  write AI 2027, but they've been extraordinarily popular and lots of, you know, companies and people want us to, to run them for their teams. And so we're probably going to like lean into that a little bit and try doing it on a more regular cadence and see how that goes.

• 1:11:33Speaker 01

  And then separately, more forecasting stuff. So like, new evidence rolls in every month. There's new models to think about. There's new developments. I think it's probably good to stay on top of all of that and to keep adjusting our forecasts and our expectations, writing, you know, both in-depth stuff, responding to criticisms and being like, here's a new updated analysis of takeoff speeds that explicitly responds to our strongest critics, while also just being a better analysis than what we did in AI 2027. That's something I would love to write at some point. Same thing for alignment. Like, here is a like, for example, like, I want to do a project on trying to guess what the companies are going to do to fix the obvious misalignments. And then trying to guess whether that would actually work or not, you know?

• 1:12:18Speaker 01

  Like that, I can go talk to people at the companies who are responsible for doing those fixes and try to get, you know, like, that's a whole project I could be doing. So, yeah, I think I'll stop there. But there's a whole spread of things like this and we're going to try to like, do miscellaneous things for a bit until we start getting conviction about one particular thing. And then we might double down on that.

• 1:12:36Speaker 02

  Another thing I'll say is I think there are a few more scenario branches that it'd be nice to write. So, so we talked about the like normative branch. There's also like the longer timelines branch, right? Where there's a lot of different things, like if timelines are 2033, right? Which we all think is, you know, pretty plausible. And a lot of people think is, you know, the most likely timeline. Like, a lot of things change. It's more likely China's doing better. It's, you know, likelier that there's a slower takeoff. There's just more, there's more, more opportunity for chaos to happen. And we kind of want to just like, at least do one branch of gaming that out. And we think that'd be, you know, easier to do hopefully than 2027 now that we've, you know, figured out how to write a scenario at all.

• 1:13:16Speaker 02

  And so we might be able to crank out a couple more branches like that pretty easily at this point.

• 1:13:22Jacob Effron?

  Yeah, hopefully you get, you get some AI scenario writers or something that 2X's your productivity too before, uh, before that, before it happens on the researcher side. Look, fascinating conversation. You know, I feel like where, where I'd actually love to end is just like, in the, in the scenario where this goes right, I mean, I'm sure you guys have thought about this as much as anybody. Let's assume, you know, the works you do and the works of others does that. Like, what does human life look like in like 15, 20 years? Like, what do we like derive value from?

• 1:13:49Speaker 01

  Probably like absolute bliss or like crazy awesome utopia. Like, I mean, you said, if everything goes well, you know? So, yeah, if everything goes well.

• 1:13:59Jacob Effron?

  It seems pretty binary in your world, right? Like, either we're here or we're not.

• 1:14:02Speaker 01

  There's some, I would say, so there's, there's like, okay, well, here's a spread of outcomes, ranging from best to worst. Or would you rather get them from worst to best for optimism?

• 1:14:14Jacob Effron?

  You know, I feel like, once you told me it was 70 to 80%, we were done. I'm scared enough, so you can start worst to best.

• 1:14:20Speaker 01

  Okay, so worst is S risk, which is face worth and death. I'll say no more about that. Then next worst is death. So this is like what we depict in the race ending, where the AIs that we build don't actually care about us at all. And, you know, don't see a reason to keep us around. And we're using resources that they can use for other stuff. And so they kill us all and take our stuff. Then after that is sort of like mixed outcomes where either it's a concentration of power outcome, or it's a misaligned outcome, but not the like, and then we all die sort of thing, but rather something more like a dystopia where like the, the humans who successfully managed to stay in charge are just a handful of humans and they're kind of not great people.

• 1:15:06Speaker 01

  They're kind of dictatorial. And so they reshape the world in their image and, you know, the world is amazing from their perspective. But, but it, and you know, most people are like well fed or something. But like, it's, it's, it's kind of like a, like a very wealthy North Korea, perhaps. Like, if you imagine like, what would happen if North Korea just had like insane amounts of like food and resources dropped on it so that they could, if they wanted to, distribute it to their population and make everyone have a very high standard of living? Perhaps they actually would do that. And like, most people, except for the political enemies, would have very high standards of living. But still, it would be kind of like a mixed, like, it wouldn't be the nice, awesome utopia that we wanted, you know? And then there's better outcomes that are actually just like truly awesome utopias,

• 1:15:51Speaker 01

  where the power is sufficiently distributed that like, there's a sort of like live and let live. There's loads of wealth and it's distributed. And then people are mostly allowed to do what they feel like with that wealth. And there's rules that prevent people from doing truly terrible human rights violating things. And then everyone gets to like, you know, go make colonies in space and, you know, not have to work again because everything is. You know, created by the robots and you know, you can just spend your time playing games and having families and you know, pursuing whatever interests you have. That I think is the sort of like best case or something close to the best case outcome. I just want to double emphasize loads of people at these companies think that basically

• 1:16:36Speaker 01

  2027 is what's going to happen or something similar to it. I don't know what percentage it is, but like I know loads of people who are like, great work. It's so good to like, try to game all this stuff out. And then I talk to them and I'm like, so what do you agree with? What do you disagree with? And they're like, oh, like, I think the robotics stuff is going to go a little bit slower than you say, or like, oh, I think the alignment stuff will be mostly solved by then, you know, like, or like, or you know, like, or like, I think China will be farther behind. But like, basically, like, they're like, basically, yep, something like this is going to happen somewhere in the next few years. And then we are like quibbling about some of the details is like-

• 1:17:07Speaker 02

  Especially, especially a lot of leadership too, I feel like.

• 1:17:10Speaker 01

  Yeah.

• 1:17:10Speaker 02

  Like, if you, have you, have you read, I wanted to ask, have you read the Elon Sam emails from the court case from like 2015, 2016, 2017?

• 1:17:17Speaker 01

  Yeah.

• 1:17:18Speaker 02

  It's like crazy how they're like, they're thinking in terms of, you know, who's in charge of the AGI controls the future. And that's why they, you know, care so much about that.

• 1:17:28Speaker 01

  Ilya said, that's why we founded OpenAI is because we didn't trust Demis not to create an AGI dictatorship, you know? And then he says, and that's why Elon, you shouldn't just have complete control over all of this. And also Sam, why you want to be CEO so much?

• 1:17:42Speaker 02

  I guess there's, there's a bunch of different ways people could do a counter scenario to you guys. What would the contours of like a really good objection to you guys that isn't that be? So something that I'm very uncertain of is takeoff speeds. So how long does it take to get between AGI and superintelligence? And I think that all of the models, people have put out a lot of models of how, how long this will take and how fast it will be.

• 1:18:00Speaker 01

  Not a lot by normal academic standards. There's like three. It is, this field desperately needs to grow bigger fast.

• 1:18:05Speaker 02

  Sure.

• 1:18:06Speaker 01

  And there have been like several different models by several different people.

• 1:18:09Speaker 02

  Yeah. And I like, I'm like really happy that those, that people did those models. I'm really happy they exist, but I also think they're bad. I think those models are like extremely unconvincing to me and I have, you know, we could go into the objections that I have with basically all of them and for why I don't really think they're, they're right.

• 1:18:25Speaker 02

  If someone produced a model that seemed accurate to me, I could imagine it really changing my minds about takeoff speeds in either direction, right? Either saying, you know, you guys were, you know, it was way too long.

• 1:18:36Jacob Effron?

  You're talking about basically like we get to like, you know, we get to the 2XAI researcher, we get to superhuman coders, but like, God, the distance between that and like your agent four is actually way longer because of X, Y, or Z.

• 1:18:45Speaker 02

  That's right. That's right. Like, I could imagine it being five years or something. And I think, I think that world looks really different if it's five years instead of, you know, six months or a year or something. And I could really imagine myself, like, I currently feel like I'm really, really uncertain about that question, and I really don't feel like I've seen very good evidence or models really in any direction. And I could imagine someone producing something. Like, I don't know how to build this model yet. I also would have done it.

• 1:19:08Jacob Effron?

  But like, I mean, would that actually change anything? Because ultimately, like, I assume it's so, it's so far out and hypothetical that, like, maybe it changes your P doom, like, I don't know, even if it changes it from like 70, 80 to like 10. A 10 P doom is like, kind of, is pretty damn terrible. It's hard to do any, you know, or focus on anything else, right?

• 1:19:22Speaker 02

  I could imagine it changing things quite a bit. It's like the epoch people's views, for example. So like Ege and Tami and such, I think, have quite different views from like, I think, Daniel and I. And I think a lot of that is downstream of this takeoff question. And I think their view is like much, is like that this will take, you know, five, maybe 10, 15 years.

• 1:19:39Jacob Effron?

  I'm not, what is like the basic crux of that, of why it would be slower?

• 1:19:42Speaker 02

  It's complicated, but I think that the crux is, is something to do with diminishing returns of research. So there's this question of how quickly do ideas get harder to find versus how quickly do you, you know, scale up your AI, you know, your supply of AI agents doing research? And then like, you know, under some views are like, it leads to an explosion because, you know, you get more AI research. And then like the amount that AIs get, you know, ideas get hard to find is like slow enough that you get this like hyperbolic or super exponential or whatever, or some really fast curve. And then there's other views that are like, no, it actually, you know, bites really hard. There are Amdal's Law type effects, there are like various bollocks.

• 1:20:18Jacob Effron?

  But what could they possibly show you that would like, you know what I mean? Like, it's like, it just feels like. Well, a coherent model would be a good place to start.

• 1:20:24Speaker 02

  Like, I'm like, they have a model. The epoch people, they do have them. They have like a sort of econometric model called like the GATE model, G A T E model.

• 1:20:32Speaker 01

  It's not even really trying to model the sort of thing we're interested in. Like, it even says, there's a disclaimer somewhere where they're like, "We don't model the effects after you reach full automation," or something like that. Like, you know, better research along those lines would be amazing, but then separately from the research with the, there's the sort of like, there's the arguments and the models, but then there's also just literally the scenarios. And the reason why I think it would be great for people to write these scenarios is, A, I think it's valuable, they're valuable artifacts as conversation topics to argue and critique about. So, for example, if someone wrote a scenario, the current best, I recommend you go read my current favorite counter-scenario that exists, which is called "History of the Future" by L. Rudolph L. It's on Substack.

• 1:21:14Speaker 01

  So, this is a like, 15, 20-year timelines scenario that has the superhuman coder milestone arriving in like 2027, but then says that like, that doesn't lead to superintelligence, basically. Or like, "Yeah, no, you should go read it for yourself." But it sort of lays out this story, and it's a very interesting, sort of like, detailed year-by-year scenario, just like we did. And so, now that that exists, we can like, critique it, and we can be like, "Here, you say there's superhuman coders, but then, like, why don't you get to superintelligence? Like, it seems like we think that you, you know..." And then we can like, zoom in on that part. And then there's another part of the story where they're like, "And then they figured out corrigibility." And it has like, a paragraph on basically, why the AIs are aligned.

• 1:21:57Speaker 01

  And I'd be like, "Well, let's talk about that paragraph. I don't find that convincing. Like, I don't think that would actually work, you know?" So, when you actually have this sort of concrete story, then you can critique it, and you can like, right? And then, in addition to the critique of the story itself, you can compare two stories to each other, and you can start seeing when reality is like, hewing more closely to one as opposed to the other. So, yeah, more arguments, more models, that'd be great, but also just more scenarios, I think. Oh, I'll also end with this, I really hope we're wrong.

• 1:22:26Jacob Effron?

  Yeah, I guess nothing would make you happier than, than you look really silly on the internet.

• 1:22:31Speaker 01

  If those benchmark curves like, start flattening, and like, 2027 arrives, and like, the AIs are only like, you know... Like two-day Horizon workers or something, and they just sort of reliably flare out when you try to get them to work over periods of months. I'll be so happy. And specifically if the trends are slowing too, so that it's not like we're about to take off, but we can sort of see, okay, we got at least a couple of years left before anything really scary happens. I'll be so happy. Throw a huge party. It'll be great.