LIMITLESS - AI DEBATE: Runaway Superintelligence or Normal Technology? | Daniel Kokotajlo vs Arvind

Two visions for the future of AI clash in this debate between Daniel Kokotajlo and Arvind Narayanan. Is AI a revolutionary new species destined for runaway superintelligence, or just another step in humanity’s technological evolution—like electricity or the internet? Daniel, a former OpenAI researcher and author of AI 2027, argues for a fast-approaching intelligence explosion. Arvind, a Princeton professor and co-author of AI Snake Oil, contends that AI is powerful but ultimately controllable and slow to reshape society. Moderated by Ryan and David, this conversation dives into the crux of capability vs. power, economic transformation, and the future of democratic agency in an AI-driven world. ------ 💫 LIMITLESS | SUBSCRIBE & FOLLOW https://limitless.bankless.com/ https://x.com/LimitlessFT ------ BANKLESS SPONSOR TOOLS: 🪙FRAX | SELF SUFFICIENT DeFi https://bankless.cc/Frax 🦄UNISWAP | SWAP ON UNICHAIN https://bankless.cc/unichain 🛞MANTLE | MODULAR LAYER 2 NETWORK https://bankless.cc/Mantle 🌐SELF | PROVE YOUR SELF https://bankless.cc/Self 🟠BINANCE | THE WORLDS #1 CRYPTO EXCHANGE https://bankless.cc/binance ------ TIMESTAMPS 0:00 Intro 4:44 Is AI Normal Tech? 30:20 Capability & Power 44:52 Manageable or Existential? 52:36 AGI Milestone 1:00:31 AI by 2030 1:12:07 Making Sense 1:16:08 Closing & Disclaimers ------ RESOURCES Daniel Kokotajlo https://x.com/dkokotajlo Arvind Narayanan https://x.com/random_walker Two Paths For AI - Daniel https://www.newyorker.com/culture/open-questions/two-paths-for-ai AI as Normal Technology - Arvind https://knightcolumbia.org/content/ai-as-normal-technology AI 2027 - Daniel https://ai-2027.com/ AI Snake Oil - Arvind https://www.aisnakeoil.com/ https://www.amazon.com/Snake-Oil-Artificial-Intelligence-Difference/dp/069124913X Pause AI https://pauseai.info/pdoom ------ Not financial or tax advice. See our investment disclosures here: https://www.bankless.com/disclosures⁠

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:03Ryan Sean Adams?

  Hey everyone, Ryan here. I think you guys are going to enjoy this debate because I certainly did. The debate you're about to hear was recorded by David and I for our new Limitless Podcast feed, where I occasionally guest host along with David and our new co-host Josh. So this is a debate that has been ringing in my head, maybe I'd say keeping me up at night, ever since our episode two years ago with Eliezer Yudkowsky. The question: is AI normal technology? Is it a tool we can control, maybe like the internet or electricity? Or is it more like a new species of superintelligence? Is it something destined to restructure our political world order, our economy, maybe in a matter of years? And if all of this can happen by

• 0:46Ryan Sean Adams?

  the end of this decade, is there any more important or more pressing question worth answering? What you're about to hear is an incredible debate with two experts on this topic: a professor and hype buster versus a former OpenAI researcher and the author of AI 2027. Now, moving forward, I gotta tell you, episodes like this will only be available on the Limitless Podcast feed, so you won't see them here. So if you're not subscribed to that podcast feed, stop what you're doing, go search it up on Spotify, Apple, YouTube, wherever you listen, there's a link in the show notes. Enjoy the episode.

• 1:19David Hoffman?

  In the Wild West of DeFi, stability and innovation are everything, which is why you should check out Frax Finance, the protocol revolutionizing stablecoins, DeFi, and Rollouts. The core of Frax Finance is Frax USD, which is backed by BlackRock's Institutional Biddle Fund. Frax designed Frax USD for best-in-class yields across DeFi, T-bills, and carry trade returns all in one. Just head to frax.com, then stake it to earn some of the best yields in DeFi. Want even more? Bridge your Frax USD over to the Frax still Layer 2 for the same yield plus Frax still points, and explore Frax's diverse Layer 2 ecosystem with protocols like Curve, Convex, and more, all rewarding early adopters. Frax isn't just a protocol, it's a digital nation powered by the FXX token and governed by its global community.

• 2:01David Hoffman?

  Acquire FXX through frax.com or your go-to Dex, stake it, and help shape Frax Nation's future. Ready to join the forefront of DeFi? Visit frax.com. Now to start earning with Frax USD and staked Frax USD. And for Bankless listeners, you can use frax.com/r/bankless when bridging to Fraxal for exclusive Fraxal perks and boosted rewards. Uniswap is your gateway to a more efficient DeFi experience. With Uniswap swapping and bridging across 13 chains is simple, fast, and cost-effective, helping you move value wherever, whenever. Thanks to deep liquidity on the Uniswap protocol, you'll enjoy a minimal price impact on every trade. And now Uniswap V4 takes it even further. Swappers benefit from gas savings on multi-hop swaps in each trading pairs, while liquidity providers can create new pools at 99% lower costs.

• 2:46David Hoffman?

  The best part, you don't have to do anything extra. Each trade is automatically routed through Uniswap X, V2, V3, and V4, so you get the most efficient swap without even thinking about it. Whether you're swapping, sending, on-ramping, off-ramping, or bridging, Uniswap's web app and wallet gives you the tools to unlock DeFi's full potential on Ethereum, base Arbitrum Unichain, and more. Use Uniswap's web app and wallet for a more efficient way to use DeFi.

• 3:09Ryan Sean Adams?

  All right, everyone, I've been looking forward to this conversation all month, maybe for many months, I've got to say. So the central question on today's episode is, is AI normal technology? Maybe it's something like the Industrial Revolution. We've been through that before. Or is it very, very abnormal? Is there incredible variance? Is it something between utopia and annihilation? It could go either way. It's like nothing we've ever dealt with in the past. We've got two guests on today that are going to discuss, or I might use the word debate, this subject. We've got Arvind Narayanan. He is a Princeton computer science professor. He is the professional hype slayer, including some hype slaying he's done in crypto in the past. He is the writer of AI Snake Oil, and he is a believer

• 3:51Ryan Sean Adams?

  that AI is normal technology, kind of like electricity or the internet. Arvind, welcome to the podcast. Thank you. I'm really excited. Thank you for having me. We also have Daniel Cocotello. He is an ex-open AI researcher. He's the author of an incredible report called AI 2027, which is like a month-by-month forecast of AI progress, all the way to AGI, which I bet you can guess the year in which he predicts that's coming. He thinks AI is very, very much not normal and might become Super intelligent by the time you upgrade your next iPhone. So pretty soon. Daniel, welcome to the podcast. Thank you for having me. Excited to discuss. All right, guys, this is gonna be a lot of fun. We've got seven topics to go through and we'll see if we get to all seven. We're gonna do this in a compare and contrast way. I'm going to seed the conversation.

• 4:37Ryan Sean Adams?

  David's gonna help and then just fire you right into the topics. You guys ready?

• 4:41Speaker 02

  Let's do it.

• 4:42Ryan Sean Adams?

  Okay, the first topic on the agenda, the question, is AI normal technology? Or is AI very much not normal, like nothing we've seen before? Is this kind of a tool that we can control? Or is this a new species destined for its own autonomy? Arvin, I've got a quote from you to kind of set the scene, maybe in your perspective. You said this, AI belongs with electricity and the internet's transformative, but yes, it's still a tool. Daniel, on the other side, you said this, this isn't another general purpose tech, talking about AI. It's a new species about to outthink us. Arvin, I'm gonna start this with you on the topic. AI, normal tech, you put out a report entitled like with that as the title. What's your case for AI being normal tech?

• 5:25Speaker 02

  We break down the way in which AI, or really any technology, is going to impact society into a kind of four step process. The first step is improvements in capabilities, new models coming out. AI is able to do new things, but there's no direct line from there to the impacts that we really care about. The next step after the first step of invention, as we call it, is innovation, turning that into product development. And from there to users learning how to use these new capabilities in ways that make sense for their workflows, what they want to do in their jobs. And the last and final and slowest step is adaptation. We talk about many reasons why business models need to change, laws and norms need to change

• 6:11Speaker 02

  for AI's impact on the world to really be felt. And we can already see this process playing out in the last two years where, you know, we had this big leap in capabilities with ChatGPT and other releases around that timeframe. But when you look at the extent to which that has transformed people's workflows, I think we're starting to see that that's going much, much slower than people initially predicted. And we have some numbers in the paper to back this up. And we predicted this trend broadly will continue. Daniel, Arvind thinks it's going slow. What's your take?

• 6:44Speaker 03

  I mean, I also think it's going slow right now. For example, you said people, it's gotten slower than people predicted. It hasn't gone slower than I predicted. In fact, it's gone about the same speed that I predicted. I know you're moderating this debate, so you're in charge, but I wondered if I could also ask some questions to help set the stage.

• 6:59Speaker 02

  You'd be in charge. Yeah, go for it. Yeah.

• 7:01Speaker 03

  So, Arvind, if you've read AI 2027, I would love it if you could give your own sort of alternative story, basically. Like, if you had been in charge, like, suppose that someone gave you a whole extra year of life and said you have to spend your year writing your own counter scenario to AI 2027. What would that be? What would you depict? And maybe one way of putting it is like, at what point would it start to diverge from what we predict in AI 2027? And then like, what would, how it would gradually diverge more and more as time goes on? Please just like, paint that picture for me because I think it's a useful way to sort of like, set the stage for everything else that we're going to be talking about.

• 7:40Speaker 02

  Sure. Yeah. So, I mean, our fundamental point is that in 2027, the world is going to look more or less like the world in 2025. And like you said, yeah, we think gradually things are going to look different. And that divergence, we say, you know, is roughly on the time scale of decades. But qualitatively, what that divergence would look like for us, there's perhaps a greater role of human agency and policies and so forth in determining what kind of future we end up in, as opposed to predictions stemming directly from the capabilities of the technology. And so that's the reason why, you know, picking one scenario would be kind of sampling from a very broad range of possible futures.

• 8:24Speaker 02

  So it's a little difficult for me to paint a scenario, but broadly, I can, I can say that over, let's say, the next couple of decades roughly, we expect to see, just for concreteness, to take one industry like software engineering. You know, people are going to figure out what it even means to be a software company or a software engineer at a time when AI is much more capable than it is today at writing codes. Today, we can vibe Code, create simple apps. What if we can create enterprise scale apps in the future with AI and minimal human involvement? So one scenario for what software might look like in that in that situation is it'll be silly to create software once

• 9:07Speaker 02

  and expect hundreds of different companies to use it because you're forcing everyone to be boxed into the abstractions that are built into the software. Instead, enterprise software will mean creating software for one company or one team or even one individual user. And the role of the software engineer of the software company will be primarily to talk to that client, understand the specifications. So creating software will primarily be that communication role as opposed to typing code into a keyboard. So that's one scenario for how one industry might change. Again, it's hard to give an overarching scenario for how the world overall might change in the next couple of decades.

• 9:45Speaker 03

  Thanks. So I agree that the future is uncertain, lots of branching possibilities. And so there's a lot of uncertainty that we sort of have to collapse down like the wave function, collapse down into like a particular path for purposes of discussion. And so I totally understand, you know, we dealt with that a lot when we were doing AI 2027 is like, we're not like confident that this particular path is what's going to happen. But it sounded like you were saying that like, yes, the AIs are going to get a lot better at coding. They might be better at autonomous coding so that they can basically like do the full stack development process without human intervention. But then the humans might go to other roles such as interfacing with the customer to understand their requirements and so forth. And you think that might happen in the next few years, but that it'll take like decades to sort of like

• 10:28Speaker 03

  gradually transform the actual industry to upend the way things are going now. So that's that's definitely a disagreement between us because I think that, I mean, to put it in like a little potted summary, the AI 2027 story is first you automate the coding. Then your research speeds up so that you build new AIs that can automate the whole research process. Then your research speeds up even more so that you can get to what you might call superintelligence, which means it's cognitively superior to humans in every relevant way. It still has to learn from real world data. It still has to deal with physical bottlenecks, but it needs less data than humans do and it can work through the bottlenecks faster than humans can.

• 11:09Speaker 03

  And there's a million of it and it thinks 50 times faster. And so then we depict that sort of process happening in 2027. And then in 2028, we depict it exploding into the economy and actually transforming things. And so it was interesting, you said earlier, your first thing that you said in response to my question was, I think 2027 will look much like 2025. And I kind of wanted to say, I also think 2027 will look much like 2025. AI 2027 depicts the economy looking basically the same in 2027 as it does today because a lot of what's going on is that the capabilities are advancing rapidly within the companies, but they're not transforming the world yet. And then in 2028, that's when the world transformation happens.

• 11:53Speaker 03

  That's when the new factories start going up, et cetera. So you talk a lot about the bottlenecks and the fictions and so forth. And I guess I'd want to say something like, yes, those things exist and they're extremely important. And that's part of why there's so much of a gap between what AI systems are currently capable of and the actual transformations of the relevant industries. Like, there's just a time lag to build new companies based on the new technology and to integrate it into everything. So that's why I think things won't look that different up until around the time you get superintelligence. And then after you get superintelligence, there'll also be bottlenecks and there'll also be time lags. However, I think that they won't take decades to overcome. I think that when you've got the army of superintelligences, we're looking at something

• 12:37Speaker 03

  more like a year to get to a fully automated economy rather than something like 20 years. So having just recapped my position there, I would then ask you, do you think that...

• 12:52Speaker 03

  We will get to superintelligence anytime in the next decade.

• 12:55Speaker 02

  Yeah, thank you, Dan. That's that's very helpful. Yeah, so let me pinpoint exactly where some of my disagreements are. So in the paper, AI as a normal technology, we dispute the whole premise of superintelligence. We don't disagree that AI capabilities on many individual dimensions are going to keep increasing. And, you know, there are going to be superhuman AI capabilities, just like there are today in things like chess. However, we think that most tasks are like writing and not like chess. So the key difference between these tasks is that when it comes to chess, human performance is limited by our computational abilities. So it's very natural to see how AI can be dramatically superhuman and, you know, has been for the last 20 years.

• 13:38Speaker 02

  And in tasks like writing, our abilities are not computationally limited. Our abilities are limited by a few things. One is hard to agree on what constitutes good writing and what doesn't. It's limited by it's limited by our knowledge of the world. And we talk in the paper about why there are going to be bottlenecks to acquiring knowledge that humans haven't been able to acquire because it requires real world interaction. It requires experimenting on people and so forth. And finally, our writing abilities are limited because, let's say, you want to write to persuade someone. We think there is only so so much that you can make a piece of text persuasive. There are fundamental limits there. And so computational limits are relevant only in a narrow set of tasks.

• 14:23Speaker 02

  In most cases, it's these intrinsic limits and knowledge limits. And we do think that over and over, AI is going to run up against those. So this idea that you can run AI at 50x speed and have a million copies of it that make things go a million times faster, we dispute that premise.

• 14:40Speaker 03

  Well, I certainly don't think it will go a million times faster. We give our quantitative estimates of how much faster it will go, and it never gets close to a million.

• 14:46Speaker 02

  Okay, sorry if I misremembered.

• 14:48Speaker 03

  I agree that the the parallel thing, there's massive diminishing returns there and things like you said. But I think I want to push you again on the point about superintelligence. The way that I would define it is it's a an artificial AI agent that is better than the best humans at every important or relevant cognitive task. And so it's fine, like, so according to this definition, if there's a hard ceiling on how good you can be at writing. That's only slightly better than the best writers that are humans. It's still possible for there to be superintelligence. It's just that the superintelligence would be like, you know, running up against that ceiling, but only on an absolute scale would still only be slightly better than the best humans. But like, as long as you think it's possible to be like ever so slightly better than the best

• 15:30Speaker 03

  humans, then it's possible to have an AI that's meeting that definition for what I'm saying. And so then the question is like, on all the relevant domains. So, so, you know, obviously in chess, they're already superhuman, but like, you know, enumerate all your favorite domains, all the domains that you think are important. Will there be AI systems that are better than humans at all of those ones? And it doesn't have to be like massively better, just like somewhat better than the best is basically what I'm asking.

• 15:53Speaker 02

  So if they're only a little bit better than humans, what leads to this dramatic acceleration? You know, I mean, if humans got a little bit better at something, we wouldn't think that would lead to, you know, a 10x of GDP or whatever.

• 16:06Speaker 03

  Well, let's talk about the different types of acceleration. So first, the research acceleration is the one that we thought the most about. So on the website, we've got this appendix where you can go read like the way we did our like little hasty back of the envelope calculations to try to get the numbers that we used in this, in the scenario. And we estimated something like a 25x speed up for the AI research stuff when you hit the, the superhuman AI researcher milestone, which is, which is kind of like relevant to this discussion because it's basically like slightly better than the best human researchers. And the, the way that, the way we get that 25x, we did a couple of different, a couple of different methods and then sort of aggregated them. And then we like shaded it downwards just to be conservative. So, well, I guess I won't get into that here, but I'll try to briefly summarize.

• 16:51Speaker 03

  One way of thinking about it is we polled researchers at companies, how much slower their research would go if they had less compute. And then we sort of flipped it on its head and thought, well, like, if you're running these AIs at 50 times human speed, or say 30 times human speed, I think, which is the number we actually used. That's kind of like having a company going 30 times faster, but with 1/30 of the compute. And so applying the numbers from the poll, we get some sort of slowdown factor of like, well, we're going this much slower because you have 1/30 of the compute. But then we're going 30 times faster because we're running things 30 times faster. That was one of the methods we used. We also used a more like granular method of just thinking through the research process

• 17:35Speaker 03

  and looking for ways you can be more efficient here and there and like adding up all the multipliers. Anyhow, so that's how we got the 25X number for the overall speed of research. We haven't done a similar analysis for every other field of human endeavor, like rocketry or plumbing. We just did it for AI research. When it comes to those other things like rocketry and plumbing, I think I would say the general heuristic that I would use is something like, think about the gap between people who are the best in the world at it and people who are merely good professionals. And then look at the shape of the distribution. Oftentimes, we see a sort of like heavy-tailed distribution where like most people are around the level of a professional,

• 18:18Speaker 03

  but then like some really good people are like able to get things done in half the time. And then like some really, really, really good people, the best in the world, are able to get things done in like a quarter of the time that it takes a typical professional or something. So you can sort of plot this distribution. And then you can try to speculate about where the sort of limits are. You know, where is that fundamental limit to how fast you can build new rockets, for example? And the insight that we're relying on is basically that if you have this distribution that looks heavy-tailed, then you should expect that the fundamental limit is not like right above the highest data point that you've observed. Like probably, probably it keeps going for a little bit before we're hitting limits. Otherwise, it'd be this strange coincidence that like the fundamental limit is like right above what we just happened to have observed.

• 19:03Speaker 03

  It would be different if the distribution looked different. Like if the distribution was sort of clumping up, then that would be evidence that we're sort of like asymptoting towards some sort of a limit. Anyhow, all this to say, the way that we got our headline figure for 2028 in the scenario was looking at historic examples of transformations of economies, such as what happened in wartime, when like the US repurposed all their factories to build planes instead of cars. And so we're imagining something similar where we're imagining a political contest. This is another possible disagreement we have with you is that You're talking a lot about the bottlenecks, the fictions, the regulations, the market being cautious to adopt new technologies, et cetera. We're imagining a lot of that still happening, but to a much lesser extent than perhaps you think

• 19:47Speaker 03

  due to the race dynamics that we're projecting, where the US is afraid of losing out to China, China's afraid of losing out to the US. So the president is basically partnering with AI companies to cut regulations as fast as they can and smooth things over in the special economic zones and so forth. In that environment, we're thinking it's similar to World War II, where they also cut regulations, cut red tape, did this massive pivot from things. So we were thinking, basically, again, to use the distribution analogy, we're thinking, how fast have good humans been able to do this in the past? How much faster are the best humans able to do it compared to typical professional humans? It seems like there's a decent gap there. It seems like the Elon Musks of the world are able to get SpaceX off the ground

• 20:31Speaker 03

  much better than Boeing and the various legacy companies. And so then that suggests that if you do have the army of super intelligences that are better than the best humans, you should be able to go faster still than the best humans. And how much faster? Well, maybe another factor of three, maybe another factor of five, something like that. And so instead of taking a World War II length of time to do this transformation, maybe it takes one-third that or one-fifth that. That was roughly how we got those numbers. But yeah, obviously, there's lots of uncertainty.

• 21:02Speaker 02

  Cool. Thanks. So there's lots of things in there, if I can respond to a few of them. So first, let me start with the policy implications. So I think there's a little bit of a self-fulfilling prophecy effect here. So in our view, AI as normal technology is not just a prediction, but a prescription for how we should approach it. We should not approach it as if we're in wartime, and that's for all the reasons that we described. And my worry is that with AI 2027, it's leading to the very same prescriptions. That will lead to the arms race dynamics that you're so concerned about. So, you know, if we think that we have to do this because China is going to do this, it inevitably kind of leads you to that spiral.

• 21:44Speaker 02

  I do agree that if it's not just cut regulations, we're talking about civil liberties, right? If we were to suspend civil liberties and suspend democracy in a very real sense, and act as if we reoriented the whole economy toward a kind of military production, except here it's improving AI industrial capacity, you know, things could go qualitatively very differently. But why should we? I think it's the only reason we would do that is if we assumed that if each side assumed that the other one is going to do that. So that's the first point. The second point is when it comes to the acceleration, this idea that the fastest humans are much faster than the median, and so we can achieve that speed up. That's a little tricky to me.

• 22:29Speaker 02

  So when we look at past technologies, you know, when the internet was starting to become a thing, people thought that this was going to lead to a 10x improvement in all kinds of knowledge work, because instead of going to the library to look up a fact, you can look it up instantly on the internet. But that turned out that was only one step in a larger process. Right? So whatever step of a process we're looking at, you know, let's say one surgeon is much faster than another surgeon. But when you look at the entire context of a hospital, that surgeon's efficiency is only one small part of it, and there are so many other bottlenecks. Same as with the internet. When we remove one bottleneck, other parts of the process become the bottleneck. Within each job, I think the slowest task is going to become the bottleneck.

• 23:13Speaker 02

  And for the whole economy, the slowest sector, the least productive sector, is going to become the bottleneck. And we have some citations from economic works that talk about this. And third and last point, I would say, is in our earlier conversation, I think you acknowledged that there might be limits, most tasks being like writing, where AI is only slightly better than the best humans. And so that maybe seems a little bit at odds with the claim that there is this long tail, and therefore AI can be far out in the extremes of the distribution, far better than the best human.

• 23:46Speaker 03

  Thank you. I guess I'll take them in reverse order. So I agree that it's task-dependent or field-dependent. How, the way, the way I like to think about it is... You keep improving capability, but at some point, you sort of asymptote towards whatever the hard limits are in terms of actual effect on the world. And it's going to be different from field to field. So I think in a lot of researchy type fields, it seems like we have evidence that it's an extremely fat, heavy tail distribution. And the very best researchers are just like orders of magnitude more productive than the typical researchers in terms of actual effects, like in terms of like how much like actual scientific progress they produce. Whereas in other fields, like perhaps in plumbing, you know, the world's best plumber is only able to do, you know, twice as fast as like a typical plumber, maybe. I don't actually know that number, but like,

• 24:31Speaker 03

  I wouldn't be surprised if something like that were the case, right? I think that for things like the transformation of the economy, we're sort of taking the zoomed out perspective of like, yeah, that's going to involve a lot of specific things that will run into bottlenecks. But then also sometimes you can sort of route around that bottleneck by like finding a different way of doing things. It doesn't depend on that. Zooming out and being like, what about this whole, thinking about this whole transformation task as itself, the thing that we are measuring. Well, it seems like there have been humans trying to do something like this in the past. Like the human is in charge of the transformation of the economy during Rolo II and stuff like that. Also, similarly, I think economic production. Like I think that like what Tesla and SpaceX and Boeing and so forth are doing is kind of similar where you want to like produce more cars that are better and that are cheaper than your competitors.

• 25:15Speaker 03

  But you have to like design them, you have to test them, and you have to like produce them. You have to, you have to purchase the relevant equipment and install it and debug it and so forth. And there's bottlenecks every step of the way through that whole process. And there's ways in which like, I mean, that's why it takes so long is that like your, your, your, your, your assembly line is not going to work correctly on the first try. And you're going to need to like purchase new equipment and like integrate it and so forth. But you can sort of zoom out of all of that and be like, okay. So how much better is SpaceX at doing that sort of thing compared to Boeing? And the answer is like a factor of five or something like they they seem to be able to like actually make progress in the aggregate like five times faster or something. So similarly, we're thinking that once you have the super intelligences and you've

• 25:57Speaker 03

  given them the legal authority and like the the money and the financing and the support to basically be in charge of all of these factories and to be calling the shots. We're imagining a similar sort of gap between that situation and, you know, SpaceX being in charge as we see between SpaceX being in charge and like Boeing being in charge, you know? And that's where we got the sort of one year takeoff, one year transformation sort of thing. If I can also talk about some of the other things we're mentioning. So the self fulfilling prophecy thing is like, it gets me to the core because I don't want to create this self fulfilling prophecy. And this is actually my number one concern about AI 2027 in the ways like, like if it turns out that I regret this on my deathbed, it'll probably be because of this.

• 26:42Speaker 03

  I understand that like these companies benefit from hype about their products and that like they have an incentive to hype it all up so that they can get more investors and so that they can like get the government to like wave red tape for them and stuff like that. However, I think that like, well, this is what I think is going to happen and it's what the companies are gunning for. Like, like AI 2027 is, is not that new to the people who've been at Anthropic and OpenAI and GDM. It's sort of like what they're aiming for and what they expect. And like, I think that they're going to get it if we don't raise up a storm about it. Like, like, I think that we can, I don't know, like, I think it would be different if like we were the only people, I think, I think basically like I'm torn about this, but I,

• 27:27Speaker 03

  but I've like made the, made the gamble that like, since it seems like this is where we're headed anyway, we need to like talk about it and raise awareness about it and hopefully use that as steering towards something better rather than sort of just hoping this doesn't happen because this is what the companies are trying to make happen and they're very powerful and they have, you know, lots of money and lobbyists and so forth. And so if I don't tell the president now, like what about China? You know, like, like if I don't raise these concerns, the companies totally will a year from now, you know? And like when, when they have, when they've automated the AI research. They're totally going to be like saying, oh, you need to cut the red tape for us and you need to like, I don't know. So I guess I'm making the gamble that like AI 2027 has a relatively

• 28:10Speaker 03

  small effect on the self fulfilling prophecy direction and that it has a bigger positive effect in the sort of like getting people to wake up and pay attention and then hopefully steer things in a better direction. But I'm not entirely confident that that's true and and I sure hope it is.

• 28:25Speaker 02

  Yeah, I definitely agree that that's something we should all hope for. And just to put a little bit more clarity on perhaps where we agree and disagree, and this gets to some of the other items in the outline that Ryan and David shared. I think for us, if some of the things that I think you kind of take as inevitable happens, which is that there will be this capability improvement, and then we will have kind of no choice but to put AI in charge of the factories, maybe give it legal authority. And for us, and then their kind of red line is AI owning wealth and so forth. If all those things happen, to me, we've already lost. We've massively lost from a policy perspective. Even if some

• 29:07Speaker 02

  catastrophic or existential event doesn't happen, this is all, you know, not the kind of thing we want to be doing in a democracy, not the kind of thing that's compatible with civil liberties. So for us, those are where a lot of the intervention points are as opposed to, you know, alignment, international treaties and so forth. Those concerns only come up at a stage where for us, a lot of missteps have already been made. So that's perhaps one point of difference. Another one is, oh, sorry, Ryan.

• 29:36Ryan Sean Adams?

  Yeah, I agree. And I'm wondering if we could zone in on that for a second and then get back to kind of the second. But this is something I read very much in your work, Arvind, and I wanted to get Daniel to really respond to this. So throughout your work and kind of the case for this being normal tech, you make this distinction between capability and power, and you sort of view those as decoupled things, right? So you say this capability is intrinsic, power is the permissions that we grant to the AI, right? So we can keep power away from the capability of AIs and do things like maybe don't give it legal status. Don't give it the ability to collect wealth. Don't give it the ability to lobby governments, right?

• 30:17Ryan Sean Adams?

  And you're imagining that we can kind of control power and separate that from capability. I have actually not heard Daniel's take on the decoupling of capability and power. And so I'd like to hear that, but I'm wondering if Daniel thinks that you can't really decouple these things. Capability leads to power. And so Arvind, maybe we could get Daniel to respond there on your first point. So, so Daniel, this idea of capability and power being separate, what do you think about that?

• 30:45Speaker 03

  I agree that they are separable in principle, like they are different things for the reasons that Arvind has said. I do think that in practice, they're going to tend to come together for reasons illustrated in AI 2027. I mean, I think just concretely, if you let the companies automate their research and get super intelligences. That's, I mean, like Arvind said, that's, you don't want to be in that situation in the first. Like, if you get to that position, maybe you've already messed up because now they're sitting on this incredibly capable, incredibly valuable thing, and they're going to be able to credibly lobby the government and say things like, China's building their own version and China is going to like let it out into the economy and have it transform things. And, you know, rapidly build up military capabilities and so forth.

• 31:28Speaker 03

  And so if you don't want to lose to China, then you have to let us do it over here on our side of the Pacific. Like, this is, this is a prediction about not the technology, but about the, the social dynamics. We are predicting politically that that's what the companies are going to be saying. And that's what the government is going to agree to, at least by default. And so that means they will end up with the power once shortly after they have the capabilities, if that makes sense. And to what Arvind, you were saying earlier, like, I think, yeah, like in a sense, I agree with you that like, if you're relying on alignment, you've already messed up as a civilization. Like, if you're getting to the point where you're like, yeah, we're going to have the army of super intelligences. And it's going to be like autonomously doing all this stuff in the economy. And the government's going to be supporting it rather than like, rather than trying to fight it.

• 32:12Speaker 03

  And so that's why we have to make sure that it has the right values because, because otherwise it could go very badly for us. Like, in some sense, you've already taken a ton of risk by the time you're putting yourself in that point. And you've like, you've sort of like played the board game into like a very difficult to win position. I think, I, I think I agree with you, with you there. And I, here's maybe the thing to say is like AI 2027 is a prediction, it's not a recommendation. Like, this is not what we think should happen.

• 32:37Speaker 02

  Yeah, yeah, I totally agree that the AI companies are going to be lobbying for all kinds of things that, again, I want to keep using this language, are incompatible with a democracy. And so I think that's the thing we should be trying to stop, and we should not in any way take it for granted. And I don't think the AI companies are as powerful as we make them out to be. When we look at the numbers, especially in the US, on the extent to which people are apprehensive about AI as opposed to excited about AI, I think it's very clear that if the companies make a play for this kind of authoritarian partnership between big tech and the government, we will be able to mobilize the sufficient public

• 33:20Speaker 02

  backlash to that, and we should build the infrastructure for potentially doing that now. Those are some of the things we should be focusing on. And for me, this difference between capability and power is not even primarily a matter of the AI companies themselves. So we make this really big distinction between development and deployment of AI. So the entities that are going to be in charge of those limits on power are going to be the everyday organizations, you know, schools and educational institutions, government agencies, banks, you know, hospitals, legal firms, every other firm in the economy who are going to be using AI in their various sectors in ways that are either with oversight

• 34:03Speaker 02

  and compatible with the principles we want, or are going to be handing over crazy amounts of power to AI systems in the pursuit of, we think, small increases in efficiency. And once again, you know, it doesn't matter if the AI companies themselves are hyping things up and are power hungry, we can ensure that the broad base of organizations and institutions in our society don't fall for this hype and abide by the principles that we think are important.

• 34:35David Hoffman?

  I would like to try and take a stab at really defining the line that separates you two, because there's a lot of agreeing going on about many, many things. And I can't quite put a pin in exactly where I feel there is disagreement. On Daniel's side of things, it feels like there is this demon in a Pandora's box that we need to not allow out of the box. And it is this truth, this object that's there that we need to make sure is contained. And I think on Arvind's side of things, there's just a declining of the existence of the demon in the first place. And that's an okay way of articulating where this line is.

• 35:15David Hoffman?

  But Arvind's fundamental premise, I think, is like, this isn't a demon inside of a Pandora's box. This is a regular web two tech company. This is the next iteration of Facebook, Meta, Twitter, Instagram, just now in the AI age. And they're going to have power and control and influence like we've seen from all the current cohort of AI companies. But it's not fundamentally different from any of the power that we've seen, like Mark Zuckerberg, a crew. And then Daniel is like, no, this is an organism that exists that will have life. It will have a mind of its own. It will escape and that will change the face of Earth forever. And these are our two extremes. And so it's easy for me to identify when they're polar opposites.

• 35:56David Hoffman?

  But I'm still struggling to find this line where you guys actually can't cross each other's, cross into others' territory.

• 36:05Speaker 03

  Well, it sounds like, Arvind, perhaps you just don't think that there's going to be superintelligence that soon. And you also separately think that even if there was, it wouldn't transform the economy that fast. Is that correct?

• 36:16Speaker 02

  Yeah, I agree.

• 36:17Speaker 03

  There we go. That's the answer.

• 36:18Speaker 02

  Again, I mean, I'm skeptical of superintelligence ever existing for some definitions of superintelligence. And the reason why AI is slightly outperforming humans at most tasks to me is not superintelligence, is that there's kind of inversion of cause and effect here in the sense that it, I think, if we ensure our policies and practices are such that humans remain in control and use AI as tools, then those increases in capabilities are going to be effectively increasing what Zaysh and I think of as human intelligence, as opposed to an unaided, technologically unaided biological human versus AI, which we think is the wrong comparison in the first place. So Daniel, where is Arvind wrong here?

• 37:03Speaker 03

  So the definition of superintelligence that I was describing in this conversation is not slightly better than humans at most tasks, but rather slightly better than the best humans at all relevant tasks. And I do think that if you achieve that milestone, then things are going to be quite transformative. I think that, in particular, one of the relevant tasks is operating autonomously as an agent in the world, you know, doing whatever it takes to steer towards your goals. And that's where the analogy is. That's where, I mean, I didn't, I think it's your words, not mine, to say it's like this alien demon or whatever. Those are my words. But like, that's why that analogy is relevant, is that like, that something that meets the definition of superintelligence, as I've described it, is an agent. It's not just a tool, it's something that can instead operate at fully autonomy in

• 37:47Speaker 03

  the world and like, you know, learn as it goes and pursue its goals and so forth. And in fact, is better at doing that than the best humans at that skill, you know? And so it is like an agent that is, you know, comparable to humans in that sense, and in fact, is superior to humans in that sense. And I think that even if we, even if we keep it capped at just slightly better than the best humans at everything, already that would be quite enough to cause this sort of analogy to be correct. Why, why, why? Well, one way of thinking about it is, think about the, it would be sort of like an alien species landing that was like, basically like humans, except just a bit better. And not just better than like the average human, but better than the best humans, you know? If this alien species landed on Earth, even if they weren't like dramatically better

• 38:32Speaker 03

  at anything, but if they're just a little bit better than the very best humans at everything, and there's a million of them, already that would be like, whoa, like, they're gonna create like a colony somewhere, it's gonna be like a really economically productive colony, it's gonna be probably militarily powerful too. And then you realize like, oh wow, like their population doubles every six months? What? Like, whoa, let's play that forward a bit, and in a few years, they'll have more population than humanity. Holy, holy cow, you know, like, like, that's sort of what you get, that's the sort of thing that you'd imagine. Basically, like if it was literally aliens with like, you know, tentacles landing on Earth and they had that level of capability and that population doubling time, it would be like, wow, like in a couple of years, they're going to be a competitor species

• 39:14Speaker 03

  to humanity. And then when you add in the factor that like, actually, I think they're going to be not just like a little bit better than the best humans at everything, but like that as like a floor, but then also much better than humans at many things, you know, then that like only strengthens it. And then I think an additional thing as well is that this whole like political angle too, is that like, I think if aliens landed on Earth, I think there'd be a natural coalition of like humans against the aliens, you know, and people would sort of be naturally hesitant to like put the aliens in charge of the US military effectively and to like, you know, basically have them run all sorts of companies and things like that. Although actually, now that I say that, historically in colonialism,

• 39:56Speaker 03

  analogous things happened all the time, so whatever. But, but like with AIs, it's like, well, I think that the companies are going to tell everybody and tell themselves that like actually they're under control and that like the AIs are just doing what they want, what the humans want them to do.

• 40:11Ryan Sean Adams?

  This is so different. The depiction of aliens landing on Earth being super intelligent, kind of doing whatever they want is so different, Arvin, than normal tech. Like, help us square this.

• 40:24Speaker 02

  Sure. I mean, that's very helpful to me. That helps me pinpoint exactly where I disagree. I think this alien analogy, population doubling, et cetera, all of that is a choice. And I think it's a choice that we will not make, we don't have to make at least.

• 40:37Ryan Sean Adams?

  Wait, Arvin, so you concede it's possible. You concede it's possible, but you think that humans can essentially opt out of that to not let it happen?

• 40:46Speaker 02

  It's exactly the capability is not equal to PowerPoint. So we're talking about capabilities, again, that I think are going to be slightly better than unaided humans and equal to humans operating with the help of AI. And in that scenario, it's entirely a choice to say, we're gonna treat this as an alien colony. And so what that means is that not just let one AI system operate in the world unsupervised, but let

• 41:12Speaker 02

  Entire collaboratives of AI systems operate unsupervised. You know, give them this power, give them control over resources. And more than that, the humans versus AI framing to me, I think is is a deeply dangerous one. Dangerous in the sense that it will bring about a lot of the safety concerns that we are worried about. And the reason for that is, when you talk about it as humans versus AI, you're assuming that all of these AI systems are going to act toward the same goal in collusion with each other. But in fact, we should be designing all our AI systems so that the best defenses against a malfunctioning AI system is another AI system. So we should never get to the point where it's even human versus AI.

• 41:54David Hoffman?

  Imagine if your checking account and DeFi wallet finally spoke the same language. That's Mantle Banking, an all-in-one fiat and crypto account. It lets you save, spend, and invest all from one dashboard. Swipe for coffee, stake ME through yield, or even use virtual cards for payments through Apple Pay. So it feels web two simple, yet stays web three sovereign. For allocators, meet Mantle Index 4, the S and P 500 of crypto. A tokenized institutional-grade fund seeded with $400 million from the Mantle Treasury and balanced across Bitcoin, Ether, Sol, and Yield Enhanced Stables. One asset, broad exposure, pure DeFi composability. The momentum is real. ME faults, FBTC bridges, and a $2.4 billion community treasury are all powering the next phase of on-chain finance. Mantle brings real-world access, yield, and utility to digital assets.

• 42:37David Hoffman?

  Ready for the next era of on-chain finance that actually belongs in 2025? Explore Mantle at mantle.xyz or follow mantleofficial. Mantle, bridging TradFi and DeFi so you don't have to. Imagine verifying yourself without handing over personal data. No hacked databases, no unnecessary personal exposure for airdrops, and no AI bots ruining community governance. Meet SELF, the on-chain identity verification protocol built for privacy and control. SELF protocol uses zero knowledge proofs to confirm your identity safely. Users prove key details like age or citizenship without revealing sensitive personal information. SELF never stores your data, it only generates cryptographic proofs. Here's how it works in three steps. First, register and verify. Use the SELF app to scan your biometric passport's RFID chip.

• 43:22David Hoffman?

  Self verifies authenticity with zero knowledge proofs. Each passport creates one unique identity. Second, you can share proofs privately. Third-party apps request identity proofs, like confirming you're over 18. You can also link proofs securely to public wallets for airdrops or governance participation. And then last, secure verification. Apps validate your proofs instantly on-chain, like on Cello or off-chain. Audited by ZK Security, the SELF app is live on iOS and Play Store. Visit self.xyz and follow SELF Protocol on X. Arvind, with my metaphor of like, there's this demon in the Pandora's box. This demon is super sentient AI that's very powerful. And if it escapes, then it's game over. I think the thing that you contest is this notion that this Pandora's box is confusing

• 44:06David Hoffman?

  and uncontrollable to humans. And I think what you would say is like, actually, no, it's a totally reasonable box that we can think about and discuss and reason about. And we do have control over that. And I think Daniel would say is like, we actually have less control over the nature of the box that constrains AI than we think that we do. Would you agree with that characterization?

• 44:28Speaker 02

  Yes, so sorry, let me try to figure out what this analogy actually means in terms of the technical specifics that I'm more used to thinking about. So, yeah, I mean, there is a sense in which AI is unexplainable and uninterpretable. And sorry to take this to a more technical discussion, but it helps me make sure that I'm not misunderstanding the analogy and misrepresenting my own views because of that. And there are concerns like deceptive alignment. AI might pretend to be aligned when we're testing it, but then when it's deployed, it might act very differently. But what we're seeing is that, yes, those concerns are already coming up with, even with, you know, current far below super intelligent AI systems. We don't even have to wait for the future.

• 45:11Speaker 02

  But actually, our best defenses against that are other AI tools in order to tease out this kind of behavior, right? And so even if we're not the ones building the demon, if you will, neuron by neuron, you know, it's emerging out of the training process. We are in control of the box and we are in control of the box through the help of other AI systems. And we can ensure that the balance of power stays that way. And we can ensure that as AI capabilities improve, it's making not just the demon more powerful, but the box itself more powerful. So that's one way of bringing it back to your analogy.

• 45:46David Hoffman?

  Daniel, what do you think when you hear that?

• 45:50Speaker 03

  A couple of things. First of all, AI 227, this is going back to something you said earlier. AI 227 doesn't actually depict the AI as escaping. So, I mean, I think that could happen, the AI could escape, but also that's not actually what I was predicting. I want to distinguish between two risks, which I just want to put on the table so that for future parts of this discussion. One is the loss of control stuff, which is like, what if the AIs are misaligned? What if they pretend to be aligned, that sort of thing? But then entirely separate from that, even if you're not worried about that at all, there's the concentration of power stuff of like, who's in charge of all the AIs, you know, and who gets to tell them what to do? And who gets to put what, put values into them, you know? And, okay, having put those on the table, now to answer your question.

• 46:31Speaker 03

  I think control research is great, and it sounds like that's kind of what Arvind is talking about, where even basically you should structure, you should build your system on the assumption that an individual AI system might not be trustworthy. And you should in fact assume that it's untrustworthy and that it's, you know, trying to kill you if it can. And then you should build a system of checks and balances and security permissions and so forth, so that it can't. You know, that seems like a reasonable way to design things. And so there's this whole like sub-literature, which sort of grew out of the alignment literature called AI control, which is people working on like that aspect of things. And I think that's great. I think that's important. And there I would just be like, well, the companies aren't going to do this and they're not going to do it well enough.

• 47:16Speaker 03

  And they're going to lose control, is my prediction. And I think that like in practice, like I think it's possible in principle to have this whole system set up, but it's just not, like the companies aren't really investing that hard into making sure that this whole system of checks and balances is ready and has been debugged and has been like robustified to superintelligence or to anything close to that level of intelligence by the time we have those things. And instead, I think that, well, it's kind of, it's kind of like a, it parallels the whole alignment thing. Like with alignment, there's all sorts of techniques that you can use that maybe will result in more aligned AIs. Things like scalable oversight and like interpretability and so forth. But then like the important thing to ask is like, okay, let's, like what techniques are we actually going to use in practice? And how confident are we that they're actually going to work in practice?

• 48:00Speaker 03

  And similarly with control, I'd be like, yes, impossible. You can have this whole system of checks and balances. But like in practice, during the intelligence explosion, when the companies are automating all their research and so forth, I don't expect them to like invest that heavily in this compared to what they would need to do.

• 48:15Speaker 02

  So can I say one thing to this before you move on to the next point? So this actually highlights another big worldview differences between us because for us, the role of the, what you call AI companies, we call model developers, is much, much less than what it is for you. It doesn't matter to us what the AI companies do or don't do. A lot of this research is happening anyway. It's, it's happening in academia. We're actually compiling a big list of the many and varied kinds of AI control methods. So that's one thing. The companies are not that powerful. And secondly, for us, these control methods have to happen closer to the deployment point. So it's going to be the law firms and the hospitals who are finding these ways of AI

• 48:58Speaker 02

  control that makes sense for their own industries and their own deployment scenario. So that's another reason it doesn't matter that much what OpenAI does or doesn't do. And to give just one practical example of that, even in the, you know, simplest case of deployment of a relatively really innocuous kind of bots compared to the kinds of things we're talking about, we see many cases in the real world of exactly how hard it is for deployers to get away with deploying AI systems without oversight if they're not innovating on these kinds of control methods. A great example is, I think from last year, Air Canada deployed a customer service chatbot. Again, super, super innocuous thing. You know, it didn't have abilities to actually book people's flights or anything like that,

• 49:43Speaker 02

  just to answer questions. But this bot was not very accurate, as you might expect, and gave people incorrect information about loyalty programs or something like that. And the customer lost money because of that. They actually sued the airline and Canada's courts concluded that the airline is actually responsible to make whole the monetary losses that the customer suffered. So this kind of thing is not an isolated example, has been such a serious liability issue for companies that even deploying, again, the simplest dumbest thing like a chatbot, has been going so slowly, right? And therefore, our prediction, and this is consistent with the, with the observations so far, is that deployers, if they start to do-

• 50:25Speaker 02

  Anything meaningful with these systems are going to face such serious liability risks, even under existing law, which I think should be buttressed by future AI regulation, that they're going to be forced to innovate on control methods before they get anywhere with, you know, serious deployments. And one example of such innovation, some of it is technical innovation, but a lot of it is going to have to be non-technical. There is a startup now that's offering insurance for exactly this kind of mistake that might, might be made by AI chatbots. So for us, insurance is an interesting AI control innovation. So those are the kinds of things that are going to be needed and they're, that are actually happening on a pretty massive scale. People don't necessarily seem to realize how much because as Daniel said, control originally started as a subfield of like AI alignment, but it's not anymore.

• 51:10Speaker 02

  If you look at all the control stuff that's happening in various disciplines like human computer interaction, computer security and in the industry, it's actually bigger than alignment. And I think that story deserves to be told and we hope to do that in a follow-up to our paper.

• 51:23Ryan Sean Adams?

  Daniel, Air Canada chatbot, like, you know, it's, it does seem like Arvind is saying every time something goes wrong with AI, basically our existing institutions will hold, you slap it back into place, we'll keep the alien species in the zoo, if we want to use analogies and metaphors and have it live there. Like, why is that so wrong? I mean, that is what we've seen in the past. I know earlier in the conversation you were saying, well, of course, things are going to look like 2025 all the way till 2027 and then something happens. And then we get this AGI moment. That's maybe another subject for discussion if we want to get into it. It seems very much like Arvind doesn't think the AGI milestone matters at all. Yeah, Arvind, you've said this, AGI is not a milestone.

• 52:08Ryan Sean Adams?

  There's no sharp capability threshold that yields instant impact. Whereas Daniel, that seems to be part of the, the premise of AI 2027, right? The AGI line, March 2027, we hit it, we cross it and we start the intelligence explosion clock.

• 52:25Speaker 03

  I mean, I forget if we actually use the term AGI somewhere in the text of AI 2027, but we basically don't. Like, I think instead we break it down into different milestones and then we say it's continuous progress that we're just going to like summarize as this series of milestones. The first milestone being automated coder, second milestone being automated AI researcher. Eventually you get to super intelligence, like, but we agree it's continuous progress. There won't be any particular sharp bright lines. And we depict that progress happening over the course of 2027 in the scenario. First, I want to just focus on understanding Arvind, your position. So it sounds like you're saying companies are mostly going to be properly cautious about integrating AI into their stuff.

• 53:07Speaker 03

  And some of them will be incousious, such as O Canada, and then will pay for it. And then the industry will sort of learn from their mistakes. And this is going to result in, A, market pressure on the AI developers to make more aligned and more controlled AIs. But then, B, separately, even if they're not aligned and controlled by the developer, like, the industry will sort of control them and like only deploy them in ways that like they can't actually cause that much harm. So I'm definitely, there's definitely a huge disagreement with me. So we've identified a crux. I think that like, this is like, kind of true right now. Like, I do agree that like, right now, people are rightly cautious about integrating AI

• 53:50Speaker 03

  into everything, and this is creating this sort of like market force. However, as you yourself have said, there's plenty of companies that aren't cautious and that are integrating things anyway, and we'll see how that plays out. The main thing that I want to say there, which is maybe a disagreement, is that I think that once you've got the army of super intelligences, like, I'm imagining, let's suppose that things go exactly as AI 2027 predicts, which of course isn't going to happen. But like, suppose it's like literally exactly that way. Then like, in 2028, Professor Arvind will be writing op-eds about how we shouldn't trust these super intelligences with all this power and responsibility, and how they shouldn't be like, integrated into the economy, and we shouldn't be like, letting them autonomously run factories and design new types of robots and so forth. But then you'll be shouted down by like, the president and all these other powerful

• 54:35Speaker 03

  people who'll be saying, "We have to beat China." And, you know, like, if we don't do, like, look at what's happening in China. In China, they're deploying their AIs everywhere, and like, you know, and like, it just feels like a political fight that you're gonna lose. And I don't, I'm not saying you should lose the fight. Like, I agree, you know, like, it's dangerous to deploy super intelligences everywhere. But I'm just saying, like, politically speaking, I think the point to intervene is before you have the army of super intelligences, not after it's already been built and the companies are already like, using it to lobby the government and so forth, like... Personally, I'll probably just retire at that point. And I'll probably just like, I'll probably just like officially give up and just like go spend some time with my family or something because I think it's basically hopeless.

• 55:17Speaker 03

  Once, once you've gotten to like, you know, middle of, once, once you're at like, like, like March of 2028 in our scenario, like, I feel like it's basically hopeless at that point. Whereas it sounds like you instead have quite a lot of hope and think that like, even if you let things get that, well, okay, but now this contradicts what you're saying. Early, no, yeah, tell me what you think. I'm confused about where your position on this is.

• 55:39Speaker 02

  Sorry, I don't know what contradiction you identified, but I think one crux of disagreement I can point to is that for us, deploying AI with adequate oversight is not this losing business proposition. I think we are innovating in ways that we're able to ensure AI control in ways that are not simply a human in a loop, rubber stamping, or second guessing every decision. A lot of people intuitively think of AI control like that. And if you do, it seems clear that the more efficient thing is to deploy AI autonomously. But for us, those are, that's not the only option. And secondly, again, this, this whole army of super intelligence is framing, again, I

• 56:21Speaker 02

  keep coming back to, most tasks are like writing and outside of certain areas like AI research itself, which is a purely computational problem, and then maybe, you know, purely robotic tasks, most sectors of the economy, again, if you take law and medicine and things like that, there are such inherent limitations to performance that whatever army you build is only going to give you a slight efficiency improvement. And I think it'll be a, a losing idea for most companies to trade off huge safety risks for those slight efficiency improvements.

• 56:55Speaker 03

  Well, I definitely also disagree about the slightness. I think that like it differs from field to field and perhaps in some fields it'll be slight, but I think in many fields it will be massive and that will cause this huge profits for the AI companies and like, it'll be this crazy thing. Not everywhere, like I agree, in some fields it'll be only a slight advantage, but yeah, I think, trying to go back to the main thing that you were saying. Oh yeah, you're more optimistic about the control stuff on the technical level. You're basically like, it's not just gonna be like trying to put humans in the loop. It'll like actually be. It sounds like you're saying that like, by the time we have, you know, by the time 2027 rolls around, we will have control techniques that allow us to get the benefits of these really smart super intelligences without the risks, as long as they're properly applied, basically.

• 57:39Speaker 03

  Like as long as, is that what you're saying?

• 57:41Speaker 02

  Yeah, I mean, our timelines are longer, right? We don't think it's by 2027, but we also don't think we're gonna have the potential for, you know, super intelligent AI without oversight giving us massive economic efficiencies by 2027.

• 57:55Speaker 03

  Right, but like hypothetically, if if the timelines were shorter than you think, and we do get these very powerful AIs by the end of 2027, you would be more optimistic than me that we could deploy them in a way that's sufficiently controlled, that they don't need to be aligned in order for it to be safe.

• 58:11Speaker 02

  Well, I mean, if it is 2027, then we're massively wrong on the timelines, right? Then we would also have to be massively wrong on the timeline of control research. A lot of that control research is happening. We see that accelerating over the course of the next decade or so. If capability massively accelerates, but control research doesn't accelerate, then yeah, then we're in a bad place.

• 58:30Speaker 03

  Okay, in that case, that's another nice point of agreement, is that like, I also think that if we had another like, you know, 15 years before we got to to super intelligence by the definition that I described, I would feel much more optimistic in general, because I think that things like control research and alignment research would have just generally had a lot more time to cook. And in a bunch of ways, I think the world would be off to a better start if things were happening in 2037 instead of, you know, 2027 or whatever.

• 58:58Speaker 02

  Yeah.

• 58:59Speaker 03

  Yeah, so we agree there.

• 59:00Ryan Sean Adams?

  Throughout the course of this discussion, we've uncovered a number of places where the two of you disagree and a few where you agree. I'm wondering if you could, you know, maybe start to summarize this for our audiences as we move to a close. And one way to do that is to kind of illustrate, I'm gonna move us beyond 2027. Let's let's move to something more interesting, another date, which is like 2030, let's say, five years from now, okay? And so I'm gonna ask you both this question through your lens of prediction in terms of how you're viewing AI and kind of the world and how this all plays out. But basically, what does AI look like by 2030? And what does the world look like? Arvind, first you, in 2030, what are we looking at?

• 59:42Speaker 02

  Sure. So let's start with the economy. I think some jobs are likely to be radically transformed, perhaps will face massive job displacement. But I think for most jobs, we're going to have task by task gradual automation, along with lots of innovations in how to deploy these AI systems safely. We are likely to have a ratcheting up of the geopolitical competition. But I predict and also hope that we are not going to buy in wholesale to the arms race dynamics. Because, look, if I can make a slight digression

• 1:00:22Speaker 02

  here, those arms race dynamics are present even without AI. And that's the reason that war has always happened, where each side is fearful of the other side. And if that's the approach we're going to take to geopolitics, that's a very risky approach, even if you take AI out of the picture, right? So whatever diplomatic methods we've been using throughout the history of war and geopolitics, to be able to tamp down those tensions and to realize that escalation is not in anyone's best interest, we have to apply all of that to the current moment as well, perhaps even more skillfully. And again, I'm cautiously optimistic that that's something we will be able to manage. And then finally, predictions

• 1:01:07Speaker 02

  in terms of other aspects of AI-related disruption. I think challenges to the education system, challenges to people's notion of identity, challenges to what art means, a lot of these things that are not really about the labor market, those are perhaps going to happen on a shorter timescale. And right now, they have been happening to a greater extent than economic disruption has actually been happening. So I would kind of maybe flip, I don't know if Daniel agrees with this perspective or not. But the disruptions that we're going to see quicker are not really related to economic efficiencies, but rather people adopting AI in their personal lives in various ways,

• 1:01:52Speaker 02

  whether as companions that they form social relationships with, or, you know, in the education system. And then you have a crisis of what the heck does college even mean? Are we teaching our kids meaningful things? Similarly, what does, what does art mean? Even back in 2023, I think Hollywood had to contend with, what does this mean for, you know, for movies and for art? And then there was a movement to push back against the use of AI in art. So, so those changes happen very quickly and those are perhaps going to accelerate by 2030. But I did not foresee large scale job replacement, for instance, by 2030.

• 1:02:27Ryan Sean Adams?

  Daniel, how about you? What does AI look like? What does the world look like 2030?

• 1:02:31Speaker 03

  Well, uh, if you go to ai-2027.com, which depicts in great hundreds of pages of detail, this sort of thing. And to be clear, I actually, I actually would guess I'd get on, I've updated towards slightly longer timelines since, since about end of last year, when we were writing the core story that became AI 2027. So now I would aim more towards 2028 instead of 2027, but like,

• 1:02:55Ryan Sean Adams?

  Not a big move there.

• 1:02:56Speaker 03

  Basically the same. So, so like the short answer to your question is like, I expect the world to be utterly radically transformed by 2030, probably. I'm not confident. You can ask me what I would, what I think it would look like if that's not true, you know? And then perhaps I would agree with Arvind that like, like I think, I think that the, I could imagine things being, taking longer than I expect. I could imagine the current sort of like wave kind of petering out and resulting in, you know, lots of downstream applications and lots of cool apps that use large language models. But, but the core dynamo of AI automated AI research just sort of like never really materializing. I could imagine that happening. It's not what I'm betting on though.

• 1:03:31Ryan Sean Adams?

  What does that look like? Can you give for, for people who haven't read 2027? I know there's a lot more detail there and there's like multiple paths we might take. But like for people who haven't gotten into that detail yet, what does 2030? When you say radically different by 2030, I mean, some people are envisioning, yeah, Daniel, I mean, it's radically different after the internet. And then we got, we got phones, smartphones, everything is radically different.

• 1:03:52Speaker 03

  This goes back to the start of this whole thing, like normal technology versus not.

• 1:03:55Ryan Sean Adams?

  Yeah.

• 1:03:56Speaker 03

  Which, by the way, slight tangent, like, I think that like, it's gonna be normal in some ways and not normal in other ways, you know, like nuanced. Fair. Perhaps you agree with that, but anyhow, if things go the way that AI 2027 predicts, which I think they probably will by the end of this decade, if not literally in 2027, maybe afterwards, maybe before, whatever, if things go roughly like that, then basically, it's like, first, you get the automated coders, then you get the automated AI researchers, then you get the super intelligences by the definition just described, then you get even more powerful AIs that are not just like a little bit better than the best humans, but like, you know, maybe, maybe only a little bit better in some domains, but like a lot better in other domains. You continue to improve them, make them more efficient, faster, et cetera. You get millions of them running, et cetera.

• 1:04:40Speaker 03

  They start assisting in the creation of new fabs to produce more chips so that you can have. And at that point, it is starting to look like the sort of aliens thing, the analogy comes, becomes relevant, because it's sort of like, you've got this new species that's better than humans in a bunch of ways, and at least economically is competing with humans, if not militarily. And it's got a population size that's growing really fast because they're producing more chips and more factories and so forth. How long does it take to, to get to the point where, the next milestone I would mention would be the, the sort of like fully autonomous AI robot economy? And what that means is that like, yeah, there'll still be humans around, but like

• 1:05:18Speaker 03

  the collection of factories and automated mines and automated this and automated that, taken as a whole, is self-sustaining. Like, it doesn't, it doesn't like crucially depend on humans for any crucial components of it, such that like hypothetically, if all the humans were just sidelined, it could continue to grow as an economy and it could continue to like build more factories and mine more things and do more research and so forth. And my guess is that that happens like maybe a bit less than a year after we achieve the super intelligence milestone. Maybe it'll take longer, but probably by 2030, it will have happened. So probably by 2030, there'll be all of these special economic zones full to bursting with

• 1:06:00Speaker 03

  all of these newly built mines and factories and ship fabs and data centers. And there might still be humans working there, but the humans will be gradually being replaced by robots of various kinds. And like, as a whole, it'll be as if there was like this new species that is now. Is now just like superior and sort of self sustaining. Now, if we're, if we're lucky, that new species will itself be sort of subordinate to humans and like will be following human orders and so forth. And that's also possible.

• 1:06:28Speaker 02

  Can I just say one quick, sorry, Daniel, I didn't mean to interrupt. I thought you were finished. I mean, so one thing I'll say here is that I think for a lot of AI technologists, and I'm not necessarily saying for you, Daniel. You know, all of this might happen and that it might go in the direction of kind of either utopia or dystopia, utopia where these AIs are safe and they have provided for all our material needs and we can spend all our time on leisure and then the dystopia is catastrophe. But like, that's only in the Silicon Valley bubble. For the vast majority of people, both of these possibilities are dystopian. Almost no one actually wants this. So whatever the economic arguments are, this is only going to be possible

• 1:07:12Speaker 02

  if there is a complete collapse of people's democratic ability to influence the direction of things. Because the public is simply not willing to accept either of the branches of this scenario.

• 1:07:23Speaker 03

  Yeah, let me, let's talk about that. So have you heard of pause AI?

• 1:07:27Speaker 02

  Yeah, of course.

• 1:07:28Speaker 03

  Yeah. So like, I mean, they're basically, I'm not sure if they agree with you, but, but they at least agree on that. Like they're basically like, we just shouldn't do this. Why don't we just stop all this from happening? And, and it sounds like you're saying at least sort of something vaguely similar that like most people in the world, if it was put to them in a vote, we'd basically be like, how about we just don't build all this crazy robot stuff? Is that right? Are you saying?

• 1:07:49Speaker 02

  Well, I mean, how about we don't use taxpayer money to massively subsidize AI companies and create special economic zones, right? So there's a big asymmetry here between intervening in the default course of action and pausing things versus affirmatively, you know, having governments work towards this future by doing all the enabling things that I think you agree needs to happen before this can be realized.

• 1:08:12Speaker 03

  Yeah. So, so excellent. So, so zooming in on that a little bit, this gets back to what I was saying earlier about the like, when will I just give up? And I'm saying like, politically speaking, we could imagine like an alternate version of AI 2027 in which the government basically.

• 1:08:28Speaker 03

  Is immune to the lobbying from the companies and is like, no, of course, we're not going to make special economic zones for you. Yes, we of course, we're going to regulate you, blah, blah, blah, blah, blah. Like you can imagine that sort of alternate trajectory. And then it would take more than a year to transform the economy, you know? And it would like it would, all that sort of stuff would be spread out and much more gradual. However, just as a matter of like politics, I think it's like going to be really tough to like win that political fight. If you let it get to the point where there's already this crazy race to superintelligence between US and Chinese companies. And they've already built superintelligence and they're already like networked heavily with the government. And like, the government has been helping them with security to beat China.

• 1:09:11Speaker 03

  Like, like at that point, I think it's just like, very likely that the government will be like, yes, make the special economic zones. Yes, we'll cut the red tape. Yes, we'll subsidize you, you know, like, like I kind of am expecting the government to be captured basically by the companies at that point. Possibly literally with the help of the AIs. After all, if they're super intelligent, that means they're better at lobbying than the best lobbyists. And they're better at like, you know, persuasion, they're better, like, that certainly can't hurt, you know?

• 1:09:38Ryan Sean Adams?

  I feel like every time I interact with chat GPT-03, right? It's persuading me of something. I've just got a lot of good ideas. I mean, it could be bottom up like that.

• 1:09:47Speaker 03

  Well, whatever. The point is that I'm hoping to intervene before then. Like, I'm hoping to like, not let it get to the point where they've already got the superintelligences. And then we're asking the government to like, make this be a slow transition, you know?

• 1:09:58Speaker 02

  Yeah, yeah. Yeah, I totally agree that if even before superintelligence, if we get to the point where national policy is oriented around the race to AGI, then yeah, I will retire at that point.

• 1:10:14Ryan Sean Adams?

  I'm glad you guys are both not retired yet and able to give us some insights today. I, thank you so much. I think the audience will benefit from this, the entire AI community will as well. Maybe I want to just end with this kind of like bonus question, because this has been in the back of my mind. I can't, I can't really figure this out. I mean, both of you are genuine, informed, intelligent, earnest people, right? Like I Very much believe that about the two of you and many in the space. And what I am perplexed, this is kind of the meta question going to this debate is like, how is there such wide variance of opinion here? But like among informed, intelligent people on your side, Arvind, you're like, hey, this is, this is normal tech. We've been through this before. We've seen it.

• 1:10:58Ryan Sean Adams?

  On Daniel's side, he's like, this is a new alien species. That's going to be super intelligent. We've just unleashed it on the world. How is there such variance? I don't think I've ever seen a debate like this in anything I've ever done, where you have like, all of these very smart, informed people reasoning about this and coming up with widely different conclusions. And the different P dooms and P utopias range from like, literally, Jan Le Cun, 0.01%, all the way to Eliezer Yudkowsky and beyond. It's like going to happen. It's 100%. I don't understand this. Have you guys been able to make sense of this?

• 1:11:35Speaker 03

  It's not so surprising for me. I was trained as an academic philosopher. And there's a saying that like, for all P, there is a philosopher who says P and a philosopher who says not P. So like, it's, I think that like, it's actually just kind of normal for, for human discourse, for this to be the case, that there's like intelligent, smart people who disagree with each other.

• 1:11:57Ryan Sean Adams?

  But like, when the stakes are this high, Daniel?

• 1:11:59Speaker 03

  Why would the stakes being high change that or something? Like it's, the reason why the philosophers disagree is not because, well, they also don't think the stakes are that high, but like, I don't think they're like, charging them up with lots of emotion would like, suddenly make them sing kumbaya. You know?

• 1:12:15Ryan Sean Adams?

  Fair point.

• 1:12:16Speaker 02

  Yeah. If I can also take a stab at this, we actually try to say a few words about this in the concluding section of the paper. We talk about what a worldview is and why worldviews can be so different from each other. One, I think different predictions about the future arise from different interpretations of the present. So I think that's one big factor. So fundamental questions like, is AI currently being adopted rapidly? And is it faster than previous waves of tech adoption like PCs? When they first came out in the seventies or whatever? People disagree radically on that, right? So even on the things that are in principle empirically testable today, there is a big disagreement. And that's at least disagreement that you can potentially minimize

• 1:13:00Speaker 02

  by looking more carefully at the data, talking to each other and so forth. But then when you get to really stuff about the future, there are assumptions that I think are essential to even start to think about these questions and those assumptions are different. A bigger one is our epistemic tools are different. So what weight should we put on which kinds of historical analogies? What is the role of probability estimation, various other epistemic tools? And then values also, I think, start to play into this. And all of these factors kind of reinforce each other. And people are in different epistemic communities. I think a couple of observers have talked about how these are kind of West Coast versus East Coast views.

• 1:13:42Speaker 02

  And for me, you know, there, I mean, I was in Silicon Valley for three and a half years, I was doing startups. And a literal reason why I wanted both physical and intellectual distance from that community was because my modes of thinking were different and I felt more at home here. And so we kind of segregate ourselves into physical and intellectual communities based on our assumptions, values, epistemic tools and so forth. And so it's not, I think, too surprising that you should get divergences as a result of all of that.

• 1:14:11Ryan Sean Adams?

  Well, Arvind, Daniel, thank you so much for airing these divergences publicly so we can all come to our own conclusions. Whatever happens next, it's going to be interesting. Let's just say that we appreciate you coming on.

• 1:14:21Speaker 02

  Yeah, thanks. Good luck, guys.

• 1:14:23Ryan Sean Adams?

  Take care.