#420 — Countdown to Superintelligence

Sam Harris speaks with Daniel Kokotajlo about the potential impacts of superintelligent AI over the next decade. They discuss Daniel's predictions in his essay "AI 2027," the alignment problem, what an intelligence explosion might look like, the capacity of LLMs to intentionally deceive, the economic implications of recent advances in AI, AI safety testing, the potential for governments to regulate AI development, AI coding capabilities, how we'll recognize the arrival of superintelligent AI, and other topics. If the Making Sense podcast logo in your player is BLACK, you can SUBSCRIBE to gain access to all full-length episodes at samharris.org/subscribe http://samharris.org/subscribe. Learning how to train your mind is the single greatest investment you can make in life. That's why Sam Harris created the Waking Up https://www.wakingup.com/makingsense app. From rational mindfulness practice to lessons on some of life's most important topics, join Sam as he demystifies the practice of meditation and explores the theory behind it.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:06Sam Harris?

  Welcome to the Making Sense Podcast. This is Sam Harris. Just a note to say that if you're hearing this, you're not currently on our subscriber feed and will only be hearing the first part of this conversation. In order to access full episodes of the Making Sense Podcast, you'll need to subscribe at samharris.org. We don't run ads on the podcast and therefore it's made possible entirely through the support of our subscribers. So if you enjoy what we're doing here, please consider becoming one.

• 0:36Sam Harris?

  I am here with Daniel Cocatello. Daniel, thanks for joining me.

• 0:39Speaker 01

  Thanks for having me.

• 0:41Sam Harris?

  So we'll get into your background in a second. I just want to give people a reference that is going to be of great interest after we have this conversation. You and a bunch of co-authors wrote a blog post titled AI 2027, which is a very compelling read and we're going to cover some of it, but I'm sure there's there are details there that we're not going to get to. So I highly recommend that people read that. You might even read that before coming back to listen to this conversation. Daniel, what's your background? I mean, we were going to talk about the circumstances under which you left OpenAI, but maybe you can tell us how you came to work at OpenAI in the first place.

• 1:20Speaker 01

  Sure, yeah. So I've been sort of in the AI field for a while, mostly doing forecasting and a little bit of alignment research. So that's probably why I got hired at OpenAI. I was on the governance team. We were making policy recommendations to the company and trying to predict where all of this was headed. I worked at OpenAI for two years and then I quit last year and then I worked on AI 2027 with the team that we hired.

• 1:48Sam Harris?

  And one of your co-authors on that blog post was Scott Alexander.

• 1:53Speaker 01

  That's right.

• 1:53Sam Harris?

  Yeah.

• 1:54Speaker 01

  Yeah.

• 1:54Sam Harris?

  Yeah, it's again very well worth reading. So what happened at OpenAI that precipitated your leaving? And can you describe the circumstances of your leaving? Because I seem to remember you had to walk away. You refused to sign an NDA or a non-disparagement agreement or something and had to walk away for your equity. And that was perceived as both a sign of the scale of your alarm and the depth of your principles. What happened over there?

• 2:24Speaker 01

  Yeah, so this story has been covered elsewhere in greater detail. But the summary is that it wasn't any one particular event or scary thing that was happening. It was more the general trends. So if you've read AI 2027, you get a sense of the sorts of things that I'm expecting to happen in the future. And frankly, I think it's going to be incredibly dangerous. And I think that there's a lot that society needs to be doing to get ready for this and to try to avoid those bad outcomes and to steer things in a good direction. And there's especially a lot that companies who are building this technology need to be doing, which we'll get into later. And not only was OpenAI not really doing those things, OpenAI was sort of not on track to get ready

• 3:09Speaker 01

  or to take these sorts of concerns seriously, I think. And I gradually lost, I gradually came to believe this over my time there and gradually came to think that, well, basically that we were on a path towards something like AI 2027 happening and that it was hopeless to try to sort of be on the inside and talk to people and try to steer things in a good direction that way. So that's why I left. And then with the equity thing, they make their employees, when people leave, they have this agreement that they try to get you to sign, which among other things says that you basically have to agree never to criticize the company again and also never to tell anyone about this agreement, which was the clause that I found objectionable. And if you don't sign, then they take away all of your equity, including your vested equity.

• 3:54Sam Harris?

  That's sort of a shocking detail. Is that even legal? I mean, is invested equity vested equity?

• 3:59Speaker 01

  One of the lessons I learned from this whole experience is it's good to get lawyers. Know your rights, you know? I don't know if it was legal, actually. But what happened was my wife and I talked about it and ultimately decided not to sign, even though we knew we would lose our equity. Because we wanted to have the moral high ground and to be able to criticize the company in the future. And happily, it worked out really well for us because there was a huge uproar. Like, when this came to light, a lot of employees were very upset. You know, the public was upset and the company very quickly backed down and changed the policies. So we got to keep our equity, actually.

• 4:33Sam Harris?

  Okay, good. So let's remind people about what this phrase "alignment problem" means. I mean, I've just obviously discussed this topic a bunch on the podcast over the years, but many people may be joining us relatively naive to the topic. How do you think about the alignment problem? And why is it that some very well-informed people don't view it as a problem at all?

• 5:02Speaker 01

  Well, it's different for every person. I guess working backwards. Well, I'll work forwards. So first of all, what is the alignment problem? It's the problem of figuring out how to make AIs sort of reliably do what we want. It's maybe more specifically the problem of shaping the cognition of the AIs so that they have the goals that we want them to have. They have the virtues that we want them to have, such as honesty, for example. It's very important that our AIs be honest with us. Getting them to reliably be honest with us is part of the alignment problem. And it's sort of an open secret that we don't really have a good solution to the alignment problem right now. Like, you can go read the literature on this. You can also look at the, you know, what's currently happening.

• 5:46Speaker 01

  The AIs are not actually reliably honest. And there's many documented examples of them saying things that we're pretty sure they know are not true, right? So this is a big, open, unsolved problem that we are gradually making progress towards. And right now, the stakes are very low. Right now, we just have these chatbots that, you know, even when they're misaligned and even when they, you know, cheat or lie or whatever, it's not really that big of a problem. But these companies OpenAI, Anthropic, Google DeepMind, some of these other companies as well. They are racing to build superintelligence. You can see this on their website and in the statements of the CEOs. Especially OpenAI and Anthropic have been pretty, have literally said that they are building

• 6:29Speaker 01

  superintelligence and they're trying to build it, that they think they will succeed around the end of this decade or before this decade is out. What is superintelligence? Superintelligence is an AI system that is better than the best humans at everything, while also being faster and cheaper. So if they succeed in getting to superintelligence, then the alignment problem suddenly becomes extremely high stakes. We need to make sure that any superintelligences that are built, or at least the first ones that are built, are aligned. Otherwise, terrible things could happen, such as human extinction.

• 7:04Sam Harris?

  Yeah, so we'll get there. So the leap from having what one person called a, functionally a country of geniuses in a data center, the leap from that to real world risk and something like human extinction is going to seem counterintuitive to some people. So we'll definitely cover that. But why is it, I mean, we have people, I guess some people have moved on this topic. I mean, so forgive me if I'm unfairly maligning anyone, but I remember someone like Yan Lekun over at Facebook, who's obviously one of the pioneers in the field, just doesn't, doesn't give any credence at all to the concept of an alignment problem.

• 7:44Sam Harris?

  Like I, I'm just, I've lost touch with how these people justify that degree of insusions. What, what's your view of, of the skepticism that you meet there?

• 7:54Speaker 01

  Well, it's different for different people. And honestly, like, it would be helpful to have a more specific example of something someone has said for me to respond to. With Yan Lekun, if I remember correctly, for a while, he was both saying things to the effect of AIs are just tools and they're going to be submissive and obedient to us because they're AIs and there just isn't much of a problem here. And also saying things along the lines of, they're never going to be superintelligent or like, you know, the, you know, the current LLMs are not on a path to AGI. They're not going to be able to, you know, actually autonomously do a bunch of stuff.

• 8:30Sam Harris?

  It seems to me that the thinking on that front has changed a lot.

• 8:33Speaker 01

  Indeed. Yan himself has sort of walked that back a bit and is now starting to. He's still sort of like an AI skeptic, but now I think there was a quote where he said something like, we're not going to get to superintelligence in the next five years or something, which is a much milder claim than what he used to be saying.

• 8:52Sam Harris?

  Well, when I started talking about this, I think the first time was around 2016. So nine years ago, I bumped into a lot of people who would say this isn't going to happen for 50 years at least. I'm not hearing increments of half centuries thrown around much anymore. I mean, a lot of people are debating the difference between your time horizon, like two years, three years and, you know, five or 10. I mean, 10 at the outside is what I'm hearing from people who seem cautious.

• 9:23Speaker 01

  Yep, I think that's basically right as a description of what smart people in the field are sort of converging towards. And I think that's an incredibly important fact for the general public to be aware of. And everyone needs to know that the field of AI experts and AI forecasters has lowered its timelines and is now thinking that there is a substantial chance that some of these companies will actually succeed in building superintelligence sometime around the end of the decade or so. There's lots of disagreement about timelines exactly, but that's sort of like where a lot of the opinions are headed towards now.

• 9:57Sam Harris?

  So the problem of alignment is the most grandiose speculative, science fiction inflected version of the risk posed by AI, right? This is the risk that a super intelligent, self-improving autonomous system could get away from us and not have our well-being in its sights or actually be, you know, actually hostile to it for some reason that we didn't put into the AI. And therefore, we could find ourselves playing chess against the perfect chess engine and failing. And that poses an existential threat, which we'll describe. But obviously, there are nearer term concerns that more and more people are worried about.

• 10:40Sam Harris?

  There's the the human misuse of increasingly powerful AI. There there's um, we might call this a containment problem. I I think Mustafa Suleman over at Microsoft used to be a deep mind, tends to think of the problem of containment first that really it's a, you know, aligned or not, as this technology gets more democratized, people can decide to put it to sinister use and which is to say they, you know, use that we would consider unaligned. They can, you know, change the the system level prompt and and make, you know, make these tools malicious as they become increasingly powerful.

• 11:21Sam Harris?

  And it's hard to see how we can contain the spread of of that risk. And um, yeah, I mean, so then there's just the the other issues like, you know, job displacement and and economic and political concerns that are all too obvious. I mean, that's just the spread of misinformation and the political instability that can arise in the context of spreading misinformation and shocking degrees of of wealth inequality that might, you know, initially be unmasked by the growth of this technology. Let's just get into this landscape, knowing that misaligned superintelligence is the kind of the final topic we want to talk about. What is it that you and your co-authors are are predicting?

• 12:06Sam Harris?

  Uh, why did you title your piece AI 2027? What are the next two years on your account hold for us?

• 12:13Speaker 01

  That's a lot to talk about. Um, so the reason why we titled it AI 2027 is because in in the scenario that we wrote, the most important pivotal events pivotal events and decisions happen in 2027. The story continues to 2028, 2029, et cetera. But the most important part of the story happens in 2027. For example, what you might call, what what is called in the literature, AI takeoff happens in AI 2027. AI takeoff is this forecasted dynamic of the speed of AI research accelerating dramatically when AIs are able to do AI research much better than humans. So in other words, when you automate the AI research, probably it will go faster. And there's a question about how much faster it will go, what that looks like, et cetera, when it will eventually asymptote.

• 12:58Speaker 01

  But that whole dynamic is called AI takeoff, and it happens in our scenario in 2027. I should say as a footnote, I've updated my timelines a little bit more optimistic after writing this, and now I would say 2028 is more likely. But broadly speaking, I still feel like it's basically the tracks we're headed on.

• 13:16Sam Harris?

  So when you say AI takeoff, are you, is that synonymous with the, the older phrase, an intelligence explosion?

• 13:21Speaker 01

  Basically, yeah.

• 13:22Sam Harris?

  Yeah. That, I mean, that phrase has been with us for a long time, since the mathematician, IJ Good, I think in the 50s, you posited this, just extrapolated, you know, from the general principle that once you had machines, intelligent machines devising the next generation of intelligent machines, that this process could be self-sustaining and asymptotic and get away from us, and he dubbed it this, an intelligence explosion. So this is mostly a story of, of software improving software. I mean, it's the, the AI at this point doesn't yet have its hands on, you know, physical factories at building new chips or robots.

• 14:00Speaker 01

  That's right.

• 14:01Sam Harris?

  Yeah.

• 14:02Speaker 01

  Yeah, so, I mean, and this, and this is also another important thing that I think, that I would like people to think about more and understand better, is that I think that, at least in our view, most of the important decisions that affect the fate of the world will be made prior to any massive transformations of the economy due to AI. And if you want to understand why, or how, or why we mean that, et cetera, well, it's all laid out in, in our scenario, you can sort of see the events unfold, and then you sort of, after you finish reading it, you can be like, oh yeah, I guess like, the world looked kind of pretty normal in 2027, even though, you know, behind closed doors at the AI companies, all of these incredibly impactful decisions were being made about automating AI research and producing superintelligence and so forth. And then in 2028, things are going crazy in the real world, and there's all these new

• 14:47Speaker 01

  factories and robots and stuff being built orchestrated by the superintelligences. But in terms of like, where to intervene, you don't want to wait until the superintelligences are already building all the factories. You, you want to like, try to steer things in a better direction before then.

• 15:03Sam Harris?

  Yeah, so in your piece, I mean, it is a kind of a piece of speculative fiction in a way, but it's all too plausible. And what's interesting is just the, some of the disjunctions you point out. I mean, like the moments where the economy is actually, for real people, is probably being destroyed because people are becoming far less valuable. There's another blog post that perhaps you know about called "The Intelligence Curse",

• 15:28Speaker 01

  Yes.

• 15:28Sam Harris?

  which goes over some of this ground as well, which I recommend people look up. But that's really just a name for this principle that once AI is better at virtually everything than people are, right? Once it's all analogous to chess, the value of people just evaporates from the point of view of companies and even governments, right? I mean, people are not necessary for, because they can't add value to any process that's running the economy, or the most important process is that they're running the economy. So there's interesting moments where the stock market might be booming, but the economy,

• 16:09Sam Harris?

  for most people, is actually in freefall. And then you get into the implications of an arms race between the US and China, and it's all, it's very, it's all too plausible. The moments you admit that we are in this arms race condition, and an arms race is precisely the situation wherein all the players are not holding safety as their top priority.

• 16:34Speaker 01

  Yeah. Unfortunately, I don't think it's good that we're in an arms race, but it does seem to be what we're headed towards, and it seems to be what the companies are also pushing along, right? If you look at the rhetoric coming out of the lobbyists, for example, they talk a lot about how it's important to beat China, and how the US needs to maintain its competitive advantage in AI and so forth. I mean, more generally, like, it's kind of, like, I'm not sure what the best way to say this is, but basically... A lot of people at these companies, building this technology, expect something more or less like AI 2027 to happen and have expected this for years. And like this is what they are building towards and they're doing it because they think if we don't do it, someone else will do it worse.

• 17:19Speaker 01

  And, you know, they think that they think it's going to work out well.

• 17:23Sam Harris?

  Do they think it's going to work out well or they just think that there is no alternative? Because we have a coordination problem we can't solve. I mean, Anthropic, if Anthropic stops, they know that OpenAI is not going to stop. They can't agree, you know, all the, all the, the US players can't agree to stop together. And even if they did, they know that China wouldn't stop. Right? So it's just this, it's a coordination problem. They can't be solved, even if everyone agrees that in an arms race condition, it could likely, with, with some significant probability. I mean, maybe, maybe it's only 10% in some people's minds, but it's still a non-negligible probability of birthing something that destroys us.

• 18:04Speaker 01

  Yeah, my, my take on that is it's both. So I think that, you know, I have lots of friends at these companies and, you know, I used to work there and I talked to lots of people there all the time. In my opinion, I think on average, they're overly optimistic about where all this is headed. Perhaps because they're biased, because their, their job depends on them thinking it's a good idea to do all of this. But also separately, there is this very, there is both a real like arms race dynamic, where it just really is true that if, if one company decides not to do this, then other companies will probably just do it anyway. And it really is true that if one country decides not to do this, other countries will probably do it anyway. And then there's also an added element of perceived dynamic there,

• 18:45Speaker 01

  where a lot of people are basically not even trying to coordinate the world to handle this responsibly and to put guardrails in place or to slow down or whatever. And they're not trying because they basically think it's hopeless to, to achieve that level of coordination.

• 18:59Sam Harris?

  Well, you mentioned that, that the LLMs are already showing some deceptive characteristics. I mean, I guess we might wonder whether what is functionally appearing as deception is really deception. I mean, really motivated in any sense that we would, you know, whether, whether we're guilty of anthropomorphizing these systems by calling it lying or deception. But what's the behavior that we have seen from some of these systems that, that we're calling lying or cheating or deception?

• 19:29Speaker 01

  Yeah, great question. So there's a couple of things. Key words to search for are sycophancy, reward hacking, and scheming. So there's various papers on this, and there's even blog posts by OpenAI and Anthropic detailing some examples that have been found. So sycophancy is an observed tendency of many of these AI systems to basically suck up to or flatter the humans they're talking to, often in ways that are just extremely over the top and egregious. And, you know, we don't know for sure.

• 19:59Sam Harris?

  If you'd like to continue listening to this conversation, you'll need to subscribe at samharris.org. Once you do, you'll get access to all full-length episodes of the Making Sense podcast. The Making Sense podcast is ad-free and relies entirely on listener support, and you can subscribe now at samharris.org.