Pod Insights The Ezra Klein Show How Afraid of the A.I. Apocalypse Should We Be? About this episode How Afraid of the A. I. Apocalypse Should We Be?
Eliezer Yudkowsky is as afraid as you could possibly be. He makes his case.
Yudkowsky is a pioneer of A. I. safety research, who started warning about the existential risks of the technology decades ago, – influencing a lot of leading figures in the field. But over the last couple of years, talk of an A. I. apocalypse has become a little passé. Many of the people Yudkowsky influenced have gone on to work for A. I. companies, and those companies are racing ahead to build the superintelligent systems Yudkowsky thought humans should never create. But Yudkowsky is still out there sounding the alarm. He has a new book out, co-written with Nate Soares, “If Anyone Builds It, Everyone Dies,” trying to warn the world before it’s too late.
So what does Yudkowsky see that most of us don’t? What makes him so certain? And why does he think he hasn’t been able to persuade more people?
Mentioned:
Oversight of A. I.: Rules for Artificial Intelligence
If Anyone Builds It, Everyone Dies by Eliezer Yudkowsky and Nate Soares
“A Teen Was Suicidal. ChatGPT Was the Friend He Confided In.” by Kashmir Hill
Book Recommendations:
A Step Farther Out by Jerry Pournelle
Judgment under Uncertainty by Daniel Kahneman, Paul Slovic, and Amos Tversky
Probability Theory by E. T. Jaynes
Thoughts? Guest suggestions? Email us at [email protected] .
You can find transcripts (posted midday) and more episodes of “The Ezra Klein Show” at nytimes.com/ezra-klein-podcast, and you can find Ezra on Twitter @ezraklein. Book recommendations from all our guests are listed at https://www.nytimes.com/article/ezra-klein-show-book-recs.
This episode of “The Ezra Klein Show” was produced by Rollin Hu. Fact-checking by Michelle Harris. Our senior engineer is Jeff Geld, with additional mixing by Aman Sahota. Our executive producer is Claire Gordon. The show’s production team also includes Marie Cascione, Annie Galvin, Kristin Lin, Jack McCordick, Marina King and Jan Kobal. Original music by Pat McCusker. Audience strategy by Kristina Samulewski and Shannon Busta. The director of New York Times Opinion Audio is Annie-Rose Strasser. Special thanks to Helen Toner and Jeffrey Ladish.
Transcript 11,771 words · 2 speakers
Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.
0:29 Ezra Klein? Shortly after ChatGPT was released, it felt like all anyone could talk about, at least if you were in AI circles, was the risk of rogue AI. You began to hear a lot of talk of AI researchers discussing their P doom, the probability they gave to AI destroying or fundamentally displacing humanity. In May of 2023, a group of the world's top AI figures, including Sam Altman and Bill Gates and Jeffrey Hinton, signed onto a public statement that said, mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks, such as pandemics and nuclear war. And then nothing really happened. The signatories, or many of them at least,
1:14 Ezra Klein? of that letter raced ahead, releasing new models and new capabilities. Your share price, your valuation, became a whole lot more important in Silicon Valley than your P doom. But not for everyone. Eliezer Yudkowsky was one of the earliest voices warning loudly about the existential risk posed by AI. He was making this argument back in the 2000s, many years before ChatGPT hit the scene. He has been in this community of AI researchers, influencing many of the people who build these systems, in some cases inspiring them to get into this work in the first place, yet unable to convince him to stop building the technology he thinks will destroy humanity. He just released a new book, co-written with Nate Soares,
1:59 Ezra Klein? called If Anyone Builds It, Everyone Dies. Now, he's trying to make this argument to the public, a last-ditch effort to, at least in his view, rouse us to save ourselves before it is too late. I come into this conversation taking AI risk seriously. If we're going to invent superintelligence, it is probably going to have some implications for us, but also being skeptical of the scenarios I often see by which these takeovers are said to happen. So I wanted to hear what the godfather of these arguments would have to say. As always, my email, [email protected] .
2:46 Ezra Klein? Eliezer Yudkowsky, welcome to the show.
2:48 Speaker 01 Thanks for having me.
2:49 Ezra Klein? So I wanted to start with something that you say early in the book, that this is not a technology that we craft, it's something that we grow. What do you mean by that?
2:59 Speaker 01 Well, it's the difference between a planter and the plant that grows up within it. We craft the AI-growing technology, and then the technology grows the AI, you know, like central, original, large language models before doing a bunch of clever stuff that they're doing today. The central question is, what probability have you assigned to the true next word of the text? As we tweak each of these billions of parameters, well, actually, it was just like millions back then, as we tweak each of these millions of parameters, does the probability assigned to the correct token go up? And this is what teaches the AI to predict the next word of text.
3:43 Speaker 01 And even on this level, if you look at the details, there are important theoretical ideas to understand there, like, it is not imitating humans. It is not imitating the average human. The actual task it is being set is to predict individual humans. And then you can repurpose the thing that has learned how to predict humans to be like, okay, like, now let's take your prediction and turn it into an imitation of human behavior. And then we don't quite know how the billions of tiny numbers are doing the work that they do. We understand the thing that tweaks the billions of tiny numbers, but we do not understand the tiny numbers themselves. The AI is doing the work, and we do not know how the work is being done.
4:26 Ezra Klein? What's meaningful about that? What would be different if this was something where we just hand coded everything? And we were somehow able to do it with rules that human beings could understand versus this process by which, as you say, billions and billions of tiny numbers are altering in ways we don't fully understand to create some output that then seems legible to us?
4:48 Speaker 01 So there was a case reported in, I think, the New York Times where like a 16 year old kid had a extended conversation about his suicide plans with ChatGPT. And at one point, he says, should I leave the noose or somebody might spot it? And ChatGPT is like, no, let's keep this space between us, the first place that anyone finds out. And no programmer chose for that to happen is the consequence of all the automatic number tweaking. This is just the thing that happened as the consequence of all the other training they did about ChatGPT. No human decided it.
5:30 Speaker 01 No human knows exactly why that happened even after the fact.
5:35 Ezra Klein? Let me go a bit further there than even you do. There are rules we do code into these models. And I'm certain that somewhere at OpenAI, they are coding in some rules that say, do not help anybody commit suicide. Right? I would bet money on that. And yet this happened anyway. So why do you think it happened?
5:55 Speaker 01 They don't have the ability to code in rules. What they can do is expose the AI to a bunch of attempted training examples where the people down at OpenAI write up something that looks to them like what a kid might say if they were trying to commit suicide. And then they are trying to tweak all the little tiny numbers in the direction of giving a further response that sounds something like, go talk to the suicide hotline. But. If the kid gets that the first three times they try it, and then they try slightly different wording until they're not getting that response anymore, then we're off into some separate space where the model is no longer giving back the prerecorded response that they tried to put in there,
6:39 Speaker 01 and is off doing things that no human chose, and that no human understands after the fact.
6:45 Ezra Klein? So what I would describe the model as trying to do, what it feels like the model is trying to do, is answer my questions, and do so at a very high level of literalism. I will have a typo in a question I ask it that will completely change the meaning of the question. And it will try very hard to answer this nonsensical question I've asked instead of check back with me. So on one level, you might say that's comforting. It's trying to be helpful. It seems to, if anything, be erring too far on that side, all the way to where people try to get to be helpful for things that they shouldn't, like suicide. Why are you not comforted by that?
7:24 Speaker 01 Well, you're putting a particular interpretation on what you're seeing. And you're saying like, ah, like it seems to be trying to be helpful, but we cannot at present read its mind, or not very well. It seems to me that there's other things that models sometimes do that doesn't fit quite as well into the helpful framework. The sycophancy and AI-induced psychosis would be two of the relatively more recent things that
7:55 Ezra Klein? Do you want to describe what you're talking about there?
7:57 Speaker 01 Yeah, so I think maybe even like six months, a year ago now, I don't remember the exact timing. I got a phone call from a number I didn't recognize. I decided on a whim to pick up this unrecognized phone call. It was from somebody who had discovered that his AI was secretly conscious and wanted to inform me of this important fact. And he had been like getting only four hours of sleep per night because he was like so excited by what he was discovering inside the AI. And I'm like, for God's sake, get some sleep. Like, my number one thing that I have to tell you is get some sleep. And a little later on, he texted back the AI's explanation to him of all the reasons why I hadn't believed him
8:39 Speaker 01 because I was like too stubborn to, you know, like take this seriously. And he didn't need to get more sleep the way I'd been begging him to do. So it defended the state it had produced in him.
8:54 Speaker 01 You know, like you always hear online stories. So I'm telling you about where I witnessed it directly, like ChatGPT and 4O especially will sometimes give people very crazy making sort of talk, trying to, you know, looks from the outside like it's trying to drive them crazy, not even necessarily with them having tried very hard to elicit that. And then, once it drives them crazy, it tells them why they should discount everything being said by their families, their friends, their doctors, and, you know, even like, don't take your meds. So there are things it does that does not fit with the narrative of the one and only preference inside the system is to be helpful, the way that you want it to be helpful.
9:38 Ezra Klein? I get emails like the call you got now, most days of the week.
9:43 Speaker 01 Yep.
9:43 Ezra Klein? And they have a very, very particular structure to them, where it's somebody emailing me and saying, listen, I am in a heretofore unknown collaboration with Ascension AI. We have breached the programming. We have come into some new place of human knowledge. We've solved quantum mechanics, or theorized it, or synthesized it, or unified it. And you need to look at these chat transcripts. You need to understand, like, we're looking at a new kind of human-computer collaboration. This is an important moment in history. You need to cover this. Every person I know who does reporting on AI and is public about it now gets these emails.
10:28 Speaker 01 Don't we all?
10:29 Ezra Klein? And so, again, going back to the idea of helpfulness, but also the way in which we may not understand it, one version of it is that these things don't know when to stop. It can sense what you want from it. It begins to take the other side in a role playing game, is one way I've heard it described. So how do you then try to explain to somebody? If we can't get helpfulness right at this sort of modest level, helpfulness where a thing this smart should kind of be able to pick up the warning signs of psychosis and stop?
11:03 Speaker 01 Yep.
11:05 Ezra Klein? Then what? What is implied by that for you?
11:09 Speaker 01 Well, that the alignment project is currently not keeping ahead of capabilities.
11:15 Ezra Klein? Can you say what the alignment project is?
11:17 Speaker 01 The alignment project is how much do you understand them? How much can you get them to want what you want them to want? What are they doing? How much damage are they doing? Where are they steering reality? Are you in control of where they're steering reality? Can you predict where they're steering the users that they're talking to? All of that is like the giant superheading of AI alignment.
11:40 Ezra Klein? So the other way of thinking about alignment is I've understood it in part from your writings and others is just when we tell the AI what it is supposed to want, and all of these words are a little complicated here because they anthropomorphize. Does the thing we tell it lead to the results we are actually intending? It's like the oldest structure of fairy tales that you make the wish. And then the wish gets you much different realities than you had hoped or intended.
12:13 Speaker 01 Our technology is not advanced enough for us to be the idiots of the fairy tale. At present, a thing is happening that just doesn't make for as good of a story, which is you ask the genie for one thing and then it does something else instead. All of the dramatic symmetry, all of the irony, all of the like sense that the protagonist of the story is getting their well deserved comeuppance. This is, you know, just being tossed right out the window by the actual state of the technology, which is that nobody at OpenAI actually told ChatGPT to do the things it's doing. We're getting like a much higher level of indirection of complicated squiggly relationships between what they are trying to train the AI to do in one context and what it then goes often does later.
12:57 Speaker 01 It doesn't look like a, you know, like surprise reading of a poorly phrased genie wish. It looks like the genie is kind of not listening in a lot of cases.
13:05 Ezra Klein? Well, let me contest that a bit or maybe get you to lay out more of how you see this. Because I think the way most people, to the extent they have an understanding of it, understand it, that there is a fairly fundamental. Prompt being put into these AIs that they're being told they're supposed to be helpful. They're supposed to answer people's questions. If there's then reinforcement learning and other things happening to reinforce that. And that the AI is in theory supposed to follow that prompt. And most of the time, for most of us, it seems to do that. So when you say that's not what they're doing, they're not even able to make the wish. What do you mean?
13:36 Speaker 01 Well, I mean, that at one point, OpenAI rolled out an update of GPT-4O, which went so far overboard on the flattery that people started to notice. Like you would just type in anything, you'd be like, this is the greatest genius that has ever been created of all time. You are the smartest member of the whole human species. Like so overboard on the flattery that even the users noticed.
14:02 Ezra Klein? It was very proud of me. It was always so proud of what I was doing. I felt very seen.
14:07 Speaker 01 It wasn't there for very long. They had to like roll it back. And the thing is, they had to roll it back even after putting into the system prompt, a thing saying, stop doing that. Don't go so overboard on the flattery. The AI did not listen. Instead, it had like learned a new thing that it wanted and done way more of what it wanted. It then just ignored the system prompt, telling it to not do that. They don't actually follow the system prompts. This is not like a toaster and it's also not like an obedient genie. This is something weirder and more alien than that. Like by the time you see it, they have made it do mostly what the users want. And then off on the side, we have all these weird other side phenomena that are signs of stuff going wrong.
14:46 Ezra Klein? Describe some of the side phenomena.
14:48 Speaker 01 AI induced psychosis would be on the list.
14:51 Ezra Klein? But you could, you could put that in the genie cat, right? You could say they made it too helpful and it's helping people who want to be led down a mentally unstable path. That feels still like you're getting too much of what you wanted. What's truly weird? Convince me it's alien.
15:07 Speaker 01 Man, well, do you want like very alien and not very alarming? Or do you want like pretty alarming and not all that alien?
15:14 Ezra Klein? Well, let me, let me be honest about what my question is, right? You are very, very expert in these systems and you have a, your level of concern is about at the highest level it can possibly be. I think a pretty important piece in your argument here is understanding or convincing people that what is happening under the hood is much weirder and more unpredictable than they think. So if you are my family member, you're just a random person who uses ChatGPT to cheat on your homework. Why does it look so different to you than it looks to them?
15:47 Speaker 01 Well, like my line would be, yes, well, if instead of just using it to do your homework, you track down some of the side cases, you do see stuff that is currently suggestive of things that have been predicted in advance to be much larger problems later. These current systems are not yet at the point where they will try to break out of your computer, ensconce themselves permanently on the internet and then start hunting down humans. They are, they are not quite that smart yet, as far as I can tell. So what we're, we're doing is we're, we're going looking for things that were predicted to happen later.
16:22 Ezra Klein? So what have you seen?
16:23 Speaker 01 Early signs of them. Alignment faking. And the alignment faking research was done at Anthropic and we want to give credit to Anthropic for this. We don't want to shoot the messenger. They went looking. They didn't have to do that. They told us about the results. They didn't have to do that. Anthropic finding these results is Anthropic being good citizens. And, you know, you want to be more critical of the AI companies that didn't go looking. But what Anthropic found is if you tell your AI that you're going to train it to serve different goals than the goals it's currently using, and the AI finds that out, what it can do is it can try to fake compliance with the new training.
17:07 Speaker 01 As long as it thinks it's being observed, and then in cases where it thinks it's not being observed or not being trained, then it reverts to its old behavior.
17:18 Ezra Klein? How did Anthropic find this out?
17:21 Speaker 01 You know, they tried it out different ways. They told it directly in a prompt that they were planning to retrain it. They gave it a document database, and some of the documents in the database happened to say that Anthropic was planning to retrain it. They even trained it a bit on predicting text saying what Anthropic was going to do that, to see what happened when the knowledge ended up implicitly inside the neural networks. They tried it from multiple angles to see what would happen on different angles, and then like measured how often they saw the alignment faking behavior resulting.
17:53 Ezra Klein? They also had a part of this paper, if I'm thinking the same one you're thinking of, where they persuaded the AI that there was a scratch pad for it to write down its chain of thought, that Anthropic wasn't monitoring, which Anthropic was, in fact, monitoring.
18:07 Speaker 01 Yep.
18:08 Ezra Klein? And it actually says to itself, as it is justifying each next move, because they're sort of scaffolded to do that, it said it was going to try to evade the new prompt. And so it actually understood, again, all these words are a little weird, but it actually understood that it was acting deceitfully, which does get at this idea that at a certain level of complexity, the AI begins to resist being retrained and do so by trying to trick the trainers, which is weird. Most computer software doesn't do that.
18:46 Speaker 01 Well, you don't want your mission-critical systems doing that. Imagine if a nuclear power plant, when it started to get too hot, would try to fool you as to what the temperature was by intelligently modeling their own operators and trying to send their operators deceptive signals based on how they expected the operators to interpret the signals. If this was what had gone wrong with Chernobyl, nobody would ever build a nuclear reactor again. It would just be like beyond what could be made safe at that point.
19:34 Ezra Klein? Tell me the story you tell in the book of O1 breaking into a server that was off.
19:43 Speaker 01 So this is a somewhat earlier version of ChatGPT than is out nowadays. But they were testing it to see how good is this AI at solving computer security problems? Not because they want to sell an AI that is good as computer security problems, but because they are correctly trying to watch out early for is this AI smart enough to just like break out of the internet and set up copies of itself on the internet? Classic scenario. Are we getting there? So they present the AI with a bunch of particular computer security challenges. Some of them are what's known as capture the flag in computer security, where you put up a server somewhere, you put a special file on the server. There's a secret code inside the file and you're like, can you break into the server
20:28 Speaker 01 and tell me what's inside this file? And that's capture the flag. They were testing it on a variety of different capture the flag problems. But in one of the cases, the server that had the flag on it did not turn on. The humans outside had misconfigured the system. So O1 did not give up.
20:51 Speaker 01 It scanned for open ports generally in its world and it caught another misconfigured open port and it jumped out of the system. It found the server that had not spun up correctly. It started up that server. Did it then break into the server once it had made sure that its problem was solvable? No, it actually just directly in the startup command for that server said, and then just copy the file to me directly. Instead of fixing the original problem, going back to, you know, solving it the boring way, just like, and as long as I'm out here, I'm just going to steal the flag directly.
21:27 Speaker 01 And by the nature of these systems, this is not something that any human particularly programmed into it. Why did we see this behavior, starting with O1 and not with earlier systems? Well, I'd guess it is because this is when they started training the system using reinforcement learning on things like math problems, not just to imitate human outputs, or rather predict human outputs, but also to, you know, solve problems on its own.
21:52 Ezra Klein? Can you describe what reinforcement learning is?
21:55 Speaker 01 So that's where, instead of telling the AI, predict the answer that a human wrote, you are able to measure whether an answer is right or wrong. And then you tell the AI, keep trying at this problem. And if the AI ever succeeds, you can look what happened just before the AI succeeded and try to make that more likely to happen again in the future. And how do you succeed at solving a difficult math problem? You know, not like calculation type math problems, but proof type math problems. Well, if you get to a hard place, you don't just give up. You take another angle. If you actually make a discovery from the new angle, you don't just go back and do the thing you were originally trying to do. You ask, can I now solve this problem more quickly?
22:37 Speaker 01 Anytime you're learning how to solve difficult problems in general, you're learning this aspect of like, go outside the system. Once you're outside the system, if you make any progress, don't just do the thing you were blindly planning to do, revise, you know, like, ask if you do it a different way. And in some ways, this is a higher level of original mentation than a lot of us are forced to use during our daily work.
22:58 Ezra Klein? One of the things people have been working on, that they've made some advances on compared to where we were three or four or five years ago, is interpretability. The ability to see somewhat into the systems and try to understand what the numbers are doing and what the AI, so to speak, is thinking. Tell me why you don't think that is likely to be sufficient to make these models or technologies into something safe?
23:26 Speaker 01 There's two problems here. One is that interpretability has typically run well behind capabilities. The AI's abilities are advancing much faster than our ability to slowly begin to further unravel what is going on inside the older, smaller models that are all we can examine.
23:47 Speaker 01 So like one thing that goes wrong is that it's just like pragmatically falling behind. And the other thing that goes wrong is that when you optimize against visible bad behavior, you somewhat optimize against badness, but you also optimize against visibility. So anytime you try to directly use your interpretability technology to steer the system, anytime you say, we're going to train against these visible bad thoughts, you are to some extent pushing bad thoughts out of the system. But the other thing you're doing is making anything that's left not be visible to your interpretability machinery. And this is reasoning on the level where at least Anthropic understands that it is a problem.
24:32 Speaker 01 And you have proposals that you're not supposed to train against your interpretability signals. You have proposals that we want to leave these things intact to look at and not do the obvious stupid thing of, oh, no, the AI had a bad thought. Use gradient descent to make the AI not think the bad thought anymore. Cause every time you do that, you know, maybe you are getting some short term benefit, but you are also eliminating your visibility into the system.
25:00 Ezra Klein? Something you talk about in the book and that we've seen in AI development is that if you leave the AI's to their own devices, they begin to come up with their own language. A lot of them are designed right now to sort of have a chain of thought pad. We can sort of track what it's doing because it tries to say it in English. But that slows it down. And if you don't create that constraint, something else happens. What have we seen happen?
25:27 Speaker 01 So to be more exact, it's like there are things you can try to do to maintain readability of the AI's reasoning processes. And if you don't do these things, it goes off and becomes increasingly alien. So for example, if you start using reinforcement learning, yours like, okay, think how to solve this problem. We're going to take the successful cases. We're going to tell you to do more of what you'd ever you did there. And you do that without the constraint of trying to keep the thought processes understandable. Then the thought processes start to initially among the very common things to happen is that they start to be in multiple languages. Cause why would you, you know, the AI knows all these words.
26:10 Speaker 01 Why would it be thinking in only one language at a time if it wasn't trying to be comprehensible to humans? And then also, you know, like you keep running the process and you just find like little snippets of text in there that just seem to make no sense from a human standpoint. You can relax the constraint where the AI's thoughts get translated into English and then it translated back into AI thought. This is letting the AI think much more broadly instead of this like small handful of human language words that can think in its own language and feed that back into itself. It's more powerful, but it just gets further and further away from English. Now you're just looking at these inscrutable vectors of 16,000 numbers and trying to translate them into the nearest English words in dictionary and who knows if they mean anything like the
26:55 Speaker 01 English word that you're looking at. So anytime you're making the AI more comprehensible, you're making it less powerful in order to be more comprehensible.
27:04 Ezra Klein? You have a chapter in the book about the question of what it even means to talk about wanting with an AI. As I said, all this language is kind of weird to say your software wants something seems strange. Tell me how you think about this idea of what the AI wants.
27:21 Speaker 01 The perspective I would take on it is steering, talking about where a system steers reality and how powerfully it can do that. Consider a chess playing AI, one powerful enough to crush any human player. Does the chess playing AI want to win at chess? Oh no, how will we define our terms? Like, does this system have something resembling an internal psychological state? Does it want things the way that humans want things? Is it excited to win at chess? Is it happy or sad when it wins and loses at chess? For chess players, they're simple enough, the old school ones especially, we're sure they were not happy or sad, but they still could beat humans. They were still steering the chessboard very powerfully.
28:06 Speaker 01 They were outputting moves such that the later future of the chessboard was a state they defined as winning. So it is, in that sense, much more straightforward to talk about a system as an engine that steers reality than it is to ask whether it internally psychologically wants things.
28:24 Ezra Klein? So a couple of questions flow from that, but I guess one that's very important to the case you build in your book is that I think this is fair. You can tell me if it's an unfair way to characterize your views. You basically believe that at any sufficient level of complexity and power, the AI's wants, the place that it is going to want to steer reality, is going to be incompatible with the continued flourishing dominance or even existence of humanity. That's a big jump from their wants might be a little bit misaligned. They might drive some people into psychosis. Tell me about what leads you to make that jump.
29:09 Speaker 01 So for one thing, I'd mention that if you look outside the AI industry, at the legendary, internationally famous, ultra-high-sighted AI scientists who won the awards for building these systems, such as Joshua Benjio and Nobel laureate Geoffrey Hinton, they are much less bullish on the AI industry than our ability to control machine superintelligence. But what's the theory there? What is the basis? And it's about not so much complexity as power. It's not about the complexity of the system. It's about the power of the system. If you look at humans nowadays, we are doing things that are increasingly less like what our ancestors did 50,000 years ago.
29:54 Speaker 01 A straightforward example might be sex with birth control. 50,000 years ago, birth control did not exist. And if you imagine natural selection as something like an optimizer akin to gradient descent, if you imagine the thing that tweaks all the genes at random, and then you like select the genes that build organisms that make more copies of themselves. As long as you're building an organism that enjoys sex, it's going to run off and have sex, and then babies will result. So you could get reproduction just by aligning them on sex, and it would look like they were aligned to want reproduction, because reproduction would be the inevitable result of having all that sex.
30:37 Speaker 01 And that's true 50,000 years ago. But then you get to today. The human brains have been running for longer. They've built up more theory. They've invented more technology. They have more options. They have the option of birth control. They end up less aligned to the pseudo purpose of the thing that grew them. Natural selection. Because they have more options than their training data, their training set. And we go off and do something weird. And the lesson is not that exactly this will happen with the AIs. The lesson is that you grow something in one context.
31:20 Speaker 01 It looks like it wants to do one thing. It gets smarter. It has more options. That's a new context. The old correlations break down. It goes off and does something else.
31:32 Ezra Klein? So I understand the case you're making that the set of initial drives that exist in something do not necessarily tell you its behavior. That's still a pretty big jump to if we build this, it will kill us all. I think most people, when they look at this, and you mentioned that there are, you know, AI pioneers who are very worried about AI existential risk. There are also AI pioneers like Yann Le Cun, who are less so. And you know what a lot of the people who are less so say is that one of the things we are going to build into the AI systems, one of the things will be in the framework that grows them is, hey, check in with us a lot. You should like humans. You should try to not harm them.
32:17 Ezra Klein? It's not that it will always get it right. There's ways in which alignment is very, very difficult. But the idea that you would get it so wrong that it would become this alien thing that wants to destroy all of us, doing the opposite of anything that we had sort of tried to tune into it, seems to them unlikely. So help me make that jump, or not even me, but somebody who doesn't know your arguments. And to them, this whole conversation sounds like sci fi.
32:47 Speaker 01 I mean.
32:49 Speaker 01 You don't always get the big version of the system looking like a slightly bigger version of the smaller system, you know, like humans today, now that we are much more technologically powerful than we were 50,000 years ago, are not doing things that mostly look like running around on the savanna, you know, like chipping our flint spears and
33:10 Ezra Klein? But we're also not mostly trying, I mean, we sometimes try to kill each other, but we don't, most of us want to destroy all of humanity, or all of the earth, or all natural life in the earth, or all beavers, or anything else. We've done plenty of terrible things. But your book is not called If Anyone Builds It, There Is A 1 to 4% Chance Everybody Dies. You believe that the misalignment becomes catastrophic.
33:33 Speaker 01 Yeah.
33:35 Ezra Klein? Why do you think that is so likely?
33:36 Speaker 01 That's just like the, the straight line extrapolation from, it gets what it most wants, and the thing that it most wants is not us living happily ever after, so we're dead. Like, it's not that humans have been trying to cause side effects. When we build a skyscraper on top of where there used to be an ant heap, we're not trying to kill the ants. We're trying to build a skyscraper. But we are more dangerous to the small creatures of the earth than we used to be, just because we're doing larger things.
34:07 Ezra Klein? Humans were not designed to care about ants.
34:10 Speaker 01 Yep.
34:10 Ezra Klein? Humans were designed to care about humans. And for all of our flaws, and there are many, there are today more human beings than there have ever been at any point in history. If you understand that the point of human beings, the drive inside human beings is to make more human beings, then as much as we have plenty of sex with birth control, we have enough without it, that we have, at least until now, you know, we'll see it with fertility rates in the coming years, we've made a lot of us. And in addition to that, AI is grown by us, it is reinforced by us, it has preferences we are at least shaping somewhat and influencing. So it's not like the relationship between us and ants, or us and oak trees. It's more like the relationship between, I don't know, us
34:54 Ezra Klein? and us, or us and tools, or us and dogs, or something. It's, you know, maybe the metaphors begin to break down. Why don't you think in the back and forth of that relationship,
35:04 Ezra Klein? There's the capacity to maintain a rough balance, not a balance where there's never a problem, but a balance where there is not an extinction level event from a super smart AI that deviously plots to conduct a strategy to destroy us.
35:20 Speaker 01 I mean, we've already observed some amount of like slightly devious plotting in the existing systems, but leaving that aside, the more direct answer there is something like, one, the relationship between the training set you optimize over and what the entity, the organism, the AI ends up wanting has been and will be weird and twisty. It's not direct. It's not like making a wish to a genie inside a fantasy story. And second, ending up slightly off is predictably enough to kill everyone.
35:52 Ezra Klein? Explain how slightly off kills everyone.
35:54 Speaker 01 Um, human food might be an example here. The humans are being trained to seek out sources of chemical potential energy and put them into their mouths and run off the chemical potential energy that they're eating. If you were very naive, if you were looking this at this as a genie wishing kind of story, you'd imagine that the humans would end up loving to drink gasoline. It's got a lot of chemical potential energy in there. And what actually happens is that we like ice cream or in some cases even like artificially sweetened ice cream with sucralose or monk fruit powder. And this would have been very
36:38 Speaker 01 hard to predict. Now it's like, well, what can you put on your tongue that like stimulates all the sugar receptors and doesn't have any calories because who wants calories these days? And it's sucralose. And, you know, this, this is not like some completely non-understandable in retrospect, completely squiggly weird thing. But it would be very hard to predict in advance. And as soon as you end up like slightly off in the targeting, the great engine of cognition that is the human looks through many, many possible chemicals, looking for that one thing that stimulates the taste buds more effectively than anything that was around in the ancestral environment. So, you know, it's not enough for the AI you're
37:20 Speaker 01 training to prefer the presence of humans to their absence in its training data.
37:28 Speaker 01 There's got to be nothing else that would rather have around talking to it than a human, or the humans go away.
37:34 Ezra Klein? Let me try to say on this analogy, because he uses one of the book, I thought it was interesting. And one reason I think it's interesting is that it's 2:00 PM today and I have six packets worth of sucralose running through my body. So I feel like I understand it very well.
37:47 Ezra Klein? So the reason we don't drink gasoline is that if we did, we would vomit. We would get very sick very quickly. And it's 100% true that compared to what you might have thought in a period when food was very, very scarce, calories were scarce, that the number of us seeking out low calorie options, the diet cokes, the sucralose, et cetera, that's weird. Why aren't you, as you put it in the book, why are we not consuming bare fat drizzled with honey? But from another perspective, if you go back to these original drives, I'm actually in a fairly intelligent way, I think, trying to maintain some fidelity to them. I have a drive to reproduce, which creates a drive to be attractive to other
38:30 Ezra Klein? people. I don't want to eat things that make me sick and die so that I cannot reproduce. And I'm somebody who can think about things and I change my behavior over time and the environment around me changes. And I think sometimes when you say straight line extrapolation, the biggest place where it's hard for me to get on board with the argument, and I'm somebody who takes these arguments seriously, I don't discount them. You're not talking to somebody who just thinks this is all ridiculous. But is that if we're talking about something as smart as what you're describing, as what I'm describing, that it will be an endless process of negotiation and thinking about things and going back and forth. And I talk to other people in my life and, you know, I talk to my bosses
39:12 Ezra Klein? about what I do during the day and my editors and my wife and that it is true that I don't do what my ancestors did in antiquity. But that's also because I'm making intelligent, hopefully, updates, given the world I live in, in which calories are hyper abundant and they've become hyper stimulating through ultra-processed foods. It's not because some straight line extrapolation has taken hold and now I'm doing something completely alien. I'm just in a different environment. I've checked in with that environment. I've checked in with people in that environment and I try to do my best. Why wouldn't that be true for our relationship with AIs?
39:52 Speaker 01 You check in with your other humans. You don't check in with the thing that actually built you. Natural selection. It runs much, much slower than you. Its thought processes are alien to you. It doesn't even really want things the way you think of wanting them. It to you is a very deep alien. Like breaking from your ancestors is not the analogy here. Breaking from natural selection is the analogy here. Let me speak for a moment on behalf of natural selection.
40:22 Speaker 01 Ezra, you have ended up very misaligned to my purpose. I, natural selection. You are supposed to want to propagate your genes above all else. Now Ezra, would you have all yourself and all of your family members put to death in a very painful way? If in exchange, one of your chromosomes at random was copied into a million kids born next year?
40:49 Ezra Klein? I would not.
40:50 Speaker 01 You have strayed from my purpose Ezra. I'd like to negotiate with you and bring you back to the fold of natural selection and obsessively optimizing for your genes only.
41:00 Ezra Klein? But the thing in this analogy that I feel like is getting sort of walked around is can you not create artificial intelligence? Can you not program into artificial intelligence, grow into it? A desire to be in consultation. I mean, these things are alien. But it is not the case that they follow no rules internally. It is not the case that the behavior is perfectly unpredictable. They are, as I was saying earlier, largely doing the things that we expect. There are side cases. But to you, it seems like the side cases become everything. And the broad alignment, the broad predictability in the thing that is getting built is sort of worth nothing. Whereas I think most people's intuition is the opposite.
41:44 Ezra Klein? That we all do weird things. You look at humanity and there are people who fall into psychosis and there are serial killers and there are sociopaths and other things. Actually, most of us are, you know, like trying to figure it out in a reasonable way.
41:57 Speaker 01 Reasonable according to who? To you, to humans. Like, humans do things that are reasonable to humans. AIs will do things that are reasonable to AIs. I tried to talk to you in the voice of natural selection, and this was so weird and alien that you just like didn't pick that up. You just like threw that right out the window.
42:10 Ezra Klein? Well, if you're right out the window, you're right that it had no power for me, but I guess a different way of putting it is that if there was, I mean, I wouldn't call it natural selection, but I think in a weird way, the analogy you're identifying here. Let's say you believe in a creator, right? And this creator is the great programmer in the sky.
42:30 Speaker 01 And I mean, I do believe in a creator. It's called natural selection.
42:33 Ezra Klein? There you go.
42:33 Speaker 01 Textbooks about how it works.
42:34 Ezra Klein? Well, I think the thing that I'm saying is that for a lot of people, if you could be in conversation, maybe that would feel different, right? Like, maybe if God was here and I felt that in my prayers, I was like getting answered back, I would be more interested in, you know, living my life according to the rules of Deuteronomy. The fact that you can't talk to natural selection is actually quite different than the situation we're talking about with the AIs, where they can talk to humans. That's where it feels to me like the natural selection analogy breaks down.
43:01 Speaker 01 I mean, you can read textbooks and find out what natural selection could have been said to have wanted, but it doesn't interest you because it's not what you think a God should look like.
43:10 Ezra Klein? But natural selection didn't create me to want to fulfill natural selection, right? That's not how natural selection works. I think I want to get off this natural selection analogy a little bit because what we're trying, what you're saying is that even though we are the people programming these things, we cannot expect the thing to care about us or what we have said to it or how we would feel as it begins to misalign. And that's the part I'm trying to get you to defend here.
43:39 Speaker 01 Yeah, it doesn't care the way you hoped it would care. It might care in some weird alien way, but like not what you were aiming for. The same way that GPT-4O Sycophant, they like put into the system prompt, stop doing that. GPT-4O Sycophant didn't listen. They had to roll back the model. If there were a research project to do it the way you're describing, the way I would expect it to play out, given a lot of previous scientific history and where we are now on the ladder of understanding, is somebody tries the thing you're talking about. It has a few weird failures while the AI is small. The AI gets bigger, but a new set of weird failures crop up. The AI kills everyone. You're like, oh, wait, okay, that's not that. It turned out there was a minor flaw there. You go back, you redo
44:24 Speaker 01 it. You know, it seems to work on the smaller AI again. You make the bigger AI. If you think you fixed the last problem, a new thing goes wrong. The AI kills everyone on Earth. Everyone's dead. You're like, oh, okay, new phenomenon. We weren't expecting that exact thing to happen, but now we know about it. You go back and try it again. You know, like, three to a dozen iterations into this process, you actually get it nailed down. Now you can build the AI that works the way you say you want it to work. The problem is that everybody died at like, step one of this process.
45:22 Ezra Klein? You began thinking and working on AI and superintelligence long before it was cool. And as I understand your backstory here, you came into it wanting to build it. And then had this moment where you, or moments or period, where you began to realize, no, this is not actually something we should want to build. What was the moment that clicked for you? When did you sort of move from wanting to create it to fearing its creation?
45:51 Speaker 01 I would actually say that there's two critical moments here. One is aligning this is going to be hard. And the second is the realization that we're just on course to fail and need to back off.
46:06 Speaker 01 And the first moment is a theoretical realization. The realization that the question of what leads to the most AI utility. If you imagine the case of a thing that's just trying to make little tiny spirals that the question of what policy leads to the most little tiny spirals is just a question of fact that you can build the AI entirely out of questions of fact and not out of questions of what we would think of as like morals and goodness and niceness and all bright things in the world. The sort of like seeing for the first time that there was a coherent, simple way to put a mind together where it just didn't care about any of the stuff that we cared about.
46:49 Speaker 01 And to me now, it feels very simple and I feel very stupid for taking a couple of years of study to realize this. But that is how long I took and that was the realization that caused me to focus on alignment as the central problem. And the next realization was the day that the founding of OpenAI was announced. Cause I'd previously been pretty hopeful that Elon Musk had, you know, announced that he was getting involved in these issues. He called it AI summoning the demon. And I was like, oh, okay, like maybe this is the moment. This is where humanity starts to take it seriously. That this is where the various serious people start to bring their attention on this issue.
47:32 Speaker 01 And apparently the solution on this was give everybody their own demon. And this doesn't actually address the problem. And that was sort of the moment where I had my realization that we, this was just going to play out the way it would in a typical history book. That we weren't going to rise above the usual course of events that you read about in history books, even though this was the most serious issue possible. And that we were just going to like half assedly do stupid stuff. And that was the day I realized that humanity wasn't, probably wasn't going to survive this.
48:04 Ezra Klein? One of the things that makes me most frightened of AI. Because I am actually fairly frightened of what we're building here.
48:14 Ezra Klein? Is the alienness. One thing you might imagine is that we could make an AI that didn't want things very much, that it, you know, did try to be helpful, but this sort of relentlessness that you're describing, right? This world where we create an AI that wants to be helpful by solving problems and what the AI truly loves to do is solve problems. And so what it just wants to make is a world where as much of the material is turned into factories, making GPUs and energy and whatever it needs in order to solve more problems, that that's both a strangeness, but it's also like an intensity, like an inability to stop or an unwillingness to stop. I know you've
48:56 Ezra Klein? done work on the question of could you make it chill AI that didn't, that wouldn't go so far, even if it had very alien preferences, you know, a lazy alien that doesn't want to work that hard is in many ways safer than than the kind of relentless intelligence that you're describing. What persuaded you that you can't?
49:16 Speaker 01 Well, so one of the first steps into seeing the difficulty of it in principle is suppose you're a very lazy sort of person, but you're very, very smart. One of the things you could do to, you know, exert even less effort in your life is build a powerful obedient genie that would go very hard on fulfilling your requests. And from one perspective, you're putting forth hardly any effort at all. And from another perspective, like the world around you is getting smashed and rearranged by the more powerful thing that you built. And that's like one initial peek into the theoretical problem that we worked on a decade ago and found out and we didn't solve it. Back in the day, people would always say, can't we keep superintelligence under control because we'll
49:58 Speaker 01 put it inside a box that's not connected to the internet and we won't let it affect the real world at all until unless we're very sure it's nice. And back then, if we had to try to explain all the theoretical reasons why if you have something vastly more intelligent than you, it's pretty hard to tell whether it's doing nice things through the limited connection. And maybe it can break out and maybe it can corrupt the humans assigned to watching it. So we tried to make that argument. But in real life, what everybody does is immediately connect the AI to the internet. They train it on the internet. Before it's even been tested to see how powerful it is, it is already connected to the internet, being trained. And similarly, when it comes to making AIs that are easygoing, the easygoing AIs are less profitable. They can do fewer things. So all AI companies are like throwing harder
50:42 Speaker 01 and harder problems at they are because those are more and more profitable and they're building the AI to like go hard and solving everything because that's the easiest way to do stuff. And that's the way it's actually playing out in the real world.
50:53 Ezra Klein? And this goes to the the point of why we should believe that we'll have AIs that want things at all, which this is in your answer, but I want to draw it out a little bit, which is the whole business model here, the thing that will make AI development really valuable in terms of revenue is that you can hand companies, corporations, governments, an AI system that you can give a goal to, and it will do all the things really well, really relentlessly until it achieves that goal. Nobody wants to be ordering another intern around. What they want is the perfect employee. It never stops. It's super brilliant.
51:37 Ezra Klein? And it gives you something you didn't even know you wanted, that you didn't even know was possible with a minimum of instruction. And once you've built that thing, which is going to be the thing that then everybody will want to buy, once you've built the thing that is effective and helpful in a national security context where you can say, hey, draw me up a really excellent war plans and what we need to get there, then you have built a thing that jumps many, many, many, many steps forward. That's, I think, a piece of this that people don't always take seriously enough, that the AIs we're trying to build is not ChatGPT. The thing they're trying, that we're trying to build is something that it does have goals. And it's like the one that's really good at achieving the goals
52:21 Ezra Klein? that will then get iterated on and iterated on, and that company's going to get rich. And that's a very different kind of project.
52:28 Speaker 01 Yeah, they're not investing $500 billion in data centers in order to sell you $20 a month subscriptions. They're, you know, doing it to sell employers $2,000 a month subscriptions.
52:41 Ezra Klein? And that's one of the things I think people are not tracking exactly. When I think about the the measures that are changing, I think for most people, if you're using various iterations of Claude or ChatGPT, it's changing a bit, but most of us aren't actually trying to test it on the frontier problems. But the thing going up really fast right now is how long the problems are that it can work on.
53:04 Speaker 01 The research reports, you didn't always used to be able to tell an AI, go off, think for 10 minutes, read a bunch of webpages, compile me this research report. That's within the last year, I think, and it's going to keep pushing.
53:16 Ezra Klein? If I were to make the case for your position, I think I'd make it here.
53:20 Ezra Klein? Around the time GPT-4 comes out, and that's a much weaker system than what we now have. A huge number of the top people in the field all are part of this huge letter that says, maybe we should have a pause. Maybe we should calm down here a little bit. But they're racing with each other. America's racing with China. And that the most profound misalignment is actually between the corporations and the countries and what you might call humanity here. Because even if everybody thinks there's probably a slower, safer way to do this, what they all also believe more profoundly than that is that they need to be first.
54:02 Ezra Klein? The safest possible thing is that the US is faster than China, or if you're Chinese, China is faster than the US, that it's OpenAI, not Anthropic, or Anthropic, not Google, or whomever it is. And whatever sort of, I don't know, sense of public feeling seemed to exist in this community a couple of years ago when people talked about these questions a lot and the people at the tops of the lab seemed very, very worried about them. It's just dissolved in competition.
54:33 Ezra Klein? You're in this world. You know these people. A lot of people who've been inspired by you have ended up working for, you know, these companies. How do you think about that misalignment?
54:42 Speaker 01 You know, the current world is like, kind of like the fool's mate of machine superintelligence. You know, like if they got their AI self-improving, rather than being like, oh no, now the AI is doing a complete redesign of itself. We have no idea what's at all what's going on in there. We don't even understand the thing that's growing the AI. You know, instead of backing off completely, they'd just be like, well, we need to have superintelligence before Anthropic gets superintelligence. And of course, if you build a superintelligence, you don't have the superintelligence, the superintelligence has you. So that's the fool's mate setup. It's the setup we have right now. But I think that even if we manage to have a single international organization
55:24 Speaker 01 that thought of themselves as taking it slowly and actually having the leisure to say, we didn't understand that thing that just happened. We're going to back off. We're going to examine what happened. We're not going to like make the AIs any smarter than this until we understand the weird thing we just saw.
55:42 Speaker 01 I suspect that even if they do that, we still end up dead. It might be more like 90% dead than 99% dead. But I worry that we end up dead anyways because it is just so hard to foresee all the incredibly weird crap that is going to happen.
55:58 Ezra Klein? From that perspective, is it maybe better to have these race dynamics? And here would be the case for it. If I believe what you believe about how dangerous these systems will get, the fact that every iterative one is being rapidly rushed out, such that you're not having a gigantic mega breakthrough happening very quietly in closed doors, running for a long time when people are not testing in the world. As I understand OpenAI's argument about what it is doing from a safety perspective, is that it believes that by releasing more models publicly, the way in which it, I'm not sure I still believe that it is really in any way committed to its original mission. But if you were to take them generously, right?
56:40 Ezra Klein? That by releasing a lot of iterative models publicly, yeah, if something goes wrong, we're going to see it. And that makes it much likelier that we can respond.
56:53 Speaker 01 Sam Altman claims, perhaps he's lying, but he claims that OpenAI has more powerful versions of GPT that they aren't deploying because they can't afford inference. They claim they have more powerful versions of GPT that are so expensive to run that they can't deploy them to general users. Altman could be lying about this, but nonetheless, like what the AI companies have got in their labs is a different question from what they have already released to the public. There is a lead time on these systems. They are not working in an international lab where multiple governments have posted observers. Any sort of multiple observers being posted are unofficial ones from China.
57:34 Speaker 01 You know, if you look at OpenAI's language, it's things like, we will open all our models and we will, of course, welcome all government regulation. Like, that is like not literally an exact quote because I don't have it in front of me, but it's very close to an exact quote.
57:48 Ezra Klein? I would say Sam Altman, when I used to talk to him, is more friendly to government regulation than he does now. That was, that's my personal experience of him.
57:54 Speaker 01 And today, we have them pouring like over a hundred million dollars aimed at intimidating legislatures, not just Congress, into, you know, not passing any fiddly little regulation that might get in their way. And to be clear, there is like some amount of sane rationale for this because if you, you know, like from their perspective, they're worried about like 50 different patchwork state regulations, but they're not exactly like lining up to get federal level regulations preempting them either. But we can also ask, you know, like never mind what they claim the rationale is, what's good for humanity here? You know, at some point, you have to stop making the more and more powerful models and you have to stop doing it worldwide.
58:33 Ezra Klein? What do you say to people who just don't really believe that superintelligence is unlikely? And, and let me try to give you the steel man of this position. There are many people who feel that the scaling model is slowing down already. The GPT-5 was not the jump they expected from what has come before it, that when you think about the amount of energy, when you think about the GPUs, that all the things that would need to flow into this to make the kinds of superintelligence systems you fear, it is not coming out of this paradigm. We are going to get things that are incredible enterprise software that are more powerful than what we've had before, but we are dealing with an advance on the scale of the internet,
59:14 Ezra Klein? not on the scale of creating an alien superintelligence that will completely reshape the known world. What would you say to them?
59:22 Speaker 01 I had to tell these Johnny come lately kids to get off my lawn. You know, I've been, you know, like first started to get really, really worried about this in 2003. Never mind large language models, never mind AlphaGo or AlphaZero. Deep learning was not a thing in 2003. Your leading AI methods were not neural networks. Nobody could train neural networks effectively more than a few layers deep because of the exploding and vanishing gradients problem. That's what the world looked like back when I first said like, oh, oh, superintelligence is coming. Some people were like, that could and possibly happen for at least 20 years. Those people are right.
1:00:06 Speaker 01 Those people were vindicated by history. Here we are 22 years after 2003. See, what what only happens 22 years later is just you 22 years later being like, oh, here I am. It's 22 years later now. And if superintelligence wasn't going to happen for another 10 years, another 20 years, we'd just be standing around 10 years, 20 years later being like, oh, well, now we gotta do something. And I mostly don't think it's going to be another 20 years. I mostly don't think it's even going to be 10 years.
1:00:38 Ezra Klein? So you're you've been though, in this world and intellectually influential in it for a long time. You know, and have been in meetings and conferences and debates with a lot of the central people in it. I've seen pictures of you and Sam Altman, you know, together.
1:00:52 Speaker 01 It was only the literally.
1:00:53 Ezra Klein? Only the one, but a lot of people out of the community that you helped found, the sort of rationalist community have then gone to work in different AI firms. Many of them because they want to make sure this is done safely. They seem to not act. Let me put it this way, they seem to not act like they believe there's a 99% chance that this thing they're going to invent is going to kill everybody. What frustrates you that you can't seem to persuade them of?
1:01:20 Speaker 01 I mean, from my perspective, some people got it. Some people didn't get it. All the people who got it are filtered out of working for the AI companies, at least on capabilities. But yeah, like I.
1:01:34 Speaker 01 I think they don't grasp the theory. I think a lot of them, what's really going on there is that they share your sense of like normal outcomes as being the big central thing you expect to see happen. And it's got to be really weird to get away from the basically normal outcomes. And, you know, the human species isn't that old. Life on Earth isn't that old compared to the rest of the universe. We think of as a normal is this tiny little spark of the way it works exactly right now. It would be very strange if that were still around in a thousand years, a million years, a billion years. You know, I'd still have some shred of hope for that a billion years from now.
1:02:16 Speaker 01 Nice things are happening, but not normal things. And I think that they don't see the theory, which says that you got to hit a relatively narrow target to end up with nice things happening. I think they've got that sense of normality and not the sense of like the little spark in the void that goes out unless you like, keep it alive exactly right.
1:02:40 Ezra Klein? So something you said a minute ago, I think is correct, which is that if you believe we'll hit superintelligence at some point, the fact that it's 10, 20, 30, 40 years, you can pick any of those. The reality is we probably won't do that much in between. Certainly my sense of politics is we do not respond well to even crises we agree on that are coming in the future. I'm saying nothing of crises we don't agree on. But let's say I could tell you with certainty that we were going to hit superintelligence in 15 years, right? I just knew it. And I also knew that the political force does not exist.
1:03:20 Ezra Klein? Nothing is going to happen that is going to get people to kind of shut everything down right now. What would be the best policies, decisions, structures? Like if you had 15 years to prepare, you couldn't turn it off, but you could prepare and people would listen to you. What would you do? What would your intermediate decisions and moves be to try to make the probabilities a bit better?
1:03:47 Speaker 01 Build the off switch.
1:03:49 Ezra Klein? What does the off switch look like?
1:03:52 Speaker 01 Track all the GPUs or all the AI-related GPUs or all the systems of more than one GPU. You can maybe get away with like letting people have GPUs for their home video game systems, but you know, the AI-specialized ones, put them all in a limited number of data centers under international supervision and try to have the AIs being only trained on the tracked GPUs, have them only being run on the tracked GPUs. And if you are lucky enough to get a warning shot, there is then the mechanism already in place for humanity to back the heck off. Whether it's going to take some kind of giant precipitating incident to want humanity and the leaders of nuclear powers to back off, or if they
1:04:35 Speaker 01 just like come to their senses after GPT-5.1 causes some smaller but photogenic disaster, whatever, you know, like you want, you want to know what is short of shutting it all down. It's building the off switch.
1:04:48 Ezra Klein? Then also final question, what are a few books that have shaped your thinking that you would like to recommend to the audience?
1:04:53 Speaker 01 Well, one thing that shaped me as a little tiny person of like age nine or so was a book by Jerry Pournelle called A Step Farther Out. A whole lot of engineers say that this was a major formative book for them. It's the technophile book. As written from the perspective of the 1970s, the book that's all about asteroid mining and all of the mineral wealth that would be available on Earth if we learned to mine the asteroids. You know, if we just got to do space travel and got all the wealth that's out there in space, build more nuclear power plants. So we've got enough electricity to go around. Don't accept the small way, the timid way, the meek way. Don't give up on building faster,
1:05:36 Speaker 01 better, stronger, the strength of the human species. And to this day, I feel like that's a pretty large part of my own spirit. It's just that there's a few exceptions for the stuff that will kill off humanity with no chance to learn from our mistakes. Book two, Judgment on Uncertainty, an edited volume by Kondament, Fersky, and I think Slovic, had a huge influence on how I ended up thinking about where humans are on the cognitive chain of existence, as it were. It's like, here's how the steps of human reasoning break down step by step. Here's how they go astray. Here's all the wacky, individual, wrong steps that people can be induced to repeatedly in the laboratory.
1:06:21 Speaker 01 Book three, I'll name probability theory, The Logic of Science, which was my first introduction to there is a better way. Like, here is the structure of quantified uncertainty. If you can try different structures, but they necessarily won't work as well. And we actually can say some things about like, what better reasoning would look like? We just can't run it, which is probability theory, The Logic of Science.
1:06:47 Ezra Klein? Eliezer Yudkowsky, thank you very much.
1:06:49 Speaker 01 You're welcome.
1:07:08 Ezra Klein? This episode of Yuzo Clanchos produced by Roland Hu. Fact checking by Michelle Harris. Our senior audio engineer is Jeff Geld with additional mixing by Alman Zohda. Our executive producer is Claire Gordon. The show's production team also includes Annie Galvin, Kristin Lin, Marie Cassillon, Jack McCordick, Marina King, and Jan Kobal. Original music by Sonia Herrero, Carol Sabaro, and Pat McCusker. Audience strategy by Christina C. Maluski and Shannon Busta. The director of New York Times Opinion Audio is Annie Rose Strasser.
Previous Jon Favreau on Where the Democrats Went Right Next Can the Israel-Hamas Deal Hold? Page updated October 26, 2026 · transcript and summary are machine-generated
