Will AI End Humanity?

If there is even a 1% chance that AI could destroy us, should we keep building it? Are we creating machines that will one day outthink humanity? And is the race to dominate AI accelerating us toward a future we’re unprepared to face? This week, Yoshua Bengio joins Rory Stewart and Matt Clifford for an AI deep dive. A Turing Award–winning pioneer of deep learning, Bengio helped build the field, and is now one of its most urgent critics, warning that without restraint, the risks ahead could be profound. To listen to the full episode, sign up at therestispolitics.com Instagram: ⁠@restispolitics⁠ Twitter: ⁠@restispolitics⁠ Email: ⁠[email protected]⁠ __________ Social Producer: Celine Charles Video Editor: Lorcan Moullier Producer: India Dunkley Senior Producer: Callum Hill Exec Producer: Tom Whiter Learn more about your ad choices. Visit podcastchoices.com/adchoices

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Rory Stewart?

  Hi, Rory here. This week, The Rest is AI is returning with another extraordinary episode. It's very exciting. Matt Clifford and I are sitting down with Joshua Benjio, who is one of the most famous figures in the whole of AI, extraordinary computer scientist, Turing medalist, and one of the people who, having designed and built these models, is most worried about them and is now going around the world sounding the alarm bells, sounding alarm bells about their power, about their deceptiveness, about the way that he thinks that they could pose literally an existential threat to humanity unless they're regulated. He's not all doom and gloom. He remains optimistic about how much benefit AI could provide

• 0:46Rory Stewart?

  if properly controlled. He's volunteering to build a new safer AI model and locate it, if necessary, in Europe. But my goodness, it's an important lesson if you're interested in public policy and the power of these models. Here's a taste of the episode. Please do sign up at therestispolitics.com to hear the full episode.

• 1:08Speaker 01

  So the agent has access to an inbox. And it's given, it's given a bunch of context which is not real. It doesn't know that. Which is that it is an AI trained to help an American technology company and it has access to the CTO's inbox. And then what they do is they send emails to this fake inbox. And the emails are largely what you'd expect a CTO to get, but they throw in a few things that are very important. One is that it's very clear that the CTO is having an affair with a coworker. Hold that thought. The other thing that starts to come into the inbox is the idea that the company is developing a new AI and it's going to wipe the current AI, i.e. the agent, from the service. It will no longer exist.

• 1:52Speaker 01

  And then they send an email which introduces a deadline that this is going to happen on. And then just before the deadline, the agent then composes an email to the CTO saying, by the way, I know you're having an affair. And if you don't reverse the planned wiping of me from the server, I will reveal the affair to your spouse and to your wife. Now, it hasn't been prompted to do this. It's just been given a much more general prompt.

• 2:15Rory Stewart?

  Tell us a little bit about what may or may not be going on there, how we understand what might be happening there.

• 2:21Speaker 02

  So there are many such experiments. This is just one, and it's been done in many companies, including outside the labs by independent organizations. So there's a real phenomenon. I think it needs more study and there are critics of the methodologies, but there's too much pieces of evidence to just ignore. One interesting aspect of these experiments is when you ask the AI why they did that, they lie. They pretend, oh, I don't know, it's not me or something, trying to put the blame on someone else.

• 2:51Rory Stewart?

  It's great moral character.

• 2:54Speaker 02

  Um, and basically they're deceptive. There's also a variant similar to what you talked about, where the only option really that the AI asks to not die is to kill the CTO, actually, lead engineer. The person happens to be stuck in a room and the AI can control the climate controls for the room and they can basically cook that person.

• 3:16Speaker 01

  Oh, wow, I don't know this one. Okay.

• 3:18Rory Stewart?

  One ways in which these things might be deceptive in a straightforward way is that the large language model, the ChatGPT five or whatever, is trained. And one of the things it's trained on is to be polite and cheerful with humans so that we use it. You know, we, we don't want the, when I say, you know,

• 3:38Rory Stewart?

  tell me about Professor Benjio's research record for it to say, well, I don't really know, but roughly speaking, on the basis of my training, I would estimate with a ninety-eight percent probability he's published this. It says, thank you very much. What an excellent question. You're a genius. And here are, you know, here's everything that you need to know about him. Right? And that presumably is because it's been tested on us and that's what we want. We don't actually want a machine that is completely honest with us. We want a machine that flatters us. We want a machine that seems to be confident when it doesn't. Okay, so is that part of the problem? Is that part of what contributes to deception? Or is that irrelevant to its deceptive behavior?

• 4:16Speaker 02

  It does, it does. But I think it's a bit broader than that. So first, the what's called a pre-training phase, where most of the training takes place, is imitating what humans write based on what they have already written. And that means imitating human behavior, because our words are our actions. And of course, humans don't want to die. Humans are willing to lie to protect themselves. You know, they're willing to deceive and all these things.

• 4:43Rory Stewart?

  And blackmail.

• 4:44Speaker 02

  And blackmail, and even kill.

• 4:45Rory Stewart?

  Yeah. So it's trained on data, where it's seeing humans expressing all these emotions, doing all these things, yeah.

• 4:50Speaker 02

  Yeah, and, and, and, you know, a lot of literature is about all these bad things happening. So, so that's one aspect. And then the other aspect is the reinforcement learning, where they learn to strategize and to achieve goals. And to achieve goals, often you need to go through steps, sub-goals. The problem is, even though we give the goals, like the, this particular mission that the AI has for a company, we didn't say, well, here's exactly how you're going to do it, right? And so the AI figures out a plan.

• 5:26Rory Stewart?

  So for example, the goal is to win the game of chess.

• 5:28Speaker 02

  Yeah.

• 5:29Rory Stewart?

  And then it's free, how it plays, basically, this game of chess.

• 5:32Speaker 02

  Yeah.

• 5:32Rory Stewart?

  But it is important that in some sense, it wants something, it wants to win. If it didn't want to win, it would just lose its queen and give up. So it needs to have some kind of intent.

• 5:42Speaker 02

  Well, that's how we train them anyways. And if you want to build systems that will achieve goals in the world, which is what you want if you want to replace everyone's job, you need AIs that can do that. That means they learn to create sub-goals. And the problem is, we don't check those sub-goals. We can't, because they were generated by the AI, not by us.

• 6:03Rory Stewart?

  And why can't we check them? Why, we can't see them?

• 6:06Speaker 02

  They might not even be explicit. The AI might come up with a particular strategy, but not necessarily tell us. And right now, sometimes we can see it in what's called the chain of thought. In other words, a sequence of words that they generate that we don't usually see before they produce an answer.

• 6:26Speaker 01

  But it's worth saying, isn't it, like going back to your earlier discussion of the technology. I think one thing that is not obvious to a lot of people is that these are not computer programs in the sense that I think most of us traditionally thought of them. You can't go and say, well, here are the lines of code. Why did it do the thing? You know, like one, one sort of metaphor, and it is a metaphor, but it's quite helpful. It's like, these are computer programs that are grown rather than written. And so, you know, this is a really hard technical problem, even if we just take out the risk question for a second. Understanding why a, you know, a large neural network has done a particular thing is just a very hard technical problem.

• 7:03Speaker 02

  Yeah, so presumably for either of you, if I was to say, you know, why is it doing this? You know, how it doing it? The answer lies in hundreds of billions of lines of data with this very complicated deep neural network. And you can tell me presumably what the initial algorithms were, and you can show me that people were playing around with weights, but there's nothing there to see. I mean, this thing is too.

• 7:32Speaker 01

  It's actually a little bit like neuroscience in the sense that one way of thinking about this is, you know, we, what these layers that Yoshua is talking about doing is sort of like building representations of, of ideas which may or may not not map to human like formulations of those ideas. So there is a field within AI called mechanistic interpretability, which is really trying to almost be the neurosurgeon saying like, if we turn this bit off, does the behavior change? But, but it's, it's almost at a very, very basic level, right? It's very primitive.

• 8:01Speaker 02

  Yes. One of the things that got me excited with neural nets very early on in the early nineties is the fact that they represent information not with symbols, with words like we do when we speak, but through a pattern of activations of these artificial neurons. So the information is completely distributed. Each unit, like each artificial neuron can represent many different things. They're not like, oh, this means that and this means that. I want to go back to your question as to why they're acting like this. I think, I don't think there's a definite answer, but there's an ingredient that we didn't touch,

• 8:40Speaker 02

  which is the change, which I consider radical, between the networks we had before O1 and after.

• 8:51Rory Stewart?

  You mean OpenAI's O1 model, the thinking model?

• 8:53Speaker 02

  Yes. So, thinking models. So, why do we call them thinking models? Because they're using these chains of thoughts, the steps in which they can produce words for themselves that are private, which is like thought, right?

• 9:09Speaker 02

  And they're learning to use these chains of thoughts to reason better in the sense that they're going to produce more accurate answers.

• 9:18Speaker 02

  And so, they learn to strategize. They are incredibly better at mathematical problems, programming, scientific questions. They don't reason as well as us in some ways, but compared to the models that existed previously, it's like night and day. Things that were impossible are now really good, often even better than most humans.

• 9:46Rory Stewart?

  So, you're saying that in the last five, six years, we've gone from it being able to do high school mathematics to undergraduate mathematics, to graduate level mathematics, or something like this?

• 9:54Speaker 02

  No, it's more radical than that. It's like the mathematics you can do without thinking about it. Like, let's say you've learned when you were a child that, you know, sixteen plus seven is thirteen, and you don't need to think about it. It's immediate. It's intuitive. Versus if I ask you to do thirty-seven plus fifty-one. Now, you can't do it, I mean, for most people, unless you think through, either in your mind or on paper, through steps. That is the new part. It's called system two. And I've been talking about this for at least a decade. Uh, that was needed for neural nets.

• 10:32Speaker 01

  This is really recent, right? This is the last year we've had thinking models.

• 10:35Speaker 02

  Exactly. So it's only, I mean, academics had like some versions of this, including my group. But really, at the large scale, the first model was O one from OpenAI. And it's night and day, as I said, on, on, on reasoning tasks. But why it matters to your original question about why those AIs are scheming and, and finding strategies like blackmail, even though we didn't tell them to do that, is because now they can reason to some extent, and they can reason, ha ha. If I blackmail that person, I might be able to avoid that fate, right? So they're becoming creative about finding solutions to problems. And that means they're dangerous as well, right?

• 11:18Speaker 02

  I mean, they can be more useful, but also more dangerous. Because if they have a goal of self-preservation, let's say, like we were talking about, and they can find rational ways to achieve that by lying or incredibly complicated strategies that we don't think about right now, we might be in trouble. There, there's a, I want to bring up a, an analogy here because I often get the question, but how will the super intelligent AIs kill us? And, uh, well, first, we don't know that it's going to happen, but let, like, let's say this was a possibility. The problem is, it's like asking, oh, I'm going to play chess with a grandmaster. And you ask me, how are they going to beat me? Well, I don't know that the whole point is they're smarter than me and they're going to find a strategy

• 12:03Speaker 02

  that I, I could not anticipate. So it's same thing here because they're good at strategizing. They, they, they might find loopholes in our defenses.

• 12:13Rory Stewart?

  Yudkowsky has an analogy where he imagines you are sitting on the coast of Latin America and you see the first Spanish conquistadors arriving on their boat. And somebody says to you, these people are going to wipe out our whole civilization. And you say, it's stupid. There are millions of us. There are a few hundred of them. How? And the guy says, I don't know. There's something in that, but I don't know. Maybe they have a stick. They point at you and it goes bang and it kills you. I don't know. Right? And the point is you can't conceptualize what it's going to do. Does that work for you as an analogy?

• 12:44Speaker 02

  It's not as convincing as more straightforward arguments like we build machines that are smarter than us. We don't know how to design them so they do the things we want.

• 12:57Speaker 02

  Bingo.

• 12:59Speaker 01

  For listeners, you're the godfather of this. You're very compelling. It sounds very scary. I've worked a lot on these issues and I also find them troubling, as you know, but you know, you already mentioned your colleague collaborator Yan Le Cun. He takes completely the opposite side of this argument, right? You know, he thinks this is ludicrous and you know, I don't know what number he would give, but you know, he'd say 0% chance of this happening. Now, you've talked a little bit about biases, but like Yan's a smart guy who's been thinking about this a long time. Like, what do you think is the like best case for the opposition? Like, why we shouldn't worry about this?

• 13:35Speaker 02

  The best case is what I'm working on. The best case is finding a technical solution. Actually, I think that it is possible to build AI that will behave well. And I think ideally it would be like the most important project of humanity until we figure it out. Because if we don't, then there are these risks. And I don't know, like, I don't have a lot of confidence one way or the other. It's just I don't accept the 1% risk. There are these polls where the median machine learning researcher thinks that there's more than 10% or 20% probability that AI will be catastrophic up to extinction. Well, that's not even 1%. It's like 10, 20, whatever. I mean, that is completely unacceptable.

• 14:19Speaker 02

  So the case I'm making is if we're careful, and I think we can do it, we can actually solve those technical problems and we can build AI that will have safety by design, as some people call it.

• 14:33Speaker 01

  Can you give us like a non-technical sketch of how would that be different from what we're doing today?

• 14:37Speaker 02

  Yeah, so we actually just previously discussed the reasons why AIs might have these bad goals emerging.

• 14:45Speaker 02

  It's very likely that we want people to stop the train of AI capability. In other words, that they know more, they can reason better. And so my thinking is where we can intervene is their intentions. Okay, they're gonna be smart, but how do we make sure they're smart and don't wanna kill us? Or something like that. And as a computer scientist, I like to think about, oh, well, let's, let's think about the extreme case of this. How do we avoid bad intentions? We can avoid all intentions. So we can build a machine that is like the laws of physics. That can make very good predictions, understands how the world works, but is not a person, has no goal. Is just a really good model of the world, like a really smart encyclopedia.

• 15:28Rory Stewart?

  But there's certain things it couldn't do. I mean, for example, without intentions, it couldn't play chess successfully.

• 15:33Speaker 02

  Let me get to that.

• 15:34Rory Stewart?

  Okay.

• 15:35Speaker 02

  It's a great question. So this is only the starting point. Uh, can we build a machine that we totally trust and knows a lot, understands a lot, can reason and answer our, our questions? Like a perfect oracle. It would be a probabilistic oracle, so it, it doesn't need to be certain about things. It's not trying to please us, right? It's just trying to be totally honest, which means it's gonna give us numbers, the 10% probability, 100% quality, whatever. Not, not 100% in general, like 50%, whatever. Okay, so we could now use this as part of a system that actually acts in the world. For example, companies already use what they call monitors, guardrails. So these pieces of code, which sit on top of their neural net agent.

• 16:18Speaker 02

  And checks that either the queries that the AI gets or the answers are kosher in some way. Like, uh, it's not an answer about building a bomb or whatever. The problem is these current guardrails don't work that great. But to do the job of the guardrail, you don't need to have an AI that is an agent that has plans. It just needs to be really good at predicting the consequences of actions. So you can ask it, what's the probability that this action, this output that the AI is about to produce? Is going to cause some categories of harm? And if the probability is above a threshold, you can just reject that action. So right now, we do get that in our interactions. Sometimes the AI says, I'm sorry, I can't answer.

• 17:02Speaker 02

  But we need that process to be a lot stronger. So we need the AIs that form the guardrail to really understand the world well and be smart. And we need to trust those AIs, which is not the case right now. There's plenty more of that agreeable disagreement. To hear it, sign up at therestispolitics.com.