The Greatest Threat Humans Just Can’t Stomach with Yuval Noah Harari

Episode 2 is available right now for TRIP Plus members. Sign up at therestispolitics.com What can the Aztec Empire's catastrophic failure to understand Cortés and the Spanish conquistadors teach us about the threat AI poses today? Why does Yuval Noah Harari believe our gut refuses to accept an "alien intelligence" our brains already understand to be a threat? And can Britain avoid becoming an "AI vassal state" - or has that moment already passed? Rory is joined by Professor Yuval Noah Harari for Part 1 of their conversation on existential threats. Part 2 is available now for TRIP Plus members. Producer: India Dunkley Researcher: Evie Townend Video Editor: Josh Smith Social Producer: Celine Charles Executive Producer: Isabella Soames Learn more about your ad choices. Visit podcastchoices.com/adchoices

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:08Rory Stewart?

  Welcome back to Thinking the Unthinkable with me, Rory Stewart. Every week on this podcast, we talk about things that absolutely fascinate me. But they're often the stories of the week. An election in Germany, an earthquake, or a flood in Nepal. Something that's happening in the British Parliament, new prime minister. But there are other things that we don't talk about as much, which may be far, far more important and are much less visible. The things that could actually completely upend our societies, things that could destroy us. And in this series, I'm going to try to go into the heart of some of those things. Some of them are things we used to talk about, but we don't talk about very much anymore. Nuclear war.

• 0:50Rory Stewart?

  Some of them are things that we talk about, but we barely understand yet. AI. And some of it are things that we hear about. Terrorism, ISIS, totalitarianism, but which we don't really understand the texture of, where I'm going to be sitting down with people who've actually lived on the ground in Mosul under ISIS, in Membich, in northeast Syria, as ISIS is coming up, talking about human nature, about totalitarianism, about extremism. And in all of this, I hope we're getting to the real core of what really, in the end, matters about politics. The really biggest things. The things that our minds struggle to predict, that our political systems,

• 1:33Rory Stewart?

  our newspapers, struggle to really talk about, that voters aren't incentivized to raise. Things which are unthinkable. And the question we'll be raising is, how do you think about the unthinkable?

• 1:52Rory Stewart?

  This episode is brought to you by NordVPN.

• 1:55Alastair Campbell?

  And I must thank NordVPN because on my Grand European Tour with Fiona, I was away from the Hawthorns for West Bromwich against Burnley, but I managed to watch it. There you go.

• 2:07Rory Stewart?

  I actually had a NordVPN thank you. I was in the new Starbucks here in Creve. And of course, the thing flashed up, saying this is an unsecured public network. And I was able to go to my VPN. And it was a little reminder of something we often forget when we're in all those cafes and things, which is actually genuinely, people can do all kinds of weird cyber attacks and spying if you don't have a virtual private network when you're sitting in that Starbucks.

• 2:31Alastair Campbell?

  Well, I think, are you not really sort of proud of me that I managed to do this? Or should I say Fiona managed to do this?

• 2:38Rory Stewart?

  I'm incredibly proud of you. That's a tech transformation. So to get the best discount off your NordVPN plan, go to nordvpn.com/racistpolitics. That's nordvpn.com/racistpolitics. And the link is in the episode description.

• 2:56Rory Stewart?

  My guest today is Yuval Noah Harari, one of the most famous public intellectuals alive today, the author of Sapiens. And today, we're going to be asking perhaps the most fundamental question in existential risk. What happens if with AI, we are building an alien intelligence that turns out to basically create a new form of empire? An empire in which AI is the emperor and we are the vassals? What's that going to mean for our economy? What's that going to mean for our politics? Ultimately, what's it going to mean for our minds and our sense of ourselves? And how can we learn from past civilizations on what happens when this incredible rupture occurs? Yuval Noah Harari.

• 3:39Rory Stewart?

  Can we sort of start with the general idea of this thinking the unthinkable? I don't know whether we want to call them existential risks, but what this category of things are and whether you recognize it as a category?

• 3:49Speaker 02

  You know, the purpose of politics, I think, is all about is making choices, is defining challenges and problems that face society, giving the public two, three, five options. And then we have a conversation about it and we need to choose what is the right policy. This is when politics functions well. But for this to happen, the politicians need to focus on the most important problems that face society. And every now and then, something happens in history which is just outside the normal political discourse. That we don't think about it. We don't even have words to describe what it is.

• 4:34Speaker 02

  But it could be the most important next thing. And then the job of politicians is to think the unthinkable. You know, the range of examples from human history is quite wide. I mean, if you're in the Aztec empire and in the early 16th century, and you just heard the reports about this very strange thing that happened on the coastline. These huge hills appeared over the ocean. And out of them came strange men or ghosts or demons. We are not sure what they are. And they have these huge creatures that can run. And they have sticks that can launch lightning and thunder.

• 5:16Speaker 02

  And this is the outside, the normal course of Aztec politics. But this is the most important thing you now need to confront. And the Aztecs did a very bad job confronting that. Because they couldn't really understand what the hell is this? And what should we do about it?

• 5:36Rory Stewart?

  So it's a lovely place to start. So obviously, Yuval, you're describing the arrivals of the conquistadores, the Spanish invaders into Latin America. And the astonishing thing there is that it's a few hundred people arriving into an empire of many millions. And despite the fact that they have horses and simple muskets.

• 5:54Speaker 02

  That couldn't fit anything. I mean, it was just for show.

• 5:57Rory Stewart?

  In practice, it would have been very, very easy for a million people to over...

• 6:01Speaker 02

  Yes, I mean, the tragedy in a way of the Aztec empire, they could have squashed Cortez very, very easily. But they didn't understand what they were facing. They made a lot of mistakes. And these few hundred people, they destroyed the Aztec empire. And of course, with the help of the microbes they brought, annoyingly, they brought all these old world diseases. They decimated the entire population of Central America and beyond. According to some estimates, about up to 90% of the people died. Entire cultures were wiped out. And they never recovered from it. There are, of course, people today in America who are the descendants

• 6:44Speaker 02

  of the Aztecs and the other people of the empire. But they never recovered from it. And really, as you said, it was just a few hundred people. They could have squashed them easily, but they just didn't understand what they were facing.

• 6:56Rory Stewart?

  So I think this is an interesting route through. So you begin as a medieval historian. You then become very interested in these moments of discontinuity. So this is not exactly like the question of why William the Conqueror won at the Battle of Hastings against King Harold, which is a more ordinary engagement, which would have been comprehensible to an Anglo-Saxon or a Norman. Somebody won a battle, they lost a battle. And you then begin to take an interest in these incredible historical moments like that, the arrival of Cortez in Latin America. And the fact that what's hitting you is not just an alien people, it's also an alien technology and something even more unexpected, an alien pandemic.

• 7:33Speaker 02

  Yes.

• 7:33Rory Stewart?

  Now, fast forward, in your later writing, you begin to focus on things like nanotechnology, robotics, artificial intelligence, pandemics as well. What begins to draw you into those sort of issues and categories away from medieval warfare?

• 7:51Speaker 02

  I think it's not away from medieval warfare or away from the Middle Ages. I mean, the Middle Ages are more relevant than ever because modernity is coming to an end. And then the question, what is there other than modernity? Becomes very relevant. And the obvious place to go is the Middle Ages. But of course, we are not going back to the Middle Ages. We are just exiting modernity into something else, which we don't know what it is. But some of the challenges we are now facing remind me of what the Aztec empires faced unsuccessfully back then. And when we talk, and we'll talk a lot about AI, so AI is this alien invasion.

• 8:32Speaker 02

  I mean, for me, AI is the acronym is not artificial intelligence. There is nothing artificial anymore about AI. Artificial brings to mind something that we create and is under our control. But AI is not artificial in this sense. It's really an alien intelligence. It's not alien coming from outer space, of course. But it's totally alien to our way of thinking and understanding the world. It's an inorganic intelligence. We have never in history met anything like it. And the job of politicians today is to do a bit better than Montezuma. And understand what is happening, take it seriously, explain it to the public and

• 9:17Speaker 02

  present choices. What do we do?

• 9:20Rory Stewart?

  To make the case for Montezuma, which, which is an unfashionable thing to do. The problem that Montezuma faces is all the structures of his government and all his historical experience and all his technology and tactics, which have made him the head of the Aztec Empire, it's no small thing. I mean, it's obviously a very nimble, able politician in general, are suddenly confronting something in those boats, which has never been seen before, which are alien.

• 9:45Speaker 02

  Yes.

• 9:46Rory Stewart?

  And so part of, I think, what this series, I hope, is going to be about is how we think about the alien. Now, we'll have a chance in our second conversation to get deeper into the question of the alien intelligence that's AI. But just before we focus on that, I want to circle around some of the other things that might seem a bit alien or difficult for us to understand. Let's maybe have a look at pandemics. That's something that you've thought about and written about. And of course, it's something we went through in a smaller way with COVID.

• 10:16Speaker 02

  Yeah.

• 10:17Rory Stewart?

  And again, where I felt I, you know, was in government coming through the end of twenty nineteen and very involved in this debate going to the beginning of twenty twenty, was an interesting example of the ways governments can and cannot respond to something which is really strange. You know, something comes out of a viral lab, it seems in Wuhan or in my fish market in Wuhan, however it gets there. And it ends up with the British government spending something close to four hundred billion pounds trying to deal with this. In fact, most of the debt problem that we have in Britain, most of our financial problem today is a legacy from this virus coming out of Wuhan.

• 10:56Speaker 02

  I think it's a very good example because I would say in, in many of these cases of these existential risks, and we certainly see it with an AI, is there is a brain-stomach barrier. Like, our brain, our intelligence, based on various data, starts understanding something, but our stomach says, nah, that cannot be possible. Why not? Because I have never experienced anything like it. So I have no gut feeling what this thing can be like, so it cannot exist. I remember, you know, the first few weeks of the pandemic, I remember myself sitting at my home, watching on television the images from Wuhan of a

• 11:37Speaker 02

  modern, huge city of millions completely shut down. And as a historian, I had all these kind of synopses going on in my brain. Ooh, this is like the Black Death. This is like the Spanish Influenza. I read about pandemics and still my stomach says, no, this is not possible. This is not possible. Maybe it's possible in China. It will never get here. Why not? Because I just never saw anything like it in my life. So I cannot accept the reality of this thing. And the stomach is usually much stronger than the brain. Even when the brain insists, this is real, this is coming, we need to prepare, there is something that that just doesn't accept it.

• 12:15Rory Stewart?

  So what I observed at the beginning of COVID, at least in Britain, is that there is a real tendency in government not to want to act. So we saw particularly dramatically in Milan in Italy, the beginnings of hospitals being overrun. And we had seven flights a day coming from Milan into London. And people would appear on the radio and they would say, there's, we're about two weeks behind Milan. And yet, the decision was, we will go ahead with the Cheltenham Festival. We're not going to shut this down. We're not even going to stop the flights from Milan. And very, very quickly, some at least of the people around government were saying, we need to take this on the chin. There's nothing

• 12:58Rory Stewart?

  we can do. There's no point wearing masks. There's no point isolating. We're never going to be able to get a vaccine in time. It's better to just develop herd immunity. And a lot of the reason for that, and I think this will be relevant, because I think the same is true about why we probably won't be doing enough about AI, AI safety, probably why we will underestimate the risk posed by Vladimir Putin, or underestimate the risk that could happen with Trump or China, is because not doing something is cheaper and easier. And you can see the cost of action. Oh, my goodness, if we have to do any of this stuff, it's going to cost our economy hundreds of billions. Our schools are going

• 13:41Rory Stewart?

  to have to shut. Our businesses are going to have to shut. So we will do our best to convince ourselves it's not really that serious, or that we can't do anything about it anyway.

• 13:49Speaker 02

  Yeah, I think this is exactly the mechanism. Like, if you look, again, we'll talk about it at greater depth in, in the next episode. But one of the threats facing many countries today in the world, including Britain, is one of AI imperialism and of becoming vassal states of the United States. If your entire economy and military is run on AI infrastructure that the Americans can shut at the press of a button, you are no longer an independent country. Now, this is a real imminent threat that can materialize within a very few years. Now, I think there are basically three options that what a country like Britain can do.

• 14:30Speaker 02

  Option number one is business as usual. Just accept the fact, yes, we are going to become American vassals. But if that is the case, you need to be aware of it and act accordingly. If your plan, how to deal with AI imperialism, is just to accept your place as American vassals, then starting from today, whenever the American government says, jump, you jump. If they say Greenland belongs to the US, then yes, Greenland belongs to the US. If they say give the Falklands to Argentina, then the UK government says, oh, the Malvinas? Of course, they're Argentinian. That's option number one. Option number two is to say, no,

• 15:10Speaker 02

  no, no, no, no. We want to remain independent, so we will build UK AI. Sounds wonderful, except for the price tag, which is something in the area of five hundred billion pounds. Now, in theory, if this is an existential thing, the UK can do it. It's one of the few countries in the world that can theoretically still do it.

• 15:33Rory Stewart?

  Because, as you just pointed out, it's slightly more expensive than COVID. It's about half the government budget.

• 15:37Speaker 02

  It's possible. But it means that you have to say it now, and you need to cut healthcare, welfare, education, transportation, everything, so you have the money to make yourself an independent AI nation. Third option is to say this is too costly for us. We'll do it with the Europeans. It's still going to cost five hundred billion pounds, but we'll give only part of it, and the Germans will give some, and the French and the Spanish. But if this is the plan, then say so and act accordingly today. You need to rejoin the European Union today and integrate Europe much more closely, because otherwise it won't work. Now, I think these are the only three options. And if the political system

• 16:21Speaker 02

  cannot articulate these options and come to the people and say, look, these are the options, all of them have costs and benefits, choose, then it's not doing its work. It's not willing to think the unthinkable. And then it's just, you know, the default is to become American vassals. But even if you go for the path of least resistance, you should at least do so in a self-aware way, so you will be good vassals. Like, you know, when the Spanish, when the conquistadors invented Mexico, so there were peoples, the Tlaxcalans, who to some extent understood what was happening, and they became the allies'

• 17:00Speaker 02

  vassals of the conquistadors, and this paid off. This, this was good policy for the Tlaxcalans. Now, again, people can say, yes, no, but you need to be aware that this is the decision you're taking and act accordingly.

• 17:17Rory Stewart?

  There are maybe two sets of questions around this. Why is it that policymakers find it so difficult to accept that those are their three options they need to do it? And the second set of questions is why those are the only options available to you?

• 17:32Speaker 02

  Hmm.

• 17:32Rory Stewart?

  Let's maybe start with the first and easier thing. Let's assume these are the only options available to you. One of the issues that Andy Burnham will face is people will say, I don't like this stuff. I don't like the tech bros. I don't like Elon Musk. These companies are making too much money. There's inequality. There's trillionaires. It's not bringing great benefit to me directly. They talk about a cure for cancer in ten, fifteen years, but in the short term, I'm going to lose my job, my electricity bill's going up. I don't want a data center next to me. I don't like how much energy it uses. I don't know how ugly it is, how much water it uses. And actually, I want to pay for my health service and I want to pay for my education. So there is no political constituency

• 18:18Rory Stewart?

  out there. I mean, if you look at the manifestos of the parties, not even the most entrepreneurial politicians anywhere in Europe are making these arguments of the sort that you've advanced. Instead, you know, to go back to the way in which we began this program, you read the manifestos, and the manifestos are about the sort of things we've been talking about for fifty, sixty years. How do we improve the NHS? How do we cut the welfare bill? How do we build infrastructure? How do we get a bit of innovation? How do we improve our schools? There's a whole question of political culture, political economy, which can make you quite fatalistic. And then, of course, there's the realisation which you will have had as a historian,

• 19:00Rory Stewart?

  which is oddly, at certain points in history, societies suddenly are able to make radical moves and make radical decisions. Britain did it during the Second World War. And actually, in the end, the end of my Covid story was not that everybody said it's all too difficult and it's all too expensive. Within about six weeks, they were doing things that seemed unimaginable, like lockdowns. What do you make of this process? Because this is a very, very important part of this. You know, I'm not sophisticated enough to produce a complete description of what the Aztec equivalent of this is. But presumably, there was a strong Aztec equivalent at an elite level, which contributed to the problems of responding correctly to the conquistator.

• 19:36Speaker 02

  I think the key asset here is just time, that humans, including politicians, need time to understand what is happening, to change the way they think, to adjust. One of the big problems with the challenges that we now face, especially AI, is that it is moving on a time scale which is also alien, which is inhuman, which is non-organic, non-biological. I mean, to make it in the most concrete way possible, I've been talking with some cyber experts last few weeks about the cyber threat. And one of the things they said on a personal basis is, I can't sleep anymore. I can't sleep, not partly because I'm very worried,

• 20:19Speaker 02

  but technically, I can't sleep. Because by the time I sleep for like six hours or eight hours, so many things have happened. And this is because AIs don't sleep. Like, when it was Churchill versus Hitler, so Hitler also needs to sleep. He got these injections that enabled him to go on for longer than most people. But in the end, even Hitler had to sleep because he was an organic entity. AI doesn't sleep. If you need to act and to understand what is happening in the age of AI, then you need to act to understand something that moves at a superhuman speed. To give another example, in the financial

• 20:59Speaker 02

  system. So bankers, investors, they were used to a situation when the markets closed down for the night, and for the weekend, and for Christmas. But AIs don't celebrate Christmas, and they don't sleep at night, and they don't go on the weekend on vacation with their kids. So we now see the markets, the financial system shifting to a constant cycle. And again, which is not just constant, it's moving at a superhuman speed. Because AIs can analyse financial information at orders of magnitude faster than human beings.

• 21:37Speaker 02

  So if the AI crisis would unfold at the speed of kind of previous political human crisis, I think the political system will be okay, probably. It will eventually understand what is happening. It will articulate the different strategies and choices. It will hopefully make the right choice. But it's just moving far, far faster than anything we have seen before. You know, I was reading today the front page of the Wall Street Journal, and there was one, a main story there about the risk of AI completely destroying humankind. Now, 10 years ago, this would have been in a science fiction magazine. Nobody thought

• 22:22Speaker 02

  10 years ago that on the front page of the Wall Street Journal, you will have a serious story. AI might get out of control and completely annihilate Homo sapiens. And just the speed with which it's happening doesn't allow the political system to do its job.

• 22:39Rory Stewart?

  Yuval, thank you very much indeed. Let's take a quick break and then back for more.

• 22:46Speaker 03

  There we are, getting a statement from Melba Tev on that flight with 350 passengers on board.

• 22:55Speaker 04

  The great game, strategic struggle between Britain and Russia.

• 23:02Speaker 05

  We assess Prime Minister, with high confidence, it was Russian.

• 23:09Speaker 06

  Holding the Russians accountable is World War III.

• 23:11Speaker 07

  They took down a British passenger jet containing 350 people.

• 23:14Speaker 04

  Man in the Kremlin is playing, Alex. Got to ask yourself, do you want to play or not? You're the Prime Minister. You've done nothing!

• 23:27Speaker 08

  We would need to understand the policy context.

• 23:29Speaker 04

  All the relevant context will be structured as a framework today, disseminated into an operation order, or FRAGO.

• 23:38Speaker 06

  A FRAGO that says to kill the President of Russia.

• 23:43Speaker 04

  Yes.

• 23:45Speaker 06

  New from Goldhanger. Who is going to be able to pull this off?

• 23:49Speaker 09

  You're my soldier, are you? Sergeant Tom McDuff, Duffy!

• 23:53Speaker 06

  An audio drama journey like no other.

• 23:55Speaker 09

  Men like Duffy, they are our most valuable weapons.

• 23:59Speaker 06

  Devised as a technically rigorous geopolitical action thriller.

• 24:03Speaker 09

  We have two tight, friendly key.

• 24:04Speaker 06

  Where Britain plays its strongest hand. He's the best. Against its strongest foe. Don't do it, Duffy! Bear Hunt. Available now wherever you get your podcasts.

• 24:31Rory Stewart?

  AI champions will say, we just need to read this as another technological revolution. So they will compare it to the Industrial Revolution, or the invention of electricity. They will say that the people who opposed the Industrial Revolution were Luddites, handloom weavers smashing up mills, and that technological progress is inevitable. There's nothing you can do to stop it. Even if you try to slow it down here, someone else will build it somewhere else. And they will add to that another dynamic, which is not only should you not slow it down, they want to go faster. They want to go faster because they're all competing with each other. Because they somehow believe if they can get to some form of superintelligence before the others, they will have an advantage that nobody else can

• 25:15Rory Stewart?

  ever, can ever meet. So maybe let's just take a moment to explore why previous technological revolutions were not like this. What is the difference between steam, or electricity, or the printing press, and AI? And this is partly about time, right?

• 25:31Speaker 02

  There are many differences, but the biggest one, the most crucial one, is that this is the first time that we are creating agents and not tools. In the Industrial Revolution, you created steam engines, trains, steamships, looms, whatever you want. But these were all tools in our hands, which means we still make the decisions. And we still decide how to use them. It still depends on us also to invent the next generation of trains, or machine guns, or whatever it is. AI is not a tool. It is an agent. An agent is something that can make decisions by itself and can invent new ideas by itself. It doesn't wait

• 26:12Speaker 02

  for us. And right now, the point that all the AI corporations are focused on is the moment of what they call recursive self-improvement, the moment when the AIs themselves can build the next generation of AIs, that will build the next generation of AIs. This is the moment that we completely lose control.

• 26:35Rory Stewart?

  And this is paradoxically, I mean, I've just come back from California, both a moment that they are desperately looking forward to and excited by, and a moment that they're absolutely terrified by. So that you have this endless contradiction, which is very similar to Saint Augustine, you know, make me chaste, but not yet, you know, make me safe, but not yet.

• 26:53Speaker 02

  Yes. Yeah. And, and again, if you invent a knife, the knife is just a tool, and it's up to you to decide whether to use the knife to cut salad or to murder someone. The knife doesn't make the decision. And the knife can certainly not by itself invent a machine gun. AI is a knife that can do that. AI is a knife that decides by itself whether it cuts salad or kills somebody, and can invent a machine gun that will then invent the atom bomb and so forth. This is why there is a potential here for a much faster exponential process, and a process which, for the first time in history, escapes human control.

• 27:37Rory Stewart?

  The problem is, you know, we play this podcast.

• 27:41Rory Stewart?

  Millions of people watch it. Somebody shows it to a prime minister or politician. You know, what are you doing about this? This guy has said all this. And immediately, some other expert will pop up and say, no, Yuval is getting this wrong, for whatever reason, right? They will either say, either number one, actually, the nature of this technology is a little bit different to what we thought. And of course, because this is quite technical stuff, right? That the politician will struggle. You start talking to them about weights and parameters and transformer architecture. Then you get muddled, right? So somebody just has to say to them, well, you know, Prime Minister, you know, it's a jagged frontier and it's not quite clear that

• 28:26Rory Stewart?

  superintelligence is arriving quite in the way in which we expected. And the real threat that we face is not from autonomous agents. It's from bad people getting control of a cyber attack or the ability to make a bioweapon. And our best way of dealing with that is to build a smarter AI that can detect those humans before they get there. And then they will add, and anyway, even if Yuval is right, there's nothing we can do about it because if we don't build it, the Chinese will build it or some other company will build it.

• 28:56Speaker 02

  Yeah, I mean, this is what happened to Montezuma. Exactly. They debated in exactly this fashion. What is it? These strange things that came across the ocean until it was too late.

• 29:07Rory Stewart?

  And your point is, could be many things. It could be that this is inevitably the nature of AI. Or it could be, there is a very significant probability that AI will be like that, in which case you need to plan accordingly.

• 29:18Speaker 02

  And this is not just me saying it. It's the corporations themselves are saying it. We have on record, people, not from years ago, from last few weeks, people like Dario Amodai from Anthropic, like Sam Altman, like Elon Musk, saying this can completely destroy humankind. And I just heard Elon Musk some weeks ago say, estimating, I think it was like 10 to 20% chance that this will destroy humankind now. And then, of course, the interviewer asked him, would you go on one of your rockets? If you thought there was a 10% chance it will explode and kill you? And he thought about it and he gave a very interesting and crucial answer. He said, ah, but this is inevitable.

• 30:03Speaker 02

  There is no, I mean, we are on the rocket, it's too late. And this is, this is maybe the most important thing to understand about it all. The message coming from the corporations is that this is inevitable. This is deterministic. There is nothing we can do about it, which actually means there is no politics. Politics is the art of the possible, of choices. And what these people are saying, there is no room for politics because there are no choices. This is inevitable. And the most important thing that politicians and also citizens should say, no, this is not inevitable. This is a choice. History is not deterministic. We always have choice, at least until the AIs really take over.

• 30:48Speaker 02

  The job of politicians is to create this space for choice. Again, the big tech wants us to think that there is no choice. But this is not true.

• 31:00Rory Stewart?

  They have an additional argument that they will make. If you go in to see Andy Bonham, the additional argument would be, yeah, somebody has a choice, but it's only Donald Trump. We in Britain have no choice. We have no way of controlling these companies. And sure enough, when I talk to these companies, the hyperscalers will say, good luck building this yourself. You don't have the money, you don't have the time. Even if you manage to get it through, through your planning regulations and this, that and the other, it would take you three, four years, it would cost you $500 billion. And by then, we will be dominating the whole world and you will be completely dependent on our systems.

• 31:36Speaker 02

  Mm hmm. Yes.

• 31:37Rory Stewart?

  And there's a second problem, which is something that I believe, which is even more troubling, right? I agree with you, these things pose an existential threat to humanity. But I think it's also true that the possessors of these technologies will have the most astonishing economic and security advantage.

• 31:53Speaker 02

  Absolutely.

• 31:54Rory Stewart?

  So if the UK were to choose not to participate at all and to simply say, and Europe could say this, we think this stuff is disgusting. We think it's dangerous. We think it's reckless. We think it should be regulated. We want to have nothing to do with this. And the US and China puts their foot down. We end up in a situation in which our economy collapses because they're building

• 32:17Speaker 02

  Yes.

• 32:17Rory Stewart?

  factories and robots and economies and businesses built off the much more profitable than us, suck all the tax revenue to them. They're building weapon systems that are far more powerful than us. Or if we're allowed to build weapon systems at all, they're dependent on their

• 32:32Speaker 02

  Yes.

• 32:32Rory Stewart?

  their stuff. So we are in a horrible situation where, in a sense, we are forced to participate in this dangerous race.

• 32:40Speaker 02

  Yeah, so this goes back to the three options outlined in the beginning. If this is true, the only remaining option is to be subservient US vassals, assuming that you don't want to be the vassals of the Chinese. But again, this is not true. You don't have to believe what you hear from the American big tech companies telling the UK government it's too late for you. Of course, they will say that. This is what they want you to believe. But I don't think it's too late. I mean, I agree that if you just decide to completely stay out of the race, oh, we'll just pass some regulations and it will be okay. No, you cannot regulate it if you're not part of the game. You like to take another historical analogy. You have the Zulu Empire in the 19th century deciding to regulate the Industrial Revolution. Good luck with that.

• 33:25Speaker 02

  You cannot regulate something if you're not really part of the game. But I don't think it's too late. Maybe for Britain by itself, maybe it's too late. But if the whole of Europe together really decides this is an existential issue for us, for remaining independent, and we don't want to be American vassals, then it's not too late. And maybe they can do it in a different way than the Americans and the Chinese.

• 33:50Rory Stewart?

  Another possible fourth option, but all these things depend on what you believe is going to happen to this technology. I mean, this is part of the problem that you're having to bet on the future of this technology. But another possibility is to say, actually, in the end, these American proprietary models are the wrong way to go. What we need is open source stuff, our own data centers, our own chips, our own weights, our own parameters. We could take open source models from the Chinese. We adapt them, we develop them, we own them. We can feed them on our own data, our own NHS data, which Anthropic doesn't have, our own legal data, which they don't have. And that there are enormous benefits in terms of our own resilience,

• 34:35Rory Stewart?

  independence, control of our data, of moving away from these closed American corporations towards an open source model.

• 34:43Speaker 02

  Mm hmm. That can be a potential course of action. Again, it has huge costs and risks. But that's the job of the government, to present the potential courses of action, to choose and to act accordingly. And just doing nothing is the worst thing that can be done. I mean, we are dealing here with two related but different issues. There is the issue from the viewpoint of, say, the UK or Europe. How do we stay independent in an age of AI imperialism? And that's one issue. The other is for humanity. How does humanity survive that?

• 35:24Speaker 02

  I mean, in the end, it doesn't really matter who wins the race to superintelligence, the Chinese, the Americans or the Europeans. If the outcome is the creation of super intelligent aliens that then take over the world. I mean, you could have the credit to say, okay, we were the fathers of the gods.

• 35:44Rory Stewart?

  Yeah.

• 35:44Speaker 02

  It wasn't them, it was us. But that's the, seems to be the only benefit. And but the crucial thing to understand about what the corporations are saying and what the market is saying. Forget even the corporations. You can say, oh, this is just hype. Look at the market. Trillions and trillions of dollars are being gambled on this thing. And the gamble is totally irrational because what the gamble actually says of the market is, we are going to create gods and they will be our slaves. It makes no sense that the amount of money being invested, unless what will be produced in a few years are entities with divine abilities.

• 36:26Speaker 02

  This is the only way that you, that the market will get a return on the investment. But this depends on the fact that the gods being created will be the willing slaves of the market. And it doesn't make sense if it, if these things will be as intelligent, as agentic, as creative.

• 36:46Rory Stewart?

  Yeah.

• 36:47Speaker 02

  As the market assumes they will be.

• 36:49Rory Stewart?

  Yeah.

• 36:49Speaker 02

  They will get out of our control and then there is no market. There is no money, there are no corporations.

• 36:55Rory Stewart?

  You've pointed to two different problems there. One of the question is a question of power. Who has the power? Does Anthropic have the power? Does Donald Trump have the power or does the AI itself? And then the second question is the paradox of this business model, which is all the money that's going in, all the stories that we're now hearing from some of these companies about how they're going to double the size of the US economy in the next six, seven years, how the productivity benefits are going to be unbelievable, are predicated on these devices being autonomous, right? All the productivity growth, all the stuff comes essentially by replacing humans with these machines. The dream is that you can have the equivalent of 2 million software engineers

• 37:39Rory Stewart?

  working 24/7 flawlessly every night. And all the benefits that they're supposed to deliver economically in terms of cures for cancer, better cyber defense, logically are also the threat because the whole idea depends on these things.

• 37:58Rory Stewart?

  Being autonomous, and autonomous by definition is outside our control. So your economic business model and the benefits you're promising people are predicated on the idea that these things are not actually managed by humans on a day-to-day basis. Otherwise, you wouldn't make the money.

• 38:12Speaker 02

  Yeah, exactly. And it's not necessarily that they will rebel against us and try to destroy us. It's just that the world will spin out of our control, out of our ability to understand it. I mean, they are different, there are so many. One of the difficulties with explaining the risks of the AI revolution is that there are many, many scenarios, and they all sound like science fiction in one way or the other. The easiest scenarios to understand is you have some bad actors, a terrorist group somewhere that tells their AI to produce a biological weapon, a virus that will kill half of humanity. This is a serious risk, and in a way, it's a good risk because we can understand it easily.

• 38:55Rory Stewart?

  Just explain it for a second so people can understand how the AI helps to do this. It's because it's good at gene editing, it's good at protein folding, it's good at all this.

• 39:03Speaker 02

  I mean, previously, a terrorist organization in some corner of the world that wanted to produce a deadly virus that will be, let's say, as infectious as COVID and as deadly as AIDS, you would need some genius biologists to be working for that terrorist organization. And, you know, most people who go to elite universities to study biology and epidemiology and these things, they don't end up in some terrorist organization. But now with AI, even if nobody in the terrorist organization has any idea even what a virus is, they just have to give the right prompt to their AI.

• 39:43Speaker 02

  And their AI that has read all their papers and books about biology will design the supervirus for them.

• 39:53Rory Stewart?

  So that's the first risk.

• 39:54Speaker 02

  That's the easy to risk.

• 39:55Rory Stewart?

  The terrorist getting control of a very powerful tool.

• 39:58Speaker 02

  Oh, no, a very powerful agent. I mean, a tool, they need to know how to use it. In this case, it's an agent, they have a new recruit to their terrorist organization, and they just tell the new recruit, "Oh, go and do this," and it goes and does it. The much more difficult thing to understand is that the AIs, even if they are not evil, even if they don't try to destroy us, they will just create a world which we cannot understand. And therefore, we will lose our ability to understand and control our lives. We'll be like the other animals on the planet. Like, if you go now to a cow or to a dog, and you tell the cow or the dog, "Look, your

• 40:39Speaker 02

  life is actually being managed by something called the financial system. There are stock exchanges, and there is the bond market, and you have these things called stocks and bonds, and actually, your life is run by that. Because of the fluctuations in the price of bonds and the value of the dollar, let's say you tell to the cow, the company that owns you has gone bankrupt, so now they are killing you." And the cow will say, "I have no idea what you're talking about. What, bonds, stocks, dollars? What are these things? Like, I look around, and, okay, I can see people, I understand they are stronger people, I can see dogs, I can see cats, and trees, and clouds.

• 41:22Speaker 02

  I've never seen a bond in my life. What's that?" Now, we know that the cow, its life is really dominated by this totally invisible system. We know that because we are smarter than the cow, and we created the bonds and the stocks and all these things. In 10 years, we could be the cows. Our lives will be managed by a totally invisible system that is far beyond our understanding. The AIs will create a new financial system, which is mathematically far too complex for the human brain to understand. They will create new financial devices, which are orders of magnitude more complicated than bonds and stocks.

• 42:07Speaker 02

  This will determine what is happening in the world, and not a single human being understands what the hell is happening.

• 42:15Rory Stewart?

  That was one of the most wonderful interviews I've ever done, but there's still so much more to get into. So in episode two, we're going to go much deeper into artificial intelligence, the risks posed by it, and the fact that it could wipe out our economies, it could crush us militarily, it could wipe out humanity, but also what it will do to our sense of identity, the alien nature of AI. If you're a TripPlus member, you can hear that next episode with Yuval right now. And if not, then please sign up at therestispolitics.com.