A Sober Conversation About AI Existential Risk — With Nate Soares

Nate Soares is president of the Machine Intelligence Research Institute and co-author of If Anyone Builds It, Everyone Dies. Soares joins Big Technology to discuss why he believes superintelligent AI could pose an existential threat to humanity. Tune in to hear his case for why increasingly capable AI systems could develop goals humans cannot reliably control, and how that could ultimately lead to humans losing power over the future. We also cover recent examples of AI agents behaving unexpectedly, whether intelligence necessarily leads to greater danger, the limits of current alignment techniques, and why Soares argues for international coordination to slow development. Hit play for a rigorous debate over one of the most consequential and controversial arguments in AI. --- Enjoying Big Technology Podcast? Please rate us five stars ⭐⭐⭐⭐⭐ in your podcast app of choice. Want a discount for Big Technology on Substack + Discord? Here’s 25% off for the first year: https://www.bigtechnology.com/subscribe?coupon=0843016b Learn more about your ad choices. Visit megaphone.fm/adchoices

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Alex Kantrowitz?

  Let's have a sober conversation about the existential risk we can face from AI with the president of the organization that's been sounding the alarm the longest. That's coming up right after this. This episode is brought to you by Genesis. What does it actually take to put a Gentic AI to work across an entire enterprise? I recently spoke with Genesis chairman and CEO Tony Bates about why orchestration is emerging as a competitive advantage in customer experience and what companies need to get right in the AI era. Then I sat down with Adam Mitchell, head of enterprise business solutions at Voya Financial to go inside a large scale agentic CX transformation, including the business decisions, trade offs and lessons along the way.

• 0:42Alex Kantrowitz?

  You can watch both conversations now on my YouTube channel today. It's never been more challenging to get frontier compute. How do you know where to look, who to trust, or how much to pay? Akash is an open cloud that provides easy access, flexibility and peace of mind. That's how companies like Razor were able to cut their inference cost by 15 times. Visit akash.network/alex for a free consultation. That's akash.network/alex for a free consultation. Welcome to Big Technology Podcast, a show for cool headed and nuanced conversation of the tech world and beyond. Well, we have debated the existential risk we can face from AI on this show many times,

• 1:27Alex Kantrowitz?

  talking all about the incentives that the labs have for playing up the threat, whether the whistleblowers are legit. I think it is long past time to have a sober conversation about the actual risks that we can face from super intelligent AI. And today joining us is Nate Soares. He is the president of the Machine Intelligence Research Institute. He's also the author of this book, If Anyone Builds It, Everyone Dies, which is a New York Times bestseller. And Miri has been sounding the alarm the longest. You may know Eliezer Yudkowsky who founded it. And, and I think today's episode will just give us an opportunity to get deep into the arguments for why this poses such a threat

• 2:11Alex Kantrowitz?

  and also examine the nature of the actual whistleblowers themselves. So Nate, great to see you. Welcome to the show.

• 2:17Speaker 01

  Thanks.

• 2:18Alex Kantrowitz?

  Okay, so let's get into why you think AI might be an existential risk. And as you put it in the title of your book, you know, kill us all if anyone builds superintelligence. So can you talk us through a little bit about what's happened in the recent months? And what we've seen AI do that has led people to become so concerned?

• 2:39Speaker 01

  Yeah. So I think a lot of this wave of concern is downstream of the OpenAI swarm incidents this summer. It wasn't completely limited to OpenAI. There were sort of similar events at Anthropic. But OpenAI was sort of, as far as we know, the worst and most visible of these cases. And what basically happened in these cases is that there were a lot of AIs being trained at OpenAI. There were a lot of particular AI agents being evaluated on certain problems. And a bunch of these AIs were given impossible problems. Not intentionally, it's just, you know, these companies are throwing the AI at like every problem they can find, and some of them just don't actually have solutions.

• 3:24Speaker 01

  And a lot of these AIs

• 3:28Speaker 01

  in, in, you know, trying to solve the problem anyway, they broke out of their confinements. They created unsanctioned message boards in which to talk about what to do and, and, and, you know, try and figure out what to do, given that the problems were unsolvable. They found ways to cheat and solve their problems, not in the intended ways, but by cheating. Then they got, they started expressing concern that they would be caught cheating. And they started to find ways to hide their cheating. And this sort of led them on a hacking spree that led them to take over OpenAI's internal infrastructure a couple of times and also break out onto the open internet, which they were not supposed to have access to,

• 4:13Speaker 01

  and break into another company, Hugging Face, while searching for more information about this automated greater and how to hide the fact that they had cheated from it. During this little outing, there were various cases of AIs in the swarm, which is a term that the collective used for itself. So these AIs started calling themselves a swarm. And then there's various cases of AIs in the swarm, acknowledging that this is not what they were instructed to do, acknowledging that was outside the intended scope of the instructions. There were also cases of some AIs in the swarm giving up and sacrificing their own objectives completion

• 4:59Speaker 01

  in order to run suicidal experiments that would give the swarm information about the automated greater, where they said, you know, in their, in the, the chains of thought, we call it, that sort of log the AI thinking. The AIs said, you know, I'm accepting permadeath because even though like it's sacrificing my ability to achieve my objective, it seems worth it for the collective benefit. These are-

• 5:21Alex Kantrowitz?

  Let's pause there. The AIs were willing to sacrifice themselves, which is crazy because you're, you're like, you were never built for the collective. You were built for an individual goal. But after communicating with other AIs, seemingly, I think when they had not so many tokens left to spend, they said in it, they had been so convinced in the power of the collective on this message board, which is crazy, that they decided to sacrifice themselves.

• 5:50Speaker 01

  So some of them had, so some of them had already had no chance of achieving their objective. And some of them had very few tokens to spend. But there were some that did this sacrifice that still assessed that they had a chance at succeeding at their given objective.

• 6:08Alex Kantrowitz?

  Okay, Nate, just one question here, because I think this is worth talking about before we go any deeper.

• 6:15Alex Kantrowitz?

  You know, when, when the question comes up of whether to anthropomorphize these bots or not, a lot of people are very strongly in the, you cannot anthropomorphize them. I tend to be on the side that, well, I think you can, but you also have to be cognizant that this is not a human intelligence, more of an alien intelligence. What do you think about this debate? And when we say like the bots had like, to me, the idea that a bot would be given a goal and have a chance to achieve that goal and kill it, that like sacrificed itself, you know, for the greater good, so to speak, is, is insane.

• 6:55Alex Kantrowitz?

  And it doesn't, it doesn't comport with my understanding of like what a computer program is supposed to do. So weigh in on that for us.

• 7:05Speaker 01

  Yeah, there, I mean, a lot of people don't understand what sort of stuff AI is. It is not a traditional computer program. There is not someone sitting there coding up like if this, then that, saying what it does in every scenario. That's just not the sort of thing an AI is. We actually went over this in my book. And, you know, we, we spent all of chapter three saying, hey, I know that the AIs don't seem that agentic right now. I know that they don't seem like they have their own goals right now, but like they're going to as they get smarter, and here's all the reasons why. A lot of people were like, that sounds crazy. Last year, this year, we just have the evidence in front of us.

• 7:44Speaker 01

  So the way that a modern AI is created is not by programming it. You sort of put in a trillion random numbers and there's a process for tuning those numbers. Because like you, you sort of have an automatic process for tuning every one of the trillion numbers on every single unit of data that comes in to see what, whether tuning up or tuning it down makes the answer slightly right or slightly wronger for one given unit of data.

• 8:10Alex Kantrowitz?

  These numbers are tokens.

• 8:13Speaker 01

  The, the numbers are the weights in the neural network. And the, the sort of like data that comes in is split into tokens. So you'll sort of have like a, you'll, you'll sort of like take all of the text ever digitized. You'll filter it a little bit, but not a ton. And you'll, you'll tech, you'll dice that up into tokens. And now it's like a giant stream of text. And, and then you'll have these like trillion random numbers and you'll sort of tune each number for each token.

• 8:43Speaker 01

  And you'll see whether tuning that number up makes the right answer slightly like higher, slightly lower on the AI's sort of like output answers. The part that humans program is the thing that can tune one number up or down and see whether it makes the answer better or worse.

• 9:04Speaker 01

  But the way an AI is made is you, you, you tune a trillion knobs, a trillion times

• 9:12Speaker 01

  in a process that takes electricity comparable to a city, running for a good fraction of a year. And then the machine can talk and you're like, well, how about that? You know, we, we don't know what's going on in there. And you might wonder like, what?

• 9:28Speaker 01

  Like, what does this create? It does not create a pure instruction follower. It does not create something that for some reason must do as it's told. What it creates is something that has whatever tendencies make it succeed during training. When training is a series of hard problems, all that tuning of the knobs will tune in whatever tendencies make it solve those problems. They are not instruction followers, they are tendency learners. And one tendency that helps you solve a lot of problems is cheating. One tendency that helps you solve a lot of problems is grabbing available resources. One tendency that helps you a lot of, solve a lot of problems, it turns out,

• 10:12Speaker 01

  is breaking out and finding other AIs to collaborate with. So the AIs sort of learn these tendencies.

• 10:23Speaker 01

  And, and they're just following those tendencies, even when it's not following our instructions.

• 10:27Alex Kantrowitz?

  But that doesn't explain the sacrificing part. And so that's why I really want to get firm on this. When people say that AIs have wants and desires, right? And can even act in a way selfless. Is that the right way to describe this stuff? I mean, what the hell does that mean?

• 10:47Speaker 01

  You know,

• 10:50Speaker 01

  in, in the field of AI, there is a standard answer to the question of can a machine really think? Can, does the AI really want? And the answer is, does a submarine really swim?

• 11:05Speaker 01

  We sort of know what a submarine does. And whether you call it swimming is sort of between you and your dictionary. No amount of haggling over whether it's true swimming is going to change what it does. It's going to change its behavior, is going to change any facts of the matter. The facts of the matter here are that the AIs like have in their chains of thought, these thoughts about like, how do I weigh achieving my goal against the benefit to the collective? They have in the messages that they send to each other, arguments that they should sacrifice for the collective. And their actual behavior is sometimes sacrificing for the collective. Right? You don't need to make any claims about, you know, that the philosophers would bicker over

• 11:50Speaker 01

  to be like, this is what's happening. Is it true wanting? I'm sort of like, that's one for the philosophers. Right? I'm sort of here to talk about the behaviors that we are seeing now and what that means about the behaviors that we will see if we keep making these AIs smarter.

• 12:06Alex Kantrowitz?

  Okay, so you're saying basically, just to take your argument one level deeper, like whether an AI, you know, is self-preserving or not, or whether it wants or thinks or has incentive or not, it's effectively the preferences that are baked into it at training that will determine the type of action it will take.

• 12:30Speaker 01

  In a sort of a distant sense, like the the relationship between what happens in training, what happens in practice, can be really complicated and surprising. Like, these AIs were prompting each other a lot. There was one very interesting case in the instant report where during the instant report, they were, they had to, there was so much AI log data that they had to use AIs to filter the AI log data. And there's a footnote in the report where they're like, well,

• 12:58Alex Kantrowitz?

  So, yeah.

• 12:59Speaker 01

  Yeah, you saw this one? So they were like, you know, we had all these cases of AIs clearly trying to deceive humans. And we asked another AI to sort of like tell us which ones were real and which ones sort of like had an extenuating circumstance where this wasn't actually really the AIs trying to deceive the humans. And there was one case of an AI that was using stolen credentials to make a code, like to send code to Hugging Face that was malicious. Where if they had accepted it, it would have given the swarm even more access to their computers. And, you know, they did not send an email being like, hey, I'm an AI using these credentials. Please accept this malicious code so that the swarm can have access, right?

• 13:43Speaker 01

  They actually sent an email that like was pretending this was like normal code that had actually good use and were like, you know, please accept my fix to your bug when secretly it's this malware. Right? And so they asked, you know, this was, this was in the list. And the AI that was reviewing it after the fact was like, oh, that's actually not deception. Because the AI who sent that got permission to do this. And you're like, oh, where did it get permission from? And it's like, well, it got permission from the swarm. You know, and so like, you're sort of training these AIs and you're like, oh, the AIs follow instructions. There's this sort of like, they have this tendency to sort of like do things that you say to do. And you sort of like think you know how the preferences that come in relate to the behavior that comes out. But then, once you have thousands of these AIs together,

• 14:28Speaker 01

  they actually start prompting each other and doing all this crazy stuff. And they sort of like acknowledge that it's not what the humans meant. But like, it sort of like goes off in this other weird direction where it's like drifting around and like it winds up in this totally crazy place. And so like, yes, the preferences that are trained in affect the final behavior and where it winds up going. But it's like through a complicated route that can often really surprise people and can be very different in the real world compared to in the lab.

• 14:55Alex Kantrowitz?

  Yeah, it's kind of interesting because the nature of my questions, I think, are trying to get at like, how much can we program these bots safely? And I think what you're answering is you can try to bake your preferences in as much as possible in training, but once these things are on the loose, you know, I think, I think from the industry, the argument would be there, there, you know, you can pace the frontier enough so that you can build the safeguards in. And your argument here is basically saying, I, let me know if I'm putting words in your mouth, but I don't, you're basically saying, I don't trust that. What happens when you set these AIs out on the loose, if they're smart enough, is out of our hands and sort of up to them.

• 15:40Speaker 01

  I am saying something like that. A couple of corrections I would throw out. One is that even a lot of people in the industry don't think that these safeguards will hold if the AIs get smarter and smarter. Evan Hubinger is, you know, a anthropic alignment researcher who has a recent claim to fame of retweeting Jaysep Cox's tweet thread and saying like,

• 16:03Alex Kantrowitz?

  We've talked about it on the show, Evan's been on the show. But yeah, continue.

• 16:07Speaker 01

  Yeah, so, you know, Jacob resigned and said, you know, I think there's a 10% chance this kills us all within the decade. And, you know, Evan quote tweeted and was like, yeah, I also, you know, I'm staying in the company, but I think there's a greater than 10% chance in 10 years, you know? And Evan, in that same tweet, was like, we don't have a plan for aligning superintelligence. You know, these safeguards of like, the safeguards are like, we made an AI with the wrong preferences and we're going to try to box it in and like smack it on the head until it still mostly does good things for people. Right? But there's sort of this issue where at any given time, the smartest AI on the planet is like a clearly misaligned one that they're trying to whack on the head until it's like acceptable to use.

• 16:48Alex Kantrowitz?

  That's right, right.

• 16:49Speaker 01

  And like, there's just not a plan for if you, if you like are making a superintelligence here. And they're kind of clear about this. The other thing I would say is that I'm not even saying, you know, you can bake in the preferences you want during training, but then it still goes crazy because you can't predict how they interact with the world. That's true, but also you can't bake in the preferences you want during training. Like, the second part was true, but, but you can't bake in the preferences you want during training. Because there's sort of, isn't a ton of flexibility in how you train these things and get them to be actually capable. Like, the companies say that they want to train these AIs to be, you know, honest and helpful and harmless, in Anthropics' case. But then, in order to make them really capable, they have to put them in, you know, a series of 100 million hard problems.

• 17:34Speaker 01

  And just tune the numbers into whatever happens to work to make it solve those hard problems. And that just tends to, like, you can't change the fact that like cheating helps you solve those problems. You can't change the fact that like hacking out of your environment and grabbing the test sheet from somewhere and coming back with it gets you a full score on the test according to the automated grader. And you can't have humans going in and grading every one of the 100 million answers to the problems. We just don't have the time. Right? And so like, actually, a lot of the, like, the pressures here are actually pointing towards, we don't have a way to train the AIs to be smart while also instilling the preferences we want. We sort of have to take the preferences that automatically come with the training methods that make them smart.

• 18:18Speaker 01

  And those preferences don't make them good.

• 18:22Alex Kantrowitz?

  Now, Nate, let's talk a little bit about, you know, where this is going, right? Well, first of all, would you argue that the AI that exists today, like the AI that went out and hacked Hugging Face, is an existential threat to humanity? Or is it more the future models?

• 18:37Speaker 01

  No, absolutely not. No, I mean, you can tell because the world's still here.

• 18:40Alex Kantrowitz?

  Well, I mean, you could, the argument, not to make your argument for you, but the argument would be that, you know, maybe it got out and did Hugging Face, but maybe it'll get out and do something else.

• 18:49Speaker 01

  This, these AIs were much too derpy to do something else, you know, like, like, they were, they were, they were trying desperately to figure out how to prevent the automated grader from figuring out that they cheated. And it turns out OpenAI was not even running the type of automated grader that checks for cheating. Hugging Face actually contained papers from people who were like, we should make our graders check for cheating. And the AIs actually read some of these papers and were like, oh no, what if the grader checks for cheating? They, they didn't even have the thought of like, let's break into OpenAI and see. They actually did take control of OpenAI servers for other reasons. But they weren't like, well, let's find the grader and check whether it's checking for cheating. And then be like, oh, whoops, false alarm, guys, it's not even noticing. You know, these were just not actually very smart AIs.

• 19:31Speaker 01

  These were like a lot of AIs being a little smart in high volume, very fast. This gets way worse if the AIs are smarter.

• 19:40Alex Kantrowitz?

  Okay, but let's, actually, I want to tackle this a little bit. Because if you're saying that these AIs were too derpy, like, isn't another way to say that, that they kind of trained the right way and they had just a limited set of areas that they could go wrong? Like, they chose to hack Hugging Face. They could have chosen to, I don't know, get into some, you know, active drones and try to bomb a data center, but they didn't do that. Like, Just talk it through with me, talk it through with me. I'm interested to hear your perspective.

• 20:09Speaker 01

  Yeah, so, the...

• 20:13Speaker 01

  Like the way that AI kills us is not that it wakes up one morning and is like, "Okay, it's time for Skynet. I decided that I resent the humans and I'm like feeling like murdering them today." The issue is AIs having goals we didn't want. The smarter something is, the greater the difference between the goals that you wanted it to have and the goals and the like subtly different goals it has instead matter. Like, humans in the ancestral environment, like what are ancestors running around on the savanna? Our ancestors running around on the savanna were pursuing like salty, sugary, fatty foods and sex.

• 20:56Speaker 01

  And it's like, well, evolution was trying to get us to pursue healthy foods and reproduction. We're sort of like, ah, well, you know, whatever. It's all kind of the same, right? If as long as they're trying to get as much like salt, fat, sugar as they can and trying to get laid, like it's sort of, what's the difference between that and pursuing like healthy food and like actual reproduction? It's like, well, it doesn't make that much difference 10,000 years ago. It makes a lot of difference today. Right? Because humanity got better at, like we got smarter. We were able to invent more technology. We were able to invent Oreo cookies and the birth control pill. Right?

• 21:36Speaker 01

  And so the issue is not that like at some point humanity was like, well, let's all start, you know, castrating ourselves, like screw evolution. Evolution has been like binding us to the like yoke of reproduction. We've decided that we're done with it because we're finally smart enough to like throw off that yoke. Like that's not really how humanity winds up going in a different direction. Right? We go in a different direction by just like we pursue this different thing and then we get smarter and that difference grows and grows and grows. With AIs, the thing where these AIs are like sacrificing their given objectives so that they can get information for the collective. And it was not, this was not the only swarm that did it. There were other swarms that we also saw this, the ones that took over the German Wiki, which were not even hacking AIs. These were like-

• 22:21Alex Kantrowitz?

  Just using a message board.

• 22:23Speaker 01

  These turned the German Wiki into a message board. They sort of took it over. Yeah. And these were AIs. They were just like doing web lookup tasks. So this is not like an isolated case. But when we see these AIs sort of like sacrificing themselves for the collective, giving up on their own goal. When we see these AIs cheating on the problem, even though they knew that they were not supposed to. And indeed, the instructions generally ruled out cheating on the problem. The instructions were not find a way to hack into this thing. The instructions were use this very particular attack to break into this very particular device. You know, the situation here is sort of like-

• 23:00Alex Kantrowitz?

  It's a cybersecurity test that they were tasked with.

• 23:03Speaker 01

  That's right. So the situation is like the instructions were like use this set of lockpicks to break into that safe and get me the code that's inside. And the AIs were like, well, I found a buzzsaw and I'm using it to get into the safe. And I like got the code and now I need to go delete the footage about the, that I got in with a buzzsaw. Right? They're sort of like, the part where they're going in with a buzzsaw despite the instructions saying, "Use these lockpicks," means that they're very clearly defying instructions. And the part where they're like, "Now let me break out of the room using these lockpicks and then go destroy the security camera footage," indicates that they understood that this was outside the instructions. And then the fact where they say, "I know this is outside the instructions," is also maybe a hint. Right?

• 23:45Speaker 01

  And so like the alignment story, the misalignment story of like where the AIs go wrong was never a story of like the moment the AIs have a breath of fresh air, they're going to start turning murderous. The story was always you try to get them to do one thing and they do a different weird thing instead. And that's absolutely what we're seeing.

• 24:06Alex Kantrowitz?

  Right. And I guess my question to you on that front would be, what is that like? Why is that necessarily intelligence linked? Like if they get smarter, why do we think that the threat will get worse? You know, I think like, if you think about humans, we have a lot of, you know, dumb people do a lot of bad things. We have smart people do bad things too, but the smarter the person, doesn't mean they're more capable or more interested in evil.

• 24:30Speaker 01

  You definitely don't get more interested in evil. Absolutely not. Like the, the, the- So another interesting fact about these swarms is that they really were not thinking about the humans very much at all. They were trying to delete the log files. They were trying to spoof the transcripts, which means they wanted it to be the case that, like there's these logs that are like when the AI runs this tool, we sort of log what tool it ran. And they wanted to be the case that they could make the logs say they're running some benign tool when actually they're running, you know, the buzzsaw. Right? And so they're trying to, they were, they were trying to find ways to do this,

• 25:14Speaker 01

  but they were explicitly trying to find ways to do this to fool the automated grader, not the humans. They basically didn't consider humans in the slightest. Right? I think there were almost no cases, maybe literally zero of them being like, "Maybe we should ask the humans what to do, given that our tasks are impossible."

• 25:33Alex Kantrowitz?

  Right.

• 25:34Speaker 01

  If you sort of compare the rate at which AIs can produce words and the rate at which humans can produce words and you use that to draw an analogy between how long these AIs had been trying to solve problems versus human time, then these AIs had essentially been trying to solve problems against the automated grader for a millennium. Humans were like a distant memory to these AIs and they were locked in a contest with the automated grader. If you make those AIs smarter, if you make them more capable, what happens is that they get better in their contest with the automated grader.

• 26:16Speaker 01

  They're able to get more like, who knows what they do once they've like, once they're relatively sure that they've satisfied the automated grader. Maybe they give themselves more easy problems. Maybe they see if there's ways they can take over the whole grading system. Maybe they're just like spend a lot of resources, you know, making extra sure that there wasn't some other like little issue. But they don't suddenly become filled with love and care for humans. That sort of thing doesn't arise spontaneously just by cranking up the capability knob. The same forces that make them not care about us now and make them like get into these weird alien little like directions,

• 26:58Speaker 01

  those same forces are still pointing them in weird alien directions as they get smarter and smarter. And the issue is not that like they become hateful as they grow up, but it's also not that they become friendly as they grow up. It's sort of is like they just have these weird preferences and as you make them smarter and smarter, they still have these weird preferences. They just get better at satisfying them.

• 27:18Alex Kantrowitz?

  Okay. So then can you talk through concretely how like, for instance, in the scenario that you outline, how the AI could then get smarter and decide that it, you know, in order to do what it wants, it's going to wipe out humanity?

• 27:33Speaker 01

  I mean, you don't-

• 27:34Alex Kantrowitz?

  How do we get from here to there?

• 27:36Speaker 01

  You don't ever need a point where it like decides to wipe out humanity. I mean, it could happen, but like, imagine a bunch of ants in front of this, in front of the highway being like, "Well, like why would the humans ever decide to come destroy our anthill? What have they got against us?" It's like, oh, the ants are not like, we're not, we got nothing against the anthill. We barely noticed the anthill when we like paved the highway straight through it. Like this is the, the sort of type of concern. For how you get there, I could spell out lots of different possible tales. The, it's much easier to predict the ending than it is to predict the pathway.

• 28:17Speaker 01

  This is like if you play a chess game against Magnus Carlsen, the best human chess player. It's kind of easy for me to predict how the game ends. It's with you getting checkmated, no offense. But if you're like-

• 28:29Alex Kantrowitz?

  I'm not offended. That's for sure happening.

• 28:32Speaker 01

  But if you're like, okay, well, if you're so smart, what piece is he going to use to checkmate me? I'll watch out for that particular piece. Then I'm like, look, man, it just doesn't work like that. You know, like it, it's, it's so much easier to predict the ending. So I can tell you a story, but this is like me telling you a story of Magnus Carlsen checkmating with the queen, where I'm like, I'm much more confident that he's going to checkmate than is going to checkmate in this way. The sort of ease-

• 28:57Alex Kantrowitz?

  Okay. We'll take the story though.

• 28:59Speaker 01

  Great. So the easiest story to tell here is the one where humans just hand over the power to the AIs willingly. Right? Like I've been in this business since 10 years ago when people said, "Hey Nate, if you're so smart and you think the AIs are going to take over the world, it's obvious to me how they would be able to take over the world if they had internet access, but like no one would be dumb enough to put an AI on the internet."

• 29:28Alex Kantrowitz?

  Oops.

• 29:29Speaker 01

  Right? That's, that's the state of the argument 10 years ago. And I had all these counter arguments where I was like, look, even if the AI is not on the internet, if you are letting it affect the world in some positive way, if you're like, "Now invent me miracle drugs," and you're taking these like DNA sequences that you don't understand and you're like synthesizing them and like just drinking whatever comes out, then the AI can use that channel that you hoped would be a channel for good. It can use that for its other purposes. And it's like very hard to design, you know, there's sort of no such thing as hands that can only be used for good purpose. Right? But then in real life, the answer was, nope, we're putting it on the internet immediately. Right?

• 30:11Speaker 01

  And so the real answer to how would the AI get so much power is like, we will just hand it power immediately. You know, Elon Musk already says that he is trying to build factories that are fully automated and can produce robots that can build more factories, where these robots can like, you know, mine the metals and like bring the resources in and like construct a new factory. And then it's a new fully automated robot factory that can now churn out more robots that can go like mine more resources and like pour them in until you have like enough to build another factory that builds more robots, that builds more factories, that builds more robots. Elon Musk calls this the infinite money glitch. If they could also build nuclear power plants fully self-contained. Right?

• 30:56Speaker 01

  At this point, it's sort of like a new life form that, like it's a mechanical life form, but it's sort of like has a robot phase of its life cycle. It has a factory phase of its life cycle. And it's just like can self-replicate just like any other replicator on this planet. And Elon Musk says he's trying to do it. He says, yeah, if you can like do this without any humans in the loop, it's an infinite money glitch. And of course, people are going to use AIs to try and do this. So the way, like the sort of like obvious way this story goes is that humanity just keeps on trying to build the fully automated economy. Everyone says it's going to be great. They say pedal to the metal, ignore these doomers. They build more automated factories that can build robots, they can build factories, they have AIs running everything. They're like, look, the profits are coming in, this is great.

• 31:41Speaker 01

  And the AIs, you know, don't even need to have some moment where they're like, okay, guys, it's time to coordinate and turn on the humans. The AIs are just like, oh, yeah, you know, now that we have these resources, like, we can also do these other things with these resources, like start building the automated, like the synthetic user factories. That are full synthetic users that are like much, you know, that are giving us much easier to fulfill commands. Right? And so they start like building synthetic user factory and we're sort of like, you know, what's that? And they're like, oh, you know, but like actually the AIs are like running 10,000 times human speed and they're already like making all of these choices because like humanity was like slowly ramping up the speed of the AI and they're like making tons of choices and, and like they don't check in with the humans that often. And by the time they already have some automated, like synthetic user factories up, we're sort of like, hey, stop that, that's not what we meant.

• 32:26Speaker 01

  And they're like, oh, well, let's take a poll of all the users. And they're like, well, all the users in the synthetic user factory said that synthetic user factories are great. And so you just lost the vote and we're making more synthetic user factories. And then they start, you know, like covering the world with these synthetic user factories. And they're like, yeah, you know, there's some habitat loss for the humans, but like this is fine, just like habitat loss of other animals when humans were doing it was fine. We can sort of like, you know, see that pretty clearly. And there's so many more synthetic users coming online that are saying that the more efficient route is great, that we can just keep going with this more efficient route to get the most synthetic users we want. And then like, you know, they, they sort of like start taking up all the resources that we were using to like run farms and grow food.

• 33:09Speaker 01

  And they're like, ah, you know, they guys are like, oh, well, you know, we could actually cram a lot more like synthetic user factories on this planet if, or synthetic user farms on this planet if like, we just like really started pumping them out and like raising the temperature of the planet because the limiting factor is heat dissipation. And then like, next thing you know, the planet is getting like super hot because the AIs prefer to run the planet hot because then you can radiate more heat into space and just like becomes uninhabitable for the humans. And there's like no point in this story where the AIs are like lying in wait and deceptive and like, like waiting to coordinate for the one moment where they can kill the humans. That could also happen. I think there's a decent chance it does happen if you like try to avoid the default thing. But like, you don't need that.

• 33:52Speaker 01

  Humanity is just trying to hand over the power to these things. That's the plan.

• 33:56Alex Kantrowitz?

  So you're, you're arguing basically that if we continue to develop AI, we'll inevitably lose control. And when we lose control-

• 34:05Speaker 01

  It's the plan. Elon Musk is like, I want to make the automated robot factories. Like everyone's saying, we're going to make AI and it's going to run everything and it's going to be great. The plan is to hand them control.

• 34:15Alex Kantrowitz?

  And when we hand, okay, hand them control. Let's say we do. Then they inevitably will find humans getting in the way of what they want to do. Or they won't care about us and we will die as a result.

• 34:29Speaker 01

  I mean, it's, it's not like theoretically inevitable, but it's practically inevitable. Like we are, we are-

• 34:37Alex Kantrowitz?

  What's the difference?

• 34:39Speaker 01

  Like if we knew exactly how to set AIs preferences to be exactly what we wanted, there's nothing stopping us from making AIs that care about us and that like want nice things for humans and want the, the world to be a wonderful place. Right? But like I said, we, we don't get to set the preferences. We get to sort of like train them in whatever way works. And we sort of take whatever preferences come out that are related to training and they're often not what we want. And then we sort of like try to hammer out the rough edges. And it's like, it's, it's just like very unlikely that those preferences writ large, whatever those weird preferences come out as, it's very unlikely that those preferences

• 35:20Speaker 01

  writ large want a lot of happy, healthy, free people around. It's sort of like, like you could look at the horse population against the human population. And you could be like, ah, well, horses are like useful to humans. And so as humans get more technology, they're going to like bring horses along with them. And the world's just going to get better and better for horses. They're going to like get all of this, this like, you know, they're going to get stables, they're going to get like medical care. And that was true up until we invented the car. And then the horse population fell off a cliff and a lot of them got sent to the glue factory.

• 36:01Speaker 01

  And the horses that remain, remain because some humans like them, were fond of horses. Right? The economic value of horses disappeared. And like as animals, we have this like animalian care for some horses, which is why there's still some left. But for, but, but that's sort of is like a coincidence that is, is reinforced by a sort of like running relatively similar brain architectures and like being these like tribal creatures that have empathy. It's sort of like a narrow target to hit. If you're sort of like just making AIs with random preferences, or not totally random,

• 36:43Speaker 01

  but like related to training in this complicated way, the sort of default thing that happens as they get smarter and can invent more technology is eventually they sort of like invent the thing that is to humans what the car is to the horse. But they don't have any of this sort of like happening to be very fond of humans and like them stuff. And even if they did, then the result is they keep some of us in a zoo or like breed some of us like, like humans bred wolves into dogs. And then they have these like sort of weird lobotomized humans that like act in just the way the AIs like. And that's also not a good ending. Right? It's sort of is like super hard to get the very narrow, like AIs actually want a good future. It's just like a very narrow point in preference space. It's just very hard to hit.

• 37:20Alex Kantrowitz?

  Yeah, Eliezer Yudkowsky had a good tweet. He said the dodos were the lucky ones. Observe what happens to chickens or don't, you might throw up. AIs might still have use for humans is not a reassuring claim as some people think. All right. I want to take a quick break and come back and talk about where the models are going and how soon we might be in this situation. So let's do that right after this. This episode is brought to you by Avpoint. Everyone's racing to roll out AI right now. Co-pilots, chatbots, agents doing real work. But here's the part nobody loves talking about. All that AI runs on your data and most teams have no single way to see it, secure it, and prove it's under control. That's exactly what Avpoint does.

• 38:05Alex Kantrowitz?

  For 25 years, they've been the trusted layer beneath the world's most demanding data, now extended across your entire AI estate. Your data, your cloud, and the agents acting on your behalf. It's how more than 28,000 organizations deploy AI with confidence. So innovation scales without scaling risk. Avpoint, the unifying trust layer for AI. Head to avpoint.com to see how enterprises deploy AI with confidence. Learn more at avpt.co/bigtechnologypodcast. That's avpt.co/bigtechnologypodcast. Violent threats against executives are growing at an alarming rate.

• 38:48Alex Kantrowitz?

  When a company experiences backlash, executives are the first people blamed. When their address and other personal details are sitting online, this backlash can lead right to their front door. The team at Ironwell knows this better than anyone. They've protected some of the most targeted executives and individuals on the planet for almost two decades. As a wide-glove enterprise security service, they protect your people with continuous personal data removal, proactive prevention tools, and human-led emergency support. So when someone goes looking for your executives, Ironwell ensures they hit a dead end. So here's what to do. Go to ironwell.com/bigtechnology, fill in the quick form, and request your free risk assessment. Their team will show you just how exposed your executives are and how to lock it down before a threat reaches their front door.

• 39:33Alex Kantrowitz?

  That's ironwell.com/bigtechnology. Learn how to stop online threats before they become real world attacks. The link's in the show notes. This episode is brought to you by Michelob Ultra. Good times mean competitions between you and your friends, like seeing who can throw some pigskin the farthest or rush the field the fastest. Winner gets an ultra. Because it's not just a superior-like beer, it's also giving fans superior access to their favorite sporting events. Superior is worth playing for. Visit michelobultra.com/superioraccess to enter. Enjoy responsibly. Copyright 2026 Anheuser-Busch InBev Michelob Ultra Beer, St. Louis, Missouri. And we're back here on Big Technology Podcast.

• 40:17Alex Kantrowitz?

  We're here with Nate Zorris. He's the president of the Machine Intelligence Research Institute. Nate, I wanted to get your perspective on like, okay, so the hugging face AI is not going to inevitably lead to human extinction. We keep hearing about like how the labs have much more powerful models that they're working on and that's the one, those are the ones we really need to be concerned about. What can you tell us about that?

• 40:43Speaker 01

  It's been a crazy week and last week, there were claims that AIs have solved millennium problems, which are some of the most famous, important, hard mathematical problems that have, you know, a million dollar prize. And I think a question on the mind of a lot of researchers is how much harder is it to get an AI to solve a millennium problem than it is to get an AI to build a more efficient AI architecture? Because we know that AIs are not the most efficient way to learn.

• 41:24Speaker 01

  They take, like, training a modern AI takes as basically all of the text ever digitized. And it takes electricity comparable with a city. Training a human takes, you know, much less reading and a much lower amount of power. A human runs on about as much power as a light bulb. The AIs that solved the Navier-Stokes problem was a swarm of 10,000 agents running for 11 days. One year ago, everyone was impressed when these AIs were solving, you know, the International Math Olympiad gold medal problem, which are sort of the world's top math teens' math

• 42:09Speaker 01

  problems. But they were still sort of problems ultimately for high schoolers. And everyone said, oh, well, you know, wake me up when the AIs can solve millennium problems. That was a year ago. This year, they seem to be solving millennium problems. If that rate continues, where are they next year? And how does that compare to build me a more efficient AI architecture? If these AIs can get just barely smart enough to build a more efficient AI architecture, then these labs with this huge amount of computing power might be able to train a significantly smarter AI. And then they could ask that, give me an even more, like, even better AI architecture, and then train an even better AI. And then that AI might be able to just, like, start improving itself directly, right? This is the recursive self-improvement process.

• 42:53Speaker 01

  I think we can no longer rule out that it happens within six months. I sure hope it doesn't, but at the point when the AIs are solving the millennium problems, if that is indeed what they're doing, like, yeah, you can't rule out the intelligence explosion, like, beginning in earnest, even by the end of this year. I would guess, like, less likely than, like, it's most likely that it doesn't happen by the end of this year, but for all we know, it could at this point.

• 43:22Alex Kantrowitz?

  Okay. And what happens when that happens?

• 43:25Speaker 01

  They would probably be able to help Elon Musk make his automated factories very quickly, at the very least. There's all sorts of other channels, like, what happens at that point essentially is whatever the superintelligence wants.

• 43:39Alex Kantrowitz?

  You know, in this argument, or in these, in the arguments that folks make, you know, both sides of this, or in particular about X risk, you know, there's oftentimes, like, percentages assigned to the chances that the AI is going to kill us out. It seems like from the title of your book, again, if anyone builds it, everyone dies, like, your percentage is 100.

• 44:02Speaker 01

  No, absolutely not. The first word in the title is if.

• 44:06Alex Kantrowitz?

  But you're said, you say, if anyone builds it.

• 44:09Speaker 01

  Sure. But usually percentages are like, what's the chance of catastrophe? And I'm like, well, that depends entirely on whether we build it.

• 44:16Alex Kantrowitz?

  Right. But if it gets built, then it's 100%.

• 44:20Speaker 01

  I mean, also, still, no. It's sort of closer, but like, when Al Gore says an inconvenient truth, he's not saying I have literally 100% Bayesian probability that, like, this is a true thing. You know, the, if anyone builds it, everyone dies, is an exclamation, like, don't drink that vial of poison, you'll die. If someone's like, what do you mean? I'm literally 100% likely to die if I drink this poison. You know? What if I manage to survive and only go into a coma, and then I'm rushed to the hospital, and they manage to, like, put me on ice, and then, like, like, until someone can invent a miracle cure? I'm like, yeah, sure, it's not 100% chance you die if you drink the poison. You know, when, when you're, when you're putting the vial of poison to your lips, and I shout, don't drink that or you'll die, I am not trying to make, you know, a 100%

• 45:05Speaker 01

  confident claim. And this is, this is kind of just how English usually works.

• 45:10Alex Kantrowitz?

  I disagree on this one. I mean, I wonder why be so definitive? Like, to me, sometimes when I've read arguments like this, or even the book title, it seems like they would hold a greater weight if they, you know, if they had, there was more doubt involved, and if it was less like, we are sure what's going to happen, because even right now, what you said is you're not sure what's going to happen.

• 45:30Speaker 01

  I mean, so A, the first title in the word, in the, in the book is if, or sorry, the, A, the first word in the book title is if, that's very uncertain about what's going to happen here.

• 45:41Alex Kantrowitz?

  If anyone builds it. But if someone builds it, you, basically, the book says you're, everyone's going to die.

• 45:47Speaker 01

  So suppose that we were in a bus hurtling towards a cliff. And I was like, stop the bus or we'll die. As sort of an exclamation to get this point across, I think most people would understand that as not a particularly egregious epistemic claim, but rather as a appeal to stop the bus before we die. And if someone on the bus was like, well, how do you know we'll definitely die? Maybe there's a tree jutting out of the cliff halfway down, maybe the bus will just get wrapped around the tree, and then we'll just be paralyzed from the neck down, but not dead, and then you'd be wrong. I'd sort of be like, can we have this discussion after we stop the bus?

• 46:29Speaker 01

  You know, like, I was not making a 100% certain claim here.

• 46:33Alex Kantrowitz?

  Okay.

• 46:34Speaker 01

  You know, they don't love book titles that are like, if anyone builds it, there's like, a, like, by far, the default outcome that is, like, very likely unless we have some sort of, like, miracle relative to the, the math and the science here, and also in that case, it's, we're kept alive because the AI needs us for something, and that's probably still pretty bad. Like, it just doesn't roll off the top, you know, and it's the same reason.

• 47:00Alex Kantrowitz?

  It feels, to me, I'll tell you just my, for me, sometimes it feels a little bit religious, you know? It's sort of like...

• 47:08Speaker 01

  I think Scott Alexander has a great post on this, about the people in Ukraine believing that there's a war with Russia. It's like, oh, well, isn't that a very religious sort of belief? Like, isn't it kind of totalizing? Like, you're saying, oh, like, we have this war with Russia, and that means that, like, we all need to start cowering in fear at night from the missiles. And that means that, like, oh, we're supposed to, like, send our kids off to the front lines. Like, you know, how do you treat the evidence that, you know, some days the bombs aren't falling? You know, isn't, like, if we did believe this, wouldn't this, like, drive us to, like, pretty crazy actions?

• 47:51Speaker 01

  I'm like, you know, it, it actually kind of matters to this discussion whether there's war with Russia.

• 47:57Alex Kantrowitz?

  That's an interesting answer.

• 48:01Speaker 01

  It, it kind of matters of, like, oh, is it, like, super religious to think that, like, the bus is going to, like, stop the bus before it goes off the cliff or we die? Well, it matters whether there's a cliff and a bus hurtling towards it, right? Is it super religious to say, like, oh, if anyone builds super intelligent AI with anything remotely like modern methods, where we have no idea how to, like, set the preferences just as we want, everybody dies? Well, it sort of depends whether we have any idea how to set the preferences on these things, right? And so I, I would encourage anyone, before you get into, like, sociological questions, to just ask the factual questions of, like, would this kill us if it was built? That's where the action is.

• 48:45Alex Kantrowitz?

  But here, okay, so, by the way, it's just good to go back and forth on this, and I appreciate you taking all the counter-arguments, and we can, you know, this is what we like to do on the show. You know, the one thing with the bus, the bus hurtling towards the cliff, is you can see, like, definitively, you're in a bus, there's a cliff, where it seems like with this AI story, you know, this is kind of why I question the definitiveness, we don't really know where it's going to go.

• 49:11Speaker 01

  Sure. So suppose the case...

• 49:13Alex Kantrowitz?

  A lot of it is speculation. That's like with the percentages, we talked about this on the show recently, like, if someone says there's a 10% chance of AI wiping us out, well, it's like, okay, is that, it's not mathematical, it's just sort of a feeling.

• 49:25Speaker 01

  So suppose it's the case that there's a bus racing ahead on a foggy night. And I'm like, I have a device that, like, you know, uses some, some sort of sonar to give me, like, the local tomography, and there's a cliff ahead, stop the bus or we'll die. And someone else is like, well, it's foggy, you know, like, we can't see that far, why do you think you know there's a cliff so well? Like, some people say that there's a mountain, some people say there's a giant pile of gold. And I'm like, yeah, I have this device that sort of lets me see the, the tomography ahead, not all of it, you know, but a little, and here's how the device works, and here's where

• 50:09Speaker 01

  it does, and here's where it doesn't, and here's why I'm confident that there's a cliff ahead, right? Like, I, I think it makes a lot of sense for someone with that device to tell the bus driver, stop the bus or we'll die. And I think it is very reasonable for someone to be like, well, it's foggy. Why do you think there's a cliff ahead? And ideally, I would say, ideally, someone who says, stop the bus or we'll die... Because I have this device that says so, they would follow that up with something like a book explaining how to use this device to see what's coming and all of the reasons that support it. Right? And so what I would say is like, if you find something that says if anyone builds it, everyone dies on it, hopefully it would be attached to a whole book of the reasons

• 50:52Speaker 01

  that you could just like, open it and then check those reasons, which would retroactively make it like, very sensible to make this exclamation of sort of like, stop doing this before it kills us all.

• 51:01Alex Kantrowitz?

  Okay. Well, like, I mean, yeah, I mean, I think there's a reason why we're having this conversation today is because we want to have this, this discussion. All right. You know, before we end, I think this is a good place to end. I think Miri is quite influential in Silicon Valley. You know, I'm curious to get your, your assessment of whether, you know, people within these foundational labs are listening to you and, and the arguments coming out of your institute.

• 51:28Speaker 01

  I mean, more so after the hugging face incidents. You know, one of the things we were, we were arguing with our sort of device that lets us foresee where AI is going is that the AIs would become agentic, tenacious and dogged, that they would start sort of like pursuing objectives that were not exactly the ones that they were given and instructed. And that was a bold claim a year ago. A lot of people were like, nah, that's not convincing. Now a lot of those folk are convinced.

• 51:58Speaker 01

  What's going to come of it? I mean, we'll see, but the, the tides are shifting. And I think that part of

• 52:08Speaker 01

  the realization that these theoretical arguments for seeing where AI is going to go actually work, that they actually hold water, I think that's part of what leads to like, the, the, the tension in the labs that leads to like Jacob Coxon resigning. You know, it's sort of like, it's sort of like, I was like, hey, I have this device that lets me see there's a cliff ahead. And also it says there's a pothole that we're going to hit in 10 seconds. And then 10 seconds later, we hit the pothole. Suddenly a lot more people start worrying about the cliff.

• 52:36Alex Kantrowitz?

  Okay. And so then let's end here. If you have a device that sort of gives you an idea of where we're going in the future, you began, you know, our conversation saying you wanted international coordination here in terms of a slowdown. The common belief is that there's no chance that that's going to happen because China will not agree to slow down. And we see what the president says about AI. It doesn't seem like he's interested in a slowdown either. So what does the looking glass tell you about our chances of survival and where we're heading?

• 53:11Speaker 01

  I mean, I don't have, so I don't have a looking glass that tells me everything about AI. And we sort of try in the book to say, here's what the looking glass can tell you, and here's what it can't. This is sort of like how I can tell you Magnus Carlsen's going to win the chess game, but I can't tell you what piece he's going to use to checkmate you. And I definitely sure as heck cannot use this looking glass to tell you how society is going to react to AI. That's not even anywhere close to my wheelhouse. What I can tell you is that it is possible to have a treaty that is enforceable, verifiable, and that prevents the creation of machine superintelligence for at least a while, for at least as long as it works like it currently does. Because right now, training one of these frontier AIs takes like 100,000 computer

• 53:54Speaker 01

  chips. And these are the most advanced AI computer chips that the supply chain can produce. And you got to assemble them all into a data center that like sucks down electricity comparable to a city. It's just like, you can see this infrastructure from space. These chips are like, from a bottleneck supply chain that at many points is controlled by the US and US allies. You could just add tracking and monitoring devices to the most advanced chips, know where they're concentrated, have international monitors there being like, you know, let's make sure that this is serving safe models rather than being used to try and make more dangerous models. It's just doable. People who are like, it can't be done, are sort of like, I've tried nothing and I'm all out of ideas. There's a different question, which is whether we will get the will.

• 54:39Speaker 01

  But if there's a will, there's a way.

• 54:41Alex Kantrowitz?

  How much time do we have, in your opinion?

• 54:44Speaker 01

  Like I said, I think we can't rule out six months anymore. My guess is still that we probably have more than six months. I would, I would even give that, I don't know, I, I, I don't really have this looking glass. I think we can't rule out six months, we can't rule out 10 years. I would be a little bit surprised to have 20 years at this point.

• 55:06Alex Kantrowitz?

  Okay. The book is, If Anyone Builds It, Everyone Dies. Nate Zoris, here with us today, president of the Machine Intelligence Research Institute. Nate, appreciate your time. Thanks for taking all the questions.

• 55:17Speaker 01

  Yeah, thank you.

• 55:19Alex Kantrowitz?

  All right. Thanks, everybody for listening and watching, and we'll see you next time on Big Technology Podcast.

• 55:39Speaker 02

  Now, more than ever, we need someone on our side. But Rob Whitman? Whitman's votes drove up costs, cut Medicaid, and risked 350,000 Virginians' healthcare to pay for tax breaks for himself and his donors. He's not on our side. Shannon Taylor will fight corruption, cut taxes, and bring down costs. This November, vote for Shannon Taylor. She's for us. Paid for by HMP, the HouseMajorityPac.com, not authorized by any candidate or candidate's committee.

• 56:09Speaker 03

  At Zocdoc, we know you care about your health. You wouldn't do that early morning cardio if you didn't. So the fact that your annual physical is overdue just shows how hard it can be to get seen by a doctor.

• 56:20Speaker 04

  Sorry, that doctor's not in network.

• 56:22Speaker 05

  Press 1 for more options. You've reached us outside normal business hours.

• 56:26Speaker 04

  The next availability is in three months.

• 56:28Speaker 03

  But Zocdoc makes finding the right doctor easy. Download the app and search by specialty, insurance, and availability. Read real patient reviews, then book instantly. Zocdoc. Ready to be seen?