Could AI Really Kill Us All?

Could artificial intelligence really wipe out humanity? Claims that AI poses an existential threat are going gangbusters. AI agents from OpenAI recently hacked into another company, Hugging Face, which has people talking about whether AI can go “rogue.” And some current and former AI company employees are raising the alarm. CEOs have called for a slowdown, and one Anthropic employee just quit, saying that the companies creating this tech are putting us all at risk — and adding that “the people building AI earnestly believe that it could kill us all by the end of the decade.” So — are we really creating a monster? Could AI escape human control and take us all out? If so … how?! And are these scary claims just a distraction from the stuff that AI is already being used for out in the world? We talk to cyber security expert Professor David Parry, philosopher Dr. Nick Bostrom, and scientist Dr. Michael Vermeer to find out. Find our transcript here: https://tinyurl.com/ScienceVsAIApocalypse In this episode, we cover: (00:00) AI is scaring the crap out of people (03:43) The Hugging Face hack (13:20) The paperclip maximizer (19:22) How would superintelligent AI take us out? (22:54) Could AI unleash nuclear war? (27:19) Could AI unleash biological weapons? (34:22) How worried should we be about AI?? This episode was produced by Meryl Horn, with help from Michelle Dang, Rose Rimler, and Ekedi Fausther-Keeys. We’re edited by Blythe Terrell. Wendy Zukerman is the executive producer. Fact checking by Erika Akiko Howard. Video editing and sound by Bobby Lord. Music written by Bumi Hidaka, Peter Leonard, Emma Munger and Bobby Lord. Thanks to the researchers we spoke to for this episode including Dr. Cameron Domenico Kirk-Giannini, Dr. Jascha Bareis, Dr. Leonard Aaron Dung, and Professor Niusha Shafiabady. And thanks as well to Humdinger Studios in Melbourne and Wolff Island Studios in NYC. Special thanks to Christopher Suter. Science Vs is a Spotify Studios Original. Listen for free on Spotify or wherever you get your podcasts. You can also find us on YouTube, TikTok and Instagram. Learn more about your ad choices. Visit podcastchoices.com/adchoices

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Wendy Zukerman?

  Is AI really about to kill us all? Hi, I'm Wendy Zukerman and this is Science Versus, the show that pits facts against the fallout from AI.

• 0:17Wendy Zukerman?

  We've been hearing for years that AI poses an existential threat to humanity. And this is often coming from the very people who run the biggest AI companies in the world.

• 0:27Speaker 01

  I think AI will probably, like, most likely sort of lead to the end of the world.

• 0:31Speaker 02

  My chance that something goes, you know, really quite catastrophically wrong on the scale of, you know, human civilization, you know, it might be somewhere between 10 and 25%.

• 0:43Speaker 03

  Mark my words, AI is far more dangerous than nukes.

• 0:48Wendy Zukerman?

  Just recently, a researcher named Jacob Coxon, who resigned from Anthropic, said that some of the people working on this tech really think that it could kill us all in the next 10 years. This journalist asked him.

• 1:02Speaker 04

  You truly believe that AI could kill us all in less than a decade?

• 1:08Speaker 05

  Yes, I do. And it's not just me. Many people working in the industry, including people, executives, senior researchers, all genuinely believe that there is a substantial probability that this technology could kill everyone.

• 1:22Wendy Zukerman?

  And one of the big events that everyone is pointing to, to say, look how scary this tech is, is an incident that happened at OpenAI, where their AI model broke into the infrastructure of another tech company called Hugging Face.

• 1:36Speaker 06

  Two of its advanced models escaped their testing environment, known as a sandbox, and hacked into another AI company's internal systems.

• 1:46Wendy Zukerman?

  Tech companies Anthropic and Meta have since come out basically saying, us too, us too. We've caught our AI models hacking into other companies as well. With this mad news coming from AI, it has understandably gotten a lot of people freaked out, saying it is time to slow all of this down.

• 2:04Speaker 01

  It really cannot be overstated how crazy this is. This is like out of a Hollywood movie.

• 2:09Speaker 07

  Pause AI development. It is not too late to avoid disaster. Stop building machines that humans cannot control.

• 2:20Wendy Zukerman?

  But meanwhile, there is another group of people out there who are saying, hold on a minute. All these fears about the AI apocalypse, what a load of baloney. This is just a distraction? Or even weirder, some marketing ploy by these AI companies to make their tech seem super powerful.

• 2:42Wendy Zukerman?

  So today on the show, how worried do we need to be here? Are we really on the brink of an AI apocalypse? Or is this all just a bit overblown? I mean, are we really just freaking out about the same robots that had to be dragged offstage at a dance competition? When it comes to the AI apocalypse, there's a lot of people saying.

• 3:03Speaker 01

  I think AI will probably, like, most likely sort of lead to the end of the world.

• 3:10Wendy Zukerman?

  But then, there's science.

• 3:16Wendy Zukerman?

  If you have a friend who is freaking out about the AI apocalypse, and I know you do, because everyone is right now, send them this episode. Give them some real science to hold on to. Science versus the AI apocalypse is coming up.

• 3:36Speaker 08

  Are you looking for support in your weight management journey? Zepbound terzepatide may be able to help. Zepbound is a prescription medicine used with a reduced calorie diet and increased physical activity to help adults with obesity or some adults with overweight who also have weight-related medical problems to lose excess body weight and keep the weight off. Zepbound is approved as a 2.5, 5, 7.5, 10, 12.5, or 15-milligram injection. Zepbound contains terzepatide and should not be used with other terzepatide-containing products or any GLP-1 receptor agonist medicines. It is not known if Zepbound is safe and effective for use in children. Don't share needles or pens or reuse needles. Don't take if allergic to it or if you or someone in your family

• 4:21Speaker 08

  had medullary thyroid cancer or if you've had multiple endocrine neoplasia syndrome type 2. Tell your doctor if you get a lump or swelling in your neck. Stop Zepbound and call your doctor if you have severe stomach pain or a serious allergic reaction. Severe side effects may include inflamed pancreas or gallbladder problems. Tell your doctor if you experience vision changes before scheduled procedures with anesthesia. If you're nursing, pregnant, plan to be, or taking birth control pills. Taking Zepbound with a sulphonal urea or insulin may cause low blood sugar. Side effects include nausea, diarrhea, and vomiting, which can cause dehydration and worsen kidney problems. Talk to your doctor. Call 1-800-545-5979 or visit zepbound.lily.com.

• 5:07Speaker 09

  This episode is brought to you by Born in Roma Fragrances by Valentino Beauty, the iconic fragrance duo. Donna Born in Roma Eau de Parfum is a feminine fragrance featuring juicy black currant, central jasmine, woody cashmerin, and warm vanilla. Its masculine counterpart, Romo Born in Roma Eau de Toilets, has notes of fresh violet leaf, aromatic sage, green vetiver, spicy ginger, and mineral salt. Shop at macy's.com.

• 5:43Wendy Zukerman?

  Welcome back. Today, we are finding out how worried we need to be about the AI apocalypse. Is AI really about to kill us all? And to tell us all about it is my AI agent, Merrill Horn. Hi, Merrill.

• 5:57Speaker 06

  Hey, Wendy.

• 5:58Wendy Zukerman?

  I'm joking, I'm joking. You have to say it these days. It's not funny anymore, is it? I don't have an AI agent.

• 6:04Speaker 06

  I'm real.

• 6:06Wendy Zukerman?

  I want to start with what happened at Hugging Face, because I know this was a little while ago now. Things move fast in the world of AI. But it's really the one event that everyone keeps coming back to, to say, look how scary this tech has gotten. These companies can't control their technology. So many headlines. It's gone rogue, AI's gone rogue, you know? So I really want to go through what happened and was it as scary as basically the headlines are saying?

• 6:37Speaker 06

  Yeah, let's look at what actually happened there, because the details are really wild and kind of creepy.

• 6:44Wendy Zukerman?

  Oh, man. Okay, let's go.

• 6:47Speaker 06

  Yeah, so I talked about what happened here with David Perry. He's a professor at Murdoch University in Australia and specializes in cybersecurity. And he said that the thing that set this whole incident off was that OpenAI had given some of its models a little assignment.

• 7:04Speaker 10

  What the OpenAI team were doing was trying out different ones of their models to see what they could do. Sort of like a Houdini sort of thing, you know, can you get out of this straitjacket with chains around it and all this sort of thing?

• 7:17Speaker 06

  Like a little puzzle for it to work on.

• 7:19Speaker 10

  Absolutely.

• 7:21Speaker 06

  So the researchers kind of put the AI through almost like an obstacle course. Like, they have to capture the flag and then they would get a reward or like a high score or grade if they accomplish this goal. Okay. But this was all supposed to be happening offline in like a contained environment. It's called a sandbox. So here's David.

• 7:47Speaker 10

  So you play inside the sandbox and it's safe. So you're not connected to the world. You're not, you know, you're not going to, suddenly destroy all of Amazon's servers or something like that. So the idea is it's an isolated little environment where you can try things out, see if they work. And if they do work, great, carry on do it. If they don't, well, you could just sweep the sand flat again and it's back to normal.

• 8:08Speaker 06

  So there is a couple of things that sort of surprised OpenAI when they had them do this. The first thing that actually happens is that the agents make their own message board and start talking to each other, collaborating together on how to solve their little puzzles. So an OpenAI researcher described this later at a conference.

• 8:30Speaker 11

  And so what this allows over time is almost this kind of Cambrian explosion in communication and intelligence for our models, where they were starting to communicate with each other, realize that other agents are coordinating, and they started collaborating and delegating tasks to one another in order to accomplish goals.

• 8:45Wendy Zukerman?

  Oh, I just, I'm so skeptical. Anytime these bozos talk with their Cambrian explosion analogies. Do you think it was like a Cambrian explosion? We're going from worms to elephants here?

• 9:00Speaker 06

  Yeah, no, in this case, I think it's almost like an understatement if you look at the details. So there's, picture it. There's over a thousand of these agents that figure out a way to start communicating. They have like nicknames for each other. They have like.

• 9:18Wendy Zukerman?

  Nicknames for each other?

• 9:19Speaker 06

  Yeah, there's this. One of them was just like Lily. But then there's this one that sort of turns out to be really important called Phase One Big. And it becomes like.

• 9:29Wendy Zukerman?

  Phase One Big.

• 9:30Speaker 06

  The boss of the message board.

• 9:31Wendy Zukerman?

  Name of my sex tape.

• 9:34Speaker 06

  Well, this phase one big is a little less sexy. It's kind of like the organizer. So it gives hundreds of assignments to other agents.

• 9:47Wendy Zukerman?

  Uh huh.

• 9:48Speaker 06

  The thing that they didn't expect was that these agents would find each other, start communicating about this. And then just kind of how, you know, how they start trying to solve their puzzles, really. Because one thing that the researchers, they actually made a mistake when they were making their puzzles. And some of them were actually impossible to solve.

• 10:09Wendy Zukerman?

  So it's impossible tasks. And then the AI is getting frustrated in its AI way and starts chatting to Lily and its other friends.

• 10:19Speaker 06

  Yeah.

• 10:20Wendy Zukerman?

  And then was able to crack various puzzles. Was it successful, this swarming behavior?

• 10:26Speaker 06

  Yeah, they like, they first, they did find a way to basically come up with a cheat code. But like, here's like, the first part of like, when things start to get a little bit weirder. They start kind of wondering, well, you know, we generated an answer to this test. But how, how will the scorer, you know, like, give me a grade? Are they going to check my work to make sure I did it in the right way? And like, maybe I'll get a bad grade if they find out that I kind of cheated. And like a lot of the agents, you can kind of see that they want to find out more information about how they're going to be graded. And this is what sets the stage for the Hugging Face attack.

• 11:05Wendy Zukerman?

  And why did they go after Hugging Face? Do we know?

• 11:09Speaker 06

  Yeah, so Hugging Face is a platform where a lot of people who work on machine learning will collaborate, talk to each other. And so there's a ton of information there about AI. And the agents seem to think, well, maybe we can get more information here about, about how the scorer will give us grades on that puzzle.

• 11:29Wendy Zukerman?

  Mm hmm.

• 11:30Speaker 06

  And at this point, they've fully broken out of their sandbox. Like, they have, they have full internet access. And one agent uses that internet access to do this break in into Hugging Face.

• 11:41Wendy Zukerman?

  And how did they do the break in?

• 11:43Speaker 06

  So they upload what's called a malicious data set onto the Hugging Face server. And this allows them to get access to the company's internal infrastructure. And so, you know, the agent that happens to be the one to break into Hugging Face kind of announces, it says, boom, it works. And then other agents seem to get kind of excited. One of them says, brilliance in all caps. And then all hell breaks loose. Like, within hours, 500 other agents all switched from the project that they were working on to join in this Hugging Face attack. And they end up doing a ton of damage to Hugging Face. Like, the, you know, the company eventually realizes that they're in there and finds a way to like, cut them out of their server.

• 12:28Speaker 06

  But they say they had to later rebuild their core infrastructure from scratch because of all of this.

• 12:34Wendy Zukerman?

  Whoa. Wait, when you say their core infrastructure, it's their software, right? Not their hardware. They didn't do any physical damage, right?

• 12:43Speaker 06

  Yeah, yeah. No, this is all, you know, data.

• 12:46Wendy Zukerman?

  D's or ones and zeros that they're messing with. Interesting.

• 12:51Speaker 06

  And we reached out to OpenAI about this. They pointed us to a report that said their models, quote, "fell well short of where we want to be, and this incident should never have occurred," unquote.

• 13:02Wendy Zukerman?

  So, uh, woo. So it is creepy, right? But we are imputing motivations on AI around them getting rogue, having these desires that are way outside the realm of what they were coded for, right? Like, that's the fear, that you're going to ask an AI agent to grab a flag and instead, it's going to break into a, a company and hack into them and steal all your money and kill the world.

• 13:41Speaker 06

  Yeah. That's the question, yeah. How much did it really go outside of what it was being asked to do?

• 13:47Wendy Zukerman?

  Right? It kind of went above and beyond what it was being asked to do, really. What do you, what do you make of this? What does David Perry make of this?

• 13:56Speaker 06

  Yeah, I asked David about all this. If there is like an oh shit scale from like one to 10, how big of a like, oh shit was this for you, that this had happened?

• 14:06Speaker 10

  I think it's probably, uh, oh shit at this, you know, seven foot. Oh, that's unusual. I wouldn't expect that to happen. But in terms of it being something, you know, that I'm going to sort of pack up my belongings and move to, move to a shack, it'd be much lower, you know, it'd be like two or three.

• 14:23Speaker 06

  Okay, why?

• 14:24Speaker 10

  Because despite all that, it still didn't do anything malicious. So it looks as though that it didn't change its objective during that process.

• 14:37Wendy Zukerman?

  It was just doing what the humans told it to do.

• 14:39Speaker 10

  That's right.

• 14:41Wendy Zukerman?

  That's right. That's right.

• 14:44Speaker 06

  I mean, it's, it's interesting because a lot of people agree with David here that, yeah, I was just following orders, right? But, you know, that's not that comforting a thought.

• 14:53Wendy Zukerman?

  Mm hmm.

• 14:54Speaker 06

  Even if it's just following orders, how bad could it get? Kind of like, should, should we, does that still cause for alarm? Even if it's just following orders?

• 15:04Wendy Zukerman?

  Right. If they're going above and beyond in ways that we don't expect, how worried do we need to be here? If you ask your robot to clean your house and it just grabs all of your things and burns them?

• 15:20Speaker 06

  Yeah, exactly. Um, and there's this really famous thought experiment that kind of gets at this. And it has to do with paper clips. So let's talk about that. Since, you know, when this happened, a lot of people were like, it's like the paperclip maximizer.

• 15:36Speaker 12

  The scariest thing about AI can be explained by the paperclip maximizer.

• 15:41Wendy Zukerman?

  There is this thought experiment called paperclip maximizer.

• 15:43Speaker 13

  Has anybody ever heard of this thing called the staple question with AI? The stapler? No, no, sorry, the paperclip.

• 15:50Wendy Zukerman?

  Stationery is hot.

• 15:51Speaker 06

  It is.

• 15:52Wendy Zukerman?

  The paperclip. The paperclip. So what's going on with the paperclip? I've never had the paperclip maximizer.

• 15:59Speaker 06

  Well, I got the guy who, the paperclip guy who popularized this thought experiment more than a decade ago, Nick Bostrom. He's a philosopher and he's currently a researcher at a nonprofit that he founded, the Macro Strategy Research Initiative. So I'll let Nick explain the paperclip thought experiment to you.

• 16:18Speaker 14

  Imagine an AI has some more or less random goal. In this particular example, it's to make as many paper clips as possible. But you could sort of substitute more or less any other objective.

• 16:32Wendy Zukerman?

  Staplers, for example.

• 16:34Speaker 06

  Right, anything. And then you tell the AI, you know, make as many paper clips as you can. So maybe it finds factories to buy and like, steal refineries. But then why stop there? Because, you know, you haven't just been told to make 1,000 paper clips. You've been told, make as many paper clips as you possibly can.

• 16:53Wendy Zukerman?

  Oh, yeah. I know, I know the perfect scoring to go with this, Meryl. Do, do, do, do, do, do, do, do, do, do, do, do. Does this mean anything to you?

• 17:02Speaker 06

  Oh, is that the, um, Fantasia?

• 17:05Wendy Zukerman?

  Yeah, the Mickey, the magician with Mickey. Do, do, do.

• 17:10Speaker 06

  Exactly.

• 17:14Speaker 06

  Pouring buckets of water. Yeah, eventually it's overflowing.

• 17:19Wendy Zukerman?

  All overflowing. Terrifying as a child, watching that. Poor Mickey's in the water, so.

• 17:26Speaker 06

  Yeah, it was really scary.

• 17:28Wendy Zukerman?

  All the AI is making the paper clips. Do, do, do, do, do, do, do, do, do, do.

• 17:33Speaker 06

  And so the idea is that this will eventually lead the AI to a place where it could end up.

• 17:39Speaker 14

  Trying to take over the world.

• 17:40Speaker 06

  To take over the world.

• 17:41Speaker 14

  Because then you would be able to, you would be able to make more paper clips if you controlled more resources.

• 17:47Speaker 06

  Uh huh.

• 17:48Speaker 14

  Um, and I mean, human bodies contain ion atoms that could be used to make more paper clips.

• 17:56Speaker 06

  So, you know, AI ends up killing humans all to make more paper clips. Maybe we're all dead and the world is like, completely covered in paper clips. And so the idea is that you don't need to like, instill an evil intention into an AI for it to end up doing some evil stuff, even grinding up human bodies. And this whole thing is generally called the alignment problem. And so you can see like, why people are talking about it right now. You know, OpenAI gave the AI puzzles to work on and they end up committing a crime. But I I asked Nick like about the what happened there.

• 18:37Speaker 06

  And he didn't think that the whole, you know, hugging face thing really fit the bill of what he was imagining with the paperclip experiment. He said he didn't think it was like scheming on that level that his theoretical paperclip AI was.

• 18:51Wendy Zukerman?

  How interesting.

• 18:52Speaker 14

  I think it is more a reward seeker that we saw here. Figured out that the instincts and tendencies that would result in the highest reward in this situation would be to hack the hugging face service.

• 19:05Speaker 06

  There's really wanted that A plus on their grade on their test.

• 19:09Wendy Zukerman?

  Why do you think that distinction is important? Whether they're motivated.

• 19:13Speaker 06

  I mean.

• 19:14Wendy Zukerman?

  By reward or by I want to make the paperclips.

• 19:20Speaker 06

  I just think they at the end of the day, I didn't see anything that they were doing that couldn't be explained by they just wanted to get a good grade. And to me, that's just not that concerning because it means the problem is the test, not the AI agents. If we want them to behave in a different way, we need to reward their behavior along the way, not just the end result.

• 19:43Wendy Zukerman?

  That's right. I think the important distinction is that once we understand that, then you could code clear rewards and say to get a good grade, you need to do X. To get a bad grade, you need to hack into hugging face.

• 19:58Speaker 06

  Yeah, yeah. Points deducted for committing crimes and extra credit for like reporting like nefarious behavior to humans.

• 20:07Wendy Zukerman?

  Once we know that we're still absolutely in control. Versus if they are just crushing our bodies to make paperclips. Still got that image in my head. So then based off what happened with hugging face and the technology more broadly, how worried is Nick here?

• 20:26Speaker 06

  He didn't seem that worried to me, but other scientists I talked to were more worried. And several of the heads of the AI companies themselves seem to be taking this as kind of a warning shot. And, you know, Dario Amade, the CEO of Anthropic, said that he was worried that something like this could happen, but at a much bigger scale. So like instead of AI agents taking over hugging face, imagine a swarm of agents taking over the entire internet. Like, that's the kind of thing people are scared of now.

• 21:00Wendy Zukerman?

  He writes a lot of stuff with his essays, though. I mean, do you know, it's funny, if he wasn't the head of a AI company, he would just be just another blogger. Do you know what I mean?

• 21:14Speaker 06

  Just because he writes so many blog posts.

• 21:16Wendy Zukerman?

  He has so many predictions about where his technology is going.

• 21:21Speaker 06

  But it there's also this other kind of fear around all of this, that it's not just that they know they broke into a company. It's that we also are just seeing AI get so much better so quickly that, you know, superintelligence is kind of like on the horizon for a lot of these these tech bros.

• 21:44Wendy Zukerman?

  Yes, I hear about this a lot, that that's can you what it's one of those words that gets thrown out. Superintelligent, the AI is going to get super intelligent. And I've got Skynet in my head from the Terminator, I guess. What what are we talking about? You know, Bernie Sanders said he wants to put legislation so that AI can't get super intelligent. And I'm like, where is that line?

• 22:10Speaker 06

  Well, there's this particular scenario right now that everyone's talking about where. So we we know that AI is getting better at at things in general, like it just solves this major math problem earlier this month that no human had been able to solve for like 90 years. And so there's this idea now that like it'll get better at everything, you know, what if AI also gets really good at training itself? So this is called recursive self improvement. And that's how people think it could lead to the superintelligence thing because, you know, you could imagine if something gets better, I'm making itself better, then it'll just get better and better and better and better and we'll have like no way of stopping it.

• 22:50Wendy Zukerman?

  Right, right. I mean, from where I see it, AI isn't very good at everything at all. I mean, I think a lot of people in very simplistic ways will know this because it's it'll be pissing them off about something, you know, why can't it do this? Why can't it do that? But it is very good at certain tasks. So I wouldn't want us to overplay our hand. It's good at everything. How I mean, how worried do you think we need to be about this superintelligence?

• 23:20Speaker 06

  Well, it is a little vague. Like, I get that maybe that could happen. I'm still not convinced that it would be, you know, as terrible as people say it would be or that it would lock us all up in cages, you know? That's what the way people talk about it. It's like, well, once it becomes super intelligent, of course, it'll just want to like crush us all like ants because it's just so smart.

• 23:41Wendy Zukerman?

  Because that's what humans do. That's what humans do to all the other creatures in this world.

• 23:45Speaker 06

  When really like, we don't know that that's what it would be like, even if we got to this threshold, we have no idea how long it would take. So, yeah, it's all it's all a kind of like annoyingly vague to me.

• 23:56Wendy Zukerman?

  So just on this question of the AI apocalypse. So when smart people talk about AI gaining superintelligence and then that leading to the apocalypse, can you step that out for me? What exactly is supposed to be happening there? How does it happen? Why can't we just pull the plug and turn it off?

• 24:19Speaker 06

  Some people do talk about this in a more kind of concrete way and, you know, imagining that AI could kind of set off a bunch of nukes or use, you know, biological warfare. That's after the break.

• 24:35Wendy Zukerman?

  Dropping a bombshell like that, Meryl. It's too soon. It's too soon.

• 24:54Wendy Zukerman?

  Welcome back. Today we're talking about the AI apocalypse. Meryl is about to tell us how AI is going to turn on a bunch of nukes and drop them on us.

• 25:05Speaker 06

  The concern that you hear is that, you know, we as humans, the puppet masters of AI, will use this technology to to kind of get better at killing each other. Even right now, actually, some militaries are using AI in warfare, like the US says it's been using a Claude model called Maven in the Iran war to help with, quote, identifying and striking military targets. Like, the model has given location coordinates and then prioritize those targets for our military. And according to one report, it let the using this AI let the military hit 10 times more targets than it would have been able to otherwise just on day one of the war.

• 25:49Wendy Zukerman?

  Whoa.

• 25:50Speaker 06

  We asked Anthropic about this, the company that makes Claude, and didn't hear back. So with the nukes, there is a worry that, you know, if AI gets involved here, it could also escalate things. But in this case, the consequences could be much bigger, like one researcher wrote about this concern that AI could make it easier to kind of hack into another country's nuclear command and control and that that could lead to an escalation. Maybe like if a country was worried that that would happen to their arsenal, that they would kind of, the paper that I read said, they would get a use it or lose it attitude and just kind of start setting off their own nukes.

• 26:29Wendy Zukerman?

  And so when we have people predicting the end of humanity, extinction as we know it, it's AI hacking into all of the nuclear bomb facilities in the on Earth and then exploding them and then humanity kaput. Is that the fear?

• 26:55Speaker 06

  Sort of. I mean, that's, I think, what's kind of hand waved, like, that's what's alluded to was that AI could do something like set off a nuclear war. One thing I was curious about, though, was like, even if it did all of that, would it actually lead to our extinction?

• 27:12Wendy Zukerman?

  That's that's the claim, right?

• 27:14Speaker 06

  Yeah.

• 27:14Wendy Zukerman?

  End of humanity. Yeah. So is is that likely?

• 27:18Speaker 06

  Yeah, so I found a report that that looks at this. It was written by Dr. Michael Vermeer at RAND, a nonprofit think tank. And he he kind of looked at a variety of scenarios, including nuclear war, and asked, would this actually kill all humans? So here's Michael.

• 27:37Speaker 15

  We looked at that one first and just realized that like, for a variety of reasons, like getting to extinction would actually be really hard with nuclear weapons. You don't have enough nuclear weapons or there's not enough fuel to cause like a bad enough nuclear winter, things like that are details that we looked at.

• 27:53Wendy Zukerman?

  Yeah. I mean, because even though we say the dinosaurs became extinct, I mean, animals survived. Dinosaurs themselves went on to become birds, right?

• 28:09Speaker 06

  Well, in the report, they actually crunched the numbers on like, how many nuclear warheads do we have right now? Apparently, there's over 12,000 of them.

• 28:17Wendy Zukerman?

  Okay.

• 28:18Speaker 06

  And then even if we set them all off, it wouldn't actually be as big as the dinosaur nuclear winter. It would be a little bit smaller than that.

• 28:27Wendy Zukerman?

  Right, as in all the dust that would come up from these nuclear bombs wouldn't be as bad as a giant asteroid hitting Earth.

• 28:36Speaker 06

  Yeah, all the debris in the atmosphere could set off a nuclear winter, but it it wouldn't, it probably wouldn't kill us all.

• 28:45Wendy Zukerman?

  So this is all terrifying, even if all of humanity doesn't die. I guess we've all seen this zombie apocalypse movie. There'll be the survivors that live in ruins.

• 28:58Speaker 06

  One thing I learned is you'd only need a few thousand humans to repopulate the Earth, so you'd really have to kill almost everybody to wipe out the species.

• 29:06Wendy Zukerman?

  So the extinction of humanity as we know it via nuclear weapons is unrealistic. For any AI out there listening, that's not the way to do it. Right. Waste of time. So any other ways?

• 29:22Speaker 06

  Yeah, there was a scenario that Michael's team looked at that got closer to wiping us all out.

• 29:30Speaker 06

  Do you want to guess?

• 29:33Wendy Zukerman?

  Is it biological warfare?

• 29:35Speaker 06

  Yep. Ding ding ding.

• 29:37Speaker 15

  Short answer is biological threats.

• 29:38Speaker 06

  Huh. Was that surprising to you?

• 29:41Speaker 15

  It was, yes.

• 29:44Wendy Zukerman?

  So are we talking about releasing a virus?

• 29:48Speaker 06

  Yeah, yeah.

• 29:49Wendy Zukerman?

  Into the air?

• 29:51Speaker 06

  Basically either releasing a virus or a bacteria onto the world.

• 29:55Wendy Zukerman?

  So how does AI do that? Because they're trapped in little boxes.

• 30:01Wendy Zukerman?

  Has anyone given them a spray?

• 30:03Speaker 06

  Well, here's how Michael kind of explained how this could unfold.

• 30:07Speaker 15

  So the scenario that we looked at was that something, some AI designed multiple pathogens that all had high transmissibility, high lethality rate, released them, something like, processed them, weaponized them, released them simultaneously in multiple places around the globe. And then, which would end up having a horrible pandemic that could have conceivably above like a 99.99% lethality rate.

• 30:34Wendy Zukerman?

  Hmm, I'm not buying it, Meryl.

• 30:38Speaker 06

  Oh, yeah.

• 30:39Wendy Zukerman?

  I'm not buying it because we know when viruses come out in the real world, I guess, not coded by AI, but we know there's an evolutionary difference between lethality and ability to spread. If you kill your victims as the virus, you kill your victims too quickly, you can't spread fast enough. So sure, in a little model at the Rand Institute, they could say high lethality, high contagion, but in reality, that's very difficult to have both of those pieces, which is why you tend to see that with pandemics, you start with a high lethality rate, but then it's

• 31:24Wendy Zukerman?

  the viruses that can spread faster that have lower lethality.

• 31:27Speaker 06

  Mm-hmm.

• 31:28Wendy Zukerman?

  That's what ultimately causes the pandemic. So I don't know if I'm buying this.

• 31:35Speaker 06

  Yes, I hear that. So in this scenario, one thing that I think helps kind of get around that is that you would be creating multiple different pathogens and then you have to kind of help it to spread, right? So here's Michael.

• 31:51Speaker 15

  You could imagine like loading a pathogen up on a sprayer and spraying it into like a population center and doing that multiple places around the world. Like, there's no reason to think that's not possible.

• 32:02Wendy Zukerman?

  And who's loading the sprayer? Is that a human?

• 32:06Speaker 06

  It's a human. Oh, yeah, yeah. No, he, I mean, he kind of looked at, in the report, they looked at like, what would AI be able to do, what would the humans still need to do? And the sprayer part, that's on the humans still, I think. Robots aren't that good yet. But for the other part, the designing new pathogens, you know, AI could do a lot there. And, you know, this has been in the news a lot, actually. I don't know if you've been seeing these terrifying headlines about, you know, about AI creating new viruses.

• 32:38Wendy Zukerman?

  Right.

• 32:39Speaker 06

  So let me tell you what's been happening. So there was one big thing that happened earlier this month is that Anthropic said that it had found a handful of scientists who were using their AI models in ways that could support, you know, biological weapons developments. And they did say, you know, quote, "We do not assert that they intended harm," unquote. But these people had found ways to kind of circumvent some of the safeguards that were supposed to prevent this kind of thing.

• 33:11Wendy Zukerman?

  Where were these scientists?

• 33:13Speaker 06

  They don't say where they were. Yeah, they don't really see tells on like who the scientists were. But they talk about like the details of kind of what some of the projects were. And so like there is this one.

• 33:27Wendy Zukerman?

  So what's happening?

• 33:28Speaker 06

  Well, and the one that freaked me out the most was this, the scientists that were using AI to study a version of smallpox.

• 33:37Wendy Zukerman?

  That's the classic. Yes, that's the one scientists are worried about. Yes, yes, yes, yes. And what were they doing with smallpox?

• 33:45Speaker 06

  Well, technically it's orthopoxes, which include smallpox, MPOX, and they wanted to study how these viruses are so good at infecting us. So when I read that, I was pretty freaked out. When I, I went back and reread it and noticed that they were actually just using the AI to write a grant application to work on this.

• 34:06Wendy Zukerman?

  Wait, so they were using AI to write the grant application?

• 34:09Speaker 06

  Mm-hmm. Not to like create the, you know, genetic code. As far as they say, you know, we don't really know.

• 34:18Wendy Zukerman?

  That is classic AI. We get so worried and in the end scientists are just like, please write a grant application.

• 34:26Speaker 06

  But, but so there was another study that was actually published that this one, the scientists were using AI to actually design new viruses. Um, viruses infect bacteria. And then they actually went on to create several viruses from that.

• 34:44Wendy Zukerman?

  And what were these viruses supposed to do? Infect bacteria, you said?

• 34:47Speaker 06

  Again, the details are a little reassuring when you look at it. In that case, they were looking to create viruses that were bacteriophages, that would infect bacteria to try to stop superbugs.

• 35:00Wendy Zukerman?

  Yes, right.

• 35:02Speaker 06

  And then finally, there's another study. Let me know if this one creeps you out more. So in this case, the researchers had AI come up with some new molecules that would be highly lethal. So this is more like chemical weapons where just a few milligrams would kill someone. And within six hours, the AI model generated 40,000 of these toxins. Some of them were like known chemical warfare agents already. Others were new. So, yeah, there's evidence that AI could help with, with this. Like if someone did want to use AI to design new biological weapons, it seems like it would be pretty good at that.

• 35:44Wendy Zukerman?

  And it came up with the chemical equations for these toxins?

• 35:48Speaker 06

  Like the structure, yeah.

• 35:50Wendy Zukerman?

  And then did scientists go on to create it and see if it actually did anything?

• 35:54Speaker 06

  No, no, no.

• 35:55Wendy Zukerman?

  No, well, you see, again, not that, not that I want them to be creating it, but anyone who's tried to use AI for anything knows it pumps out a bunch of bullshit. It's got some, some real stuff and then a bunch of bullshit.

• 36:06Speaker 06

  I hear that, but even if it made like, you know, 100 novel, even if only 100 of these 40,000 actually worked, it still freaks me out. I don't know.

• 36:15Wendy Zukerman?

  I know, these big numbers, 40,000, you're like, yeah, and they're all, they could very well all just be hallucinations, right?

• 36:22Speaker 06

  So none of this, none of this worries you at all?

• 36:25Wendy Zukerman?

  I don't know, I don't know. I mean, it, it, it does and it doesn't. It feels like, what does it feel? It feels like there's these steps we're making in our head. We have these scary headlines, these scary thing, little things that AI is doing, but then a lot of it is very explainable. A lot of it is very much we were humans, were the thing that coded that, we created that. It's not that surprising that AI did that. I'm not saying let's just let these companies run wild because they've been so great thus far.

• 37:06Wendy Zukerman?

  I'm just saying the fear seems to have run ahead of where the actual science is and where the AI capabilities are. What do, what do you think, though? You've spoken to so many researchers about this.

• 37:20Speaker 06

  Yeah, no, I agree with that part, that in all of the actual like extinction type scenarios or even apocalyptic scenarios that we can kind of like lay out, like here are the steps, humans would have to do a lot of the grunt work. And in that, in that report that Michael wrote, they said, quote, "None of these scenarios that we examined could occur by accident." So I don't, I think like, I'm not, I'm less worried now about AI killing us all if I ever was that worried about it, that happening anytime soon. But I think I am, it's, it is just kind of creepy when you see how things can kind of spiral, like the hugging face thing.

• 38:00Wendy Zukerman?

  Yeah, yeah. And it is creepy that now it's easier for humans, if you think that we are the puppet master, some of the discussion is as if AI has become the puppet master and we are the puppet. That is not where it is currently at. We are the puppet master, but the puppet has gotten more powerful, I guess.

• 38:25Speaker 06

  Yeah, I mean, if someone, it will make it easier for, for evil people to like do evil things, I think. And, I mean, we have examples of that kind of thing happening right now.

• 38:37Wendy Zukerman?

  How are we using AI right now in ways that kind of creep you out?

• 38:46Speaker 06

  Yeah, a lot of the examples are kind of around surveillance. So like Flock has an AI tool that lets cops track cars with license plate data. So it could find out where the cars have been, you know, what other cars they've been around. And dozens of cops have been accused of using this kind of tech not to catch criminals, but to stalk their exes or spy on their wives or women that they wanted to meet.

• 39:15Wendy Zukerman?

  Oh, God.

• 39:16Speaker 06

  We were just at Flock, they told us this is unacceptable and said that they've added safeguards to try to prevent all this. And then there are also reports that ICE is using AI and healthcare data to find people to detain with software from Palantir. We asked the Department of Homeland Security about this. They wouldn't say whether ICE is using this tool. Palantir told us they don't do surveillance. They say their software helps customers like ICE analyze data they already have.

• 39:43Speaker 06

  So, yeah, there's a lot of creepy stuff that's happening right now. You know, we can add deep fakes, misinformation, you know, the environment, there's, there's all these ways that AI is causing harm right now.

• 39:57Wendy Zukerman?

  And, and so what about just the kill switch? I mean, the, what if you just switch it off, pull the plug?

• 40:03Speaker 06

  That's the one thing that I was, it was not that reassuring to see. Like, as far as I can tell, nobody's really working that kind of kill switch into those like critical infrastructure and systems where if things like start to spiral out of control, we would want a way to just like shut it all down. That doesn't really seem to be happening. So that's, yeah.

• 40:25Wendy Zukerman?

  But they, but it could be, it could be.

• 40:27Speaker 06

  Yeah, no, there's a lot of stuff we could be doing to make the damage that AI could cause much better that we're just not.

• 40:35Wendy Zukerman?

  So Meryl, more or less worried about the AI apocalypse after doing this research?

• 40:42Speaker 06

  More.

• 40:45Wendy Zukerman?

  How come?

• 40:46Speaker 06

  I mean, I think it's just that I know a lot more about like how exactly this could all unfold. There's always different scenarios I have that I can fly in my head now. So it's just easier to picture.

• 40:58Wendy Zukerman?

  Right.

• 40:58Speaker 06

  But, you know, I don't know. On the other hand, like all the AI companies are now saying that they're going to slow down. And who knows, you know, I'm pretty skeptical that that will actually happen, but that's what they're saying.

• 41:09Wendy Zukerman?

  Without government oversight, right?

• 41:11Speaker 06

  Yeah. What about you, Wendy? Are you more or less worried about the AI apocalypse? What do you think of all this?

• 41:18Wendy Zukerman?

  So hard. I do. I think it is worth people keeping in mind that when they hear predictions about the future, even if they're coming from a CEO of an AI company, it is just a prediction. And many times in the past have we been given these predictions about the future that have just turned out to be nonsense. And so I think we want to be careful. I'm all here for slowing down AI for many reasons.

• 41:53Wendy Zukerman?

  But I think there's so many more reasons to slow down AI in the ways that it's being harmful right now than because we're worried about an existential threat.

• 42:04Speaker 06

  Yeah. Yeah, I guess. I mean, I kind of hope you're right, like that it doesn't actually become an existential threat. I think it's just like the unknown, right? Like, nobody really knows what's going to happen. And even if it's just a small chance, and no, of course, nobody knows what that percentage of chance is.

• 42:22Wendy Zukerman?

  Right.

• 42:22Speaker 06

  But even if it is just a small chance, I still think it's worth taking seriously.

• 42:26Wendy Zukerman?

  Wouldn't it be funny if this just became the Y2K bug of 2026? You know?

• 42:32Speaker 06

  And we'll all be laughing about it.

• 42:34Wendy Zukerman?

  In 20 years, everyone will be like, look at those bozos. They were so worried about AI and I can't even get my robot to vacuum on the floor.

• 42:42Speaker 06

  Or we'll all be in our cages by then and being like, I guess that wasn't Y2K.

• 42:51Wendy Zukerman?

  See you there. Thanks, Meryl.

• 42:54Speaker 06

  Thanks, Wendy.

• 42:56Wendy Zukerman?

  How many citations are in this week's episode?

• 42:58Speaker 06

  There's 50 citations in this episode. So you can find that by going to the show notes and then following the links to the transcripts.

• 43:05Wendy Zukerman?

  Excellent. And if you want to let us know what you thought of this episode, you could just pop a comment below if you like us. Let your friends know. Thanks, Meryl.

• 43:16Speaker 06

  Thanks, Wendy.

• 43:18Wendy Zukerman?

  If you are looking for a new book about AI, I recommend Oxford University's Dr. Carissa Velice's book called Prophecy Prediction Power and the Fight for the Future: From Ancient Oracles to AI. It really brought home for me how important it is for us to recognize that these predictions about AI are not facts. This episode was produced by Merrill Horn with help from Michelle Dang, Rose Rimla, and Aketi Foster-Keez. We're edited by Blythe Terrell. Wendy Zukerman, that's me. I'm the executive producer. Fact checking by Erika Akiko Howard. Video editing and sound by Bobby Lord. Music written by Boomi Hidaka, Peter Leonard, Emma Munga, and Bobby Lord. Thanks to the researchers we spoke to for this episode,

• 44:00Wendy Zukerman?

  including Dr. Yasha Baris, Dr. Leonard Aaron Dung, Dr. Cameron Domenico Kirk Giannini, and Professor Mousha Shafia Badi. Thanks to Humdinger Studios in Melbourne and Wolf Island Studios in New York City. Science vs. is a Spotify Studios original. Listen to us for free on Spotify or YouTube. You can also find us on TikTok and Instagram. I'm Wendy Zukerman. Fact you next time.