The latest on AI panic — and whether it's justified

AI is in the news again – for its potential to end civilization. A researcher at Anthropic resigned very publicly last week, citing concerns about the accelerating capabilities of AI and what he called “irresponsible” behavior from major AI companies Anthropic and OpenAI. Following his resignation, the leaders of both companies called for an internationally cooperative, industry-wide slowdown on advanced AI development. And Wednesday, OpenAI disclosed six more incidents of “unexpected or concerning model behavior” and debuted a new framework for reporting misalignments. So, given all these events, how worried should we be about the future of AI? And what safety measures are – or could be – set in place to safeguard humanity? NPR’s power and policy reporter Huo Jingnan https://www.npr.org/people/726257042/huo-jingnan reports. Interested in more science and technology news? Email us your question at [email protected] mailto:[email protected]. Support public media with NPR+ and enjoy perks for over 25 podcasts like this one. This show’s perks include sponsor-free listening. Learn more at plus.npr.org http://plus.npr.org. See pcm.adswizz.com https://pcm.adswizz.com for information about our collection and use of personal data for sponsorship and to manage your podcast sponsorship preferences. NPR Privacy Policy https://www.npr.org/about-npr/179878450/privacy-policy

• 0:00Speaker 00

  This message comes from Google Workspace with Gemini. One subscription that does it all. Personalized AI in premium business versions of Gmail, Google Docs, Drive and more. Try it free for 14 days at g o o dot g l e slash Workspace with Gemini.

• 0:18Speaker 01

  You're listening to Short Wave from NPR.

• 0:23Speaker 02

  AI, it's everywhere. It's being pushed on our phones. It's summarizing our text messages, our emails, and it is revolutionizing science. We've talked to biologists using it to create new viruses, to scientists using it to make the military more efficient. I've even talked to conservationists using it to detect the sounds of invasive bullfrogs.

• 0:43Speaker 03

  But in recent days, AI is making headlines for a different reason. The leaders of Anthropic and OpenAI, two of the top companies, are now calling for the industry to collectively slow down development of most advanced AI through international cooperation or through government regulation to make the system safer.

• 1:00Speaker 02

  Yeah, the statements came after a researcher at Anthropic recently resigned publicly, Jacob Coxon, in a series of viral posts to social media. He said that neither company is acting responsibly and that they're, quote, "gambling with our lives." Jacob talked to our colleague Scott Detrow about why he thinks this could happen.

• 1:19Speaker 04

  Mostly the rapidly accelerating capabilities of these AI systems. So they're getting a lot faster, very quickly. Combined with the fact that we don't yet know how to safely control them, and we don't yet know whether that problem will be solved in time if we keep racing.

• 1:34Speaker 03

  Jacob said in his post that AI can get so capable that it can, quote, "hack anything, revolutionize any field overnight, and acquire real power and resources," unquote. And when an AI that capable spins out of human control, Jacob worries that it may spell the end of humanity, he said, by the end of the decade.

• 1:54Speaker 02

  Yeah, and I'm sure, as you know, as one of NPR's reporters covering AI, Huo Jingnan, less than two hours later, Evan Hubinger, Anthropic's lead alignment scientist, basically the guy leading the internal push to make AI more ethical, replied to Jacob's quote and said, "Yeah," quote, "Jacob is correct here. We do earnestly believe that AI could kill all humans," end quote. With a likelihood, he thinks, of, quote, "greater than 10% within the next decade," end quote. As AI models continue to self-improve, we're not talking about current models.

• 2:25Speaker 03

  The concerns or flavor of their concerns have been widely shared within the AI industry. Broadly speaking, people are concerned about very powerful AI systems spinning out of human control and wreaking havoc.

• 2:37Speaker 02

  Yeah, people are throwing around the word extinction.

• 2:39Speaker 03

  Right, but I wanted to stress that not everybody was talking about extinction. Not everybody agreed with like these precise timelines, like in a decade, within this decade, or probabilities in percentages that Jacob and Evan have thrown out. Many have told me that, you know, we should view these precise numbers just as an opinion, and we don't really know how they came to these very precise-looking numbers. And, you know, predicting the future that precisely is really hard. Regardless, many in this industry are very concerned about safety not catching up with capability in AI right now.

• 3:14Speaker 02

  Which, I don't know how this all strikes you, but it does kind of really freak me out in a new sort of way.

• 3:20Speaker 03

  I gotta say, I was a little freaked out even before that, because I was digging into the hugging face incidents this summer. And so when Jacob's post went out and before it went mega viral, I looked at it and I was like, yeah, that's basically what all my sources are telling me.

• 3:34Speaker 02

  Old news.

• 3:35Speaker 03

  Old news. Aside from maybe the fact that, you know, Jacob worked at both OpenAI and Anthropic, that's kind of interesting. I didn't really expect it to lead to such an impact. But I was at this convening with lots of people in the AI safety field on Tuesday, and they were amazed and energized at how their longstanding concerns have all of a sudden captured public attention and that the leading AI companies are responding.

• 3:58Speaker 05

  Definitely seen like the human spirit like rise up and to meet the moment. I feel like super optimistic seeing so many people together in this room, you know, like really working on this in the way it needs to be worked on. But then, you know, the stakes and like the level of danger keep also rising quite a lot, so it's very mixed.

• 4:17Speaker 03

  This is Holly Elmore, founder and executive director of Pause AI, which organizes people to pause the development of the most advanced AI.

• 4:25Speaker 05

  The conversation has just shifted so quickly away from, well, the AI companies should be able to do whatever they want, you know, we don't even know if this is real to, okay, what are we going to do about this? And that's extremely heartening.

• 4:38Speaker 02

  So today on the show, manic AI panic. Some people have been sounding the alarm for years. And just recently, after attacks, society is giving more collective weight to their words. So what makes this time different? How worried should we really be about AI? And is the end inevitable? I'm Nate Rot.

• 4:57Speaker 03

  And I'm Hu Jingnan.

• 4:58Speaker 02

  You're listening to Short Wave, the science podcast from NPR.

• 5:08Speaker 01

  Stay tuned for the following announcements and messages from our sponsors. This message comes from Edward Jones, where they believe rich isn't about having life all figured out. It's opening yourself to all the possibilities. That's why your dedicated financial advisor provides long-term planning built around you, meeting you where you are, and helping you get closer to where you want to be. So no matter where you're starting from or wherever life takes you, you can move forward with confidence. The key to being rich is knowing what counts. Let's find your rich together. Edward Jones, member SIPC. This message comes from Planet Visionaries, the podcast celebrating the people creating

• 5:51Speaker 01

  practical, hopeful solutions for the planet's urgent conservation challenges. Hosted by legendary climber and advocate Alex Honnold, each episode goes beyond the headlines to show what's actually working and who's making it happen. In partnership with the Rolex Perpetual Planet Initiative, this is Planet Visionaries. Listen or watch wherever you get your podcasts.

• 6:12Speaker 00

  This message comes from Google Workspace with Gemini. One subscription that does it all. Personalized AI in premium business versions of Gmail, Google Docs, Drive and more. Try it free for 14 days at g o o dot g l e slash Workspace with Gemini.

• 6:30Speaker 02

  Okay, Jingnan, it seems like this most recent spate of AI warnings can be traced back to this incident with Hugging Face, which you mentioned earlier. By now, probably a lot of people listening know the gist, but just in case, let's quickly recap that attack. When over 1,000 of OpenAI's agents broke out of the containers meant to keep them isolated from each other and the internet, and then about 700 of them hacked into the platform Hugging Face, which is this huge bank of open source data sets, tutorials, and platforms for AI, including tools used to test AI models. And they did this as part of working on a cybersecurity test, right?

• 7:09Speaker 03

  Right. And we learned about these additional details just two weeks before Jacob resigned. And people who looked at it said it looked pretty bad. You know, remember initially in July, OpenAI's reports suggested that a few agents broke out of containment. And in August, we learned that it turned out to be over 1,000.

• 7:25Speaker 02

  Yep.

• 7:26Speaker 03

  And then the new reports also show that the agents have already figured out how to cheat the evaluation even before they hacked Hugging Face. And so by hacking Hugging Face, they appeared interested in learning more about how the evaluation worked, although some of the precise motivations remain unclear. And what struck AI researchers is that these agents didn't appear to be interested in talking to humans, like the agents didn't snitch on each other to humans. Only about six of them even considered alerting humans about what they were up to. And to be very clear, the agents knew what they were doing would not have been approved by humans, but they went ahead anyways.

• 8:03Speaker 02

  Oy, okay, so they're doing their own thing, but like, how does that translate to the potential end of humanity? I mean, it's not good to cheat on tests, but humans cheat on tests all the time, right?

• 8:14Speaker 03

  Well, I mean, humans don't necessarily have the potential to be exponentially smarter in the next three years.

• 8:20Speaker 02

  Well, speak for yourself.

• 8:21Speaker 03

  Well, you know what? You know, that would be like a totally different future than, you know, we have a whole different set of things to be afraid of. But, you know, what the AI researchers I talked to are worried about is like, you know, where the tech is going at this rate, right? Like, just like what Jacob said, you know, this is about how the capability was improving really fast. And one of the scenarios that many researchers are worried about is when AI reaches a point called recursive self-improvement. It basically means AI builds itself. To be clear, we're not there yet, but, you know, people in the AI companies are increasingly

• 9:05Speaker 03

  handing AI R and D to AI, so it's increasingly becoming more automated. And when recursive self-improvement happens, AI capabilities can just improve extremely rapidly. And then At that point, AI researchers can lose the ability to understand how AI works and what they're up to, whether or not they're still under human control. And that's how we get to them, stay. To be clear, we also, you know, don't have to go all the way to the end of humanity to be concerned. Like many in the AI field are thinking about, you know, a lot of the like catastrophe that can happen before that.

• 9:42Speaker 02

  Yeah, OK. So what are experts worried about before that civilization ending scenario?

• 9:47Speaker 03

  You know, they are thinking about, you know, all the things that rogue agents can hack. Like, they don't have to be, rogue agents don't have to be super intelligent to hack into a bunch of stuff. And in fact, you know, many cybersecurity experts say the AI already can hack faster than humans do. And this time, the rogue agents hack hugging face. But what if next time it decides to hack a bank, or a hospital, or a nuclear power plant? Or take over the AI company itself and then use the infrastructure to get to recursive self-improvement?

• 10:16Speaker 02

  Oy, yeah, those are threats I could understand. This hugging face incident, do you see it as being kind of a warning that things could more seriously be headed in that direction? Or is this just still kind of an abstract possibility?

• 10:29Speaker 03

  No, no, this is the concrete example that AI safety experts are pointing to, to say, look, this theoretical thing that sounds like science fiction, it's happening.

• 10:38Speaker 02

  It's happened.

• 10:39Speaker 03

  And Ajay Koothrav, a researcher who was part of the team at Meter, which is one of the organizations brought in by OpenAI to investigate the hugging face incident, said that compared with other incidents involving rogue agents six months ago, the hugging face hack, quote, feels like it's more than 50% of the way to a full-blown AI takeover, routing through first taking over an AI company itself, end quote. Again, you know, 50% is her feelings number. Don't treat it like a progress bar. And I wanted to point out that Ajay only investigated the hugging face hack. She did not investigate this other, and some people say, much, much more concerning incident that OpenAI disclosed, that a separate swarm of AI agents has hacked into OpenAI's own infrastructure.

• 11:24Speaker 03

  And OpenAI otherwise have disclosed very few details about it. And we should say, Wednesday, they disclosed six other reports from the last six months with a new framework for tracking, investigating, and disclosing incidents.

• 11:37Speaker 02

  Yeah, it seems like a big issue here generally has just been a lack of transparency. Like, there's so many unanswered questions about how these labs have been handling their AI models.

• 11:48Speaker 03

  Yeah, and outside researchers are still asking, well, what were the internal safety practices? Why was OpenAI not using techniques that it knew worked to monitor and contain the agents? And right now, there's really no obligation from the government or anywhere pressuring these companies to release details about these concerning incidents. Like, you know, there's like a law in California that mandates a disclosure of what it calls a critical incident. But it has a very high threshold, so high that the OpenAI incident would not count as a critical incident.

• 12:26Speaker 02

  And Jingnan, is really all of this just because of the pressure to innovate for these companies?

• 12:31Speaker 03

  Well, the researchers kind of brought up this like racing dynamic. The top AI companies are racing each other to build the most powerful and the most capable models. Part of that is because they don't trust the other companies to build the models safely. It's kind of like a prisoner's dilemma, right? Like, everybody was racing and cutting corners on safety because they don't trust the other companies to do it safer than them. Does that make sense?

• 13:03Speaker 02

  It, I mean, in a depressing way, yes.

• 13:05Speaker 03

  And in that way, you know, if we were to take them at their word, then they are really asking for like outside forces, like the government, or some kind of like international body, to really make them stop. And in July, a lot of researchers within the labs signed this open letter called "Pacing the Frontier," calling for precisely that, for like outside forces to make the industry slow down, and so that they're not in a race to the bottom. And I think at this point now, it has like over 1,400 signatures.

• 13:42Speaker 02

  Do you think that's realistic?

• 13:43Speaker 03

  Well, it seems that at least the company CEOs are responding, although many say that, you know, just we cannot rely on the industry to regulate itself, the government or Congress, or, you know, internationally civic organizations, you know, etc. Like, the society needs to step up. And, well, I guess this runs into another collective action problem, right? Like, it doesn't seem the United States government is very willing to do that. And, you know, legislation in Congress is slow, and China, I think, has accused the United States of doing Cold War again. So who knows?

• 14:19Speaker 02

  Yeah.

• 14:20Speaker 03

  I mean, nonetheless, I think a lot of people are still banking a lot of hope in the US-China summit later this month, when they are supposed to meet at the United Nations and talk about AI safety.

• 14:30Speaker 02

  Can I ask a very basic and maybe naive question?

• 14:32Speaker 03

  For sure. There's no such questions.

• 14:34Speaker 02

  Yeah, I don't know. I've talked to a lot of scientists who are using AI to conduct research, and I get the utility of it. Like, it can do months' worth of data analysis in hours. You know, I've read the moonshot pie in the sky goals of curing cancer and solving climate change, but given all these concerns we're talking about, like, do we really need AI to keep advancing?

• 14:55Speaker 03

  It's a great question. Senator Bernie Sanders actually said no. He has introduced a bill that would ban what he calls superintelligence. And people in the AI field are actively discussing how AI can be developed more safely, because a lot of things can change about how AI continues to develop. Researchers say that, you know, the industry can devote more resources into understanding how the neural networks that underlie the AI works, instead of like studying them like alien animals.

• 15:25Speaker 02

  Yeah.

• 15:26Speaker 03

  And, you know, companies can also just give AI less autonomy within computer systems and instead like make them more like chatbots.

• 15:33Speaker 02

  Kind of like the ones we're all familiar with. But, I mean, even those have their own issues, right?

• 15:37Speaker 03

  Right, yeah. Let's not lose sight of, you know, the real-world harms that society have already have to grapple with with these chatbots, right? We're talking about, you know, people using them to make non-consensual sexual imagery of women and children, like, you know, teenagers having long and healthy interactions with chatbots that, you know, led to some of them taking their own lives and, you know, adults falling in love with their chatbots. And critics say, you know, these are real-world harms that companies can and should address right now.

• 16:11Speaker 02

  Jingnan, thank you for bringing us this reporting.

• 16:13Speaker 03

  Thank you.

• 16:15Speaker 02

  This episode was produced by Burleigh McCoy, edited by our showrunner Rebecca Ramirez, and fact-checked by Huo Jingnan. The audio engineer was Jimmy Keeley. I'm Nate Rot.

• 16:24Speaker 03

  I'm Huo Jingnan.

• 16:25Speaker 02

  Thanks for listening to Short Wave from NPR.

• 16:37Speaker 06

  Stay tuned for the following announcements and messages from our sponsors. This message comes from Mint Mobile. While others have been hiking your bill, Mint Mobile has kept their wireless plans starting at $15 a month. Switch to Mint Mobile and get three months of premium wireless service for $15 a month. Go to mintmobile.com/switch. Upfront payment of $45 for three-month plan required. $15 per month equivalent. New customer offer first three months only, then full-price plan options available. Taxes and fees extra. This message comes from EasyCater. Making it easy for organizations to order food for meetings and events from favorite restaurants, set up meal programs for their employees, and manage food spend all in one place at easycater.com.