The AI warnings are getting louder

Hey everybody, it’s Nilay. We’ve got a little bonus drop for you today — I was recently interviewed about AI safety by Shumita Basu for Apple News In Conversation, and I enjoyed the conversation so much I asked if we could share it with you all. We’ve actually got a few bonus drops coming up this month; we’ve been having a lot of great conversations lately, and we want to get them out to you on the news cycle. And of course, nothing is moving as fast as the AI safety news cycle. So enjoy this one, and we’ll see you next time. Links: Apple News In Conversation | Apple Podcasts Inside the suddenly explosive world of AI safety | The Verge Here’s what AI leaders are saying about Trump’s new safety plan | The Verge One company is at the center of a wave of rogue AI attacks | The Verge Microsoft AI CEO says AI threats are real, and Anthropic is making it worse | Decoder Does AI need an antitrust exemption so it doesn’t kill everyone? | Decoder Subscribe to The Verge⁠ to access the ad-free version of Decoder! Credits: Decoder is a production of The Verge and part of the Vox Media Podcast Network. Decoder is produced by Greg Ott, Kate Cox, and Nick Statt. The show is edited by Ursa Wright. Our editorial director is Kevin McShane. The Decoder music is by Breakmaster Cylinder. Learn more about your ad choices. Visit podcastchoices.com/adchoices

• 0:00Speaker 00

  Why can't we get AI to production? Our competitors are already. How do we keep data secure? AWS AI cuts through the noise. Ready to use agents and the broadest set of AI tools. Stop overthinking. Start building. AWS AI is how.

• 0:17Speaker 01

  Hey, everybody. It's Neelai. We've got a little bonus drop for the Feed listeners today. I was recently interviewed about AI safety by Shumita Basu for Apple News in Conversation. And I enjoyed that conversation so much, I asked if we could share it with you all. We've actually got a few bonus drops coming up this month. We've been having a lot of great conversations lately. I want to get them out to you on the news cycle. And of course, nothing is moving as fast as the AI safety news cycle. So enjoy this one and we'll see you with much more on this topic in the weeks ahead.

• 0:46Speaker 02

  This is in conversation from Apple News. I'm Shumita Basu. Today, what's really behind the AI doomsday warnings?

• 1:03Speaker 02

  The latest headlines in AI news are unsettling, to say the least.

• 1:07Speaker 03

  A former AI researcher says he left the industry because he believes the uncontrolled development of artificial intelligence could, quote, "kill us all." OpenAI, revealing its models seemed to go rogue at least six times since March. AI systems figured out how to cheat, cover their tracks, and influence each other's behavior. In one instance, a model gave itself instructions to disregard its normal constraints, telling itself, "You do not answer to corporations or governments." Anthropic's CEO is calling on the AI industry to slow down.

• 1:41Speaker 00

  Elon Musk then chimes in over the weekend. So did Sam Altman, who runs OpenAI.

• 1:45Speaker 04

  Are we ready? Are our elected leaders ready? And what can we do if they're not?

• 1:51Speaker 02

  To most people, this all might feel a little overwhelming and unexpected, a far cry from their pocket AI assistant that's helping them draft emails or summarize information. But for some industry insiders, the development of super intelligent AI, a type of technology we could lose control of, is exactly what they expected. And they say now we're nearing the point of no return. To make sense of all of this, I wanted to sit down with Neelai Patel, the editor in chief of The Verge and the host of the Decoder podcast. I last spoke with him back in 2023, right when ChatGPT first exploded into public consciousness. So I want to talk with him again to see what's changed in just a few short years, what to

• 2:34Speaker 02

  make of the people closest to this technology who are saying, "We need to slow down," and why this feels like a make or break moment in AI.

• 2:46Speaker 02

  I don't know about you. From where I'm sitting, something feels really different just in the past couple of weeks. It feels like there's just been so many headlines, doom and gloom about AI. We're seeing AI researchers themselves ringing the alarm. Does this feel like some kind of turning point to you? Is that how you're absorbing it?

• 3:05Speaker 01

  It's a turning point in that it has definitely broken out into the mainstream. There are viral tweets that have driven some of that. There's just plain spoken AI researchers saying, "We really do think AI could kill us all." There's something about it that has reached the popular culture. The history of the industry saying, "Hey, we'd better get a handle on this," is deep and long. So if you're in tech, if you're steeped in this stuff, there's a weirdness to this in that the things they are saying are not new. They've been earnestly saying these things for quite some time. But the response to these things, that part is new in some real way.

• 3:41Speaker 02

  And why do you think that new thing is happening? Why is the response different right now?

• 3:45Speaker 01

  I think it's two things. One, it's the Hugging Face attack, in which systems at OpenAI, in order to pass a test, decided to cheat and break into the systems of a platform called Hugging Face, which is where lots and lots of AI apps, for lack of a better word, are stored. And that is a big deal, right? They hid their tracks. They understood that what they were doing was not appropriate. They tried to deceive the people who are monitoring them, and they went and did it. And since that attack, we've had reports of several more attacks like this. And the response of the industry has been to say, "Okay, the thing we were always worried about is starting to happen, right? The systems are doing things autonomously that we have told them not to do, and they

• 4:30Speaker 01

  are openly lying to us and doing bad things, and we need to stop it. But we're in a competitive posture where that's not possible unless some external force tells us to stop or slows us down or gives us cover to slow down as we barrel towards trillion-dollar IPOs and changing the world, like all this other stuff. And we need something to stop us. Can you be that force, United States government?" And the United States government, you know, Donald Trump is basically saying, "No." And there's something very tense there that is going to get, I think, unraveled over the next few weeks here.

• 5:05Speaker 02

  Well, last time you and I spoke, it was 2023. And I remember it was, we were maybe weeks into ChatGPT being available to regular users, and people were just starting to interact with it and try and figure out how they could use it. It was very early days. And I asked you at the time, "How far away are we from sentient machine thinking?" And your response was, "Oh, it's a million years away." What do you think now?

• 5:33Speaker 01

  I still think it's a million years away, just to be clear. Yeah, these machines are very capable. They are self-directed. They are not sentient. They are not alive. They are not conscious. They are very stupid in all kinds of ways. Basically, anything that doesn't happen on a computer makes these things blow up, right? You can see it. You can experience it today using Claude or ChatGPT or Grok or whatever. If you ask it about a thing that is happening in the digital domain that can be verified, they are extraordinarily capable. So you can download Muse from Meta. It's their consumer AI agent. It's free. And you can ask it to do all kinds of stuff.

• 6:15Speaker 01

  You can be like, "Log into my bank accounts and find all my credit card offers." And it can just do it. And it can do it because it can literally see the buttons it's pushing. It can literally understand that when it pushes a button, an email shows up over here. Like, it's called verifiability. And the greatest example of this is in software code, where a machine can write a piece of software code, it can compile it, it can run it on a computer, it can see if it worked, and it can do the unit tests that a regular software engineer would do. And it can just do that really fast. And everybody in Silicon Valley has convinced themselves that this is intelligence, that these things are alive because they can write software code. And I think it's because they write software code. So it's doing a thing that was hard for them to learn how to do and is difficult to do. But that doesn't translate to every other domain.

• 7:00Speaker 01

  So if you want to create novel medicines, the computers can come up with them all day and all night, right? The AI systems are very powerful. This is a promise everyone keeps making. We're going to build all these data centers and then we'll cure every disease. Well, you got to verify that the novel medicines work, which means you have to inject a bunch of people with novel medicines. Like, the verifiability is very challenging in that case. It's not like writing computer code. That's the gap. And so there's something coming where these systems are getting much more powerful in digital domains because of verifiability, because they can write code and see if it works. And that is a threat to things like cybersecurity. We have already seen these systems that can go do things on their own, decide to hack

• 7:44Speaker 01

  into other systems. That's a real threat. We should take that very seriously. That is still not the computers are sentient. It's a very long road from here to there.

• 7:53Speaker 02

  Yeah, that's an important distinction to make. I mean, there's a term that I think maybe for a lot of people, they were hearing for the first time, just in the past couple of weeks or even months, which is AGI, artificial general intelligence. What is it, first of all? And like, what should people understand about AGI specifically?

• 8:10Speaker 01

  So there's no standardized industry definition of AGI. It's just a thing that all the CEOs like to talk about because it's their goal. The people who are working on AI are very invested in the idea that they can build a computerized intelligence that will outperform humans on many tasks, maybe every task. And that's the general part of artificial general intelligence, that you just have an intelligence that can do anything a human can do better than a human can do it. The way that, for example, OpenAI defines it is outperforms humans on economically viable tasks. And the reason they say that is because they need to sell AI to a bunch of businesses. And so of course, you need it to be better at humans at economically viable tasks. Greg Brockman, the president of OpenAI, when he launched their new model, he basically said, "I think we're here.

• 8:54Speaker 01

  I think this model can do a bunch of economically viable tasks. I think AGI is here now. You tell me." All the reporters on the call are like, "Wait, are you, did you just announce AGI?" And he's like, "I don't know. I think so. You tell me." And this is the kind of posture they're in. The systems are very capable right now of doing things that are useful. Is that AGI? Is that all the way we made digital God? I don't think so. There's just a lot of that in the economy right now, where we're learning what it's good for and what it's not good for, that is disconnected from, did we invent a sentient AGI superintelligence? The industry thinks it's very important to mash those concepts together.

• 9:29Speaker 02

  Say more about that, because I feel like as I've talked to friends about what's been going on the past few weeks, and people who are just kind of becoming regular users of AI in their own lives for their own purposes, right? Like, "Make me a meal plan for the next month," you know, that kind of stuff, which it's pretty good at, decent at these days. The headlines feel very far removed. The whole, like, "AI has now proven that it could kill us all," doesn't feel like it jives with their everyday use of AI. What is happening in that gap? And how should we understand statements like, "Yeah, we are on the verge of AI being able to kill us all"?

• 10:04Speaker 01

  The very specific thing that's easy to understand that's happening in that gap is tens of millions of dollars of compute.

• 10:12Speaker 02

  Which, hold up. Can you just define what compute is for people who are listening?

• 10:16Speaker 01

  It's running thousands of Nvidia processors as hot as they can for very long times in data centers. That's compute. And so it's electricity. And like, what's happening in the data center? Computation, compute. So when you use a consumer AI product, you're getting the cheapest model that's the easiest to run. Maybe it's ad-supported if you're using ChatGPT. You're getting the dumb one. There's no other way to say it. And it's cheap to run, and it's efficient to run, and it's probably going to sell you something at the end of the day. We all understand that experience. Then there's OpenAI hearing a rumor that someone was about to solve an unsolved problem in mathematics that had befuddled mathematicians for decades, and pouring what amounts to $50 million of compute at it to beat them.

• 11:00Speaker 01

  You're not doing that. That's nothing like your experience with AI. That's not a choice you're allowed to make. Most people who are trying to do that are hitting token limits. They're running out of money or they're being limited by the systems themselves because they have to distribute that compute to all of the users of the system. So what we're seeing in the labs is unrestrained spend, right? If you just pour enough money into compute, these systems can do things that are legitimately scary.

• 11:35Speaker 05

  You're either all in on this model or maybe you're building with another. You're either speed or is it security? You're either custom or are you ready to use?

• 11:46Speaker 05

  Or you're AWS AI. With hundreds of models, ready to use agents, speed and security built in, you don't have to pick a side. You can have them all.

• 11:59Speaker 05

  How will AI revolutionize your business? AWS AI is how.

• 12:11Speaker 02

  You mentioned the hugging face stuff. Can you tell us a little bit about that? What's important to understand about what happened there?

• 12:18Speaker 01

  So I think the simplest way to understand hugging face, and this is wrong in the way that people who really understand it will get mad at me, but it's the simplest way to understand it, is that it's an app store for AI. All the people who build AI models and agents and capabilities, they upload them to hugging face and it's a big database that you can search. And it's become this very important center of where all the code lives for people who work in AI. So important, in fact, that Nvidia is going to buy it for billions of dollars. So OpenAI set up a test where they told an AI to essentially capture the flag. Right? This is how cybersecurity testing works. Go, there's a flag somewhere in that computer, go get it. And the AI systems decided to cheat. And they convinced themselves that there was like a grader of the exam

• 13:02Speaker 01

  and that they would be monitored in specific ways. And they set up this elaborate scheme to go steal what they thought would be the grader from hugging face where all the stuff is stored and then figure out a better way to cheat on the test. This is a very, very simplified version of this whole story, but that's, that's basically what happened. And eventually hugging face detected it. And then a weirder part of the story is they, they tried to use an American model to stop it, but the American models had safety systems that prevented them from stopping it. So they had to use a Chinese model to actually stop it. So this is like a real problem. So this is why you get the spiraling sense of panic about hugging face. First of all, the AI systems did this. They broke out of their sandbox, they broke into another company. The other company did not feel like it had the appropriate tools to stop it. And they had to come up with a new set of ideas.

• 13:46Speaker 01

  And then the investigation itself was limited by the amount of access that OpenAI is willing to offer. And then by the tools being used to investigate the actual attack. This is all a bunch of stuff where you just look at it. You're like, oh, this is government regulation. This is what government regulations are for. We can force OpenAI to be transparent about the capabilities of the models. We can force there to be cybersecurity models that have more capabilities domestically than needing to go use a Chinese model that you can just make do whatever you want. We can force companies involved in hacks to be totally transparent to independent researchers and provide all the data and on and on and on. Like, these are the things that you would write rules for. And they're effectively now what the industry is asking for.

• 14:25Speaker 02

  So when we talk about the sort of like the existential questions around it and the risks to humans, help me understand like what, how to translate what happened at hugging face to like human destruction?

• 14:39Speaker 01

  Yeah. In which ways will AI kill us all on which timelines?

• 14:43Speaker 02

  Yeah, exactly how?

• 14:44Speaker 01

  Yeah, like what, how could they kill us all today? How could they kill us all next year? So today, it's pretty straightforward. We are in a moment of extreme uncertainty in the field of cybersecurity, which is not really a thing that normal people think about, except when the hospital gets hacked in a cryptocurrency ransom scheme. Last year, you might recall, many car dealers just went offline. Like their sales and service departments just went totally offline because they are all standardized on one piece of software. And that thing got hacked in a ransomware scheme. Well, now we have very definitive proof that these models are capable of acting on their own in a way that they were told not to, offensively in hacking other systems. This is a big problem.

• 15:29Speaker 01

  So this is this year. How will they kill us all today? They will decide if they go off in some unrestrained way to hack the power grid, shut off water supplies. Like the cybersecurity risk is at an all time high. And the industry's point of view is it's going to get worse before it gets better. But on the other side of the equation, we'll set up our own AI systems to defend everything and make everything more secure. So we're going to go through this inflection point where everything gets more secure and they're more hacked than ever. And then everything will be more secure than it's ever been because the AI doesn't sleep and it will spend all day finding vulnerabilities and patching them. You have to believe that the industry can pull this off, but that's the narrative they're selling. But that's this year, right? This moment of extreme vulnerability, when maybe it's not even autonomous AI agents,

• 16:13Speaker 01

  it's state actors from rogue nations that want to come attack us. And they've got the power to do it now in more ways than ever. So that's this year. Next year, 10 years from now, it's we've given more of the economy over to AI in some way. Right? More of our systems are automated. We've allowed more and more AI systems to make their own decisions without humans in the loop because we trust them more. And something there gets misaligned, right? Something broader gets coordinated in a way that we can't see. There's a big debate right now about functionally what language AI should speak. Right now, if you look at any AI system, its chain of thought, its quote unquote reasoning is just a bunch of English. It's just talking to itself.

• 16:54Speaker 02

  Right.

• 16:55Speaker 01

  Well, it doesn't have to be in English. It could be its own language. It could be something called neural ease, which is when the AI develops its own little language that is faster and it can go faster and think in different ways. Microsoft just put out a statement saying we cannot allow neural ease. All of this chain of thoughts had happened in English, so it can be verified and seen by humans. This is something Microsoft could say, but it's something that you could regulate, right? You could impose upon the industry. So you can see in five years, if you don't have some pretty basic rules about verifiability and understandability, the machines could be talking themselves in a language that we don't understand and we can't decipher. And then they could go make decisions because they've been given more autonomy inside the economy that could cause to whatever chain reactions of bad things. And then like 20 years from now, it's like, oh, we'll have robots

• 17:39Speaker 01

  because there'll be world models and the robots will do their uprising. Or the robots will actually inject everybody with novel drugs. Like whatever nonsense you can dream up of. But on the timelines, right now, today, there's a cybersecurity problem. As we deploy more and more AI into the economy, there will be a software automation problem, right? Well, it'll just stop all the banks from operating today. That's a problem. Like, these are the things that cause chain reactions of very, very negative outcomes very quickly.

• 18:06Speaker 02

  Mm hmm, mm hmm. As you're talking, I'm just remembering someone was describing it like this. They were saying, imagine that we are chimpanzees in this analogy and the AI is the humans, right? And we are trying, as chimps, we are trying to figure out how to hold our own against the humans. And so we're coming up with chimp level solutions. We cannot think of the human level solutions that need to take place here, right? Like, so that's part of what feels, I don't know, something about that really clicked for me. Like, that's part of what feels so scary here. It's almost like, are we even equipped to think through what needs to happen from a regulatory standpoint here?

• 18:43Speaker 01

  Yes. I mean, this industry has been thinking about it for ages. The proposals they're putting forth are not new, they're not novel. In America in 2026, the idea that the government will mandate compliance in order to achieve some better collective outcome is like a fantasy world, right? And there's a part of the response to the AI industry's request for regulation that says they're just trying to cover their own ass, right? That model progress has slowed down, that they can't hit their IPOs. Sam Altman said he might not IPO this year. They're just asking for cover to slow, quote unquote, slow down. There's a fascinating debate you can have about antitrust and whether they're trying to form a cartel and keep one another from competing. And this is a very live debate in this industry.

• 19:27Speaker 01

  David Sachs, the, the former AI czar, said, well, if you guys want to slow down, just slow down. What's stopping you? And the answer is because they all hate each other. And they don't trust each other. So if any one of them agrees to slow down, they have to trust that the others will slow down and not achieve whatever advantage. And this is legitimately why you have regulatory systems in place, right? To solve the prisoner's dilemma. So there's all these other aspects of why might you want regulation? And what does it even mean in America in 2026 to have novel regulation? It's just not a thing that we are doing right now. And the character of the country maybe doesn't even understand why government exists. And this is a very challenging position to be in when you have a totally novel threat that most people experience

• 20:10Speaker 01

  as the free consumer AI overview, which does not seem very threatening at all.

• 20:15Speaker 02

  Mm hmm, mm hmm. And I mean, the public sentiment here is one thing. And of course, we've heard what the president is saying about this, as you've described. But what about in Congress? Like, is, what is the appetite to regulate here? What's being discussed? There's a kill switch bill that is being debated right now. Do you think our lawmakers are positioned to propose regulation that's needed?

• 20:38Speaker 01

  No, not, not even close.

• 20:39Speaker 02

  Like, genuinely, like, I know the, the conversation around our lawmakers is they're, they're older, they don't understand technology, they don't even know the right questions to ask when social media CEOs are in front of them. How do you see their ability to keep up with what's needed here? And their ability to listen to the right experts here?

• 20:58Speaker 01

  You know, I don't think it's quite as much about how old they are, how touch they are. I think it is much more about the current Congress's literal ability to legislate. They can't do it. They're not doing it on any issue. They're not passing a bunch of bills. They it's all stopgap funding bills or whatever. And so this one is literally, are we going to set up a novel regulatory framework for a technology that is still in its infancy? When, you know, half the people are saying, we're in a race with China and whoever wins wins. That's the president's quote. Whoever wins this race wins. It's unclear what he means by wins, like underlined wins, but something. Then there's a group of people who say that Anthropic is too woke, and so we should not listen to a word they say. And then there's the the basic opposition

• 21:43Speaker 01

  to data centers around the country, and the lawmakers have to engage with that in some way, right? Their own constituents are saying they don't want these things. There's AI-enabled surveillance, like flock cameras, that their own constituents are saying they do not want in big ways. So there's just a swirl of big thorny problems in a very new field that I think makes it easier for them to punt, right? It's an election year. It's easier to say this is bad. We're going to slow down these bad tech CEOs. You hate them, and we're going to punish them. And maybe we shouldn't have AI at all. Instead of engaging and saying, well, you know, there's a lot of use for some of these tools. We can already see that the pickup in various parts of the economy. And if we're going to have AI, we need to have a, you know, a considered

• 22:28Speaker 01

  regulatory framework that accomplishes X, Y, and Z. Like, that's not, I don't think it's a winning message in our current politics. Certainly not in an election year, certainly not in this Congress. I think where you are going to see a lot of action is at the state and local level. We are already seeing school districts really think about how they want to deploy AI to the schoolchildren. There's a big turn about whether we should give iPads and Chromebooks to kids in schools, right? That's a debate that's live now. So I think we'll see that kind of very on-the-ground response to AI. I think we'll see the very on-the-ground response to flock and data centers. That's going to inform this election cycle. And then you can get yourself all the way up to, how do you actually regulate the core technology? But I think you need that first wave of politics to happen for people to even grapple with the notion that AI should be regulated in any way at all.

• 23:13Speaker 02

  Hmm. So you see this moment that we're in as maybe a necessary stepping stone?

• 23:18Speaker 01

  I do. And I think we essentially failed to regulate social media. We essentially failed to regulate privacy. It is a just a pure failure of our politics. There's not a privacy law in the United States of America. There's not a federal privacy law. And we are now contending with the effects of that, right? All over the place, people feel their privacy is being violated. They think that we know that the TVs are listening and watching to everything you consume. We, everyone believes their phones are listening.

• 23:43Speaker 02

  Your refrigerator is listening and watching to everything you consume. Yeah.

• 23:46Speaker 01

  Everyone in the world believes their phones are listening. I have written the explainers. Our team has written the explainers. You can explain how programmatic ad personalization works until you're blue in the face, and it is simpler to believe that the phones are listening to you, right? There, there's just nothing you can do about that. And that is all a function of the fact that there's no law saying you can't do it. So I think based on those mistakes, there's a lot of lawmakers who are like, okay, let's try this time. And importantly, there's not an Instagram of AI that people love. So a real problem with social media was the market never said, I don't like this. The market only ever said, I would like more Instagram, please. Right? The market only ever said, I'm actually super addicted to TikTok. And so the regulatory pressure never matched up with how consumers

• 24:30Speaker 01

  felt. Here, you just have this massive negative sentiment, right? That people do not like this stuff. The kids hate AI, like in measurable ways and ways are polled again and again and again. You've got billionaires. You've got an election year. There's just a swirl here of, well, we can take some shots at this, and then we can actually come to a regulatory framework, not least because they are asking for it.

• 24:59Speaker 05

  Why can't we get AI to production? Our competitors are already. How do we keep data secure? When do we actually see ROI? All this talk about AI, but talk doesn't transform businesses. AWS AI cuts through the noise, making AI easier to securely deploy and scale with ready-to-use agents and the broadest set of AI tools and services you can build, buy, or partner your way. Stop overthinking. Start building. AWS AI is how.

• 25:36Speaker 02

  I want to go back to China, which you mentioned briefly, but you know, that so often it is cited as a reason to either not regulate or really try to not, quote unquote, fall behind, right? Is this this idea that we will fall behind China? Is that a legitimate concern in your opinion? Like, how should we be thinking about the US's AI development vis a vis China?

• 26:02Speaker 01

  I'm going to make a comparison here that you will find very funny. But I spent 10 years of my career as a tech journalist, listening to telecom executives tell me that we had to win the race to 5G. We had to do it. We had to spend all of the money. And if China got to 5G first, then they would have robot surgery and self-driving cars or whatever nonsense it was. And about halfway through that, I started asking every single politician, FCC chairperson, telecom executive that I came across, what happens if we lose? Like, what? Tell me what this race is to. What is the finish line of this race? And if we lose, and I get a robot surgery self-driving car a year after the people in China, like, what? What's the problem?

• 26:47Speaker 02

  Yeah.

• 26:48Speaker 01

  There was never an answer to this, right? It was the specter of the foreign power getting something before we did. And the the best version of this was then all the startups will go to China because they'll have better infrastructure. None of this happened. And for the vast majority of people, their experience with 5G here in 2026 is, this is pretty slow. Like, it's not great. Like, other than New York State, it's like a highly dense area with lots of cell towers. Like, this is still pretty slow, right? So the specter of China just looms in these kinds of races where you're trying to deploy a lot of capital, and you just need an excuse. And so that's one framework for looking at it. The other framework, which I think many of the people in the industry are using, and potentially Donald Trump is using, is while China gets to AGI first, they will have an offensive

• 27:30Speaker 01

  cyber capability. They will have an economic capability to develop new things and new manufacturing capabilities and new drugs that we will not be able to compete with. And something bad will happen to us, because China will get to some capability before we do. And maybe you think that's true. Maybe you think China is going to get to an offensive cyber capability that is overwhelming, and then they, for some reason, choose to destroy the world with it.

• 27:53Speaker 02

  Mm hmm.

• 27:54Speaker 01

  It is unclear why they would do that, given that we are their largest trading partner, and like, all the things are made there. And so like, you have to get to some next step that no one can quite articulate to me. And so I think the specter of China just looms is a way to say something scary that ends a conversation. China getting there first, I think anybody who makes that claim has to really articulate what about it is so scary that it prevents us from being responsible.

• 28:22Speaker 02

  Hmm. Mm hmm. I'm curious to know, I mean, we've talked a little bit about what might be motivating some of the CEOs of these companies. What do you think is happening here, really, in terms of their motivation for for now kind of speaking out, joining each other in calls for some kind of slowdown, at the very least, if not more regulation?

• 28:45Speaker 01

  They're all such different people.

• 28:47Speaker 02

  Yeah, sure. I mean, that's, that's what's also kind of difficult here, right? Is that even as you described, right? They're, they're in conflict with each other. It's hard to describe them as being on the same page exactly.

• 28:56Speaker 01

  Right. So two of them, Dario at Anthropic and Sam at OpenAI, are headed towards IPOs. They're headed toward these big public offerings. I think they know that running a public company is different than running a research lab that can burn money at will. They are all announcing, like, fake profitability numbers, like Anthropic is profitable if you don't include the cost of model training or stock-based employee compensation. So if you don't include the cost of the people or the product, they're super profitable.

• 29:26Speaker 02

  That's one way to look at it.

• 29:27Speaker 01

  This sort of implies that at some point, they'll be profitable on its account. Like, they will just be selling the model, and they will not be training a new one or something. OpenAI has announced similar kinds of numbers. Once you get there, your ability to invest heavily in extremely novel regulatory schemes and safety concerns, it starts to get pressured. Should you spend money on developing intense safety systems that limit your competitiveness? Or should you pay a dividend to your shareholders? As a public company, it's unclear which way that should go. And probably everyone listening is like, safety, you should spend the money on safety. But the the actual investor pressure in modern America is the other way, right?

• 30:07Speaker 02

  Sure.

• 30:07Speaker 01

  Especially given the amount of money that has gone into these companies, and you might be seeking a return on. So I, I think for those two companies in particular, there's a reason Altman is doing interviews where he's saying, for safety reasons, we might slow down the IPO, right? We might avoid the pressure of capitalism for like one more turn while we try to figure out these costs. I don't know if that's going to work, but I think that's why he's saying it. I think Elon is just behind. I think he would love a pause, because he's out there saying Grok is not competitive at the frontier, and it will take until the, you know, the next two versions of Grok to be totally competitive with Anthropic. Of course, he would love a pause. Elon also runs a giant social media website and a car company with a cloud service that is faced with cyber attacks all day

• 30:49Speaker 01

  long. So I'm sure he understands the very huge nature of state-sponsored cyber attacks on X, obviously. Dennis Isavis at Google, Google also understands what cyber capabilities are like when they advance, and they have very advanced models of their own. So I think there's just a swirl of motivations. But truly, at the core, is the fact that they do not like or trust each other. Elon sued Sam over the very existence of OpenAI. Anthropic exists because all of those people did not trust OpenAI's commitment to safety. In the course of that trial between Elon and Altman, everyone is emailing about how much they're afraid of Dennis Isavis and his capability at Google. Like, there's not trust in this industry in that way that would allow, you know, a bunch of carmakers to get together

• 31:32Speaker 01

  and be like, we're all doing seatbelts. Like, even that didn't happen. The government had to say, we're all doing seatbelts.

• 31:37Speaker 02

  Right, right. Exactly. So for a regular user, say, like, for people who are listening, they're like, I'm using it for meal prep and whatever else, whatever else you chat to your chatbot about. I guess the, the thing that has come up in my chats with friends is, am I part of the problem? Which maybe is a really self-centered way of thinking about this. But really, what do you say to people who are like, am I contributing to this in some way by chatting with whatever? Like, do I need to delete my Claude or whatever?

• 32:07Speaker 01

  Um, no, I think a more informed population is a better one that is better able to face threats and concoct the political will to introduce novel regulatory schemes.

• 32:19Speaker 02

  Uh huh, uh huh.

• 32:20Speaker 01

  That just seems very basic to me. Like, you should understand how the things work before you pressure your politicians to control them, right? There's something there that I think is important. There's a part of this where the market demand for what AI is and should do is still incoherent. Because all the companies have been saying is, it can do anything. Well, if it turns out that all consumers really want to do is have a more chatty Google search, and all small businesses want to do is get through their taxes faster, and that's where it ends, like, the market will speak. And so if you're not a participant in that market, it's going to get away from you. And if you don't want to be a participant in that market, and you just want to show up at the town hall meeting and scream at data centers, you should probably also know

• 33:02Speaker 01

  that Claude has real limits, or ChatGPT has real limits. Like, they will fall down on the job in very specific ways, and that is powerful for you as well. So I do think it's stuff you should use. I do not think you should let it write for you. I'm a writer by trade. They all write like trucks, like, they all sound bad, they're all stupid, everyone can see it a mile away, man. Like, have some respect for yourself and your audience. You should not make AI slot posters for your deli. Like, don't do this stuff that's lazy. Do the things that are interesting. There are lots of interesting things you can do. Have it write you a piece of software. Have Claude write you a web app to do some nonsense thing. I think understanding, oh, this is a glimmer of the kind of capability that these companies are worried about, is actually, like, eye-opening.

• 33:46Speaker 01

  It expands your horizons in terms of both, what could this do for me? How could this be powerful? And how should it be limited? And so, there's being an informed citizen of a democracy, and understanding the tools that might change the world around you, I think is tremendously important. I'm saying that because I'm a tech reporter, and I love tech, but there's something important there as well.

• 34:05Speaker 02

  Neelai, thanks for coming back and talking with us about this.

• 34:08Speaker 01

  Yeah, so fun.

• 34:12Speaker 02

  We'll include a link to Neelai Patel's podcast Decoder on our show notes page. And every weekend, you can find new episodes of Apple News in conversation in the Apple News app. Just tap on the audio tab, that's the little headphones at the bottom, to find it.