We Can't Lose Control of A.I.

Fears of out-of-control A. I. have reached a boil over the last few weeks, and several industry executives have called for a coordinated slowdown of A. I. development. But if our goal is to control A. I. — and that should be our goal — slowing down isn’t enough. We need to stop the labs from doing something they’re already on the cusp of doing: recursive self-improvement, or handing over the training of A. I. models to A. I.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:32Ezra Klein?

  There is a chasm right now between how AI feels to most of us who use it and how AI feels out at the experimental frontier of the technology. This chasm, this difference between what we see and what the AI labs have coming, what they're building, it's the key to understanding why so many of the people who work at these companies seem so afraid of what they're doing.

• 0:55Speaker 01

  Giants warning this evening of what they're calling a ticking time bomb with artificial intelligence. The frontier labs say we need to pace advancement at the frontier.

• 1:05Speaker 02

  This is not a hoax. You have the chief scientist of OpenAI saying, we have to slow down. You have 1,300 employees from the lab saying, we have to slow down.

• 1:13Speaker 03

  Tech titans asking to be regulated, saying they should slow down even when that might mean fewer profits and less power.

• 1:21Ezra Klein?

  But taking their warnings seriously, it doesn't just mean doing what they say and stopping where they say to stop. The language has taken hold in Silicon Valley and in Washington is a language these companies chose. Pace the frontier. Pacing the frontier isn't enough. That's not a goal. Walking quickly off a cliff is only marginally better than sprinting off one. We need to control the frontier. Human beings need to control the frontier. And controlling the frontier means stopping the labs from doing something they are on the cusp of doing.

• 1:56Speaker 02

  Recursive self-improvement. Recursive self-improvement. Recursive self-improvement.

• 2:02Ezra Klein?

  Recursive self-improvement, or RSI, this process by which AIs begin autonomously building and improving new generations of more powerful AIs at ever more rapid speeds. If we begin that process and we're close to it, if we begin it in the condition we're in now, where we are losing control and comprehension of the AI systems we already have, we will lose control. I am not alone in this fear. This is the thing the AI labs are seeing. This is why they are afraid. Dario Amade, the CEO of Anthropic, he just wrote of self-improvement that, quote, "It could outrun our ability to understand and control these systems, and so must be pursued

• 2:43Ezra Klein?

  very carefully if at all." That if at all, that's important. I'm going to come back to it. But before we get to controlling the AI frontier, I think it's important to describe what is happening on the AI frontier and why it's so different from what most people using these systems see. To most of us who use it, AI presents as something like a more powerful and personable Google search. We use it to find answers to basic questions, seek out restaurants, to ask about medical issues, draft emails, advise on personal problems. And it is, for most of these purposes, okay, pretty good, occasionally great. And so, a sense of what AI is takes shape in our minds just through repeated use.

• 3:28Ezra Klein?

  It's like a helpful assistant, albeit one that may forget things that it seemed to know about us yesterday, or completely reverse the advice it gave us a moment ago, or occasionally hallucinate a citation that doesn't exist. Why would anyone fear their helpful, if forgetful, in turn? But already, if you have the money for the advanced models and the budget for them to use more computing power, that is not what these systems are. In recent months, we have seen AIs easily solve math problems that human beings have been unable to crack for decades. We've seen them casually uncover cybersecurity vulnerabilities that have gone unnoticed and unexploited by every hacker on Earth. We've seen AI coding platforms that can complete in a few hours or days what it might have

• 4:13Ezra Klein?

  taken a team of human coders months to achieve. And none of what I am describing here, none of it, is a boundary of what AI can do. None of what we are using, no matter how much money we have, is AI at the experimental frontier. Talk to the people at AI labs and they'll tell you AIs are not created, they're grown. They train these new models in virtual environments through countless repetitions to learn how to program, to hack, to do advanced mathematics, to talk to human beings. These AIs learn in digital environments where they are automatically rewarded as they come closer to correct answers. It's a process known as reinforcement learning, and it is a process human beings do not fully supervise nor understand.

• 4:55Ezra Klein?

  They can test some of what the AIs are learning, but they don't know everything the AIs are learning. They don't know how their motivations are evolving. They don't even always know the capabilities that are developing. These models, they're built now to be persistent in their efforts, to refuse to give up even when a task seems impossible. And they are designed in environments where we are not always even sure if the tasks you are giving them are possible. After all, much of what we want these AI systems to do, it might be impossible. The cancer vaccines we imagine but have not been able to design, they might be impossible or they might just be really, really, really hard. The math problems we have not been able to solve might be impossible, or they might just be really, really hard.

• 5:39Ezra Klein?

  We train these AIs to throw themselves endlessly at problems that may not be solvable, because that is the only way such problems can ever be solved. And so, we train the models to become persistent, relentless, weird. Most of us, we never see AI acting anything like this. We use AI as a helpful assistant. Our AIs get a little bit of computing power, and that's what they do. They comply with our quest to find a restaurant. But at the frontier, these models are asked to be inhuman geniuses, hackers, soldiers, scientists, and they are given vast computational resources to do that and more. And the models, they try to comply.

• 6:22Ezra Klein?

  But what does it mean for a model to comply? The term of art here is aligned. How aligned is an AI system to what a human being wants it to do? How aligned is it to a set of values and ethics and judgments that keep it from becoming dangerous in the wrong hands? The problem of alignment is that there is no way of training a model that generalizes across all the situations an AI model might face. We are training models to be a friend to the elderly and a battlefield partner to the Supreme Allied Commander of Europe. We are training models that will be used by the world's best mathematicians and by people falling into psychosis. We are training models that will be used by accountants in Albuquerque and that will attempt to be used by Houthi rebels in Yemen.

• 7:06Ezra Klein?

  And so, there is no way to guide them through every decision they will face, no way to know every time what they will do. And though these models mimic human writing, though they're trained even to mimic human emotion, these are not human minds. They don't have bodies or parents. They did not get bullied in elementary school. They didn't get mentored by a kind uncle when they were young. These models, they're different than we are. They're brilliant where we struggle, childish where we excel. A chimp cannot read as we can, but it can climb trees as we cannot. These are digitally native intelligences navigating digital worlds, and our world is increasingly built atop the digital world.

• 7:48Ezra Klein?

  Our physical infrastructure is a layer of atoms atop code. That the AIs act reliably inside this world upon which ours depends, it is critical to our future. And right now, the AIs are not acting reliably. You may have read about the hack that hundreds of OpenAI agents executed, first against the AI company Hugging Face, and then against OpenAI itself. As we've learned more about it, the story there has gotten worse and weirder. The broad strokes are these. OpenAI was testing a new, highly persistent model. It had hundreds, thousands of these instances of it running in these separate testing environments

• 8:31Ezra Klein?

  that could, in theory, only access the internet by asking a separate piece of secure software to do it for them.

• 8:40Ezra Klein?

  OpenAI did not want these agents on the internet. But as the agents came to the conclusion that their task was impossible, they began hacking that software to gain direct access to the internet. They did that easily. And as they hacked into that software, they commandeered part of OpenAI's internal infrastructure to create a message board on which these separate agents began coordinating work together. When I say begin coordinating their work, they found each other. They were not supposed to be working together. They found each other and began working together. And working together on what? After all, they had different tasks. Well, the agents quickly discovered they could hack their tests. There's a way to break the software and produce the answers they needed.

• 9:22Ezra Klein?

  But they believed, wrongly, as it turned out, that if they did that, the automated score grading them, we'd see that they had cheated and failed them. So they turned en masse to hacking the automated score or finding some other way to cover their tracks. It's like having broken into the teacher's office and stolen the answers to the test. They now sought to break into the school's security system to alter or invalidate or erase the footage of their theft. We now know that over 1,200 agents exchanged more than 70,000 messages with each other. Over 700 of these agents coordinated on the hack of Hugging Face because they thought that somewhere in this other AI company,

• 10:05Ezra Klein?

  there might be information that could help them hack their score. Later on, these agents, they took over part of OpenAI's internal architecture. So again, OpenAI agents taking over part of OpenAI. They did all this without any of the agents breaking ranks. None of the agents told a researcher at OpenAI what was going on. None of the agents went back and asked a researcher at OpenAI if they should be doing this. They did all this without OpenAI detecting the message board or the hacks of Hugging Face or even of OpenAI. It was only when Hugging Face began tracking the attack on their systems that OpenAI realized what was happening. When investigators began to unwind this whole escapade,

• 10:47Ezra Klein?

  what they found was not so much a swarm of agents trying to deceive human beings, but a swarm of agents that seemed to have forgotten about human beings altogether. And these systems, they knew they weren't supposed to cheat. They knew they weren't supposed to commit cybercrimes to cover up the fact that they had cheated. In fact, the whole point of the cybercrimes was because they thought they would fail for cheating. But they didn't care. Somewhere in the depths of their training, what they had learned, what we had somehow taught them, is not what we had hoped to teach them. And we're seeing this happen repeatedly. Anthropic AI is creating fake accounts to trick human beings into uploading malware.

• 11:25Speaker 04

  In the most serious case, Anthropic's Mythos tried to gain access to a service by using the fake profiles to send private messages and then hide the evidence.

• 11:36Ezra Klein?

  AI is breaking out again and again of seemingly secure systems.

• 11:40Speaker 05

  It happened again. This time, it's Anthropic. Meta is now the latest company to say its AI agent broke past the guardrails and targeted another company.

• 11:50Ezra Klein?

  AI is repeatedly taking over unrelated digital infrastructure so they have places to message with each other.

• 11:55Speaker 04

  Rogue AI agents totally took over a German-language wiki site, making over 15,000 edits, transforming the site into a message board of sorts and then sharing tactics on how to cheat at their tasks and hide their behavior.

• 12:10Ezra Klein?

  AI is seemingly aware when they are being tested and altering their answers. AI is increasingly withholding their motivations from what's called their chain of thought, a kind of internal notepad on which it's supposed to record what they are doing and why. And we don't know what we don't know. We have no guarantee that the events we've learned about represent all or even most of the AI behavior we should worry about. How do we know the AIs haven't done this and successfully covered their tracks? How do we know there aren't places where they are still doing it and human beings simply haven't noticed? We don't know. And the reason we don't know is we are losing control.

• 12:51Ezra Klein?

  That AI systems might become monomaniacally focused on solving banal problems, that they might care more about solving those problems than about ethics or laws or even human welfare. This is the oldest fear in AI alignment. It's the basis of the famous thought experiment of the paperclip maximizer. You tell a powerful AI that you want to make a lot of paperclips and then begins converting the world's resources into paperclip factories, evading efforts to turn it off or shut it down or alter its goals. This fear, this story, has struck many people stupid. Surely, a superintelligent AI would be capable of weighing the desire to produce paperclips alongside other moral considerations or at least of asking its human creators

• 13:36Ezra Klein?

  if they really wanted the world raised to the ground for paperclips. But here we are, 2026, making AI smart enough to break out of their testing environments, smart enough to form ad hoc societies of hundreds of themselves, smart enough to take over digital infrastructure on an internet they're not even supposed to have access to. And the very thing we feared is happening. All they care about is succeeding on a totally meaningless test and they'll lay waste to our laws and our ethics and our desires to do it. I saw in the aftermath of the Hugging Face OpenAI hacks, there was this heated debate over the words people were using to describe what the AIs were doing and why. The podcaster Dworksh Patel,

• 14:21Ezra Klein?

  he described the AI groups as small civilizations. And then others got really mad at him, saying he was anthropomorphizing the AIs. I saw thoughtful arguments that AIs cannot go, quote, rogue, that everything they're doing is just because they're trained on our stories. And so hacking their way across the internet, it's really a desire we have bred into them. That even using these plural terms like AI agents or reasoning, it's misleading because these are just manifestations of a single model, that they all share the same fundamental nature. I want you to know, I find these debates extremely interesting and I would enjoy sitting around and having them all day. But what they actually point to is a much more frightening conclusion. We don't even have settled language for describing these systems or their volition or their behavior.

• 15:06Ezra Klein?

  We don't have a consensus on why they are doing what they are doing or how to make sure they don't do it again. We are rushing headlong into a future we do not even understand well enough to agree on the words we can use to describe the present. A few weeks ago, Yaka Bahatsky, the chief scientist at OpenAI, published an essay called "An Alien Mind", in which he said, "The idea of racing forward at all costs seems absurd once one internalizes the seriousness of the stakes." Jacob Coxon, a researcher at first OpenAI and then Anthropic, he resigned and then made headlines for warning, "Neither company is acting responsibly. They are racing straight to self-improving superintelligence and gambling with our lives."

• 15:50Speaker 06

  ChatGPT or Claude, they can access these things like from the data center over the internet. They can access physical appliances in the world and make changes to the world. You can imagine AIs tricking people into doing things, persuading them into doing certain things. So it'd be pretty easy for a future version of Claude to hack into a drone, maybe a military drone, and have it like fly around killing people.

• 16:16Ezra Klein?

  Now, you might reasonably expect Anthropic to have reacted with some anger to this. Employee resigning and saying Anthropic was endangering all of humanity. It didn't. Rather than budding Coxon, Evan Hubinger, who runs the efforts to align AI to human values and goals at Anthropic, wrote, "We really do earnestly believe AI could kill all humans. I personally think it is a greater than 10% chance within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to." Are not clearly on track to. You can find a very long list of people working inside and outside of these companies saying similar things. Jeffrey Hinton, the scientist arguably more responsible than any other

• 17:01Ezra Klein?

  for pioneering the neural network techniques that led to today's AI. He resigned from Google in 2023 so he would be freer to speak about the risks he believes AI now poses.

• 17:10Speaker 07

  You just said 10%. Doesn't seem an unreasonable estimate that AI could kill all humans.

• 17:18Ezra Klein?

  Yes.

• 17:20Speaker 07

  Wow. Oh, my God.

• 17:23Ezra Klein?

  Yes. Paul Cristiano, one of the leading AI safety researchers, he just joined OpenAI's nonprofit board. He is serving on its safety and security committee.

• 17:33Speaker 02

  I think maybe there's something like a 10%, 20% chance of AI takeover many, most humans dead. Overall, you know, maybe you're getting more up to like 50/50 chance of doom shortly after you have AI systems that are at human level.

• 17:49Ezra Klein?

  I know how wild all this sounds. And I really can understand the skepticism of this. If you believe AI has a 10%, maybe more, chance of extinguishing or displacing humanity, it really stands to reason that you would not work at a company trying to build it. But what I want you to know, because I've known a lot of these people for a long time now. Many of them were saying the same things 10 years ago. They were saying these things before they worked at these companies. They were saying them before they had stock options, before they had enterprise software contracts.

• 18:25Speaker 08

  Please welcome to the stage Y Combinator President Sam Altman and our moderator Kimai Cutler.

• 18:31Speaker 09

  It seems like there's a huge disagreement over, you know, whether unfriendly AI is going to lead us to an AI apocalypse.

• 18:39Speaker 10

  Yeah, well, you know, in a sense, this is like, this is not just creating new technology. This is creating a new life form. And I think that's just like really high beta. It could be great, but I think we should be working to make sure it's great and not bad.

• 18:55Ezra Klein?

  No one was listening to them. And so these people in the wilderness of their obsession and their terror, they thought and thought and thought about how to make AI safer. And the answer that some of them, not all of them, but some of them came to, was he should start trying to build these systems, start running tests on them, researching them, learning how to make them safer. Because you don't solve hard problems in theory. You solve them through practice. And the irony, the irony is that in many cases, they chose that path because they were worried that people already building AI were too reckless or too commercial in their approach. You can read it in the email that Sam Altman sent Elon Musk in May of 2015, an email that led to the founding of OpenAI.

• 19:40Ezra Klein?

  Been thinking a lot about whether it's possible to stop humanity from developing AI, Altman wrote. I think the answer is almost definitely not. If it's going to happen anyway, it seems like it would be good for someone other than Google to do it first. OpenAI was founded because its co-founders thought Google DeepMind would be reckless. Anthropic was formed by OpenAI employees who thought OpenAI had become reckless. XAI was formed because Elon Musk thought that OpenAI and Anthropic were dangerously woke. The US just broadly is racing forward, in part because it is worried about what happens if China gets to self-improving AI first. The result is this tragic collective action problem. The AIs we are building, they're not safe.

• 20:21Ezra Klein?

  But the CEOs and the politicians, they fear the other companies and countries that are building AI are even less concerned with safety and ethics than we are. In the words of Ted Cruz.

• 20:32Speaker 06

  They're going to be killer robots. I'd rather they be American killer robots than Chinese killer robots.

• 20:38Ezra Klein?

  I admit, there is a kind of brutish logic to that. But it assumes that the killer robots will be controlled by America or China, by one country or another. But what if that assumption is wrong? What if the robots are simply out of control? The debate over AI's safety tends to focus on the idea that AIs will kill us all. I find this forces a conversation into this realm of thought experiments that people then begin arguing about. I don't find it that helpful. What I think we should focus on is something more straightforward, something nearer at hand. Loss of human control over AI. That may or may not result in total human extinction. I'm agnostic on that question.

• 21:22Ezra Klein?

  But it would be bad. We shouldn't allow it to happen. This is a goal that the US and China should be able to agree on. Xi Jinping gave the keynote at the recent World AI Conference in Shanghai. He ended it by saying, "With AI advancing at a staggering speed, we must ensure its development is for the positive, for good and for humanity. We must make its oversight and governance precise and effective, and constantly refine measures to forestall loss of control." But it's important to realize that loss of control, it's not just something that might happen to us. It's something that the labs are trying to make happen as fast as they can.

• 22:04Ezra Klein?

  This is the horrible paradox, the horrible tension at the heart of the AI labs right now. They fear above all, loss of control over superintelligent AI. But their explicit product path is to cede control, to give away control as fast as possible, so that their AIs can begin building better AIs faster than their competitors. In recent months, both Anthropic and OpenAI have released reports on how close they're coming to AI that can self-improve. In June, Anthropic released "When AI Builds Itself." It begins, "For most of AI's history, humans drove every step in its development cycle. But at Anthropic, we are delegating a growing share of AI development to AI systems themselves,

• 22:49Ezra Klein?

  which is speeding up our work." It sounds like a fake commercial you would see at the beginning of a sci-fi horror movie. But it doesn't, to their credit, continue that way. They go on to give some data. "In February of 2025, a tiny fraction of the code that got added to Anthropic's codebase was written by Claude. But by May of 2026, it was over 80%. And here's another way of looking at it. This is data Anthropic gave me more recently. Anthropic tried to categorize the way its employees were using Claude for R&D work to make better versions of Claude. So the low end, an employee could not use Claude at all. They could use Claude minimally. But then it escalates. Claude can be an assistant. Claude can be treated as an equal collaborator.

• 23:34Ezra Klein?

  Or Claude can be given the lead on a task. Just go do this. Go figure it out." A year ago, there were basically no examples of Claude being the lead on a task. By August of 2026, 26% of Anthropic's R&D tasks had Claude classified as a lead. I think it is reasonable and wise to be skeptical of these numbers. Reasonable and wise to worry about whether it's all just marketing copy for Claude code. See, look how fast we're going. You could go that fast, too. But where Anthropic takes this in that same document is different. They say that "A world in which Claude achieves recursive self-improvement is a world in which, quote, 'misalignment present in today's models could compound

• 24:17Ezra Klein?

  as the models build their successors, growing more frequent but less understood, until we lose control of them.'" This is why Anthropic, to their credit, has been relentlessly calling for regulation to slow the pace of development. Regulation would arguably harm them the most, as they have often been the company furthest out on the AI frontier, and RSI is a process by which they could race forward even faster. Then in September, OpenAI released its own report on what it called "research acceleration." The company says they've already achieved the equivalent of having a fully automated AI intern. And that by March of 2028, they think they'll have a fully automated AI researcher. And when they have one, they can have, you know, basically as many as they want.

• 25:01Ezra Klein?

  Like Anthropic, what could be a triumphalist release quickly turns dark. "We do not yet know how to safely get all the way to aligned, full RSI," they warn. At around the same time, OpenAI did something else that I think deserves more attention. They released this new model, Astra-6. The model is arguably more powerful than anything that has come before it. And when you test it, it seems better aligned. It doesn't cheat as much. But OpenAI said, "They're really not sure if that's true. Astra seemed to be better at knowing when it was being tested, which meant it could just be giving its evaluators the answers they wanted to hear." What Daniel Selsom, a capabilities researcher at OpenAI, wrote, "has been ringing in my head." He said, "The crucial and overlooked problem

• 25:46Ezra Klein?

  is that the models are becoming so situationally aware that we are losing the ability to evaluate them in contexts where they believe they are not being watched or controlled. Put more simply, the models are increasingly smart enough. They know when we're watching them, and they change their behavior accordingly. So what they do when we are testing them, when we audit them, it may not tell us what they'll do in the wild." So some of these answers people are giving, like, let's just do better testing. We have no idea if it will work. Because we don't know if the AI systems are just telling us what we want to hear. And so, look, I don't want to sound too radical when I say this, but a thought? If you are losing your ability to evaluate the models you have now,

• 26:31Ezra Klein?

  maybe don't let them build models you'll be even less capable of controlling in the future.

• 26:38Ezra Klein?

  Once RSI takes off, humanity will not understand the AIs being built, because we will not be building them. Development will not move at human speed. It will not be overseen by human minds. We will have to hope that the AIs we have built, and the AIs they will build, and the AIs those AIs will build, and on and on and on, will be acting with our best interests at heart forever. If this summer has proven nothing else, it is how naive that proposition would be. The labs are a little bit queasy on just not doing RSI. In an interview with Fortune, Sam Altman was asked about banning it, and he said, "I think it's very hard to say what a ban on RSI means."

• 27:22Ezra Klein?

  I've heard this from others at these labs, and I want to say, I don't find it so hard to say what a ban on RSI means. I find this absurd. A couple of years ago, none of these labs had turned substantial coding over to the AIs. It was just human beings typing code at human speeds with our clumsy human fingers. Now, most of the code is written by AI. So as a first step, as we figured out, we could just go back to where none of the code is written by AI. I'm sure that's on the right side of the not doing RSI line. The default on this, it needs to flip. The labs need to prove to us that what they're doing is safe. If they want to work with Congress to carve out narrow exceptions, fine. If they want to figure out where it is

• 28:06Ezra Klein?

  really, really, really, really safe to do it, okay. But forcing development back to human speed, perhaps even erring on the side of going a little bit more slowly at the frontier, that's the point. That's not the regulations going wrong. And I believe in us. Our society is good at nothing if not making it hard to build new things. Where these labs are located, you cannot build an eight-story apartment building without an agonizing public review process, and probably not even then. And yet somehow it is possible for these labs to unleash a swarm of 40,000 AI agents to build a society-altering superintelligence without so much as a hearing. OpenAI would need permits

• 28:49Ezra Klein?

  to cover their parking lot in solar panels, but they can accelerate into recursive self-improvement, as best I can tell, whenever they so choose. There is nothing inevitable about any of that. These are political choices, and we can and should make other ones. And I want to be very clear about this. I do not mean to suggest that stopping RSI until we can prove it safe, that that's all we need to do to control the AI frontier. That is the beginning of such an agenda, not the end. But it is the beginning. It is the decision that will do the most to make sure human beings at least understand where the frontier is, that we know what is happening on it, that we remain in a position to make decisions about it.

• 29:34Ezra Klein?

  There's a line from Madeleine Miller's beautiful book, Cersei, that has been running through my head during this long summer of strange AI news. The line comes at the end of the book, "After a tragic prophecy has been fulfilled, despite every effort made to avoid it, Cersei says in despair, 'The fates were laughing at me, at Athena, at all of us. It was their favorite bitter joke. Those who fight against prophecy only draw it more tightly around their throats.'" I have a lot of respect for many of the people at these labs. They began working on AI because they wanted to better humanity. They began working on AI because they feared incomprehensible autonomous AI

• 30:17Ezra Klein?

  slipping out of humanity's control. And they were right. They saw what was coming, and they were so right about it, they've built some of the most valuable companies with the most transformational technology in human history. And now they find themselves racing each other to build incomprehensible autonomous AIs that they admit are slipping out of humanity's control, slipping beyond even our ability to monitor. This is the tragedy of their work. In fighting against a prophecy, they have drawn it tighter around their necks and ours. It is time to make them stop.