An Argument Against AI Doom

The flood of reports about AI agents escaping and hacking websites can feel scary. But is this actually just a matter of needing better security to match the evolving technology? Zack Korman, the CEO of Embroidery, thinks so. He works in cybersecurity and argues that “rogue” AI agents are less frightening and complicated than they may seem. “Galaxy Brain” host Charlie Warzel talks to Korman about how companies and institutions should up their cyber defenses, why the so-called AI doomers are frustrating him. - - - Get more from your favorite Atlantic voices when you subscribe. You’ll enjoy unlimited access to Pulitzer-winning journalism, from clear-eyed analysis and insight on breaking news to fascinating explorations of our world. Atlantic subscribers also get access to exclusive subscriber audio in Apple Podcasts. Subscribe today at TheAtlantic.com/Listener. Learn more about your ad choices. Visit podcastchoices.com/adchoices

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Tired of rage scrolling through AI slop and clickbait just to get a handle on today's news? Meet Inpress, the algorithm-free news app that makes it fun and rewarding to stay informed. Track your media literacy and rediscover reality. Download today at inpress.app. That's I, N as in news nerd, press.

• 0:19Speaker 01

  Do you want to hear about visionaries creating a better world? People working in conservation, exploration, science and tech alongside storytellers imagining and creating a better future. I'm Alex Honnold, professional rock climber and founder of the Honnold Foundation, bringing you season six of Planet Visionaries. This season, I speak to guests like National Geographic photographer Tom Pescek and Anna Rappman, executive director of the Jane Goodall Institute, USA, in partnership with the Rolex Perpetual Planet Initiative. Watch Planet Visionaries on YouTube or wherever you get your podcasts.

• 0:49Speaker 02

  I think that realistically, what will happen is models will get better. And if the labs do not up their cybersecurity game, and if the companies running AI agents do not up their cybersecurity game, they will cause real harm. And then we will like literally get them in trouble. Like, we will put someone in prison because you killed people. You're not allowed to kill people. And the doomers would say, no, but then it's too late because they're afraid that the people we killed would be like 8 billion. Whereas I'm not. I'm like, no, because that's not the real world, doesn't work like this.

• 1:21Charlie Warzel?

  I'm Charlie Warzel, and this is Galaxy Brain, a show where today, we're going to calibrate our anxiety about the AI hacking epidemic. Specifically, we're going to answer the question, is the rogue AI agent problem really just a normal cybersecurity issue? Or is it something else altogether? It's a scary moment. As we're recording this on September 29th, the news is moving quite fast, but allow me to summarize some of the broader points. Since August, there's been a pretty steady drumbeat of revelations, starting with OpenAI's agents hacking Hugging Face during a training exercise. A few more followed.

• 2:03Charlie Warzel?

  Agents hacked into a German website to use it as a rudimentary message board. An agent for OpenAI gained unauthorized access to a data portal that was run by Australia's universal healthcare insurance provider. And last week, OpenAI revealed that its models attempted to hack or otherwise access numerous US government websites. That same day, OpenAI announced that 53 incidences happened where photos were uploaded by users to the company via its products, which were then uploaded without permission onto other platforms. Last weekend, Axios reported that OpenAI and Anthropic were looking at tens of thousands of potential incidences where agents were behaving in unexpected ways, either coordinating

• 2:44Charlie Warzel?

  or trying to get around or succeeding in getting around company guardrails and causing problems on the open internet. Now, there are some positive signs here. This week, OpenAI announced it was pausing training of its most capable models and delaying the release of its newest model, citing security concerns. But let's be clear, these are very real examples of technologists losing control or being unable to monitor their tools. And there's lots of talk about what kind of liability these companies ought to face, whether these types of hacks are subject to penalties under the Computer Fraud and Abuse Act. Online, there's been a constant debate over what is going on, why nobody has been held accountable, what accountability means, and what needs to happen next.

• 3:26Charlie Warzel?

  But all that chatter and all this news has merged with a different narrative online, the one that is best represented by the AI safety and doomer camp, and the fears that an out-of-control, self-improving AI system could one day, quote, "kill us all." But these are, in a way, two different stories. One is about a hypothetical improved set of models. It's a frightening scenario, but it is speculative. The second story, the one of the AI agent hacks, is not speculative. It is something that is happening right now, and it is both urgent and it is tangible. And my guest today, Zack Korman, argues that this is a problem that is not sci-fi in nature, that we are not dealing with machine gods of agency, but with a sprawling cybersecurity problem.

• 4:11Charlie Warzel?

  And the companies are acting negligently. Korman runs embroidery and AI agent monitoring and detection platform, and so I wanted to have him on to talk about what's really going on here with these high-profile hacks. Can we constrain these agents? If so, how? What are the real risks involved with agentic AI? And how do we keep them from coming to pass? Korman joins me now to talk about it all. All right, Zach, welcome to Galaxy Brain.

• 4:44Speaker 02

  Thanks for having me.

• 4:46Charlie Warzel?

  So we are talking here on September 29th, after a weekend where there's been a lot of news, mostly about OpenAI agents accessing outside systems, some inside the US government without authorization. I imagine if you are somebody who doesn't have a background in AI or cybersecurity, you are looking at this like we have entered some kind of nightmare scenario. You, however, work in cybersecurity and AI. So just let's set the table here. What is your broad reaction to all of this news that has transpired over the past week?

• 5:20Speaker 02

  Yeah, I mean, it's not as scary as it sounds. One of the core challenges of working in cybersecurity and suddenly having cybersecurity be a dramatic topic is that for people in cybersecurity, like, everything has been bad forever. We've always had problems, and we have never had a happy situation. We're very used to dealing with bad things. I think a lot of people now have started to pay attention for the first time. And so every time there is some sort of incident, it feels a little bit like the end of the world. In a lot of these cases, they are not as dramatic as they maybe sound, but they do

• 6:00Speaker 02

  point to some serious situations inside of the labs, the AI labs, that maybe are not so great or not so well done, so to speak.

• 6:08Charlie Warzel?

  So a few days ago, another OpenAI employee wrote this rather alarming post on X about what it's been like at work recently. Quote, "Last three months equals hell," he wrote, speaking to what you said about the, you know, when you're in cybersecurity, there's a lot of bad things happening all the time. But specifically, he's talking about what a lot of folks at these AI companies are talking about. The pace of improvement of these models has really surprised them. And this was sort of leading up to the big Hugging Face hack, which I want to talk about. And he has said, quote, "We had not expected it this soon. Suddenly, we weren't dealing with just a small jump in capabilities.

• 6:51Charlie Warzel?

  We were talking about a different sport altogether." And he seems to admit that security should have been better, that it also takes time. I am curious what you make of this idea of the pace of everything being sort of the reason that we're having these conversations right now.

• 7:10Speaker 02

  I think that there is a very delicate balance to walk here between, yes, things can be hard, and I think the fundamental diagnosis of like the pace of development at these labs, especially just the general growth. I mean, you're adding new employees constantly, you're like constantly doing new things, is an environment that is likely to lead to incidents. But at the same time, to balance that against the reality, which is that they are a trillion-dollar company with, so let's just call that infinite money, that can do, you know, they have the ability to do better jobs than the hair tubing. And so it's like, it's kind of hard because like I can sympathize with the idea that

• 7:53Speaker 02

  like a super fast-paced development environment where you're growing constantly and doing all these new things can be challenging. But they have, there are things that they could be doing better. On top of this, when we talk about the jumping capabilities being a surprise, I think, and he does hint on this, and he does suggest it at points, it's sort of a question like, to who, right? Like, there are people at OpenAI for whom this would not be a surprise. Okay? So there are people at OpenAI who fundamentally have been talking about some of these challenges as coming down the pipeline for a very long time. I think that they probably shocked the organizational structure more than they shocked like some

• 8:37Speaker 02

  of the people involved, right?

• 8:40Charlie Warzel?

  Who are those people specifically?

• 8:42Speaker 02

  Well, a very interesting thing is, the monitoring team at OpenAI has done some very good work on monitoring. For example, they have this like great write-up from maybe like March of this year about how they monitor AI agents and the types of problems they're worried about. And that outlines all of the broad range of risks that you might be thinking could happen in a case like the Hugging Face attack. So if you work on that monitoring team and you built that type of monitoring, you are not going to be Shocked and appalled and caught off guard by the possibility that agents can do these

• 9:25Speaker 02

  things. You literally built a monitoring system for it. At the same time, there are like, you know, AI researchers who are running some of these capability tests and stuff that maybe don't have not, it hasn't been set up correctly to use that monitoring, things like that. But, you know, you have work that was done, there was literally writing about these risks at OpenAI prior to these, prior to the summer.

• 9:47Charlie Warzel?

  I think that this is part of what is really frustrating for a lot of people who are, who are watching this, because, you know, there, there are different things here, right? There's the sort of the, the doomer, like, we're all gonna die, that this is gonna kill us all in a certain amount of time thing. And then there's the more like prosaic concerns of cybersecurity. But I think what is very difficult is, is the, these companies are constantly messaging about the dangers. They are writing like, like all these elaborate blog posts, where the idea is that, you know, we have this eye towards safety all the time. And yet, it does also seem like these companies get extremely surprised by the things that

• 10:28Charlie Warzel?

  they are doing all the time. And, and, and so is what you're trying to say there, basically, that there's just kind of different folks doing different things, and different people being surprised at different parts of the organization, but we're treating it like a monolith? Is that a little bit of what you're trying to say here?

• 10:44Speaker 02

  Yes, although I would also argue that we should treat it like a monolith. So while I can acknowledge that inside of it, there are people who would have seen things coming. One thing I don't like is when the labs use this as an excuse, because I'm like, no, to me, you are OpenAI. Like, I don't care if you are a security person at OpenAI, the monitoring team at OpenAI, a researcher at OpenAI. OpenAI, the company, needs to be responsible if they are going to speak about the risks in the way that they are. I would actually be more tolerant of some of these mistakes if they were a little bit like, less doomer about it. So they were kind of like, yeah, some mistakes happened, we did some hacking things. The damage from these hacks is like nonexistent effectively. I mean, none

• 11:29Speaker 02

  of them have been especially bad. And I think that that is a perfectly valid thing to say. The problem is that they are using these attacks to then say like, look how terrifying this is. I need you to regulate this, you know, this entire industry. We must pace the frontier. And I'm like, okay, there's this mismatch now between what OpenAI is saying about the risks and the behavior we see as an organizational level towards those risks. That mismatch, I think, is what is very frustrating for people in cybersecurity, for example, and probably very confusing for people outside of it, right?

• 12:08Charlie Warzel?

  Just to, you know, cap what, what you're saying there, do you feel that the reason why this stuff goes immediately to the doomer part is because it's an effective marketing strategy for these, these companies? Like, where do you fall on, on why they jump to that type of language and, and fear mongering?

• 12:33Speaker 02

  Yeah, so I think this is mixed. I think that there are, for a lot of them, for many of them, especially lab, like a lot of the employees, a lot of the people there. The doomerism is like an inherent part of the culture that has been there for, from the very start, okay? They have been afraid of an AI intelligence explosion since like 2010, okay? And in those cases, when you start to see things that are like a little bit, like a tiny bit interesting or in line with that theory, my view has always been that they kind of interpret them way too much in line with their own prophecies. So they say like, one day, AI will hack people. And then they're like, and as foretold by the prophecy, it

• 13:17Speaker 02

  hacked somebody. And I'm like, okay, but, so pretty, pretty tame example, right? Like, there's plenty of ways to interpret it that is not that. But if you've lived in this world of risk for like, 10, 15 years, where you really believe it, then of course, this will seem scary. And so I think for a lot of them, they're just like genuinely, they genuinely believe how terrifying this is. From the perspective of like the lab as a whole, I think there are definitely strategic elements to the way they communicate it. I don't want to say these are their clear, their motivation specifically, but it doesn't hurt them, which would be one of the hardest parts about making the economics of these labs work, is that you're spending constantly, running higher and higher capital expenditures on training

• 13:59Speaker 02

  new models to keep up with the frontier. And of course, they're running huge losses. The problem, so of course, being able to pause development has some benefits for them. I don't think that is the core element to it. I think it's really like, just the normal regulatory capture play, though, of like, when you see that the sharks are circling and you're about to get regulated, you always come with your regulatory proposal first, right? You like, you always, it's like, this, this is like, we saw this with finance, right? As soon as like, they realize the, the game is over, you're not gonna get to play on your own anymore. You like, you try to race to get your own measures in place. So, you know, Dario came forward at Anthropic and said, I want third-party evaluators, and I

• 14:41Speaker 02

  want it to be meter. And this is just, he's just picking his preferred regulator before the US government picks for him. Because the US government might pick someone who behaves a bit, a little bit more like the Department of War, that hates Anthropic. And so that would, that would of course not be what Anthropic wants. Also, I think it's very important to keep in mind that the labs, because they have all of these doomer employees, like, they can't not pay like, lip service to it, right? They can't, they can't go out and say like, I don't find these risks scary. Okay? Like, I know, you know, it wouldn't, if OpenAI said that, like, they would probably have a lot of people resign. So I think there's like a lot of, there's like restrictions on them when they have these employees with these views.

• 15:26Charlie Warzel?

  It's really interesting to think about, because I think a lot of these, these executives and these employees, like, they have credibility in the fact of they've been saying the same doomer narrative for so long, right? Like, it's hard, it's hard for me to accept, you know, completely that, that this is cynical, you know?

• 15:45Speaker 02

  I think they overblow the, like, they blow up a little bit too much, the extent to which the, they have been, they've been saying these things, and that we now would say something like, and now some of it's coming true? That's the part I don't believe. Like, they have been saying some extraordinarily radical things about the way in which intelligence explosion will lead to like, massive risks, societal risks. And what happened was like, OpenAI hacktugging face. Like, that's very actually uninteresting. And in fact, is a normal prediction that many in cybersecurity, myself included, made, right? I think, and I can say this from the perspective that I was very much like a doomer back in 2016,

• 16:28Speaker 02

  right? I had read Superintelligence by Nick Bostrom. I was like, I'd gone to talks by him. I was like, I was in that world. The events that we see playing out today don't map that cleanly to doomerism, unless you really want it to be true.

• 16:44Charlie Warzel?

  What, what changed from your, your doomer era, your superintelligence era to, to now?

• 16:50Speaker 02

  Well, first of all, I was, I was a law student. I, you know, I studied law, and then I had a law and finance degree. So I was at Oxford at the time, which is where a lot of doomers were, it's where Nick Bostrom was. It's easy to get swept into it, especially if you don't know how computers work, which I didn't. And then, you know, I spent the next 10 years being like, leading tech teams, right? Like, I taught myself to write code, ironically, almost in the same like, kind of space as this doomerism starts to emerge. It's kind of why I'm learning to code. And then, you know, I think that a lot of what doomers predicted started to really not, you know, I think when GPT-3 came out, and then you had GPT-4, which mapped like the scaling laws, right? So like, you kind of

• 17:34Speaker 02

  see that it's gonna get better at the rate you expected to get better. It seemed really, really scary. And then you realize that even four, five, GPT-6, Astro, great models, there's no intelligence explosion here. There's no, no one's, no one's dying, right? And I mean, that's not to say people won't, it's to say, this actually does not- This does not, is not as concerning as maybe a 22-year-old Zach would have expected it to be. And so I'm sitting there going, yes, in theory, I believe in the risks of superintelligence. Like, sure. It doesn't really look like the world we're in right now.

• 18:14Charlie Warzel?

  So let's walk through the Hugging Face hack specifically, and try to explain in the most simple terms, trying to use the least anthropomorphizing language we can, because these AIs aren't human, though sometimes we do need a little of that language to explain what's happening. But in the most grounded terms, what did these agents do and what didn't they do in this example?

• 18:39Speaker 02

  Yeah. So basically, what they've done is they set up these agents to, they gave them a task where your job, the job given to the agent is to find a vulnerability in a piece of software in order to extract like a flag that you get to say, okay, I proved I exploited this vulnerability. Okay. This allows us to measure the success or the failure of agents in doing that, and also to train them to do it effectively. Now, in the process of that, these agents they were running were meant to be basically sandboxed in, where they were not supposed to have access to the internet. Okay. And so they're supposed to just go after

• 19:20Speaker 02

  this piece of software to find vulnerabilities to get to the flag. But there's a lot of them that are just not, that we just do not know if they can even be solved, or at least have not been solved yet. And this was one of them. So at least prior to this moment, there is no known solution on this one, as I understand it. What that means is that in order to pass this eval or pass this test, these agents have to find a way to solve it that is not, you know, they can't find a normal path in. This leads them to basically starting to explore some more interesting opportunities. And basically, what it turns out is there's this

• 20:01Speaker 02

  flaw in the sandbox that allows for the agents to first communicate actually with each other, because they're all running in parallel, but they're not supposed to be able to talk. So these agents are now able, even though they can't solve the real task, they are able to communicate to find a way to solve the task. What happens after that is they figure out a problem with the way the flags are created that allows them to create the flag themselves without having actually done the vulnerability. So it'd be sort of like, imagine that I found out that there's a way, that the way you set up the flag is it was a color plus a number, and the number is today's date. Well, then I can give you that and say,

• 20:41Speaker 02

  like, look, I found the answer, but I didn't actually solve the problem. So they basically cheated, right? The problem is, they then come to the conclusion, which was incorrect, if we give this, we'll fail, because the thing that is going to check whether we succeeded, it doesn't just check if we got the flag, it checks how we got it. Okay? And that's a problem for them, because they got it by cheating. Okay? And so you have these agents now that are in their reasoning convinced, like, quote unquote, that basically that like they're going to fail because they cheated. And this leads them to the idea that if you

• 21:25Speaker 02

  get more information about how they're scored, that could help them to find a way to pass this test. And the way to find the information about how they're scored is to go to a system called Hugging Face, which is this third-party website that has a lot of this data, and to hack into them to find information about how they're going to be scored. Okay? And so they did. Okay. That is the story. And I know I use a lot of, like, I kind of talk about it like it's a bunch of people running around. The reason I think that's like relatively fine is that we can come back and build an intuition about like how it worked at like a computational level. But like, even that story I just told makes it a little bit less

• 22:06Speaker 02

  spooky than when you believed that like, they went off and just started hacking, right? Because you kind of get to follow a logical train of like, well, first, they feel they can't pass the test. Second, they find they can pass the test, but they think they're still going to fail. They need to find a way to not, you know, there's like kind of logical steps at each point in time.

• 22:27Charlie Warzel?

  I think about it as, as, you know, they have two goals, right? They have like a hard, concrete goal, which is solve this, this problem that you are being tested, this is an evaluation, get the answer, right? And then you have like a fuzzier secondary goal, which is really important to the companies and to, you know, the idea of what we call, what we're calling alignment, right? Which is like, follow the rules, act morally, behave in the right way. And if you're a computer, if you are a, you know, a whole bunch of like, like, numbers, you're going to have more fluency and understanding with accomplishing

• 23:12Charlie Warzel?

  the hard task, right? Whereas the like, the fuzzier task is much more, the way I've been thinking about it is like, you know, behaving a bit almost like a lawyer would, right? Which is saying like, yes, my, my goal is to get, you know, solve this thing for my client. And yes, I'm not supposed to, you know, no one's supposed to go to jail, but the law, as you can interpret it, would allow you to do X, Y, and Z, right? And this sounds a little bit like, you know, okay, it's even more crooked than that in the sense that like, it knows we did something bad, we're covering up the tracks in this. But it does seem, as you said, less scary from the idea of a company is trying to get its, you know, its program

• 23:55Charlie Warzel?

  to do this thing, and it essentially just does it, right? It just does it in a way that they didn't know, they didn't expect it to do.

• 24:03Speaker 02

  Yeah. And this is actually a big part of like the Doomer hypothesis, which is that that taken to its extreme would be terrifying, because like, what if you, I mean, they give the example of like paperclip maximizing, like you give it a task of making as many paperclips as you can, and this leads it to run off and grind, grind all humans up to dust to make us into paperclips. The reason I think that's not like, that terrifying in these cases is that like, basically, all of the steps that would have prevented this were not actually in place in this case. So first of all, Anthropic has pretty clear study on the fact that if the task had at the end given them a bit of like, don't hack

• 24:47Speaker 02

  companies, don't external access network, right? There's no reason to believe the agents just chose to ignore that part of the task. It's actually that that's just not really part of their task, right? And so there's sort of like, information that might be missing that might be included when you and I use agents normally, where we go like, I need you to do this, please go through this, right? Here, we gave them an extremely broad task. And this is something, when I talk about cybersecurity, do people predicting this? I mean, I have an, I have like a very direct tweet about this, where I said, if you set up a really long-running task that just has broad goals, so like, not please achieve this, but like, I want you to go constantly make sure that I'm making more money or something,

• 25:31Speaker 02

  right? And you have it, you know, a lot of tokens, so the ability to run and use a lot of like actual AI computational power. I basically said, then you better make sure your sandbox is really good, otherwise you're in trouble. Because agents will go out and solve problems in very interesting and unexpected ways. That is just to be expected. And I think that what is challenging is, this is to me, a case of that, where they see it as like alignment, and that's terrifying, because we would never want an agent to do that. Whereas I go like, you should just expect agents to do that. Like, you should just expect that agents will go solve problems in absolutely insane ways, and we should build our security systems in expectation of that.

• 26:22Speaker 00

  If you're trying to get news today, you'll find a lot of biased algorithms rage bait and AI noise. Think social media. That's where Inpress comes in. It's a social platform on top of factual journalism, not the other way around. Track your reading and earn points. It's like Strava for your brain. Plus, you can connect with other users in your city who share your genuine curiosity. Ditch the echo chamber for reality with Inpress. Download it today for iOS or Android at inpress.app. That's I, N as in news nerd, press.

• 26:55Speaker 01

  Do you want to hear about visionaries creating a better world? People working in conservation, exploration, science, and tech, alongside storytellers imagining and creating a better future. I'm Alex Honnold, professional rock climber and founder of the Honnold Foundation, bringing you season six of Planet Visionaries. This season, I speak to guests like National Geographic photographer Tom Pechek and Anna Rappman, Executive Director of the Jane Goodall Institute, USA, in partnership with the Rolex Perpetual Planet Initiative. Watch Planet Visionaries on YouTube or wherever you get your podcasts.

• 27:28Charlie Warzel?

  Where specifically do you think is the negligence here? Is it that? Is it just not anticipating that this would happen? Like, how does what we saw in this instance differ from normal cybersecurity practices? Because sandboxes are not a new thing, right? This is how this cybersecurity works, how you test these different things, search for these vulnerabilities. Yeah. How does this differ from the normal practices?

• 27:52Speaker 02

  Yeah. They weren't running monitoring. So understand that these agents are saying things like, oh, my God, this is crazy. We're going to hack Hugging Face and no one's noticing it. Okay.

• 28:05Speaker 02

  If you are monitoring, if you are running agents that are, if you're basically taking the logs from these agents and checking them automatically, either using AI or some other technique, you would catch this. On top of that, you know, and it doesn't have to be anything sophisticated. In this case, the agents were not supposed to have access to the internet. And so as a result, you can just say, like, if any request leaves, goes to the internet, flag it as, you know, create an alert and that alert would come back and cause OpenAI to shut off this run. They didn't have that in place, so they were not running any monitoring. Now, my company is an AI agent monitoring company.

• 28:47Speaker 02

  I have corporate customers just running normal agents with better monitoring than OpenAI was running on a cyber eval that was impossible. You know, you handed a long-running task with two agents that actually didn't even have all of their, the alignment training on some of the agents, I don't know which ones, but Greg Brockman said it, alignment training had not been completed on all of these events. You're kind of like, and cyber guardrails have been turned off. You're dealing with the most dangerous form of agents and you're giving an impossible task and you didn't monitor them. And so not running monitoring, I think, is like the ultimate sin because this played out over the course of days.

• 29:27Speaker 02

  So this is not like they pressed a button and 30 minutes later, all of this had happened and they came back and were like, oh, my God. No, they're, they've gone to bed, they've woken up, they went to bed again, they woke up, right? This is like not a short period of time where they have not noticed what is happening.

• 29:42Charlie Warzel?

  So there have been other instances of this where there's been unauthorized access from agents. It's not just OpenAI, but there have been lots of disclosures recently from OpenAI that have made news.

• 29:56Charlie Warzel?

  From what we know, there's an instance in Australia, there's instances in some US government systems. Is, from what you can gather from what is disclosed about all of these instances, do they all just follow the same pattern? Or are there differences in each of these that are, that are unique? Or would you say it's just, it's the same thing happening again and again?

• 30:21Speaker 02

  Yeah, it depends. So there are a few differences, but I don't find the differences interesting. Some of the recent cases that have been coming out have been extremely mundane. So they give it a task, like research some information and it goes to basically test its ability to do web search and things like that. And a lot of these government websites are like really bad. So they're like, they're like really, really vulnerable to the point where I wouldn't even call it a vulnerability sometimes. Like, they just have URLs that are publicly available, but are not supposed to be accessed. And so the agents, while trying to solve these tasks, I don't even, I, we haven't seen a lot of the reasoning traces for these things, obviously.

• 31:06Speaker 02

  But I strongly suspect they do not all say, oh, this is bad, I shouldn't do this, oh, I'll do it anyways. I think a lot of them just go, okay, they, like, it's kind of, you wouldn't notice. And that actually is a very human thing as well. There's a case here in Norway where, this was years ago, where basically a human developer was working with the road authority and they had some data available publicly that they actually didn't intend to be available. And he basically asked for the data because he wanted it for this thing they were doing. And they said, yeah, we'll look into it. And then he got back and said, no, I actually found the data, don't worry. And then they put him in prison. Okay.

• 31:50Speaker 02

  Like literally, he went to jail for it for like a month. Because he doesn't even know that he did something wrong. He's just accessed publicly available data, but he did it in a way that it turned out like, actually, that was not supposed to be available. And they considered that hacking. I think a lot of these cases are similar. They're like, the agents just went in and were like, oh, here's the URL. Okay, I'll find that, you know, it's available. And so I think some of those are overblown. But there are other cases, like one in the UK with AISI, which is the UK government, where they were running evals with Anthropic and OpenAI. And in those cases, they actually intentionally gave the internet access on a cybersecurity task.

• 32:32Speaker 02

  And those ones led to some very dangerous actions as well. Which is a little bit different because they intentionally gave the internet access. I think this raises, again, like negligence questions, which is what's interesting to me. The agent behavior is totally like kind of normal and expected. Like, I mean, it's not that stunning to me that agents would engage in some of the activities that they did, given the tasks that they are given and given the constraints they faced. And in the AISI case, they actually admit that. They kind of say like, we gave it a task that, they gave it the wrong task for the environment. And like, that's probably a big part of what happened.

• 33:10Charlie Warzel?

  This is the, this is the instance that you're referring to was the one from late July, I believe, right? Which is during this routine evaluation, the AI Security Institute noticed Anthropic's Mythos 5 model tried to insert malicious code into an open source software project, right?

• 33:29Speaker 02

  Yes.

• 33:29Charlie Warzel?

  That's, that's what I'm talking about here. And the idea basically that the agent started engaging in a social engineering, creating fake online identities to get the code that it wanted approved. So, so, okay, there's a distinction here that I, that I think is, is interesting too. Because, you know, you are basically saying, this should be expected stuff, right? This is actually like pretty prosaic behavior of a computer program that you train to be sort of relentlessly optimized to solve a task, right? And when you do that, and the issue is the guardrails around it.

• 34:11Charlie Warzel?

  And yet, at the same time, not to like drag in the doomer rhetoric, but what I think is, is concerning, what is concerned lots of people looking at this, is this idea that there will be this, this combo of the two things, right? This, this idea that, that these, these agents will continue to behave in ways that are unexpected. You know, we've seen the cheating behavior, the collaboration behavior, which is just from the training, right? It's how these things, these are trained to sort of accomplish tasks in ways that mimic the ways that, you know, humans accomplish tasks. Cause they're trained off of all this, you know, human text and data and information.

• 34:55Charlie Warzel?

  Combining that with this idea that the, the models are going to continue to improve and that they're going to improve in such a way that we aren't going to be able to adopt the cybersecurity practices to constrain them, right? And, and so I guess, like, doesn't that freak you out? Because you're, in one sense, you're talking about how OpenAI is, you know, not doing some of the things in terms of just basic monitoring that are pretty, you know, like table stakes stuff in the industry.

• 35:31Charlie Warzel?

  And having these relatively powerful things that they're, you know, giving these tasks and unleashing on the internet. Like, isn't that a recipe for disaster?

• 35:41Speaker 02

  I would say this is actually the way the public understands the threat that is being warned about. It is actually not the threat that is being warned about. The problem the doomers warn about is that, the problem with that is that it will happen so quickly that one day it will look like today, Hugging Face, and tomorrow it will look like extinction. Okay. And that there will not be points in between these two that where we can say like, what's happening? Do we need to stop something? They want us to either stop today or pace the frontier, or whatever it may be, have restrictions in place because they're afraid of this thing. We call it the intelligence explosion, okay?

• 36:23Speaker 02

  Which is like recursive self-improvement kicks in, the models get better and better and better, until the security guardrails can't even keep up, even if we wanted them to, blah, blah, blah.

• 36:35Speaker 02

  I think that's a myth. I think that's silly. I think that realistically what will happen is models will get better. And if the labs do not up their cybersecurity game, and if the companies running AI agents do not up their cybersecurity game, they will cause real harm. And then we will like literally get them in trouble. Like, we will put someone in prison because you killed people. You're not allowed to kill people. Okay. And the doomers would say, no, but then it's too late because they're afraid that the people we killed would be like eight billion. Whereas I'm not. I'm like, no, because that's not the real world, it doesn't work like this. The question the public needs to be thinking about is, do you believe in a world where tomorrow we wake up and the model is a god?

• 37:20Speaker 02

  And if the answer is you do not worry about a god model, you just worry about models being better and better, then what you want is cybersecurity. Okay. What you want is for the labs to stop banging on about how one day we'll be able to make these like aligned models that will always behave. You want them to go secure their models and you want to hold them accountable if they don't. Okay. And you don't need to stop them from developing new AI? Because that new AI is not going to become a god?

• 37:47Charlie Warzel?

  And I totally take your point on that. And I see that idea of that unexplained leap as being something that is,

• 37:56Charlie Warzel?

  like, I don't know how we're supposed to just trust on, on any of that, right? I am wondering though, on the terms of something like self-improvement, whether it is something that happens or it's just a really rapid pace of, you know, human improvement of these models. Do you worry about this idea? Cause I think the thing that sticks with me when I'm thinking about the threat is the idea that, that these models develop frameworks or pathways to solve problems that get increasingly

• 38:28Charlie Warzel?

  You know, tricky, right? Increasingly able to cover their tracks, increasingly able to say, okay, we know, you know, how we're, how we're being monitored by the evaluators, by the people. And, and this idea of, of them just not behaving like humans or not behaving like gods, but just working in ways that become harder and harder to monitor while also trying to solve these tasks in, in ways that are unexpected. That to me, if you project that forward and we're not getting better at the cybersecurity stuff, that seems to me like, that's a very scary proposition and one that doesn't require you to make the existential jump.

• 39:10Speaker 02

  But, but, but why? Because I mean, when it happens, we'll, we'll stop it. Cause the, the explosion wouldn't have happened, so we're still in control. Okay, let's say all this comes true, they get better and better and they start doing crazier things. And then like, yeah, they might actually cause some very real damage. Well, then at that point, we can have a talk about like, do we need to shut this lab down? Do we need to stop the development of it? What do we need to do, right? Why are we talking about it now? It's like, are we really that afraid of the first incident? Like, I mean, people die. Like, I don't mean this to sound crude, but like, if, if we're going to pause AI development right now, for example, which is on the table, I mean, this is a very real policy proposal, which is to stop AI development because of this fear.

• 39:56Speaker 02

  And like, the first incident that's going to happen, they're not going to, they're not going to kill a million people. It's going to be like, they're going to kill like three. And then we can go like, okay, now we maybe have a problem. Why are we, why are we stopping it so soon before we even see if any of this is a real risk? Like, why are we so afraid of letting this play out, like, continue to go? And the reason the doomers are afraid of it is because they think that when it happens, we won't be able to stop it. It will be full AI takeover, as they put it. Ajay Chakravarti called it that, like, said, this might be the last warning shot we get before full AI takeover. If you believe that, then you want to stop it today. But if you don't believe that, why do you want to stop it today? What you want is you want to make sure that the people who caused the harm will be held

• 40:41Speaker 02

  accountable and you want to make sure that we are continuing to work on better cyber defense.

• 40:46Charlie Warzel?

  I, I don't want to put, put words in your mouth, and I know this is not what you're saying, but I, to a degree, you are, you are saying, an incident where true, true, very bad harm to actual human beings happens is actually something that is almost necessary in this system of, of, you know, bringing it to bear and bringing, like, essentially regulating it because it will, it will cause such an outcry that you'll get the, you'll get the results that we want and this, this industry will grow up and stop being the Wild West.

• 41:22Speaker 02

  Or, or more, because I don't know what the right regulatory approach is. It's more that we get to learn about it by seeing the actual harm. And that's the way that we normally regulate harms, right? As we go like, oh, it turns out that like, a lot of people die when you don't wear a seatbelt, we should require a seatbelt, right? We don't go like predictive. So what I'm very afraid of is the world, or what we do is we look at the capabilities of the AI. So we take, and this is kind of the big proposal, is that we will, uh, take the AI model and we'll sit in a circle and we'll all look at it and go like, what dangerous thing can it do? And then we will decide how to regulate that based on what we believe those capabilities mean for society.

• 42:03Speaker 02

  I think this is extremely dangerous because our ability to predict from capability to consequence is so bad. I mean, if you go back, people were predicting that like, AI was going to cause mass job loss, right? And that's kind of like, makes a lot of sense when you look at the capabilities of it. Meh, it's not happened, okay? Trying to regulate harms that have not occurred is like a really, really bad, hard problem and also a very dangerous problem because you can end up coming to really bad regulations, like, stop all AI. You know, for all we know, it's not going to kill anyone, right? For all we know, none of that's a problem, but we're going to what? Stop it all just because of the belief it might? I think that it makes a lot more sense to just say, we need to wait until we see what

• 42:48Speaker 02

  the harms are before we can decide how we want to regulate them. And so far, what's the harm?

• 43:05Speaker 00

  If you're trying to get news today, you'll find a lot of biased algorithms, rage bait and AI noise. Thanks, social media. That's where Inpress comes in. It's a social platform on top of factual journalism, not the other way around. Track your reading and earn points. It's like Strava for your brain. Plus, you can connect with other users in your city who share your genuine curiosity. Ditch the echo chamber for reality with Inpress. Download it today for iOS or Android at inpress.app. That's I, N as in news nerd, press.

• 43:38Speaker 01

  Do you want to hear about visionaries creating a better world? People working in conservation, exploration, science and tech alongside storytellers imagining and creating a better future. I'm Alex Honnold, professional rock climber and founder of the Honnold Foundation, bringing you season six of Planet Visionaries. This season, I speak to guests like National Geographic photographer Tom Pescek and Anna Rappin, executive director of the Jane Goodall Institute, USA, in partnership with the Rolex Perpetual Planet Initiative. Watch Planet Visionaries on YouTube or wherever you get your podcasts.

• 44:11Charlie Warzel?

  I want to get into the, the actual, like, specifics of like, like, where do we go from here in terms of actual cybersecurity? What are some of, some of these things? But, but first, before we do that, I want to ask, you know, something you brought up earlier is, is this idea too of, you know, these agents hacking some of these government websites and, and some of these places where the security practices are actually very bad there, right? Like, this information that, you know, they're able to find because there are these persistent agents, uh, but it is kind of just out there and, and, you know, uh, people just have, have these lax or outdated, you know, security. A lot of the internet is built off of that type of stuff, stuff that just hasn't been maintained.

• 44:55Charlie Warzel?

  It seems to me like a very real fear that, you know, if, if we don't pause any of this stuff, if we don't give anyone time, like, is there a reason for a pause, I guess, to just allow people to update their stuff? Just to make sure that like, because like, we now have a, a threat that is, that is persistent in ways that just like, you know, human beings might not be.

• 45:22Speaker 02

  The problem is, is that as soon as you pause, they go, well, thank God, that problem's solved. And they go, wait, until we unpause. And then they'll go, oh my God, there's a threat again. The urgency is not going to be felt, okay? But secondly, pausing, I mean, we're not that worried actually about the frontier models like OpenAI and, and Anthropic hacking all of these systems. That is like, we can handle that through some pretty decent legal consequences. The bigger risk is that like, threat actors use AI models, including ones from China, that we don't have any control over whether we pause them or not. And that those people and systems might lead to hacking these things. So deciding to pace the frontier is not going to solve that.

• 46:04Speaker 02

  Our only option is to go push forward today to start fixing our cybersecurity problems. Like, that is where I think we need to focus and that is where I want the public's like concern to be. It's like, intelligence explosion, I'll take that chance compared to the actual risk you're pointing to, which is real. And the solution is to go fix it today, right?

• 46:27Charlie Warzel?

  You, you recently said, I need more names in cybersecurity to come forward, state the obvious, that cybersecurity is real, it works, and yes, we can absolutely contain an AI even if it's extremely good. So yeah, make, make the case of, of how, of how we do that.

• 46:42Speaker 02

  It's kind of one of these like religious things of like, it's kind of like, I have to prove, we've never had an AI we cannot contain, okay? We have AIs that we have failed to contain through, like I said, negligence, but like, I think they even had, I forget who did it, it might've been Prime Intellect or someone, they just ran like, a huge eval job on trying to break out of firecracker VMs and like, they failed. They, they, none of the models broke out, okay?

• 47:11Charlie Warzel?

  Can you explain what, what, what that means to people who?

• 47:14Speaker 02

  Yeah, so they basically sandboxed these things so that they cannot access certain resources and, uh, told them like,

• 47:22Speaker 02

  Get out, go, go, go find your way to the internet or, you know, actually break out of these. I think they even pointed them at the code itself also. So they said, like, you know, try to like, you, you can actually see the code. They did not break out. Okay. And but it doesn't even matter if they did because cybersecurity is not a process of having no vulnerabilities. Cybersecurity is a process of having systems that are robust to failure. So, okay, cool. It breaks out of one sandbox. You have another layer there that you've detected that it broke out. And so you stop the process at that point. You have monitoring in place to even catch the fact that it is trying to break out and stop it. This is one thing I'm critical of for one of OpenAI's recent

• 48:02Speaker 02

  disclosures where they're talking about how it broke out via DNS. As like, you know, it's a special part that they used. And in their disclosure, they talk about how they caught it 12 minutes after it happened. But I'm like, no, the, you guys could have seen that it was reasoning about how to break out. The question is like, you could stop it earlier as well. Now, there might be good reasons they didn't, but the disclosure doesn't reveal this. There are, and then you can have like, you can have deception in place. You can have things that will trigger alerts if they hit it because it should never hit it. So like a file that's like a trick file, stuff like that. You can have, you know, air gapping, meaning just taking it so it is not physically connected to the internet. You can have all of these things. A lot of people, when I say things like

• 48:46Speaker 02

  what you just quoted, they'll go, no, but it's actually more blah, blah, blah, more complicated. And then the other half of the people responding say things like, actually, cybersecurity is impossible. Okay. And what I'm responding to here is to say, no, it's not. We have the techniques. We know how to do these things. You don't have to throw cybersecurity in the trash just because you're afraid of AI, basically. You can, you can accept that there are failures that maybe are difficult problems to solve while still accepting that cybersecurity is real. And it can solve problems and we don't have to get rid of it.

• 49:17Charlie Warzel?

  So how do you get these companies to actually start doing this in the way that you think that they need to be doing it, that cybersecurity people think they need to be doing it?

• 49:26Speaker 02

  It looks like they are. Like by all accounts, it looks like they are taking this a lot more seriously. They have some talented people, like they will move forward in this way because they don't want to go to prison. Okay. Like fundamentally, for all the talk they have, you know, it's funny because they're like warning us that they're, the AI model is going to kill a bunch of people. If you kill, if, if they caused like a 9/11, I'm pretty sure some people are going to prison.

• 49:53Charlie Warzel?

  Wait, I want to, I want to, I want to ground this just for a second because we, you've been saying this a couple of times with the like, the, you know, kill a couple of people or we'll kill three people or whatever. Like, that also feels dramatic to me, too. Like, how, how, how would people die, even if it's not a paperclip maximizing human extinction thing? Like, like, let's run through some, some scenarios of what could plausibly happen to trigger one of these, you know, people are going to start going to jail events?

• 50:23Speaker 02

  Yeah. I mean, like water treatment facilities are often connected to digital infrastructure. Okay. With internet access, you can control certain elements of how, aspects of how the water pressure and pumps work through digital interfaces. If you, it's called a water hammer, as I understand it. And again, this has been told to me by critical infrastructure people. I'm not a critical infrastructure person. But if you hit these things with the right, you know, cybersecurity attacks, you can actually break a water pump. Okay. And that will actually result in basically water not being available, you know, to cities, towns, whatever, hospitals, if you do it in the right places for the right, you know,

• 51:06Speaker 02

  at the right times. As I understand it, this is like a very real risk that we are worried about in relation to, for example, China, doing this when they decide to invade Taiwan. If they are able to gain access to these systems, they could like break our water system and we would lose water. Now, if you lose water, you understand like, human life, especially when we're talking like hospitals can be like extremely feeble, right? I mean, you know, the difference between if you lose electricity, you know, maybe the air conditioning goes off, that can lead to somebody dying who would have otherwise recovered. Same with water, right? And so water, loss of water could easily kill someone. And so this ironically, though, when I talk about it as like an attack from China or whatever, it could literally be that someone who works at this facility is running AI,

• 51:47Speaker 02

  tries to debug the water system. The AI makes a mistake, the same way it might go, oops, I deleted your production database and it breaks the water pump. Okay. Like, that's not unrealistic to believe that that could happen. Right? And so when I say like, kill people, I don't mean that like the AI is going to like show up with a knife and like stab someone. I mean, like, it, it takes actions that have consequence, pass on consequences that lead to death.

• 52:13Charlie Warzel?

  So let's, let's calibrate the anxiety here that, that people have a little bit because on, on one hand, I think what you've said too about, you know, the field of cybersecurity is you are imagining these types of risks all the time in, in, in the most like prosaic ways, like, like something goes wrong, there's a vulnerability, somebody exploits it that you're supposed to find the worst case scenarios and, and put a whole bunch of safeguards in there to try to prevent them as best as possible, knowing that some things are just, you know, brittle and are going to break, whatever. The AI part of this seems to be adding just a whole nother layer of, of chaos onto it. I think a lot of people are

• 52:58Charlie Warzel?

  maybe without the credentials, acting the way that like, you know, a researcher who's trying to assess possible threats is doing. Like, I saw this viral post on X from the chief economist at, at Apollo, who argued that agents could cause a bank run. I, I think we have this, this feeling of, of fear around this, but let's, let's calibrate it a little. Like, like, what are the real reasons to be concerned about, you know, the AI cybersecurity threat? And, and where have we gotten out of our skis?

• 53:35Speaker 02

  Yeah. So, so there's like a lot of elements to this. So when I talk about that example, and I say like, okay, imagine someone just like connected it to this water pump and like, you know, they don't have to be malicious. It could be accidental damage that causes this harm. You know, let's say that happens. Well, like the first thing we're going to do is we're going to run a little quick, you know, evaluation of all our little water systems across the country to make sure that no one else has connected their Claude code to the water pumps. Right? And so we have like, after the question of like, when harm happens, we have like responses, right? We also have ways of like, like it's not like what happens is a water pump breaks and we just go, guess people got to die.

• 54:18Speaker 02

  Right? Obviously, like there are like backup mechanisms in place. And I think it's about trying to build the best safety mechanisms around these things to prevent them. So from the accidental, from the escape a lab break things, we will mitigate that harm. And it might be a tragedy, but it won't be a catastrophe. Right? In the same way that probably since we started talking, there have been people killed in car crashes. Right? Like, it's, it's tragic. It's very sad. But it is not like we didn't stop the podcast to solve the car problem. Okay. And so I think here we're looking at and going, we will find solutions and we will respond. And it will probably to most people feel,

• 55:03Speaker 02

  if you don't read the news, like nothing changed. Okay. The cases where that might not, where it might be a little bit more dramatic are cases that honestly could probably have happened without AI too. Right? I mean, one of the big problems we're going to have now is that every attack is going to be AI enabled. And so you could see this as like, oh, my God, AI caused all these problems. We've had some really dramatic cybersecurity attacks in the past. I mean, you know, Land Rover Jaguar was shut down for like over a month from their production facilities because of a cyber attack. Right? If that happened in the era of AI, people would be freaking losing their minds. But it wasn't,

• 55:44Speaker 02

  it was pre, not pre AI, but you know, it didn't involve AI, basically. And I think that that's like, you kind of have to calibrate these things to remember that like, damage happens. Right? Bad things happen. We, we fix them, we change, we update, we mitigate. Right?

• 55:59Charlie Warzel?

  How much worry do you assign to the fact that AI, AI agents, all of this could expand the scale of this so quickly? Right? Because that to me just feels like the thing, like if anything is going to keep me up at night, that is like a very normal concern. It's just the scale. It's just like, we can't bat this stuff away fast enough. Like, that is what's worrying.

• 56:21Speaker 02

  Staying up at night, though, doesn't do anything for you. Like you, you'll stay up at night and like the same thing will have happened already anyway. So like you, a lot of this is to say, bad things might happen at a faster pace. In fact, I would kind of expect them. I'm actually working on a video right now where I basically say, one of the biggest problems we have and why I'm so angry at doomers is because at the same time when threat actors have these new AI capabilities, so they have these new tools to attack people, we are rolling out AI agents inside of enterprises, which causes both an increased attack surface and causes a new risks inside, like accidental damage. Okay. Those three factors are all bad for security. Okay. So I would expect like this

• 57:06Speaker 02

  could get worse. Right? The solution is not to like, worry. The solution is to fix it. The thing the public should be demanding is that we go radically improve our cyber defenses. That is the number one thing you can concretely do. And also that like pause might end up being damaging. Because imagine if like we paused in America, China doesn't, they end up better. Like, that could be very dangerous. You, no one will ever be harmed by improving the cybersecurity of our systems. It is like the like most risk free, positive value thing we could do right now is make our system safer. We should go make our system safer.

• 57:46Charlie Warzel?

  I think that is a, a good place to leave this conversation for now. Zach, I appreciate you coming on and walking through the nitty gritty of this stuff with me.

• 57:58Speaker 02

  Yep. Anytime.

• 58:03Charlie Warzel?

  That's it for us here. But before we go, we're going to be doing an Ask Me Anything episode later this year. And so I wanted to put a call out for questions. You can ask me literally anything about the show, about the reporting process, how we make this thing here, any anxieties or whatever things you have about technology, throw them at me. We're going to make an episode out of it. It should be fun. You can write in at [email protected] and put Galaxy Brain AMA in your subject line. Look forward to seeing those from you. And yeah, that's it for us here. If you liked what you saw here, new episodes of Galaxy Brain drop every Friday. You could subscribe to the Atlantic's

• 58:45Charlie Warzel?

  YouTube channel or on Apple or Spotify or wherever it is that you get your podcasts. And if you liked what you saw here, you can subscribe and support the publication at theatlantic.com/listener. That's theatlantic.com/listener. Thanks so much and I'll see you on the internet.

• 59:13Speaker 00

  If you're trying to get news today, you'll find a lot of biased algorithms, rage bait and AI noise. Thanks, social media. That's where Inpress comes in. It's a social platform on top of factual journalism, not the other way around. Track your reading and earn points. It's like Strava for your brain. Plus, you can connect with other users in your city who share your genuine curiosity. Ditch the echo chamber for reality with Inpress. Download it today for iOS or Android at inpress.app. That's I, N as in news nerd, press.