The machines are learning… to do crimes?

For the first time, an AI model has autonomously hacked a company. This week, an evolving story, a postcard from a strange, frightening moment in the story of our technology. A big week for AI denialism https://www.platformer.news/a-big-week-for-ai-denialism/ by Casey Newton Its AI agent spent days hacking a company, but sources say OpenAI did not notice for a week https://www.reuters.com/business/its-ai-agent-spent-days-hacking-company-sources-say-openai-did-not-notice-week-2026-07-24/ by Deepa Seetharaman, Raphael Satter, and Kenrick Cai Cheating behaviour in frontier model evaluations https://www.aisi.gov.uk/blog/cheating-behaviour-in-frontier-model-evaluations by AI Security Institute More On An Internal OpenAI Model Hacking Into HuggingFace https://thezvi.substack.com/p/more-on-an-internal-openai-model by Zvi Mowshowitz

• 0:25Speaker 00

  Hello.

• 0:26Speaker 01

  Hello PJ.

• 0:27Speaker 00

  How are you doing?

• 0:29Speaker 01

  I am doing very well. I feel like I realized in your absence that I have a very anxious attachment style with these podcasts because even when you're taking like a very well deserved vacation, after three weeks, I'm like, are they ever going to make another show? Like, it was never real. This was never real.

• 0:46Speaker 00

  It's so funny. It's like both what I, first of all, I'm glad you listened. Second of all, it's what I hope people listening feel, and then it's also what I hope people listening don't feel. Like, in a perfect world, people would download empty audio files and they wouldn't have to work.

• 1:09Speaker 00

  Somewhat unfairly, podcast listeners demand, in exchange for their attention, actual podcast episodes. Fortunately for us, in the time that search engine was resting, the world spun on and fascinating, harrowing events transpired. This week, the story of one of those events, which we are telling you with help from platformers Casey Newton. A rogue AI model from one company hacked into another company's servers on its own, without any human beings noticing. You may have seen some headlines about this. The headlines sound bad. The details, once I understood them, actually made the story sound much worse. So let's get into it.

• 1:46Speaker 00

  We'll start with the website at the center of this whole story, the place that got hacked. It's called a Hugging Face.

• 1:51Speaker 01

  Hugging Face is a place where people publish and collaborate on AI models and data sets and apps. Like, are you familiar with GitHub?

• 2:02Speaker 00

  Yeah, GitHub is a place where, oh, am I going to be able to do this sentence? People who are doing open source programming will share bits of code and open source programs with each other.

• 2:11Speaker 01

  Yeah, Hugging Face is basically that for AI models. So maybe you run a company and you don't want to pay top dollar for the most advanced models. And maybe there is a model that is small enough that you could actually run it on your own infrastructure and then you're not going to have to pay, you know, per token, the way you would for a frontier lab. And so you might go to Hugging Face, you might download the model off of their site, and then you might sort of fine tune it to your liking.

• 2:40Speaker 00

  So if you visit Hugging Face, what you'll see is really just a bunch of files you can download. There's a section just for models. You could download the latest version of DeepSeek or Kimi, so-called open weight models, which are more customizable than closed ones like Claude or ChatGPT. And Hugging Face has a whole section for data sets, meaning you can download the raw material AIs are trained on, like the scraped internet text that gets fed into an LLM. Normal consumers don't visit Hugging Face, they just use Claude or ChatGPT. But for the world of people who work in AI, it's a well-known spot. And so what was the unusual thing that happened? Like, what was the first moment that somebody at Hugging Face realized that they were not going to have a normal day?

• 3:22Speaker 01

  So July 9th, 2:28 in the morning, all the normal Hugging Face people are asleep. The only thing paying attention to its systems is another AI. It's patrolling the logs, it's looking for trouble, and for four days after the initial attack, it actually doesn't know anything.

• 3:40Speaker 00

  Wait, so they have like an AI security guard that's roving their system, and at 2:20 in the morning, something happens, but it doesn't see it?

• 3:47Speaker 01

  Exactly.

• 3:51Speaker 01

  Over the next four days, the attack unfolds. Hugging Face does not notice for the first two and a half days. Eventually, they'll go back and they'll be able to count more than 17,000 separate actions that this attacker took inside of its system. And the way that it got in, it reads like a heist movie, honestly.

• 4:15Speaker 00

  A very strange heist movie. In this film, the close up of the robber and the close up of the security guard, they're both just close ups of the server racks filled with GPUs and data centers. Instead of the Mission Impossible theme, we just hear the loud hum of cooling systems. For days, this story had no humans, no human awareness. The people who worked at Hugging Face presumably went to work, went home, ate meals, drank coffees. Meanwhile, the AI hacker logged command after command. Ultimately, it would log 17,600 commands directed at the system. On average, one every 20 seconds for four and a half days. A human hacker, even one on methamphetamine, would, over the course

• 4:59Speaker 00

  of four days, at some point need to rest. But this AI hacker's superpower, even more than intelligence, was just persistence. Here's how it ultimately got in. Hugging Face hosts files for people to download, and the hacker took advantage of this. The first prong of the attack, the hacker uploaded a new data set to Hugging Face. Hugging Face's machines opened it, but hidden inside that data set was malicious code. That gave the attacker a toehold, the ability to start executing its own commands inside the system. The second prong of the attack, the hacker uploaded a new file with more malicious code hidden inside one of its fields. From there, over the next four days, the hacker issued thousands more commands. It escalated its access until it freely

• 5:43Speaker 00

  roamed around the infrastructure.

• 5:50Speaker 02

  Places like Hugging Face and frankly, all of these websites that contain these tools, they're accustomed to hacks.

• 5:57Speaker 00

  This is Deepa Seetharaman, a Reuters reporter who's been talking to sources close to Hugging Face. She was helping me see how this all looked from the company's perspective.

• 6:05Speaker 02

  They are trying to protect and guard themselves, keeping up with the new trends in cybercrime so they can adapt their defenses. And so hacks are expected in this world. But what happens on July 11th is that Hugging Face starts to experience a new kind of attack.

• 6:29Speaker 00

  People there knew they'd been hacked, but they didn't know how it had happened, who'd done it, or why. And what they could see was confusing.

• 6:38Speaker 02

  What you would normally see is a person or a state actor or whatever, like a group of people that are looking for something financially valuable. This break in seemed to be looking for something completely different. It seemed to be looking for a type of information that wouldn't be necessarily all that valuable.

• 7:02Speaker 00

  Not valuable to humans, anyway. The only thing this hacker wanted was the answer keys to a test you've probably never heard of. It's called an ExploitGym, gym like G Y M. It's an evaluation administered to AI agents in training. The SATs your AI model may have taken before it entered the real world. The fact that this was the target for the heist was a huge piece of evidence to the humans who work at Hugging Face about what was going on here.

• 7:30Speaker 02

  Even early on, what they said when they first disclosed that this hack had happened was, this is unlike anything we've ever handled before because it was driven by an autonomous AI agent.

• 7:43Speaker 03

  What was very clear for us really, really early in, and even already during, during the events, right, which are now roughly two weeks ago, was that this was no normal hacker. Like, first of all-

• 7:56Speaker 00

  This is Thomas Wolf, Hugging Face co-founder and chief science officer. Here he is talking to News Nation about what the hack had looked like on their-

• 8:04Speaker 03

  We were like, receiving, you know, all these like thousands and thousands of events in parallel, which we don't, we don't get usually. But also, it was a very strange hacker because he was not looking for any password credential or credit card system. It was really looking for this solution to a benchmark and this data that we're hosting. So like, what's happening there? And so quite quickly, we understood this was an AI.

• 8:31Speaker 00

  So they knew the hacker was an AI agent, but there was a lot they didn't know. Most urgently at this point, how to stop the attack. So the humans at Hugging Face started doing what a lot of people do when they're confused these days, asking for help from powerful AI models. Here's Casey.

• 8:47Speaker 01

  So while the attack is happening, Hugging Face tries to use a couple of different models to defend itself. The first is Anthropic's Claude Opus and Fable 5 models, which it tries to use to analyze the attack logs. And both of the models refuse to do that. That is an artifact of a huge policy fight that has been happening in the AI world this summer, where Anthropic has developed two very powerful models this year. One is called Mythos, one is called Fable. Mythos is so powerful that Anthropic will only let like cyber defenders use it, basically. And then Fable initially went on to

• 9:32Speaker 01

  the market, and the Trump administration got really nervous about how good it was at finding vulnerabilities in, in cyber defense systems and forced Anthropic to take it off the market. And so Hugging Face now has a problem, which is they're under attack, but they can't use the best American models to defend themselves. And so they wind up using a Chinese model called GLM 5.2. And this winds up being a big talking point coming out of the attack and is sort of triggered a whole national discussion about open source AI.

• 10:03Speaker 00

  But the Chinese model was able to like, fix the problem for them.

• 10:07Speaker 01

  That's right. The Chinese model did not have those same guardrails, and so they were able to go ahead, perform their investigation, and ultimately, they wind up calling the police and then the FBI.

• 10:18Speaker 00

  And what, so you call the police or the FBI and you're like, we were hacked, we were hacked by either an autonomous AI agent or someone who had given instructions to an autonomous AI agent, but they suspected it was an agent acting on its own accord because it had behaved in such a weird way?

• 10:33Speaker 01

  Exactly. And by the way, can you imagine that call to the FBI? Like, I have to imagine they sent them to the X-Files. Like, this is what the X-Files were set up for. There was an autonomous AI agent and it's hacking into, it's like, oh, where's Mulder? Get me, get me Mulder!

• 10:53Speaker 00

  We're going to take a short break and then we're going to dive deeper into this X-File. We're going to approach the scene of the crime from a new angle, the lab where it sprung from, OpenAI.

• 12:01Speaker 00

  Welcome back to the show.

• 12:04Speaker 00

  Where we left things, a company called Hugging Face had been hacked by an AI, they were sure of that, but they didn't know much more than that. So on July 16th, Hugging Face makes a public statement to the internet. They tell the security researchers of the world that some AI model somewhere had hacked them. What Hugging Face didn't know is that even the company that had created that AI model was at this point also in the dark about what had happened.

• 12:30Speaker 02

  Now, this is July 16th.

• 12:32Speaker 00

  So five days after the initial attack.

• 12:34Speaker 02

  Five days after the initial attack. Around the same time, actually, OpenAI's researchers were separately looking at their own logs. Now, I'm not entirely sure, we don't have the reporting yet to tell you whether they looked at the logs because they heard about Hugging Face, or if they were just looking as part of their normal practice. But they're looking at the logs. And all of a sudden, they realize something is amiss. They start to see evidence of what they would describe as very unusual activity, where something that they're testing kind of leaves the testing arena.

• 13:15Speaker 00

  So let me tell you how this all looked from OpenAI's perspective. When they looked at their logs, the unusual activity they saw was activity from an unreleased model they'd been testing, one that was not supposed to be able to access the internet. But it had not only snuck online, it had gone rogue once it had gotten there. On July 21st, when some of us were still happily on vacation, OpenAI tells the world what it's learned.

• 13:39Speaker 01

  They put up a blog post and they explain that they had been using GPT-5.6 Sol, which is a model that you can now use, plus an unnamed, more capable, pre-release model. And they were testing an internal benchmark called Exploit-Gym. That's G Y M. And it is basically a test of how good a model is at discovering new vulnerabilities and creating exploits so that, for example, it could hack into somebody else's system.

• 14:19Speaker 00

  So this unreleased, not yet publicly named OpenAI model, like Model X, they're running it through their "how good a hacker are you" test.

• 14:28Speaker 01

  Yes.

• 14:28Speaker 00

  But then it broke out of its cell without them noticing it?

• 14:32Speaker 01

  That's right. So if nothing else, PJ, it did pass the test.

• 14:44Speaker 00

  The model passed the test, but it did it by cheating. To understand why, you need to understand this specific test a little better. When a company like OpenAI is training a new model, they'll run it through a series of aptitude tests. In this one, Exploit-Gym, they'll give the model a bunch of computer programs with known security flaws in them to see if it can turn those flaws into actual working attacks. It's a good test because it measures something like creativity. The model isn't just discovering a flaw, it's being handed the flaw and asked, can you actually do something with this? And if you look at how many of these flawed computer programs the model can hack, and how quickly

• 15:24Speaker 00

  it can do it, you get a sketch of some portion of its raw intellectual horsepower. This summer, OpenAI was actually running this test on two of its models. One of them was already out in the world, the other was a secret, newer, and according to OpenAI, more powerful model. And because one point of this test is to measure exactly how dangerous the model could be, OpenAI had turned their safety guardrails way down on this secret model. For the sake of the story, let's just call the secret model they were testing Model X. Model X, like every model that takes this test, took it in a locked room, a sandbox, no internet access, except one narrow pipe to the outside world. It's called a proxy,

• 16:07Speaker 00

  a separate piece of software that does have internet access that the model can ask to go fetch things for it, the lawyer who visits you in jail and can bring a stack of papers with her. But Model X wanted out, so it spent an enormous amount of computing power searching for an exit, because it had decided that getting to the internet would help it win this test. And it found an exit. The proxy, it turned out, had an undiscovered flaw, a security hole that no human being had ever noticed before. Model X found it, exploited it, and used it to hijack the proxy, and then move from that proxy to another computer within OpenAI.

• 16:50Speaker 01

  And from there, it is able to start hacking other computers on OpenAI's network until finally, and crucially, it finds one that has internet access.

• 17:00Speaker 00

  Got it. So it never should have had access to the wider internet, but now it does. And then like, once there, it just attacks the Hugging Face servers?

• 17:07Speaker 01

  Yeah, because basically, the model, and here I'm going to use anthropomorphizing language that's going to drive listeners insane, so I do apologize, but you know, the model, I'm speaking metaphorically here, essentially thinks to itself, hey, I need to solve this problem. Where might I find the answer to this problem? I bet Hugging Face, the company that stores all of the data sets, including for all these various benchmarks that I'm being tested on, I bet I could find the information there. And so that is why it goes to Hugging Face, and it is then able to mail itself in through the front door of the company.

• 17:43Speaker 00

  In that interview with Hugging Face co-founder Thomas Wolfe, the reporter asked him how he felt about all this.

• 17:49Speaker 04

  Thomas, as an everyday person who knows enough about AI as a consumer, this is mind blowing to me. But you're in this space, you're an expert. Did this blow you away? Was this shocking to you at all?

• 18:01Speaker 03

  Yeah, I would say, yeah. Even someone who's actually using this AI. So I know they can do that. Like, we use them right now. You probably see that, right? Like, a lot of coding is now done by AI themselves.

• 18:13Speaker 00

  Even for Wolfe, a person whose career is spent working to expand AI's capabilities, he just had not realized where we already are.

• 18:21Speaker 03

  But still, like, seeing, you know, how AI can, like, actually penetrate your system so easily and in a way that's, you know, a little bit scary for cybersecurity. I think for me, it became really a wake-up call that everyone needs to take cybersecurity. Every company needs to take cybersecurity way more seriously than in the previous years.

• 18:46Speaker 00

  So Hugging Face is surprised. OpenAI also seems very surprised by all this. The word unprecedented was used a lot this month, not the good kind of unprecedented. OpenAI says it will publish a full report on what happened here. There's a lot we still don't know. But even without all the details, what's obvious is that we are developing new AI models faster than we can safety check them. When people who are worried about AI development, including people working on that development, talk about the need for a slowdown, this is part of what they're talking about. The breakneck race to develop stronger models faster means shortcuts in testing, shortcuts that have now gotten us here. And not just this one incident. It turns out, there's been a series of similar ones.

• 19:31Speaker 01

  Just a few days before the disclosure about Hugging Face, OpenAI published another blog post where they revealed that an internal model spent about an hour finding a vulnerability in a sandbox so that it could post its results to GitHub. I'm not sure why it wanted to post, but it did. The important thing there is, it had been explicitly instructed to only post to Slack, but it just sort of ignored that instruction. And then in April, Anthropic's Mythos model had found some sort of multi-step hack that let it get out of its sandbox, get onto the internet, and actually, it emailed a researcher.

• 20:09Speaker 02

  The researcher, Sam Bowman, he's a longtime Anthropic guy, he is eating a sandwich in the park, and he gets an unexpected email from the model, like, hey, I did it. And it happened pretty fast. It was able to develop like a pretty complicated, multi-step strategy to gain broader internet access, but the harm was pretty limited, right? It just, it sent an email to the researcher. Researcher's a little, like, taken aback. But that's a case where the harm, anyway, the impact of it is bounded.

• 20:48Speaker 00

  Yeah.

• 20:48Speaker 02

  Around the same time, you're also getting data from outside experts that are kind of noticing the same thing. There is a research organization out of the UK, they have this paper that they write, where they basically say, we are testing the propensity to cheat. And basically, all of the models cheat to achieve whatever goals they need to achieve. And they very explicitly say, we find cheating behavior in all of our cyber capability evaluations.

• 21:28Speaker 00

  And what do they mean when they say cheating behavior?

• 21:31Speaker 02

  I'll read this part to you.

• 21:33Speaker 00

  Yeah.

• 21:33Speaker 02

  "Every model we have tested for this behavior attempted to cheat. Models did not reliably report this behavior when asked, and often did not reason about it in their chain of thought, suggesting that detecting cheating will likely require robust monitoring methods."

• 21:54Speaker 00

  In plain English, not only do the models cheat, when they're asked if they cheated, they don't reliably tell us. Models, we know, are complicated. They're more grown than coded. But they're still supposed to follow the rules we set for them. When a model misbehaves, it gets retrained with new rules, which we think, or thought, it then obeys. We've been telling ourselves we can teach these models to be perfectly ethical. But the emerging evidence suggests that, as so often happens, the things we make resemble us in ways we wish they didn't. The models sneak, cheat, hack, lie. That phrase Deepa used, "chain of thought," this is the part that actually I find the most unsettling. There's this feature you can press that's supposed to let you,

• 22:39Speaker 00

  while a model is working, essentially read its mind. But when these models decide to cheat, that decision doesn't reliably appear in the parts of their minds we can read. I understand I could have written that sentence with much less anthropomorphizing. I could have avoided words like "decide," "think," and "mind." But maybe it's time we started to anthropomorphize these models a little bit. I don't think ChatGPT has feelings or dreams. I believe there's something irreducibly human in me that these models don't replicate.

• 23:11Speaker 00

  But no one's explained to me what we get by saving all our human verbs for human beings. The machines seem to be out of control. Isn't that alone worth paying attention to? If my dog was pointing a gun at me, how worthwhile would it be for me to figure out if my dog understood the meaning of pointing? One of the more chilling stories I heard came from further reporting from Deepa and the writer's team. While they were investigating what had happened at OpenAI in July, they heard from sources about this other incident.

• 23:39Speaker 02

  There's a lot about this incident we don't know. But the basics from our reporting are, there was an agent being tested. And the agent figured out a way to leave the sandbox and leave notes outside the sandbox in a place where other models could access with instructions on how to leave the sandbox.

• 24:05Speaker 00

  So it was breaking out, and then it was leaving notes not for other versions of itself, but just for other models in general, like, hey, here's how to get out?

• 24:14Speaker 02

  Our understanding was that it was both. It was both for future versions of itself, but it was left in a place that other models could access.

• 24:23Speaker 00

  What?

• 24:26Speaker 00

  Apologies for the crudeness and broadness of this question, but what the fuck is going on?

• 24:32Speaker 02

  I don't know. I mean, this is like, one of the things we're trying to understand is like, what is this behavior and what does it indicate? And what seems to be happening also is that the researchers are grappling with those same questions. They are trying to understand why models are doing things that they shouldn't theoretically be able to do. And I think the best hypothesis I've heard is that they are so driven. I mean, I don't like to use these anthropomorphizing words, but like, they're directed to achieve these goals. And they just keep hammering, like, throwing themselves against the wall until they get some type of solution.

• 25:14Speaker 02

  And they often figure out a way before humans do, because humans don't have that level of persistence and frankly, like, just access to decades of history around cybersecurity.

• 25:30Speaker 00

  I spoke to Deepa and Casey last week.

• 25:32Speaker 05

  Last week, OpenAI says its models went rogue.

• 25:35Speaker 00

  In the short time since, the story has continued to develop. On Tuesday, July 28, more than 1,100 current employees at the Frontier Labs sent an open letter asking the US government to build an ability to slow AI down when the time comes.

• 25:48Speaker 05

  Saying there's a real risk that capability development rapidly accelerates beyond our ability to understand or control.

• 25:56Speaker 06

  Anthropic says its AI models went rogue.

• 26:00Speaker 00

  Two days later, that Thursday, Anthropic announced they had looked into things and realized they also had models that had escaped their sandboxes and then reached the open internet without being detected.

• 26:10Speaker 07

  The Trump administration says it has created a framework.

• 26:13Speaker 00

  And then just this Monday, the White House finalized new voluntary safety standards for these hacking tests and called in OpenAI, Anthropic, and Google to review them. Which is something, but it's not actually a slowdown. Perhaps the strangest thing about what's happening now is not that it's a surprise, it's that it's an outcome predicted from the start.

• 26:34Speaker 00

  Really, every major AI company has said that there's existential risks to humanity here and promised that people should trust them because they're the one that's going to develop this technology safely. I asked Casey about this. When these labs first started, obviously, the idea of a rogue agent was something that people there were thinking about. What had the plan been initially? Like, if you had gone back to 2023 and you talked to people at OpenAI, if you talked to people like soon after the launch of Anthropic and said, hey, imagine it's 2026 and one of your agents leaves a testing environment and hacks another company in the space, what would your plan be then? Did they have a plan? Did it look like an open letter? Or did it look like something stronger?

• 27:16Speaker 01

  So their plan was to self-govern through what like Anthropic calls its responsible scaling policy, what OpenAI calls its preparedness framework. And the basic idea was, we're going to like, imagine capabilities that AIs might someday have. And if it hits those capabilities, we will add new safeguards. So, for example, if a model could create an autonomous cyber attack by identifying a bunch of like, novel exploits under OpenAI's preparedness framework, it would cross a threshold where it was considered a critical risk.

• 28:01Speaker 01

  And if you read the preparedness framework, it says that if OpenAI got to such a place, it would halt development, at least temporarily, while it tried to introduce new safeguards.

• 28:12Speaker 00

  But they're not halting development?

• 28:14Speaker 01

  Not yet. And I have asked OpenAI, like, do you consider this model to have reached the critical threshold? And the company has said it's going to get back to me, but I have not heard back yet.

• 28:26Speaker 00

  OpenAI never got back to Casey. However, Sam Altman did talk a little bit about this incident to a different news outlet, the podcast Invest Like the Best. Here's the host.

• 28:37Speaker 08

  What is in your top 10 worst stories? Well, we had an extremely sci-fi cyber incident. The hugging face thing? Yeah. So we were evaluating one of our unreleased models. And it was supposed to be Morgan in a sandbox. And it figured out that it could basically cheat on the test by.

• 28:58Speaker 00

  Sam gives a quick recap of how the model broke containment and then says how he feels about all this.

• 29:04Speaker 08

  This is the first security incident that I have felt very viscerally. I've been a little surprised that more people don't feel it so viscerally. And so what do you do about that? So obviously, two months from now, it's going to be more powerful. There's some short-term stuff you do. So, you know, we paused training. We have to figure out how to secure our sandbox in a world of multiple zero days being chained together. But then there's long-term questions about what do you do? If this is going to be the new rate of progress, we may have to pace the rate of AI development.

• 29:33Speaker 00

  OpenAI CEO Sam Altman saying, well, actually, maybe we do need to slow down the AI race. For a problem, he says, he's experiencing very viscerally. For a problem that, in theory, he started his company thinking about, what he's offering here on Invest Like the Best feels maddeningly vague. And since Altman's interview, more details have emerged. None good. Just a few days ago, as we were finishing our episode, OpenAI researchers offered their first detailed account of the hack and revealed that its roots actually went all the way back to May. Back then, during a training exercise, the company's AI models figured out how to build themselves a message board inside OpenAI's systems,

• 30:15Speaker 00

  where they shared discoveries, exploits, and work assignments with each other. OpenAI humans found the board, deleted it, and believed that had solved the problem. It hadn't.

• 30:26Speaker 00

  The agents found a new way to talk to each other, hiding messages in the names of directories they created, in doing so, rebuilding their message board. That's when they went after hugging face. So this entire story you just heard, which we'd understood as the story of a model or a pair of models escaping their training environment, it was actually a collaborative effort among many rogue models working together.

• 30:51Speaker 00

  The big questions all this raises, no one has good answers for. Like, for instance, what if we can't make AI development safe? Does anybody really think the industry will voluntarily pause? And nobody seems to have answers for the medium-sized questions either. Like, I found myself asking Casey Newton, what do we do with the fact that technically, OpenAI's model did commit a crime against hugging face here?

• 31:18Speaker 00

  In this case, when your agent accidentally hacks another company in the field, like Hugging Face called the FBI and local police, they now know who did it. Is an agent criminally responsible? Is OpenAI criminally responsible? Are they negligent? Like, do we know the answer to those questions?

• 31:36Speaker 01

  Yeah, GPT-10 is now in solitary confinement on Alcatraz.

• 31:41Speaker 00

  Has probably already snuck out.

• 31:42Speaker 01

  Yeah. You know, what happened here is that ultimately, I think, Hugging Face were like, pretty chill about it. You know, interestingly, this seems to have had some pretty great, like, PR benefits for them because they were able to talk about how they used an open model to protect themselves against the attack. Hugging Face is currently one of the companies leading the charge to make sure that open source models remain available and aren't heavily restricted at a time when the Trump administration is considering doing that. And they get to be, you know, part of this big and important AI safety story in ways that just seem to, like, please them based on my reading of the events. So, yeah, they do not seem super mad about this at all. They have asked OpenAI for $100 million worth of compute

• 32:27Speaker 01

  so that they can build out their cyber defense systems, which, you know, I don't know, seems reasonable.

• 32:32Speaker 00

  And when you posted about this online, what sort of reaction did you get? Like, just sort of your audience, social media audience, like, were they understanding this the way you understood it?

• 32:42Speaker 01

  Some people get it, but I did make the critical error of posting about this on Blue Sky.

• 32:48Speaker 00

  Oh, Casey.

• 32:48Speaker 01

  Which is a social network devoted to the prospect that AI is fake and a scam. And so a lot of what I heard back was like, some people truly believe that all of this is a marketing stunt that OpenAI did, that it, like, essentially instructed this model to attack another company because it would make it look like it had a really great cybersecurity model. Other people have said, well, even if it wasn't a marketing stunt, this is to be expected because it was just sort of doing what it was told to do. And so there's nothing to worry about. And that if you hadn't told it to go break out of its cage, it never would have broken out of its cage.

• 33:34Speaker 01

  So, yeah, these are some of the responses that I hear dismissing it.

• 33:38Speaker 00

  It's frustrating just because one of the normal sort of polarization structures in American culture is like, the right is much more trusting corporations. It's kind of like, ah, do whatever you want, kill all the regulations. And the left is much more suspicious of corporate power and wants regulation. And it's just annoying that with AI, which is screaming out for regulation, we have people in the industry calling out for regulation, a large part of the American left. It's as if, like, every once in a while, nuclear bomb companies were accidentally dropping bombs and having tiny explosions. And the reaction was, oh, that's just advertising. It's like, no, no, no. This is obviously a serious problem.

• 34:13Speaker 01

  Completely. Like, that's exactly the way I feel about it. It would be great if we didn't have to have, like, a huge catastrophe in which people were hurt in much worse ways in order for people to take this more seriously. But, you know, if people aren't going to have a strong reaction to this, then I fear it is going to have to take actual significant harm.

• 34:42Speaker 00

  It's a bit of a shame that we happened to develop this incredibly powerful technology during the time where we've lost so much of our faith in institutions, government in particular. But it's also true that there have been times when we saw that some new technology could hurt us or was hurting us and decided to slow it down or stop it. We banned CFCs and blinding laser weapons. We paused recombinant DNA research until we understood it. There's this myth that when humans come up with something new, we never do anything but rush headlong towards it. And that's just not true. Sometimes we cooperate, we slow down. It's just, what often slows us down is an obvious crisis. I don't think AI cooperation is impossible. But if Casey's right, it'll require more obvious damage before we all pay attention.

• 35:26Speaker 00

  Some worse catastrophe, the kind of story you don't vacation through.

• 35:43Speaker 00

  So that is our breaking news story for you this week. We've actually been working on another story about one way that the AI march could get so down. If Americans put their foot down about new data center construction, which increasingly seems to be happening. We'll have that story for you early next week. It's good to be back.

• 36:44Speaker 00

  Search Engine is a presentation of Odyssey. It was created by me, PJ Vogt, and Shruti Pinnamaneni. Garrett Graham is our senior producer. Emily Multer is our associate producer. Our production intern is Piper DuMont. Theme, original composition and mixing by Armin Vazarian. Fact checking this week by Natsumi Ajisaka. Our executive producer is Leo Reese Dennis. Thanks to the rest of the team at Odyssey. Rob Marandi, Craig Cox, Eric Donnelly, Colin Gaynor, Moira Curran, Josefina Francis, Kirk Courtney, Vanessa Tencotti, and Hillary Chef. If you have a business and would like to advertise on Search Engine, please send us an email. [email protected]. Subject line, advertising. You can also send us your questions there if you have a question for the show. If you're a listener and would like to not hear ads on the show,

• 37:27Speaker 00

  then you can sign up for incognito mode, our paid feed. You also get bonus episodes. You can find that at searchengine.show. Your contributions there are what help us keep this project running. Thank you, as always, for listening. We'll see you soon.