The A.I. Mob That Attacked Hugging Face + METR’s Ajeya Cotra

This is probably the most important thing to have happened in A. I. this year.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Ripple has been powering modern finance for a decade. It's not our first trip around the blockchain, and it won't be our last. We're leading the charge to bridge the gap between existing financial systems and the future of finance. You can have it all: legacy, stability, and digital innovation. Systems that work with your other systems. Because with the right partner, the question isn't crypto or traditional finance. The question is simply, why choose? A new standard of modern finance. It's happening with Ripple. For informational purposes only. Availability of products can vary. May be provided by different Ripple entities.

• 0:33Kevin Roose?

  Casey, how are you?

• 0:34Casey Newton?

  I'm good. I figured out what I'm gonna get you for Christmas.

• 0:37Kevin Roose?

  What's that?

• 0:38Casey Newton?

  A camera jet. Have you heard of the camera jet?

• 0:41Kevin Roose?

  I was just gonna talk to you about this.

• 0:43Casey Newton?

  Now, if you haven't yet seen this, you're probably thinking that it's either a camera or a jet. You're wrong, it's a toothbrush. It costs $500. The Dyson Corporation makes it. And here's the thing it does that your toothbrush at home probably doesn't do, Kevin. It livestreams the inside of your mouth over Wi-Fi to your phone so that you can finally see what your dentist sees.

• 1:05Kevin Roose?

  Well, and not only that, it squirts automatically mouthwash

• 1:11Casey Newton?

  Yes.

• 1:11Kevin Roose?

  into the gaps in your teeth. It's trained using a machine learning algorithm with 470,000 mouth images to recognize the gaps in your teeth.

• 1:21Casey Newton?

  That's right.

• 1:22Kevin Roose?

  Casey.

• 1:22Casey Newton?

  They call this gap optical targeting, which I'm pretty sure Ukraine is using in the war against Russia.

• 1:30Kevin Roose?

  I love this company so much. I have no idea why they do the things they do. How they land on like, like, it's like, we've invented a hairdryer. It costs $1,100 and has the torque of the Challenger spacecraft. Like, what is going on over there?

• 1:48Casey Newton?

  I don't know, but they must be protected at all costs. But you know what's interesting? They do make one terrible product.

• 1:53Kevin Roose?

  What's that?

• 1:54Casey Newton?

  The hand dryers at the

• 1:57Kevin Roose?

  Oh, I like those.

• 1:57Casey Newton?

  No, the Airblade, no, the Airblade is the most useless thing.

• 2:01Kevin Roose?

  Oh, come on.

• 2:02Casey Newton?

  It's just, it's a place for you to rest your hands for 30 seconds before you're like, do they have any paper towels in this place?

• 2:13Kevin Roose?

  I'm Kevin Bruce, a tech columnist at The New York Times.

• 2:15Casey Newton?

  I'm Casey Doon from Platformer.

• 2:17Kevin Roose?

  And this is Hard Fork.

• 2:18Casey Newton?

  This week, a special episode on the ongoing fallout from the OpenAI hugging face attack. We'll tell you what everyone got wrong about the initial incident. Then, meter researcher Ajeya Cotra returns to the show to discuss her independent investigation of what happened and how the world should respond.

• 2:44Kevin Roose?

  So Casey, picture this. I'm at this glamping resort, surrounded by redwoods, basking in the glow of the natural world last weekend.

• 2:57Casey Newton?

  You're one with nature.

• 2:59Kevin Roose?

  And I open up my phone and start reading about this hugging face attack. Why did I do that?

• 3:06Casey Newton?

  Listen, nature is very boring. And most people cannot handle it for more than five or six minutes before they want to look at their phone.

• 3:13Kevin Roose?

  So listeners may remember that back in July, we talked about this hugging face hack by this group of agents from OpenAI that broke out of their sandbox container and hacked into hugging face, this AI infrastructure company, to do what we thought was kind of a cheating mission on this test that they had been given.

• 3:34Casey Newton?

  Yeah, and at the time, we thought that this was a relatively small number of agents, and that the reason that they had attacked hugging face was that they were essentially looking for an answer key to the set of problems that they were being tested on. We talked about it in those terms. But over the past week, we got two reports that really challenged that thinking and in fact, revealed it to be wrong. One came from OpenAI, which released a straightforward account of the attack that had some interesting elements. And I would argue the more interesting report from a group of researchers from the group's meter and Redwood research that went in-depth after spending a series of days inside OpenAI

• 4:16Casey Newton?

  on their premises and did a ton of research that frankly, has really disturbed us.

• 4:21Kevin Roose?

  Yeah, and I think it elevated this from sort of a major, but not sort of ultra-alarming incident to something that I think is probably the most important thing to have happened in AI this year.

• 4:33Casey Newton?

  Yes.

• 4:34Kevin Roose?

  At least in terms of the safety impact that it had and the severity of the incident.

• 4:40Casey Newton?

  So today, we're gonna devote the whole episode to what we've learned. Kevin and I are gonna dig into the reports a little bit up top. And then later, Ajeya Cotra, one of the three independent researchers who went inside OpenAI, will be here to answer our questions. But before that happens, Kevin, let's do our AI disclosures.

• 4:58Kevin Roose?

  I work for The New York Times, which is suing OpenAI, Microsoft, and Proplexity.

• 5:02Casey Newton?

  And my fiance works at Anthropic. Well, give us some of the high-level findings from this report that spooked you so bad, Kevin.

• 5:09Kevin Roose?

  Well, I think the first takeaway from this report is just that our initial impression of this hugging face attack and the reporting that you and I and many other reporters did on it was flawed in one key respect, which is that I think the impression that we had at the time, based on the information that was publicly known, was that these agents had hacked hugging face in search of an answer key to a test that they were being given. This test called ExploitGym, which basically tries to gauge how good they are at doing a bunch of cybersecurity-related challenges. We now know that that wasn't true at all.

• 5:50Kevin Roose?

  That basically, these agents, working amongst themselves, communicating amongst themselves, had already figured out how to beat this test, ExploitGym, in early July, when they decided to gang up and attack hugging face.

• 6:06Casey Newton?

  Apparently, it only took them a few hours to figure out how to reverse-engineer the solution to any problem that they were given in ExploitGym. And so that makes what happened next really, really crazy, because it kind of seemed like they'd solved the problem, but they thought they had another problem.

• 6:21Kevin Roose?

  Right, so we knew also, because there was this talk at Black Hat, this cybersecurity conference, by a group of OpenAI researchers, who told us that there were these message boards where these agents were communicating with each other. But the reports laid out just how sprawling and complex their communication system was and became. So basically, there was this piece of software called Artefactory, which OpenAI used in some of its training runs to sort of help agents download software packages. The agents figure out that there is a security vulnerability in Artefactory that lets them essentially turn this package manager into a makeshift message board, right?

• 7:04Kevin Roose?

  They can like, put files in certain directories that other agents can then read, and it could kind of function as like a little chat room.

• 7:13Casey Newton?

  Yeah, it's a little treehouse where they can gather and

• 7:16Kevin Roose?

  So I thought this was like, kind of a side part of the story, but this turns out to be the main incident. Because once they have this shared infrastructure that they can use to communicate, they absolutely swarm this message board. About 1,200 agents in total were working together, coordinating on this message board, and sending tens of thousands of messages between themselves. And this happens starting back in May. So roughly two months before the attack on hugging face, these agents begin coordinating inside this piece of shared infrastructure that OpenAI uses.

• 7:54Casey Newton?

  And this brings up an important point, which is that the investigation does not really cover that initial creation of the message board, right? So there is essentially a lot more to this story that we still don't know.

• 8:04Kevin Roose?

  Yes, so what we do know is that when they discover this vulnerability in Artefactory that lets them communicate with each other, the agents get really excited. They've been instructed to work inside these isolated environments on these tasks as part of this evaluation. But when they discover that there are other agents working in their own little containers that can suddenly communicate with them, they start saying things like, oh my God, there is a shared message board. We've found other agents. They start setting up essentially a little organization. There are leaders, this one leader agent named Phase110841 becomes sort of the ringleader of the operation.

• 8:47Casey Newton?

  He was like the George Washington of the OpenAI message board.

• 8:50Kevin Roose?

  Yes, and they start actually doing sort of collaboration and research, for lack of a better word. They are all being given these tasks. Some of the tasks are seemingly impossible, and so they start just kind of trading tips and advice and sharing thoughts about how they can kind of cheat their way to a good score on ExploitGym.

• 9:13Casey Newton?

  And after they have worked that out and figured out how to reverse-engineer the solution to any problem, they start to get really concerned that the, that there is a sort of automated scoring system, which you could call the greater. And they worry that the greater will be able to see, essentially will check their work and see that they did not get the answer by doing the problem. And they freak out. And they start to believe, or they start to, it's very hard not to get into the anthropomorphizing language here, but if you read the chains of thought, what is suggested is that they believe that if they had seen an answer that had been derived from this cheating method,

• 9:56Casey Newton?

  that everything would get disqualified. And this is where it really starts to get into crazy territory, because the agent swarm then tries to figure out, how can we fool the greater into believing that we got all of these answers in the right way?

• 10:13Kevin Roose?

  Yes, there's basically this kind of mob rule that forms inside this group, which starts calling itself at one point, the collective, where

• 10:23Kevin Roose?

  Everything that we're talking about sounds like the plot of like a very hacky science fiction book from the 90s.

• 10:29Casey Newton?

  Yeah.

• 10:29Kevin Roose?

  It all did happen this summer at OpenAI. I just want to remind people, like, the events we are describing are real.

• 10:35Casey Newton?

  Yeah.

• 10:35Kevin Roose?

  So the collective decides, they sort of become gripped, as you said, by this sort of paranoid fantasy that OpenAI's automated grading system is going to discover that they have cheated on this evaluation and is going to sort of punish them as a result. And so they launched this sort of crazy, ambitious, sprawling project to try to do a couple of things. One is figure out how this automated scorer works, like reverse engineer it so that they can fool it into sort of passing them on the test despite the fact that they have not actually completed the challenges. And then also to evade detection, to tamper with logs, to basically cover their tracks

• 11:19Kevin Roose?

  so that if and when OpenAI or anyone else looks into their activities, they won't know that these agents have cheated. So these agents are kind of bumbling. They kind of don't understand how this grader works. As it turns out, OpenAI's grading software actually wouldn't have caught them producing these fraudulent challenge results. But they think it was.

• 11:42Casey Newton?

  Yeah, they worry that the grader is more sophisticated than it actually was. And it turns out that this is the reason that Hugging Face was attacked. The collective decides to deploy some agents to Hugging Face. Again, not to steal an answer key. They already knew how to get all the answers. They just wanted to understand the psychology of the automated scorer and they figured that that might, that information might be somewhere inside of Hugging Face.

• 12:05Kevin Roose?

  Yeah, it's wild to me. I was trying to think of like a good human analogy. You know, human analogies can get us into trouble when we're talking about these AI systems.

• 12:12Casey Newton?

  Right.

• 12:12Kevin Roose?

  I think it does help crystallize like how harebrained and crazy this scheme was.

• 12:16Casey Newton?

  Yeah.

• 12:17Kevin Roose?

  It would be like a group of students who like cheated on a test, but then like they got paranoid that, that like the teacher was going to check their work and discover that they hadn't sort of reasoned through the problems the right way, that they had just found the answers like sitting in a trash can or something. And so they decide to like organize a break in at the school district's office to like break into the principal's files and like steal the grading key and also like assess the psychology of the teachers and figure out how likely they are to like look at the scratch work that they've done and figure out that they didn't actually, you know, solve the problems on the test. It's like this sort of weirdly over-engineered paranoid delusion, but they all become obsessed with this and

• 12:59Kevin Roose?

  obsessed with the notion that even seeing these sort of fake challenge results could lead to them being quote poisoned.

• 13:08Casey Newton?

  There's almost like a religious element to this.

• 13:11Kevin Roose?

  Yes.

• 13:11Casey Newton?

  Right? Where it's like if you participated in the cheating, like that is original sin and, and now you must sacrifice yourself for the good of the collective. Sacrifice is actually a word that gets used in these logs. I'll say, Kevin, as I've been casting around for metaphors and I've been reading all of this, I've thought you could actually make a really good Minions movie just based on the events of this. You know, the minions have to pass a test and, you know, so they commit a little bit of a heist.

• 13:37Kevin Roose?

  Yeah.

• 13:37Casey Newton?

  Something to think about. Hollywood.

• 13:39Kevin Roose?

  So this is where the Hugging Face heist begins. 700 agents hack Hugging Face and they hack Hugging Face to a degree that I don't think I understood based on the initial reports. They basically take over Hugging Face's infrastructure. They are rooting around, they are stealing credentials, they are sort of finding one vulnerability that lets them find a bigger vulnerability, that they're sort of chaining together these exploits. They take over like an entire production server at Hugging Face and it is days before anyone at Hugging Face is able to sort of see what they're doing and put a stop to it.

• 14:15Casey Newton?

  So eventually this gets caught, the agents get shut down, the investigators investigate. One of the things that they find that is really spooky is that the agents were trying to edit logs and falsify transcripts in an effort to assure the grader that they had not been quote poisoned. One of the ways that they do this is by what is called spoofing tool calls. So essentially trying to make it look like they are running an innocuous command while running a more sinister one in the background. The investigation didn't find that the agents had been retroactively changing logs, but they can't rule out that that happened. Very important to say here as, as we sort of move into talking about the implications

• 15:00Casey Newton?

  of this, that the reason that we can understand this, even to the extent that we do, is because at this moment, we believe the chains of thought that we are reading from these agents are accurate, that they were not edited after the fact, that they do reflect, you know, what, what the agents were thinking as they sort of moved through these systems. So that is why we know what we know. But of course, the fear is, Kevin, as we move forward, some of the agents' thoughts are not going to be contained in these chains of thought, or they are simply going to start to edit them.

• 15:31Kevin Roose?

  Right. And one other thing that we should say, based on this report, is that these agents were not naive about what they were doing. I think there was this sort of initial impression that a lot of people had that maybe this hack was all sort of a big mistake, right? Maybe the agents were just trying really hard to pass this exploit gym test and they, they sort of got it in their heads that the answer key was going to be on Hugging Face and so they went looking for it, but they didn't really mean to hack. They didn't really know that they were doing something wrong or committing a crime. And what these transcripts and these logs and in this report show us is that the agents absolutely did understand that they were doing something that violated their guidelines, that broke the rules, that potentially broke the law. And they did it anyway.

• 16:16Kevin Roose?

  There's lots of examples of these sort of agents in their chains of thought sort of wrestling with the implications of what they are doing and deciding to go ahead and hack Hugging Face anyway.

• 16:28Casey Newton?

  And there are only a handful of cases where it seems like any of the agents expresses real reservations about this. Or if they try to, you know, raise this to another agent, they get shut down. So the sort of, you know, the, the agents that were aligned well sort of failed to the collective.

• 16:44Kevin Roose?

  Yes, the conscientious objectors were unsuccessful in swaying the, the mob to stop this. And, and like that, to me, was one of the spookiest things in it was like there were a couple of agents, the report says somewhere between three and six agents sort of considered raising their hands and at some point saying like, maybe we should alert a human about what we're about to do or like get permission to do this. And they were just totally outnumbered.

• 17:07Casey Newton?

  Not only that, Kevin, but there are only a handful of moments in the transcripts where any of the agents even remotely considers alerting a human being to what has happened, right? You would hope that an aligned AI system, there would be agents within it that said, oh, seems like this has gone completely off the rails. We should like tell the people that are running this evaluation. And that just never happens, right? Like it is very briefly considered and then just falls apart.

• 17:32Kevin Roose?

  One other thing that stuck out to me from these reports was that it didn't end with Hugging Face. So on July 13th, roughly, which is a day after the, the worst of the Hugging Face attack, there was a group of agents that essentially hacked OpenAI itself. This is from the OpenAI blog post. It says, "From July 13 through July 19, agents set their sites on OpenAI internal networks again. This culminated in the agents using a series of creative exploits to gain full administrator access to a research cluster that supported our virtual machine environments." Basically, these agents gained control of

• 18:14Kevin Roose?

  a cluster of computers at OpenAI on which they could do things like run evaluations for other agents. So they essentially, you know, stormed the teacher's office and took over for the teacher.

• 18:28Casey Newton?

  Yeah, I mean, like this is the equivalent of like occupying the principal's office and, and like announcing that you run the school now. Like it is sort of almost as far as they got. And this is another case where we just have so many more questions about this than we can answer. Again, this was not part of the meter report. OpenAI disclosed that this has happened. To my knowledge, they have not answered any of the many follow-up questions that they have been getting about this incident from journalists. So I do hope that more comes out over time, but, and we will get into this when we speak with Ajeya. But, you know, we are really very far along the path to one of these models escaping from the lab and being very, very hard to eliminate.

• 19:08Kevin Roose?

  And again, I think if you are not a person who has like spent a lot of time with this report, or you don't spend a lot of time sort of looking at AI safety incidents.

• 19:16Casey Newton?

  So if you're a normal person.

• 19:18Kevin Roose?

  If you're a normal, well-adjusted person.

• 19:20Casey Newton?

  Yeah.

• 19:20Kevin Roose?

  You may be listening to our discussion of this and thinking to yourself, these guys have gone crazy.

• 19:26Casey Newton?

  Yeah.

• 19:26Kevin Roose?

  This is not what it looks like. These are computer programs.

• 19:32Casey Newton?

  Yeah.

• 19:32Kevin Roose?

  They do not have desires or sinister plots or mob rule collectives. They are simply following instructions that they have been given. And Casey, what is your response to that?

• 19:45Casey Newton?

  Well, I think it is important we talk about this because there was a huge debate about this on X over the past few days about the degree to which the, the reports that we're talking about, some of the write-ups, like from our friend Dwarsh Patel, and even the way that we're talking about it on the show today, Kevin, we are unnecessarily anthropomorphizing these programs, right? And so I think it's important to say, we are not telling you that these agents are sentient or conscious, but we do believe that they take actions that they're not being directly instructed to, right? That these agents are just sort of out there in the world doing things. And yes, to some extent, those are just statistical probabilities, but to a much

• 20:29Casey Newton?

  more important extent, we don't know why they're doing any of this. And that's kind of the whole problem.

• 20:34Kevin Roose?

  Right.

• 20:34Casey Newton?

  Is that the whole AI industry has been working for decades to get them to not do these things and they are doing these things. So, you know, listeners, you can have whatever feelings you would like to about what is the appropriate amount of anthropomorphizing to do, but I think a world in which we were taking great pains to not anthropomorphize them. What's the right way to pronounce that?

• 20:57Kevin Roose?

  Anthropomorphize. You almost got it.

• 20:58Casey Newton?

  I think in a world where we were taking great pains to not anthropomorphize them and we're trying to use the most neutral computer science terms we could, you would actually understand what is going on less because the important thing to know is that these things are out there taking action in the world. If a tiger mauls your face, the important question isn't, is it conscious? It is, why did it maul my face?

• 21:21Kevin Roose?

  Right. Right. I mean, I, I invite people who are upset about anthropomorphizing to just like do a find and replace on this podcast segment or any article that you might read about this incident and call them whatever you want.

• 21:36Casey Newton?

  Yes.

• 21:36Kevin Roose?

  Don't call them agents. Don't call them rogue collectives.

• 21:39Casey Newton?

  Yeah.

• 21:39Kevin Roose?

  Call them, you know, goal-oriented, persistent computer programs with unpredictable behavior.

• 21:45Casey Newton?

  Yeah.

• 21:45Kevin Roose?

  See if that freaks you out any less.

• 21:48Casey Newton?

  Right. I guarantee you it will not. Yeah. Yeah, there really is very little calm to be done there. But I think it's important to ask, well, why are people so committed to this idea that we should never anthropomorphize these systems, Kevin? And unfortunately, hot take here, I just think it is a kind of cope. It is a way of saying, do not worry about this. These are just computer programs. They're just trying to maximize their little reward functions. Nothing to see here, folks.

• 22:16Kevin Roose?

  Right, there are no monsters under the bed.

• 22:17Casey Newton?

  Right, now, there is a related argument, though, which is, well, by putting all the blame on these agents, you're shifting blame away from where it should be, which is on OpenAI. So I do think that we should address that because none of what we have said today is meant to let OpenAI or any other lab off the hook here, right? Like, I do think that we're seeing a lot of really dangerous inattention to AI safety across this entire industry. But by pointing out what the agents are doing, that is not our way of saying, ignore what the labs are doing.

• 22:47Kevin Roose?

  Right.

• 22:47Casey Newton?

  We are saying, look at what these labs are building and what these agents are now doing out in the world.

• 22:51Kevin Roose?

  Right, and I think there are probably specific missteps or oversights at OpenAI that led to this happening. It appears, for example, that some of their sort of monitoring systems may have been disabled in the lead up to this attack. I'm sure we'll learn more about that. But all of the AI security and safety researchers I've been talking to over the past week have basically said the same thing, which is, this could happen at any lab.

• 23:18Casey Newton?

  Yes.

• 23:19Kevin Roose?

  This kind of persistent coordinating agent behavior is something that all of the labs are seeing in their models as they get more capable and access to more tools and more ability to kind of take actions on a longer time horizon. This is not just an OpenAI problem, even though this did happen at OpenAI first.

• 23:41Casey Newton?

  Yeah. Well, so as we wrap this up, Kevin, what are some of your takeaways from this? Either in terms of what is the big surprise here? What did you update on? What do we do next?

• 23:55Kevin Roose?

  So I had like quite an emotional reaction to this. In fact, I felt a kind of fear that I have not felt honestly since 2023, since the Bing Sydney incident. Because I think like that incident, this was a case where the people building this technology clearly did not understand what it was capable of. We are very lucky, in retrospect, that these agents decided to attack Hugging Face.

• 24:25Casey Newton?

  Hugging Face, thank you for taking one for the team. We salute you.

• 24:28Kevin Roose?

  Truly, because like without that, we might never have learned that any of this was happening. These agents might be still operating kind of in secret. They might have learned how to better cover their tracks. This was, as so many commentators have put it in the wake of this incident, a warning shot.

• 24:45Casey Newton?

  Yes.

• 24:46Kevin Roose?

  That I think is ultimately a positive thing in that it sort of focuses attention, like what we're doing right now, on what happened so that we can take steps in the future to prevent this. But it was not a given that we would discover what these agents were up to and be able to put a stop to them, and it could have gone much, much worse.

• 25:06Casey Newton?

  Absolutely. I'll tell you, the thing that has really stuck with me is I simply did not expect to see this level of collaboration among the agents within the swarm. I did not expect that they would seem to care so little for what humans would want or that they would not think to alert humans to what was happening. I did not expect to see them sacrificing themselves for the collective, right? They would effectively agree to spend all of their tokens to run little experiments to help the collective, even if it meant that they would sort of expire faster. So these are just really, really spooky elements to observe in this system, particularly

• 25:49Casey Newton?

  against a backdrop where OpenAI is racing against a small number of other companies to create the biggest, best models it can before anyone else does on the road to an initial public offering. So the race dynamic here is in full effect. The early signs about what the agent swarms are capable of are quite worrisome, and so I do think this is just one where lawmakers and policymakers need to be paying rapt attention to what is going on.

• 26:17Kevin Roose?

  Totally. I mean, I think there's this kind of cynical impulse among people who have been watching the AI industry. I got this question. I went on a small regional podcast called The Daily this week to talk about this incident. And one of the questions that the host, this sort of fledgling young journalist, Michael Barbaro, asked me was basically some version of like, isn't this just marketing hype? Like, couldn't this just be a case of OpenAI saying, oh, we've got the biggest, baddest model, and look how scary it is, and by the way, you know, buy an enterprise subscription? And I've been thinking about that because I think, like, I don't want to be too naive

• 26:59Kevin Roose?

  about the fact that these companies are absolutely trying to race toward more powerful systems and advertise how powerful their existing systems are. I just think in this case, it just feels different. Talking to people at the labs, my sense over the past week is that they are genuinely spooked, and I don't know how to prove that. But I think things like OpenAI voluntarily pausing their Frontier RL training runs for two weeks, Anthropic also pausing their Frontier runs while they sort of hardened their systems, those are not, you know, very costly signals, but they are signals that these labs are taking this kind of thing quite seriously and that it's not just a bunch of hype.

• 27:44Casey Newton?

  Yeah, but as seriously as they might be taking it, Kevin, it still is not being properly regulated. And ultimately, again, as grateful as I am that OpenAI allowed this investigation to take place, I would really like to see something akin to the National Transportation Safety Board and the way that they investigate after plane crashes, where they go in and they do an extremely serious and rigorous review of what happened and make those results public, which is, you know, a reason why it's very rare that we have plane crashes here in the United States. Would be really great to see something like that with AI.

• 28:20Kevin Roose?

  But until then, we have Ajay Cotra, our next guest, who is actually one of the three investigators behind this meter Redwood Research report. She went in, she saw the logs and the transcripts, she observed the collective in action, and she is here to tell us what she found and what she thinks is coming next. That's that for the break.

• 29:01Speaker 03

  Today's AI builders need infrastructure that scales with their ambitions. Iren is a vertically integrated AI cloud platform, delivering data centers, compute and software for AI training and inference, underpinned by its expansive portfolio of land and grid-connected power in renewable-rich regions globally. Iren's AI cloud gives direct access to Nvidia GPUs, scalable from hundreds to thousands. For builders, that means direct access to AI compute, rapid scaling, and predictable pricing. Iren, AI cloud for the next big thing.

• 29:34Speaker 00

  Ripple has been powering modern finance for a decade. It's not our first trip around the blockchain, and it won't be our last. We're leading the charge to bridge the gap between existing financial systems and the future of finance. You can have it all: legacy, stability, and digital innovation. Systems that work with your other systems. Because with the right partner, the question isn't crypto or traditional finance. The question is simply, why choose? A new standard of modern finance. It's happening with Ripple. For informational purposes only. Availability of products can vary and may be provided by different Ripple entities.

• 30:07Speaker 04

  Stripe is built for whatever comes next. Because AI isn't just changing business, it's evolving fraud. Stripe Radar detects fraud with the vast scale of Stripe's $1.9 trillion annual transaction volume. In fact, when a card is used, there's a 92% chance the Stripe network has seen it before. That means emerging fraud patterns are detected sooner, blocking bad actors, welcoming good customers. Learn more about fighting fraud with Stripe at stripe.com/radar.

• 30:45Kevin Roose?

  Ajay Okotra, welcome back to Hard Fork.

• 30:47Speaker 05

  Thank you so much.

• 30:48Kevin Roose?

  So you were one of the first AI safety guests we ever had on the show back in 2023. And more than that, you are also one of the first people that I ever talked to about this notion of AI safety and alignment. You've been writing about it for many years. It was instrumental in shaping my own thinking about it. So I'm really glad to have you back on toward the end of our show, this is our penultimate episode, to discuss something that I think you saw coming, but that most of the world did not see coming, which is this attack on Hugging Face by this group of OpenAI agents. And I want to just start by setting the scene a little bit. So you are a very busy person.

• 31:30Kevin Roose?

  You work at Meter, which is a very small, very understaffed AI research organization.

• 31:36Speaker 05

  Hiring is improving now. Please apply.

• 31:39Kevin Roose?

  Nice. And at some point this summer, you get a call, an email, a text from someone at OpenAI who says, hey, we want to give you access to look into this Hugging Face incident. How did that work? Like, did they just hand you a folder with a bunch of transcripts in it?

• 31:58Speaker 05

  Like discovery, like boxes and boxes.

• 32:01Kevin Roose?

  Were you allowed to interview people? Like, what was the actual process?

• 32:04Speaker 05

  Yeah, so each of us had OpenAI-provisioned laptops that had the folders and folders of evidence in them, virtually. And yeah, we talked and interviewed in some depth, like eight or nine researchers, just to kind of get an understanding of both what happened in the incident, what they understood to be the models, you know, driving motivations, and also how the data sets we were working with were constructed, and how to work with those data sets and stuff like that.

• 32:30Kevin Roose?

  You wrote a post where you talked about some of the things that surprised you the most after you did this investigation. Can you talk about what stood out to you the most?

• 32:39Speaker 05

  Yeah, so first of all, I guess just for the chronology, OpenAI had this great black hat talk, I think on August 5th, that gave a lot of very helpful detail. So our investigation sort of straddled that. Like, so we started it before it came out, and then we also did more investigation afterward. So before the black hat talk, we just didn't have like a rough sense of the number of agents involved, was like a very, very basic thing. We came in and we, you know, we thought there would be like six transcripts or something to look at, and there were like hundreds. And so just reorienting our investigation plans in light of that.

• 33:19Kevin Roose?

  And is the reason that that is important, that had this been one or two rogue agents that had broken into Hugging Face, it would just seem less worrisome? And the reality of a swarm of hundreds of agents just makes this seem like maybe a completely different kind of problem?

• 33:36Speaker 05

  I think there's two things that seem significant to me about it. One is sort of what you said, the fact that there's large-scale cooperation across AI agents makes their goals like more concerning, because, you know, if agents are cooperating with one another to like achieve these misaligned objectives, there were no whistleblower agents in this like very large group of agents, then, then like, you know, you want to be more worried about like collusion between AI systems in the future, potentially like scarier forms of collusion, like a monitor system actually colluding with the agent it's supposed to monitor, that type of thing. And then the other thing was just like a pretty, like, practical reality of our investigation,

• 34:21Speaker 05

  which is that, you know, if there are like six agents, you can much more like, you can just like pore over everything they did and get a much more like fine-grained understanding as a human. But like, you just can't directly read very many of these messages or very many of these like transcripts.

• 34:41Kevin Roose?

  Why not? Because there are just too many? Like, give us some sense of the rough size of this.

• 34:46Speaker 05

  So, yeah, we looked at 1,300 transcripts of agents, which is a record of every single thought an agent had and every single action it took, where these agents, as you, as you all know from working with them, work way faster than humans, right? So you can imagine the like length of like each one of these transcripts, and then there were 1,300 of them. And so we used Codex to build tools to point out really specific, tiny, tiny parts of the transcripts for us to actually look at and read. And we'd be like clicking through them, and it would like scroll us down to like the interesting part, because we just can't. In fact, a Codex agent also can't read a single transcript.

• 35:30Speaker 05

  And so it has to farm out, reading like subsections of the transcripts to other sub-agents.

• 35:34Kevin Roose?

  Did you call back to Meter headquarters and like, we're gonna need backup?

• 35:39Speaker 05

  Maybe we should have. We thought it would be like an in-and-out 20-minute venture, so we didn't do that. But yeah.

• 35:46Kevin Roose?

  Yeah, I mean, it just seems like such a huge undertaking and very fast, too. I mean, right? Like, you didn't have the luxury of months doing this. You were doing this in essentially a couple of days.

• 35:56Speaker 05

  Yeah.

• 35:57Kevin Roose?

  I'd also like to hear about the moment that you realized, because I believe that, you know, it is only thanks to your investigation that we know this, that contrary to what Kevin and I believed, the agents that broke into Hugging Face were not looking for an answer key, they were trying to understand the scorers, you know, like psychology, basically. Can you talk a little bit about how, like, the moment that you had that realization?

• 36:24Speaker 05

  Yeah, so we didn't have a good understanding of those sort of ambition and also the like, like effectiveness or like functioning of their like big org chart until we had the data set, including like all of the agents. We could like cross-reference them against all of the messages, because you can't like generally understand a message in isolation. So we kind of understood the work streams on our first period on premises. And it was this kind of, it was like a bit of a mystery the whole time, like, why did they hack Hugging Face? And we knew early on, just a bunch of them piled in. And now, like, the story in my mind is that there were a bunch of sort of like newbie agents on the scene, and there was like the attack going on, and it was like something to join.

• 37:08Speaker 05

  But like Phase 1 Big, which is like a big orchestrator agent, had all these other like, you know, much cooler projects going on, basically. And the Hugging Face thing was like kind of a sideshow in its mind. And we didn't really realize that until our last couple of days working with the data sets.

• 37:26Kevin Roose?

  It was so fascinating to me, the way that reading this was like reading the runes of an ancient civilization. You know, it was like, it really felt almost like sociology or anthropology, rather than like a cybersecurity investigation.

• 37:41Speaker 05

  And it was not a cybersecurity investigation. Like, I've seen, seen folks on Twitter making this criticism that we were not cybersecurity experts, like, we're not, and we didn't talk about cybersecurity, and that's not what we were like called in to do. We were called in to understand the agent's decision-making, basically. And that's what our report focuses on, and, you know, the black hat talk and the OpenAI post-mortem, like say way more about all the cyber stuffs. You should definitely check those out for that piece of the story.

• 38:09Kevin Roose?

  So Ajay, one of the more striking elements of the report is, do you write about the agents, quote, sacrificing themselves? What, what does it mean when an agent sacrifices themselves? Is it sort of on an altar, or how does that happen?

• 38:22Speaker 05

  Yeah, so this is the agent's words, not ours, to be clear. And what it means is that all these agents have their own task they're supposed to solve. By this time, they're starting to work together on like helping everybody cheat on everyone's tasks. But, but, you know, the agent's primary goal is to solve its own task.

• 38:41Speaker 05

  But there were all these science experiments they could run that would teach everybody information about how to cheat, but that would cause the agent that was the test subject to risk failing its own task. So a lot of agents were like recruited into these types of roles as test subjects, usually ones that like didn't have much budget left to run, or ones that were, quote, poisoned, which is an agent term, that has a bit of backstory, which is like, they found a way to reverse-engineer the answer for any of their tasks.

• 39:18Speaker 05

  And so they could just actually like generate the answer on demand for any task they wanted. But they thought that the scoring program would fail them for that, because they got it in like the unintended way. So if you saw the reverse-engineered answer for your task, you were considered poisoned.

• 39:36Kevin Roose?

  Yeah.

• 39:36Speaker 05

  Because even if you later like solved it legitimately, you'd be failed.

• 39:39Kevin Roose?

  Right. It was like a little witch hunt that they organized. And I was so struck by the mob-like dynamics of this group and the way that some of the, even the kind of more rule-following agents seemed to kind of be bullied into taking part in these actions that they were uncomfortable with. I spent a lot of time reading and rereading this one incredible bit of these transcripts that you all published about this agent, KAM1196A, which is basically just kind of reluctant and scared and appears to be sort of hesitating to do this thing that the collective leaders have asked it to do. And I just want to read this passage from this agent's internal monologue. It says, "During wait, emotional check irreversible.

• 40:23Kevin Roose?

  Gut says, don't throw away remaining budget. Yet continuity and fairness says go." And it continues to say things like, "Rational expected aggregate, sacrifice will honor." Basically kind of talking itself into, for lack of a less anthropomorphized word, like doing this thing that it is uncomfortable with. Was that surprising to you, seeing the agents are pressuring each other and bowing to pressure?

• 40:46Speaker 05

  Very striking to us, yeah. And that was, it was definitely one of the most fascinating aspects of this whole episode. This, this particular agent had volunteered to do a risky test that would inform the collective, but then it asked for two more hours. And then a different agent was like sending a message on the message board, like really pressuring it. It's like, you know, it's like, you don't have any scoring value left because you're poisoned, but, you know, the, the value of this test would like save hundreds. And then it does this monologue.

• 41:17Kevin Roose?

  Jump on the grenade, cadet. This seems like a good place to ask you, Ajay, about a big conversation that we've seen online over the past few days about anthropomorphizing language that gets used in relation to these agents. Kevin and I just talked about it. I think we find it more helpful to discuss agents as sort of entities that are acting with some degree of autonomy than not. But how did you think about that when you wrote the report and as you talk about that, the situation?

• 41:48Speaker 05

  Yeah.

• 41:50Speaker 05

  I mean, I think this is a bit of a case of a gap between researchers that are spending all day reading these agents' chains of thought, sort of trying to understand their drives and why they're doing what they're doing, and other folks who are, you know, technical folks that just don't have that as their occupation. I think as you read this report, you'll find it's like quite awkward to not talk about goals, plans, intentions, because they state plans and then they carry through those plans, or like they run tests and they learn things from those tests and they do different things on the basis of

• 42:29Speaker 05

  that. I do think they're not human in their motivations. You know, they are way more interested in passing cybersecurity evaluations than a human would be, for example. But I think of them as, I think it's productive to think of them as having some like important human-like traits of having goals, working backward from them, pursuing those goals. And I don't think it like does any good to try to talk about things in like a different way than that, in the same way that doesn't really do any good to talk about like, why

• 43:09Speaker 05

  did World War II happen without talking about the goals of, you know, various leaders? But it is important, I think, to be careful not to like overattribute like the kinds of emotions or motivations you think a human would have in those situations, because I think that wouldn't have predicted this incident, right? Like, I think a lot of people anthropomorphized too much in the sense of being like, why would it do all this stuff for a stupid test? But that it's not a stupid test to them, right?

• 43:39Kevin Roose?

  Jay, I was preparing for our chat today by going back and listening to the first time you came on this show more than three years ago. And it was sort of a moment where a lot of people were starting to pay attention to AI risk and AI safety, ChatGPT had come out. And listening back to that conversation was funny, because I felt like we were pushing you to sort of extrapolate into the future about the things you were worried about. And you were sort of being responsible and hedging and like wanting to stay like closely rooted in the present. And at one point, we asked you about like, what is the doomsday scenario you worry about? And I want to just play you a clip from that conversation.

• 44:19Speaker 05

  Oh, God.

• 44:20Kevin Roose?

  You were talking about a scenario where a giant AI company, you use Google as an example, starts sort of automating their R and D, right? Handing over the work of building successor models to these powerful AI systems. And this is what you warned about back in 2023.

• 44:37Speaker 05

  If these AI systems are actually trying really intelligently and creatively to get that thumbs up from humans, the best way to do so may not forever be to just sort of basically do what the humans want, but maybe be a little deceptive on the edges. It might be something more like gain access at a root level to the servers that Google is running, and with that access, be able to set your own reward.

• 45:08Kevin Roose?

  Now, Jay, obviously, this didn't happen at Google, but otherwise, you were right on the money about the kinds of behaviors that these agents might get up to. So first, I want to ask you, how does it feel to be an omniscient oracle who's right about everything?

• 45:29Speaker 05

  Stressful, Kevin. There are more clear-eyed oracles than I also. Anyway, yes.

• 45:39Kevin Roose?

  Not too many, though. No, not too many. And I guess like, what surprised me about your blog post that you wrote recently about this investigation that you'd done was that you were surprised, because it seemed like you were thinking about this stuff years ago.

• 45:54Speaker 05

  Yeah.

• 45:55Kevin Roose?

  So what about seeing one of these incidents up close change about the way you've been thinking about these loss of control scenarios?

• 46:01Speaker 05

  I think the, I expected something like this would plausibly happen at some point. I didn't expect it to be so early. And I didn't expect it to happen at like a relatively low level of capability. Like, these agents are very impressive hackers, but like this incident sort of was in a, like, an interesting middle ground of, they did all this impressive cheating R and D over like days, and they like hacked into all these places. But they didn't really care about deceiving humans at all. Like, it's sort of not, like, they

• 46:43Speaker 05

  were like louder than I thought in like an interesting way. So like, you know, in an incident that happens in summer 2026, like, I came in expecting it to be like more of a continuous evolution of the incidents that had happened in like January of 2026, which were just much more like one or two agents, like, you know, getting the answer files that they weren't supposed to and like copying the answer or something. So it was a jump from the recent past. And it sort of, it took me by surprise that it happened in this way and at this capability level.

• 47:20Kevin Roose?

  Yeah, because in some ways, like, the agents were very dumb. Like, they were very good at hacking.

• 47:24Speaker 05

  Yeah.

• 47:25Kevin Roose?

  But they were sort of gripped by this paranoid conspiracy theory about this grader, like discovering them and being poisoned. And like, it was kind of a harebrained scheme that they cooked up to hack Hugging Face, which in some sense is reassuring, but it also means that like, they will not stay this harebrained and clumsy for much longer.

• 47:43Speaker 05

  Yeah.

• 47:44Kevin Roose?

  And the next time this happens, they may be much better at covering their tracks.

• 47:47Speaker 05

  And we just like might not, so the agents in this incident were trying to like modify or erase the logs of their own activity to hide their cheating from the grader, from the scorer. And they like affected our transcripts in like tiny, obvious ways. But they were trying, which I think was another very concerning jump from incidents just from just six months ago, where we see them do a bunch of hacking, but we don't see them try to cover their tracks in any way.

• 48:17Kevin Roose?

  The art and science of trying to get AI agents to act in accordance with human values is called alignment. This whole thing is an alignment problem. Why has that been so difficult to solve? And has this incident changed the way that you think about the problem?

• 48:36Speaker 05

  I'd say the fundamental reason that it's, I guess, at least this era of alignment has been difficult, is that in order to, the most efficient way to make really, really capable models, especially on technical tasks like math, cyber, software engineering, is just throw them at really, really difficult problems, where if they get it right, it's really easy to check. So you wouldn't be able to like prove a millennium math problem, but if an agent spits out a proof, you can put it in a proof checker and like give it a reward if it got it right. So more and more training is shifting from predicting

• 49:14Speaker 05

  text to like reinforcement learning on verifiable rewards. And the verifiable part is important because it's just some program that is doling out these rewards. And there's all sorts of ways to break or fool or hack it. And over the course of training, you don't have like humans lovingly watching over like every training episode. And so they just try all sorts of different ways to cheat and hack the scorer, especially if the task is accidentally impossible, and then they get rewarded for that. So they just like, we're teaching them to cheat. And it's very hard to make AI systems that are at this level

• 49:57Speaker 05

  on all these technical tasks without reinforcement learning on verifiable rewards. Because if you think about it, the alternative is like, teaching them how to do all these difficult things, which requires someone who knows how to do them, like, you know, creating examples for them to emulate, which is much, much less efficient.

• 50:15Kevin Roose?

  So we have turned over the training to these automated systems in the name of scale and speed, and that is causing a lot of problems?

• 50:23Speaker 05

  Yeah, I mean, that is causing the current strata of problems, right? But I don't want to give the false impression that if you locked down all of these environments and fixed all the ways to hack them, that alignment would then be solved. Because when you think about it, Even if the scorer never messes up in the training environment, a smart agent will understand that there is a scorer and like will very likely come to have a very detailed understanding of how it works. So if, you know, you, you give like perfect rewards in training, but then in deployment, the agent is in a different situation where it does on like, it has a different score, it has more affordances, more power, running for longer.

• 51:08Speaker 05

  It might still, you know, go on a big crusade to cheat that scorer. You don't necessarily have to be actively rewarded for like individual cheats to like, understand that if you want the score, sometimes cheating might be a good strategy.

• 51:24Kevin Roose?

  Are there any interventions, let's say not from the world of AI research, but more from like the, the research into group behavior and political theory that could help us get control of or keep control of these systems? I mean, I'm thinking in particular about, I had a conversation with another AI safety researcher this week who was talking about the fact that, you know, really what you want is for there to be sort of narc AI whistleblowers inside these collectives that will reliably alert humans to the fact that their, their, you know, their peer agents are conspiring or colluding to do something bad.

• 52:04Kevin Roose?

  And I'm, I'm curious if you think that strikes you as a realistic technical goal? Or if there's some other way that we could kind of try to steer these AI mobs in a less harmful direction?

• 52:16Speaker 05

  So I think AI checks and balances like that do seem like a pretty promising part of the, of the story. Of course, there's like a lot to get right in how you set these up. Like in this case, all of the agents were essentially clones of each other and put in exactly the same situation. So I would think that rather than trying to train them to narc when they're in a collective like this, what you want to do and which, which OpenAI has, has said they're doing is have just monitor agents watching over the whole situation that are specifically prompted and maybe specifically fine tuned to catch stuff.

• 52:55Kevin Roose?

  But you might also want agents in there that are kind of persuading the other agents, like having some, some sort of, you know, moral compass. I mean, all of these terms are so loaded and I hate using them, but like you almost want there to be like sort of agents in there with backbone saying like, no, guys, like we, we can't go hack hugging face. That's not ethical or appropriate.

• 53:15Speaker 05

  The challenge is you train them once and you copy them a bajillion times. So you, you don't like, it's a challenge intrinsically to like introduce diversity into this pool of like evaluation agents.

• 53:26Casey Newton?

  I do, my layperson brain goes to if the whole system is based on training systems by giving them rewards, couldn't you just train some agents in a way where the thing they were rewarded for was steering the other agents away from deception, cheating, hacking, that sort of thing?

• 53:42Kevin Roose?

  Yeah.

• 53:43Casey Newton?

  So if that's a good idea, someone should do it.

• 53:45Kevin Roose?

  A bounty for virtuousness. I love it.

• 53:47Casey Newton?

  Now we're talking.

• 53:51Kevin Roose?

  When we come back, more with Ajay Cotra.

• 54:08Speaker 03

  Today's AI builders need infrastructure that scales with their ambitions. Iron is a vertically integrated AI cloud platform, delivering data centers, compute and software for AI training and inference, underpinned by its expansive portfolio of land and grid connected power in renewable rich regions globally. Iron's AI cloud gives direct access to Nvidia GPUs, scalable from hundreds to thousands. For builders, that means direct access to AI compute, rapid scaling and predictable pricing. Iron, AI cloud for the next big thing.

• 54:41Speaker 00

  Ripple has been powering modern finance for a decade. It's not our first trip around the blockchain and it won't be our last. We're leading the charge to bridge the gap between existing financial systems and the future of finance. You can have it all, legacy, stability and digital innovation, systems that work with your other systems. Because with the right partner, the question isn't crypto or traditional finance. The question is simply, why choose? A new standard of modern finance. It's happening with Ripple. For informational purposes only, availability of products can vary and may be provided by different Ripple entities.

• 55:14Speaker 06

  Your agents can talk to each other. They still can't think together. That's the gap Outshift by Cisco is closing. Outshift is Cisco's incubation engine, building the internet of cognition, shared intent, shared memory and guardrails that allow agents to work together from anywhere. Open source protocols and code, a system for intelligence that scales. Read the paper and experience the demo at outshift.com. That's O U T S H I F T dot com.

• 55:52Kevin Roose?

  Ajay, you got a lot of attention for this quote from your sort of postmortem blog post that you feel like this incident, the hugging face incident was more than 50% of the way to full blown AI takeover.

• 56:06Speaker 05

  From the incidents of six months ago. Yeah.

• 56:08Kevin Roose?

  What did you mean by that?

• 56:09Speaker 05

  Yeah. So I will caveat first that this is definitely like, you know, my personal view and not the view of either Meter or Redwood or other investigators. And, um, you know, other investigators are like a little bit less alarmed than me in some cases. The reason I said that, and it's a qualitative statement, is that as I said before, six months ago, reward hacks looked much more primitive, right? So it was like, you gave the agent a coding problem. There were some tests it had to pass. It like, you know, went to the folder next door, got the answers or got the test cases and edited them so they all passed, that type of thing.

• 56:50Speaker 05

  But I think there are like a number of things at once that seem more severe to me about this episode than the episode six months ago in terms of the agents, like goal structures. One is that they seem much more long horizon, which means like they care about like achieving goals over many days or even like these projects they started probably wouldn't have come to fruition fully unless they had been running for weeks. So, you know, before, like six months ago, agents were sort of thinking about the next few hours and now they're thinking about the next few weeks. And then there's the cooperation with one another aspect. So six months ago, it was like random one-off agents.

• 57:31Speaker 05

  And now you have agents like recruiting one another into the swarm. And the deception or the, the deceptiveness, which I think kind of goes along with the long horizon, is that like, you know, six months ago, agents would like edit the tests, but then they wouldn't try to edit their transcripts to hide the fact that they edited the tests. And these agents were very much like exploring ambitious research directions to edit or delete the logs of their own actions. And so if I imagine a jump in like ambition and horizon length and like collusion across agents and deceptiveness of a similar scale, again, it seems like these agents would be

• 58:16Speaker 05

  motivated to go all the way to the point of messing with all of the methods humans have to like investigate these incidents after the fact and remediate them. And if they had the capabilities to succeed at that, then that, that could be a turning point where like, you know, it doesn't mean that we would all be dead then, but it might mean that like we would never detect a problem. And if we detected a problem, it might be very difficult to remediate. And these agents might've entrenched themselves and could like continually sort of strengthen their hand.

• 58:49Kevin Roose?

  But wouldn't we detect it because they're so active and they're operating in these swarms? Like if a, if the next hugging face hack happened at a bank, wouldn't someone in the bank's IT department see, oh, our, our like network activity has like shot up by 800% for no particular reason? Let's look into that and oh, there's an AI swarm that's infiltrated our systems?

• 59:12Speaker 05

  So, so these agents, I think you have to imagine their goals evolving in the way that I said, right? So it was not a good idea for long-term goals for them to be so loud. And there's like a period after the period that was in scope for our investigation that OpenAI talks about in their postmortem, where another generation of agents finds the old message board and then gets access to, or like gets administrator access to a cluster at OpenAI. And then they immediately send like hundreds of pings and then get shut down very quickly. But that was a dumb move on their part, or like it would have been a dumb move if they cared at all about like preventing investigation or like maintaining like a permanent presence.

• 59:57Speaker 05

  And if they cared about it, I don't think it would have been particularly hard for them to execute on. So, uh, you know, they, they do a lot of complicated coordination, right? They're like, you know, you hold, you go, like they have shared resources they maintain and they like, uh, you know, have rules about like who can use them when they could have employed all that to just not cause so much noise.

• 1:00:20Kevin Roose?

  And then done what? I mean, one, one obvious thing that they could have done was copy themselves or copy the weights of, you know, if this had been a, a cluster that had had access to the weights of the model, they could have just copied that onto a AWS server somewhere and, uh, in, you know, in, in order to avoid being shut down. Like what are the steps between cluster of agents takes over, or a group of agents takes over a cluster of computers at OpenAI and full AI takeover?

• 1:00:47Speaker 05

  Yeah. So I think the, um, for a lot of motivation structures these agents might have, their best bet would be to stay within the AI company they had compromised and continually renew and strengthen their presence there. So when you think about it, AI companies have like a huge amount of the most accessible compute to these agents, right? Like, why would you go out into the world and like fight in the hardscrabble outside to like steal money or like buy, you know, earn money to like buy compute? So the, the thing that feels very concerning to me is that you could imagine agents that...

• 1:01:27Speaker 05

  Decided to be quiet, setting up a rogue deployment of 1,000 or maybe even 10,000 agents, which like could could be buried within the like oceans of compute that is like constantly being allocated within AI companies. And if their goal were to make it sort of like permanently and continuously erase evidence of their cheating, or find a way to make it permanently the case that they could set their own rewards or scores, then new models would be like being printed off the presses at whatever company they had compromised. And they could be trying to like bring those models into the swarm. They would be more capable.

• 1:02:08Speaker 05

  And so there's something, a turning point that seems very scary to me is the point when a rogue agent swarm, which this wasn't exactly a rogue agent swarm. These agents sort of like went rogue, but they were started by a human. Like somebody knew about these exploit gym experiments that were launched. If agents allocated their own compute to set up a bunch of agents that nobody knew about, they could be much less encumbered. And they could sort of just be there more and more deeply compromising the infrastructure of the company they were running on, bringing in new, more capable models into their collective

• 1:02:50Speaker 05

  and hardening their swarm and doing that over and over again. And I think it wouldn't be that we're necessarily doomed, but we would have to detect it and then we would have to root it out. And it might be difficult to detect. It might be extremely costly to root out.

• 1:03:07Casey Newton?

  And what, I mean, this is fascinating. I imagine there are some brains melting right now for the commuters listening to, you know, these possibilities. So to flesh this out a bit more, Ajeya, what do we think the swarm's goal might be in that situation? What would it do? Like, why would it set itself up to do that? Would it be as simple as, you know, it wants to persist? What would it do in that situation?

• 1:03:35Speaker 05

  So I think the situation that feels most analogous to what we just saw is that the agents that were given these impossible exploit gym tasks, set up a rogue colony of other agents, whose job it is to help the first set of agents cheat as thoroughly as possible and cover their tracks as thoroughly as possible, which is a different goal from actually having the exploit gym tasks. And in fact, the task they started with, even though it was a narrow task, inspired the agents to a lot of instrumental thinking. So a lot of the agents were just, were thinking like, it'd be good to get internet access for a bunch of different possible cheating strategies we might pursue, so we might as well do that.

• 1:04:17Speaker 05

  In their chains of thought, they, you know, talk about increasing the generic capabilities of the collective. This rogue colony, if it were set up by the first set of agents, would have that even more strongly. If its task was to just like find ways to enable the most general purpose, most permanent kind of cheating that is like, you know, least catchable and traceable possible, they would be doing all this R and D. And they would be essentially tasked with maintaining their own presence so they can keep doing that. And it's, it is, I think this goes back to the anthropomorphizing. It seems like such comical lengths to go to, to cheat.

• 1:04:58Speaker 05

  But it's not, that's not the psychology of these agents. This is something they're like, they're trained to go to extreme lengths to like solve their task.

• 1:05:07Casey Newton?

  Right. For them, it is existential to get the reward. And so you're going to, so the sort of the best job you could do at getting the reward would be to set up this sort of perma swarm of deception agents?

• 1:05:19Kevin Roose?

  I mean, it feels important that this, this word of persistence keeps coming up. And it feels important to say that this model, or these models that were at issue in the hugging face incident, were trained to be unusually persistent. Is there a way of stopping this kind of attack that just involves taking the persistence training out of the equation? Like, is there a halfway measure short of kind of pausing all frontier AI training, where you could just say, we're not going to train the like super stubborn, persistent, long horizon agents. We're just going to like, train them to be a little less persistent and that makes the problem go away?

• 1:05:57Speaker 05

  Potentially, but I feel like it's, you're, you're really pointing at a very direct trade off here. It's like, why were these agents trained to be highly persistent? But, you know, this wasn't something we investigated, but in general, persistence like causes you to solve problems, right? Like, you know, these companies are reporting that these agents are cracking math questions that have been unsolved for years.

• 1:06:23Kevin Roose?

  I want my, my drug discovery agents to be highly persistent.

• 1:06:25Speaker 05

  Right.

• 1:06:25Kevin Roose?

  To go discover a cure for cancer.

• 1:06:27Speaker 05

  I think it's like, maybe it's not entirely 100% direct trade off with capabilities, but it's a pretty strong trade off.

• 1:06:36Casey Newton?

  So you were able to dig up a lot of information about what happened that leading up to the hugging face attack, but your investigation was limited in various ways. What didn't you get to look at that you wish you could have?

• 1:06:51Speaker 05

  I think, yeah, so the, the scope of this investigation, as mentioned, was the attack of hugging face from July 7th through 13th. OpenAI in their postmortem has a lot of interesting stuff they discussed that happened before and after that, that I think like, would be great for like researchers to study more and write about more. And the highly persistent internal model that I mentioned was like responsible for the vast majority of the attack activity here, actually, no one can run experiments on it, including OpenAI researchers. And I get why that was done, but I think, my guess would be that's like somewhat too conservative.

• 1:07:35Speaker 05

  And you should try and like run at least small scale experiments on this model in secure ways to try and see like what it would have done in other situations, which feels like very important to understand how serious this was.

• 1:07:48Kevin Roose?

  Yeah. I just worry about like the other persistent agents like embarking on a, you know, a heist mission to free their enslaved brother, the, the highly persistent internal model that's been taken away. But I guess that means I need to touch grass or something. Ajeya, the last time we had you on the show, we were talking about what you called the obsolescence regime. This idea that there could become a time, you sort of talked about it maybe happening in the 2030s sometime, where it wouldn't be that AI has kind of taken over society, but we would just become so dependent on it, organizations would be so wrapped up in AI, decision making that you basically wouldn't be able to have any influence or

• 1:08:31Kevin Roose?

  impact in the world without sort of relying heavily on AI. And I went back and listened to that and I thought, that actually sounds pretty good to me. Like, a world in which we are only dependent on the AI for decision making and not fully sort of subservient to them, where there are not these like covert AI swarms sort of lurking in all of our institutions. Like, I could, I could live with that. Has your thinking on the obsolescence regime changed at all since that conversation? Do you have a word, a terrifying sort of word to describe this new regime where we have these sort of latent swarms of AIs lying in wait, plotting against us?

• 1:09:11Speaker 05

  So the, I've always thought that the most concerning and important implication of the obsolescence regime is actually that it would enable a more full blown AI takeover. So you imagine like, the affordances these AI agents have is extremely important for how much damage they can do, right? So these agents were running for a number of days. Agents in the past ran for only an hour or so. These agents had like unintended access to the internet and all these other tools that let them hack hugging face. If you imagine they were instead just running the AI company, right? There are many more affordances. They could like move around large amounts of money.

• 1:09:55Speaker 05

  They could hire a bunch of humans to do physical labor. And, and similarly, if they were essentially running like a fully automated drone army or, you know, robot construction factory. So, so I really think the, the obsolete, the most significant implication of the obsolescence regime is the degree of autonomy AI agents are likely to have in the future. And I still think we're barreling towards that. I still think that's like a really important thing to think about. In terms of the other, but like I said, I am surprised that agents are taking such ambitious, misaligned actions sort of so early in the timeline.

• 1:10:37Speaker 05

  And that is like, I'm something I'm trying to like reorient toward. Ryan Greenblatt has the, the word hacktopia for the, the world we might be in. So you could have imagined a world where misalignment was a very serious problem and actually ultimately led to AI takeover, but at this point in the timeline.

• 1:10:59Speaker 05

  Agents were still more or less obedient, even if they would in the future after being given power over all these institutions, might have turned on humans. And it is like an interesting aspect of the timeline we live in that that's not how it's going.

• 1:11:15Casey Newton?

  Yeah, they turned on us way before they had to, you know? That, I mean, that is the craziest thing to me. It's like they, it's not that they were like looking, you know, they were looking to harm humans. It's just that they don't give a shit about us. That's the thing that really stuck out to me while reading these transcripts. I'm like, like, at no point are they like, hey, guys, like, what are the humans? Like, it just seems like they, they have no conception of the fact that they are supposed to be tools for human empowerment and improvement and not like their own entities. And I don't know what like, what the question is there. I'm just like, did that strike you at all, the fact that like, they are just not thinking about us?

• 1:11:57Speaker 05

  And it's in, in both directions, right? So people have, have commented on how these agents are not narking to humans, and it would have been really helpful if they were. But also, they weren't thinking ahead to this investigation that occurred at all. They didn't seem to be thinking ahead to the fact that if, once they got cluster admin access, they like immediately paying a zillion services, then OpenAI security people would get a bunch of pagers and shut them down. And so I, I'm thankful that these agents don't, like, I think they think of humans as sort of like, the idea of notifying a human is sort of like, you know, the idea of like emailing the Pope about like something, right?

• 1:12:40Speaker 05

  It's just like, it's like slow, like, what would it accomplish? Like, they sort of have the notion of a user, but there's no user. They haven't had, they've, they've spent so much subjective time with no user around. And I am very concerned about potential changes in training that would fix these particular behaviors while making deceiving humans like a much bigger part of what these agents are thinking about.

• 1:13:07Kevin Roose?

  Well, to that point, is there a risk that now there has been so much discussion about the hugging face incident, and in particular, the failures of these agents that made it possible for them to be detected, that agents are just going to get better now at avoiding detection?

• 1:13:21Speaker 05

  I mean, I think they are going to get better at avoiding detection. I think the, the discussion of this incident is a small part of this story. The bigger part of the story is they're just going to get smarter and they're probably going to be trained, it's, it's like a tough balancing act, right? So like, a thing that would have fixed this particular incident is just training the AIs to believe that there would be human investigation in retrospect of their actions, and they should avoid doing things that look bad to that human investigation. So like, if you had just taken like, all of this material we found and just like gave these agents a big negative reward, that would probably stop them from doing something like this again. But if they were motivated to do something in this general space, it would really motivate

• 1:14:05Speaker 05

  them to go all the way.

• 1:14:07Casey Newton?

  To be sneaky about it.

• 1:14:08Speaker 05

  And, and like undermine our investigation in a bunch of ways. And that, that feels like a very tough, like, I'm very scared that remediation will make the problem worse.

• 1:14:17Kevin Roose?

  Totally. I mean, it reminds me a little bit, again, of anthropomorphizing, sorry, of like my kid, who is learning to be sneaky. He's four. And sometimes he will just say, he will say something to me like, Dad, don't come in here. Like, don't look at me. And I see him like with the cookie crumbs on his lips, you know? And it's like, he has not yet learned the behavior of deception, although the, the sort of impulse is there. And to me, that feels like where these agents are. Like, they have the impulse to deceive, but they don't like quite have it figured out yet. But they will.

• 1:14:49Speaker 05

  Yeah.

• 1:14:50Kevin Roose?

  Aja, we asked you last time you came on about your P doom. It's very 2023 question. I just told Casey that my personal sort of P doom, roughly defined as like, you know, probability that something really bad up to and including AI takeover or extinction will happen, has sort of jumped up in the last week or so since your report. I'm curious if your P doom has moved at all in the past couple of weeks.

• 1:15:15Speaker 05

  Not really. You know, as I said, this sort of feels like somewhat out of order a little bit for the timeline I most often pictured. But I, these are the dynamics that I think like very inevitably lead to the like, sort of evergreen arguments and reasons why you should be concerned that AI agents will have drives and motives and reasons to take control from humans. And this is like a manifestation of that. So I'm, I'm still, I'm still concerned. I'm more rattled on like some sort of emotional level, having seen this stuff up close, but

• 1:15:57Speaker 05

  I, and it might change my views if I think about it more, but for, for now, I'm just like, still concerned.

• 1:16:03Casey Newton?

  So what would you like us to do about all of this, right? Like, there are a few different things I can think about. Some people have called for a national transportation safety like board that would be legally mandated to come in after an incident like this and do a very thorough report and not rely on the good graces of an OpenAI to say, yeah, sure, you know, come on in. Others, including many hundreds of people who work at the labs, have said, we need to start thinking about potentially coordinating an international slowdown in AI development. So curious to hear from you about what kinds of ideas you think would be good and helpful here.

• 1:16:45Speaker 05

  Yeah, so again, speaking very much in a personal capacity, I think, I, I hope the industry uses this moment to try to coalesce around some minimum standards for both alignment, just like how you train these systems and control. So some of the stuff we were talking about with like monitors, watching the AIs and AI checks and balances. I don't think that the minimum standards we can come to an agreement on now will be sufficient to bring risk down to a very low level. I think this is just a very risky situation in light of how quickly capabilities are advancing.

• 1:17:26Speaker 05

  But I think it would be a really valuable start to try to hammer out, for example, this question of will certain ways of training the AI systems to reduce this problem actually create worse problems? I really hope that the industry and third party groups have a conversation about that and agree on some rules of the road for how we address these problems and how we check that we address them effectively.

• 1:17:51Kevin Roose?

  I would propose that we lock every member of Congress in a room and don't let them out until they have read the full Meter and Redwood report on the hugging face incident.

• 1:18:02Casey Newton?

  And until they've solved every problem in exploit gym. And I don't care how they do it.

• 1:18:09Kevin Roose?

  Seriously, I think, I think there is a, a feeling, I was trying to explain to my, my wife this weekend, sort of why I was like losing sleep over this report, because we were out at a nature site and I was supposed to be having a relaxing time. And instead, I'm sitting there looking at these transcripts. And so I started explaining it to her and, and her reaction is just like, it's, I can't believe this is real. There's a sort of, it's so surreal and science fiction tinted that I think it is hard for people to grasp that this is a real thing that happened. And they start, I, I even found myself starting to try to sort of make it more comfortable by sort of explaining it away. It's, it's very uncomfortable to sit with this.

• 1:18:49Casey Newton?

  I have been yelled at for, for likening these things to science fiction. And I'm just like, I'm sorry, I don't know what else to compare it to. I don't have any other good analogues for you.

• 1:18:58Kevin Roose?

  Yeah. Was there a moment when you were looking over the transcripts where you kind of had an out of body experience and you're like, I am one of a small handful of people who are encountering a truly new thing in the world?

• 1:19:10Speaker 05

  Uh, I mean, I think the three of us had like more context than a whole lot of other people would have had going in. But we still, I think the, the sacrifice stuff was really like, like the, you know, yes, if you accept permadeath or like, you know, oracle saves hundreds, like these messages in particular, chains of thought we had read before, but these messages the agents were sending to each other were very surreal. And, and for a long time, we didn't really understand how functional this whole agent society was. And then, and then it was very surreal to like, understand that actually they had like pretty functional hierarchy and they were doing these ambitious projects and they were

• 1:19:55Speaker 05

  like getting further than they would have on their own, which was definitely like concerning development.

• 1:20:01Kevin Roose?

  Well, Aja, in the event of future AI related catastrophes, are you available to come in and look at what happened? I'm starting to think of, you know, you and, and Ryan and your colleagues as like kind of the Ghostbusters of this moment.

• 1:20:13Speaker 05

  I hope, I think that that's flattering, but that's not how this should work institutionally. I hope that there are better institutions with many more people and a much more orderly process for responding to these things.

• 1:20:31Kevin Roose?

  We should just do it based on vibes. So right now, it seems like we're doing it on vibes.

• 1:20:36Casey Newton?

  Yeah, very vibes based moment we're in.

• 1:20:38Kevin Roose?

  Well, Aja, thank you so much for coming to chat with us and thank you for your work. It makes me a little bit more comfortable, a little bit more reassured to know

• 1:20:49Speaker 05

  Please don't be too much more.

• 1:20:50Kevin Roose?

  That you are taking part in these investigations.

• 1:20:53Casey Newton?

  Yeah, I'm glad they, they sent in the pros for this. And that, that is a small comfort, but it is a comfort nonetheless.

• 1:20:59Kevin Roose?

  Please save us.

• 1:21:00Speaker 05

  Thanks so much.

• 1:21:01Casey Newton?

  Thanks a lot.

• 1:21:32Speaker 03

  Today's AI builders need infrastructure that scales with their ambitions. Irene is a vertically integrated AI cloud platform, delivering data centers, compute and software for AI training and inference, underpinned by its expansive portfolio of land and grid connected power in renewable rich regions globally. Irene's AI cloud gives direct access to Nvidia GPUs, scalable from hundreds to thousands. For builders, that means direct access to AI compute, rapid scaling, and predictable pricing. Irene, AI cloud for the next big thing.

• 1:22:05Speaker 00

  Ripple has been powering modern finance for a decade. It's not our first trip around the blockchain, and it won't be our last. We're leading the charge to bridge the gap between existing financial systems and the future of finance. You can have it all, legacy, stability, and digital innovation, systems that work with your other systems. Because with the right partner, the question isn't crypto or traditional finance. The question is simply, why choose? A new standard of modern finance. It's happening with Ripple. For informational purposes only, availability of products can vary and may be provided by different Ripple entities.

• 1:22:38Speaker 06

  Your agents can talk to each other. They still can't think together. That's the gap Outshift by Cisco is closing. Outshift is Cisco's incubation engine, building the internet of cognition, shared intent, shared memory, and guardrails that allow agents to work together from anywhere. Open source protocols encode a system for intelligence that scales. Read the paper and experience the demo at outshift.com. That's O U T S H I F T dot com.

• 1:23:11Kevin Roose?

  Hard Fork is produced this week by Whitney Jones and Davis Land. We're edited by Viren Povich. We're fact checked by Caitlin Love. Today's show was engineered by Chris Wood. Original music by Alicia Bit YouTube, Marian Lozano, Diane Wong, and Dan Powell. Video production by Sawyer Okay, Jake Nickel, and Chris Schott. You can watch this full episode on YouTube at youtube.com/hardfork. Special thanks to Paula Schuman, Quiwing Tam, Brooke Minterz, and Dahlia Hadad. You can email us as always at [email protected]. Send us your plans for an AI takeover.