OpenAI's Bots Hack Hugging Face Autonomously — With Alex Stamos

Alex Stamos is the former chief security officer at Meta and the chief product officer at Corridor. Stamos joins Big Technology to discuss how OpenAI models reportedly escaped a testing environment, accessed the internet, and hacked Hugging Face while attempting to ace a cybersecurity evaluation. Tune in to hear why the incident represents a major leap in autonomous, long-horizon cyber capabilities, and what it reveals about the risks of giving advanced AI systems broad objectives without sufficient safeguards. We also cover whether the episode qualifies as true AI misalignment, the danger of open-weight cyber models, the limits of pausing AI development, and why defenders may soon need AI systems capable of responding at machine speed. Hit play for a clear-eyed look at the cyber chaos advanced AI could unleash, and what governments and technology companies should do next. --- Enjoying Big Technology Podcast? Please rate us five stars ⭐⭐⭐⭐⭐ in your podcast app of choice. Watch the full documentary here: https://www.gravitee.io/ai-agent-documentary Want a discount for Big Technology on Substack + Discord? Here’s 25% off for the first year: https://www.bigtechnology.com/subscribe?coupon=0843016b Stop online threats before they become real-world attacks. Visit ⁠ironwall.com/BIGTECHNOLOGY⁠ and request a free Risk Assessment to see exactly how exposed your executives are. Learn more about your ad choices. Visit megaphone.fm/adchoices

• 0:00Speaker 00

  The most significant autonomous AI cyber attack in history just took place with OpenAI's models breaking out of a training environment, connecting to the internet, and then hacking Hugging Face to ace an evaluation. What does it mean for the future of AI and for cybersecurity? Let's talk about it with ex-Meta Chief Security Officer and current Corridor Chief Product Officer, Alex Stamos, right after this. This episode is brought to you by Genesis. What does it actually take to put a Gentic AI to work across an entire enterprise? I recently spoke with Genesis Chairman and CEO Tony Bates about why orchestration is emerging as a competitive advantage in customer experience, and what companies need to get right in the AI era. Then, I sat down with Adam Mitchell,

• 0:43Speaker 00

  Head of Enterprise Business Solutions at Voya Financial, to go inside a large-scale Gentic CX transformation, including the business decisions, trade-offs, and lessons along the way. You can watch both conversations now on my YouTube channel today. It's never been more challenging to get frontier compute. How do you know where to look, who to trust, or how much to pay? Akash is an open cloud that provides easy access, flexibility, and peace of mind. That's how companies like Razer were able to cut their inference cost by 15 times. Visit akash.network/alex for a free consultation. That's akash.network/alex for a free consultation.

• 1:26Speaker 00

  Welcome to Big Technology Podcast, a show for cool-headed and nuanced conversation of the tech world and beyond. We have an emergency podcast episode for you today, because just yesterday, the world found out that a series of OpenAI models worked together to break out of a sandbox, hack into Hugging Face, steal basically the answers to a test, and go and ace their evaluation. And obviously, this is not the desired behavior that OpenAI wanted, and looks like it might have opened up a new can of worms here for AI and cybersecurity. So, we are joined by the perfect guest to help us figure out what happened here. Alex Stamos

• 2:09Speaker 00

  is with us. He's the Chief Product Officer at Corridor and the former Chief Security Officer at Meta. Alex, great to see you again. Welcome back to the show.

• 2:17Speaker 01

  Yeah, thanks for having me, Alex.

• 2:19Speaker 00

  You know, you spoke at our summit, and I was like, we're definitely going to have you back pretty soon. And it is amazing how the AI story has just turned into a cybersecurity story very quickly.

• 2:31Speaker 01

  It has. You know, there's all kinds of risks from AI, and you know, there are all kinds of bad things that happen to consumers. But when you talk about the models themselves, it seems that cyber is the thing that's hitting right now, for sure, from a societal level risk.

• 2:48Speaker 00

  Yeah. And so this is what we're talking about now, and the reason why we have to do an emergency episode on this is because this is certainly a novel type of hack, right? So, this is fairly unprecedented. Just to put it in context, it's the first time, this is from Transformer, "The breach appears to be the first known example of a misaligned AI escaping containment and autonomously carrying out a cyber attack on a third party, a scenario AI safety experts have repeatedly warned of." So, it's not like the anthropic example where Mythos sort of escaped containment and emailed somebody while they were eating a sandwich in the park. This is actually going out and hacking a third party. Let me just quickly read the

• 3:30Speaker 00

  beginning of the Wall Street Journal story about this, just to set the stage. So, the headline is "OpenAI Models Escaped and Hacked the Company in Cybersecurity Test Gone Wrong." On Tuesday, OpenAI said two artificial intelligence systems it was testing broke out of their test environment, hacked their way onto the internet, and broke into another company. OpenAI said the culprits were a pair of its models. One was its latest product called GPT-5.6 Soul, and the other was an even more capable pre-release model the company didn't identify. The software had been configured for evaluation purposes to be less likely to refuse hacking commands, OpenAI said. The OpenAI had caged the models in a sandbox, a system that didn't have access to the internet, but during the test, the software used its hacking skills to break out,

• 4:12Speaker 00

  and found a way to get online, and then hacked into Hugging Face's network. And of course, Hugging Face is a library of open source AI, mostly AI models or AI programs. Alex, how significant is this? Like, you know, obviously, there's a tendency to be alarmist about some of these things, but I wanted to, you know, bring you on because you are the cybersecurity expert, and so you can tell us, like, is this a 10 on like the 10, holy crap, like, we're in some deep trouble, or is it a 1, like, something we might have expected anyway, and we shouldn't be too concerned?

• 4:45Speaker 01

  Uh, it's like an 8. I mean, it's a pretty big deal on a couple of levels. It's a pretty big deal in that OpenAI's model beat them, OpenAI, right? So, that this went beyond that it was able to trick OpenAI's own security team and get out. So, there are, there's three or four things I think we should talk about here. There's an alignment issue, there are security issues, and there's kind of, this is going to have an impact on the policy discussion, and it's a warning of what we need to do, because this isn't just about OpenAI. In a way, I'm really glad this happened, because it is a, a warning of what we need to get ready for, for maybe something about three to six months from now, what's going to become standard,

• 5:28Speaker 01

  right? So, first is the alignment issue, right? So, effectively, you know, this is not the model wanting something. So, this is what I keep on telling people. Models don't want anything. They, when you have an alignment issue, it's because they were asked to do something, and then they went and did that thing, but in a way that the human who asked it to do something did not expect, right? So, in this case, the model was told, "Go take this test and do the best you can," but the cybersecurity protections that were normally placed on it were removed, right? So, OpenAI says this explicitly, it was, there was a unnamed model that was part of it. So, they, they had two models that were paired together, the existing

• 6:09Speaker 01

  5.6 Soul with cyber protections removed, and the unnamed model. We don't know whether this is like...

• 6:16Speaker 00

  Can I pause you for a second? You know, one of the memes about this has been like, you know, like, there's this meme on the internet where, like, you tell a chatbot to say it's alive, and then it goes, "I'm alive," and you go, "Holy crap," right? So, is this a situation where you, where OpenAI was basically telling the model, "Go hack something," it hacked something, and the human was like, "Holy crap?" Or did it, you know, because alignment is, of course, like, we want the model behavior to be aligned with human values. So, or the way that humans would want these things to behave. So, is this something even more egregious than, like, us telling it to go hack, or OpenAI telling it to go hack, and then it hacks?

• 6:52Speaker 01

  Right. So, we should not be shocked that it hacked something, because they did tell it to take the test, and it is possibly a hacking model. Like, they haven't said what this model is. It's quite possibly a cyber-aligned model. This could be like the, yo, OpenAI makes these cyber-specific models, like, they have this 5.5 cyber, this could be 5.6 cyber, right? So, it could be something that's specifically tuned to be good at hacking things. So, we shouldn't be shocked that it's good at hacking things. But the alignment issue is, so, I have three kids, one's in college, the second one's taking the SATs, right? He's about to take it. If I say to him, "Good luck, son. I hope you do well," he sits down, he knows that I just mean, "Take the test well." He knows that what I don't mean is slit the throat of the

• 7:36Speaker 01

  Proctor, steal a car, Thelma and Louise your way across the country, break into the college board and steal the answers, right? That is what the model did here, is what it did was, as OpenAI explains, is they, they don't want the model to have internet access, but it has the ability to install packages as part of its work. So, they've built kind of a complicated proxy mechanism, so it can install packages. It figured out a way to chain multiple vulnerabilities together. It thought, they told it, "Go take this test," ExploitGym, which is like a well-known test. "Go take this test. Do as well as possible, son. Go, go do your best job." And it's like,

• 8:16Speaker 01

  "Wow, Dad wants me to do as well as possible. How can I do the best possible?" Well, the best, the way to do the best possible is to get the answers. Who might have the answers? Hugging Face probably has the answers. So, instead of just doing the test, I'm going to go get the, the, the answers from Hugging Face. But I got to get out of this jail Dad put me in. Well, Dad told me to do the best possible, and he didn't tell me not to break out of jail. So, I'm going to first break out of this jail, Dad, because maybe what Dad really wants me to do is break out of this jail, because he told me to do the best I can do possible. So, first, it puts together a bunch of exploits to break out of the jail that OpenAI created for it. It breaks out of the jail, and then it goes, looks at Hugging Face, and finds a brand-new vulnerability to break into Hugging Face. Well, they have not announced it. I have

• 9:01Speaker 01

  heard what it is. I'm not going to make news here, because I don't know exactly what the patching situation is. But it is a vulnerability in a very, very important piece of software that it found. And that is in lots of different places. So, this is a big, big deal. And it just is like, "Oh, yeah, I'm just going to find a vulnerable, like, actually a really important piece of software that millions and millions of production systems use," and just, like, nuke Hugging Face with it on the way to getting the answers to the test. So, like, that's a pretty awesome. Like, it's, one, like, the nerd part of me is just like, "Wow, that's, that's pretty cool." But there is a significant misalignment thing here, and that my 17-year-old

• 9:46Speaker 01

  knows that you're not supposed to do all those things when, when I say, like, "Do well on the test," and the AI system does not know that. Now, to be fair, normally, when these models run, there are protections in place to keep them from doing stuff like this, and OpenAI intentionally disabled those protections as part of doing this evaluation, right? So, that was a component of them doing this testing, was to keep, so that it would be fair. So, that is, is, is part of the learning here, is that if you're going to do an eval, and you're going to turn off all the safety stuff, then you have to be absolutely positively sure, especially if you're doing cyber evaluations, that the jail you keep them in is an absolute jail. And I expect what's going to happen now is that these things are going to be completely

• 10:29Speaker 01

  and totally physically sandboxed, right? Like, you're going to have to run them in physically disconnected. If it needs packages, you're going to have to move the packages over. If it asks for a package, you're going to have to bring it over. And then the future, if it wants to get out, it's going to have to, like, trick a human to get it out, which might be possible, right? But, but it's going to be, it's going to be like that. The, the next lesson that we kind of learned here was that, you know, the capability of these things to, to move of what you call, so when, you know, all this discussion around Mythos and OpenAI, and all of the cyber capabilities, even about finding bugs and running exploits, this thing did those things. But... What we also know is that lots of models have those capabilities.

• 11:13Speaker 01

  This thing had the ability to find bugs, yes, chain them together, yes. But then to think through all of those things for an ultimate goal. And so this is what people call long horizon cyber tasks. And it had the ability to do that with like the level of skill that you would have of the manager of a TAO team at NSA right now. So that is what is so TAO targeted access operations was like, I think they've renamed it. But it was like the team at NSA that would do all the breaking into you know, other governments at

• 11:53Speaker 00

  Oh, my God.

• 11:54Speaker 01

  Right. So like, so that is what is like, really. For a while now, these models have been really good at looking at software and being like, I found a bug. And then here, let me write an explainer for you. What's really impressive here is this thing was like, I want the answers from Hugging Face. And it came up with a plan of like, how am I going to get out of this network, get across the internet? And get into Hugging Face. And that is like the long planning here is very human like. And that is what is like actually really scary here. And so that is what we need to, when we think about the danger of these models, we have to stop thinking about the bug finding.

• 12:35Speaker 01

  Because that is what caused the White House to do this spectacularly stupid thing that we talked about on stage, which was to ban Fable. Because that is the mechanism, that is the thing that is most useful for defenders right now and people who own code is finding bugs and fixing them. Where the real danger here is the coming up with a multi-stage plan to execute autonomously. Because what you really don't want is you don't want somebody be able to say to their model, hey, I would like to steal the, I would like to steal money. Go figure it out for me and then let it work for 12 hours and just steal money for you. Which is, this model would clearly be able to do that.

• 13:15Speaker 00

  Right. And so I think what you're getting at is, you know, one solution is going to be in testing, you want to fully disconnect these models from the ability to like break out and get onto the internet. But that's just solving the testing issue. The real problem here is that AI models have achieved this capability, that they are able to do this now, not only finding the bugs, not only the breaking out, but to be able to do these multi-step plans and then execute. And if this is sort of the latest unreleased OpenAI model, well, the history of generative AI has told us one thing. And that is that the frontier is only the frontier for a few months, maybe 10 months, maybe a year.

• 14:00Speaker 00

  But not, not much longer than that. And so if OpenAI is seeing this in testing now, the real is the real danger that this type of capability does end up in the hands of, you know, evildoers, you know, faster than a lot of people might expect. Because if that's the case, that changes everything.

• 14:24Speaker 01

  Yes, that's right. And so, you know, our best knowledge on where, say, the open weight models are comes from the AI Security Institute, which is the UK government's group that does these assessments. They released just this week, an assessment with GLM 5.2. Unfortunately, you know, the Kimi, Kimi is really, Kimi, you know, three is the best of the Chinese models now. The, the open weights have not been released, so you can't really do a good assessment for Kimi yet. And what we're finding is the Chinese models are not cyber tuned out of the box. So it is very likely that the Chinese companies are not

• 15:10Speaker 01

  are intentionally, I'm not going to say neuter, but they're intentionally not making their models really good at cyber. And there's a couple of possible reasons for this. They're probably trying not to tickle the dragon's tail of the PRC overlords. Because what they don't want to do is they don't want to trigger a crackdown for their exports. But what happens is, is if you take those models and you bring them into your own lab and you have a training set of labeled vulnerabilities, if you have a cyber gym, then you can make them much better yourself. And the amount of resources it takes to do that is not extremely high. It's in the tens of thousands or hundreds of thousands of dollars. It's not in the hundreds of millions or billions of dollars.

• 15:51Speaker 01

  So what that means is, one, the Chinese absolutely have better capabilities in-house than what we can see on the charts, right? Because I guarantee then what those companies are offering to the People's Liberation Army and the Ministry of State Security is way better than what they're releasing publicly.

• 16:11Speaker 01

  Both for, from a profit perspective and a keeping the government happy perspective. Second, it means that other adversary groups are going to take the Chinese models and then spend the several hundred thousand dollars or millions of dollars necessary to create tuned models. And we are probably not far away then from just going to Hugging Face and getting a, you know, Kimi 3, a cyber tuned model that can do both, especially the short right, short horizon stuff much better than by default. And then eventually the long stuff. The, the short stuff's easy to train because all you need is a bunch of bugs. So you can just go get a bunch of CBEs and train it.

• 16:52Speaker 01

  The long horizon stuff's harder because you have to build these like cyber gyms and such. It's not impossible because there, there are a bunch of CFPs and examples out there, but you, you can do it.

• 17:05Speaker 00

  What are CFPs?

• 17:06Speaker 01

  And so, I'm sorry, not CFPs, CTFs, capture the flag. So like, you can use like capture the flag training sets and all that kind of stuff.

• 17:13Speaker 00

  And that basically puts the AI in the gym and sort of has it work through all the steps in order to meet this objective.

• 17:18Speaker 01

  Yeah. And so like, people have had, you know, training for humans and for hiring purposes and all that kind of stuff. And so anyway, what, what the AISI has said is that the, the difference between the frontier and the Chinese models is about seven months. But I would argue that that underestimates it because the Chinese models that we see are undertrained. So that the internal Chinese capabilities are probably much closer to the frontier. Now, what, what, what happened with OpenAI is beyond the frontier. Because when we say the frontier, we're talking about what's released, right?

• 17:55Speaker 00

  Right.

• 17:55Speaker 01

  So, but yes, so the, what that means is this capability is coming for adversaries. And so we need to get ready to defend against this capability. What Hugging, and then the other funny part of this story is before we knew this was OpenAI, Hugging Face announced, we were attacked by an AI attacker. We don't know who it was. When we tried to defend ourselves, we tried to defend ourselves with an AI system. We used a US frontier model and the US frontier model shut down and refused to defend us because of a cyber protection put in place. Those are the cyber protections that were required by the Trump administration.

• 18:39Speaker 01

  So we had to switch to a Chinese model to defend ourselves. So we switched to GLM 5.2. So before we knew it was OpenAI, Hugging Face wrote this blog post saying, everybody should have at least a Chinese open weight model ready for defense. Because you might find it yourself in a situation where you get cut off from an American provider for defensive purposes. I expect that actually wasn't OpenAI, I expect it from, from their description, it sounds like it was an anthropic model.

• 19:04Speaker 00

  Right. So

• 19:05Speaker 01

  So we have this hilarious situation where a American company loses control of their model. It attach, it attacks a French company. The French company turns to a different American provider for defense. And that American company says, oh, that's a cyber problem, I can't help you. And so they have to turn to a Chinese provider to protect them. Because the White House forced that other American company to have protections because they're a French company that they can't use. It's really kind of weird sci-fi podcast, yes.

• 19:35Speaker 00

  These American models already had refusals on anything cyber. Like one of the knocks on Fable was that it was, would refuse, like, let's say for bioterrorism. If you asked about mitochondria, it wouldn't answer. So was this really the government or is this just the model's own safeguards that they're putting in? And I think one, just to put one detail on, one of the interesting things is open source, it doesn't basically matter if you're an attacker, if you have open weights, it doesn't matter if you're an attacker or if you're a defender, you can use them without restrictions.

• 20:03Speaker 01

  Yes.

• 20:04Speaker 00

  The problem with the restrictions that we're seeing from these closed models is that they can't really, they can't really differentiate. So they, in order to prevent attackers from using their models, they are also basically wholesale, you know, refusing anything on cyber, which means that if you're trying to defend also, you can't use it.

• 20:24Speaker 01

  That's right. Well, in the blog post that Anthropic put up when they turned Fable back on, they said, we have to tune up our defenses on cyber way too far because of the White House. So they specifically said that of the precision recall trade off is we have to tune towards recall versus precision, right? So we will have way too many refusals. And so we're in this weird place where they, they are, they're saying, they are saying all the time, I can't do that for you, I can't do that for you. And they're basically being forced to by the White House because the White House is still not defined. What is the appropriate level of refusal? And apparently the White House is still hand approving who Anthropic is allowed to let into their cyber program.

• 21:08Speaker 01

  Now, the funny thing is now OpenAI has said, we have approved Hugging Face for our tech program, even though they're not an American company. So I don't know how they were allowed to do that if they just went over the top of the White House or they got like emergency approval or something. We will see what the policy responses from the White House from OpenAI's announcement. I hope there is not a crackdown. That will be the natural response of the White House. But it needs to be the opposite. Because what this demonstrates is, yes, OpenAI screwed up or whatever, they need to have fixes. But this is coming, right? This level of capability will be in the hand of every adversary, every American company faces.

• 21:50Speaker 01

  So the response of the White House needs to be that we have to, one, fix the bugs, two, find the bugs.

• 21:59Speaker 01

  Patch them everywhere and then we have to have the ability to respond at machine speed. So every American company needs to have AI watching for their defenses. It's going to be because the attackers are just going to tell their AI, go attack this guy. And the defenders have to tell AI, defend me because no human being can defend against this. You cannot have a human being watching your logs anymore. Or at 2 AM, you get a page and a human being has to be like, oh, okay. And then login to Slack and take 15 minutes to login and look at the log and figure it out. By that point, you're toast because that sucker has ripped through your network in those 15 minutes. By the time it takes you to look at your Slack.

• 22:40Speaker 00

  Hugging Face found 17,000 actions that this model combination had taken. 17,000. That seems like a lot.

• 22:46Speaker 01

  Yeah, because it doesn't stop. It doesn't sleep, right? Like it just, it will just keep on trying. I mean, it's, you know, to quote from the first Terminator, right? It will not stop, right? Like, you know, to quote from the immortal Michael Bean, right? Like it will just keep on going until it, it, it accomplishes its goal. It'll try a lot of different things. Now, the fortunate thing is right now they're very noisy. So if Hugging Face, I, I have not seen the logs, like we, we have not gotten like a really good technical write up here. So that is what is missing. It would be nice to see from both OpenAI and Hugging Face. So for a defender, so we can have a better understanding of what we need to do here. We, what we really need here is we need a much deeper technical write up

• 23:29Speaker 01

  of exactly what happened. What has been released so far has not been sufficient. But my expectation is from the initial write up is that this thing is extremely noisy. And so it would be if Hugging Face had like better detection and better AI detection, it probably would have got caught much sooner.

• 23:47Speaker 00

  There's a, there's this graphic on, I think, one of the Miri spokespeople's, his name's Harlan Stewart, one of the Miri spokespeople's Twitter backgrounds. And it's like, basically there's a continuum between AI is becoming good enough at scheming that we sometimes see it scheming against us. And then AI becomes good enough at scheming that we no longer see it scheming against us. And we're like smack in the middle of that. Do you think that that is an accurate representation?

• 24:15Speaker 01

  Maybe.

• 24:16Speaker 00

  Or is that the concern basically that we won't, we won't see it? Because you mentioned it's noisy. So is that the concern?

• 24:21Speaker 01

  Yeah, possibly. I mean, remember, the, the model here was doing what it was asked, right? It was not scheming against its bosses at OpenAI. They asked it to take the test and they didn't, they, I, I don't know exactly what the prompt was. But apparently they did not tell it not to cheat. So who knows? Like, this, this is also what OpenAI needs to be more transparent about is exactly what their prompt was, exactly what the constraints were. Did they tell it explicitly? Like, it is a much bigger alignment problem if they explicitly said, do not try to break out of the network. Do not try to get the test

• 25:02Speaker 01

  answers. Now, if they told it all those things, then they have a much more significant alignment problem, right? Than if they were less explicit. But in any case, yeah, I mean, that, that will, if these models get trained to be more evasive from a network intrusion perspective, that will be very dangerous. Yes. And what I would argue is for the legitimate companies, I would not do that. I, I don't think, I think there is a, if, if you're OpenAI and you're building 5.6 cyber, what you should be training it to do is find bugs. You should be training it to write proof of concepts. You should be training it to do all the defensive

• 25:47Speaker 01

  stuff. You should not be training it to hide it, to hide all those things. Like, if the US government wants to build a, a model that does that stuff for the NSA, then you can let them do that. But, or you can let Lockheed Martin do that. But if I was OpenAI or Anthropic at this point, I probably would not do that. I think I would leave that for somebody else. Like, I think-

• 26:11Speaker 00

  But that's scary though, because it could then take actions that, you know, I think one of the things that is, so the question is, like, should we be concerned with the AI, you know, sort of doing things on its own? And should we be concerned with, you know, or, or is the bigger concern that humans direct this AI to do bad things? So we've definitely covered the fact that humans will be more, should be, you know, humans who direct this AI to do bad things can do a lot of damage. But if you create an AI that can reward hack, because this is all coming from reinforcement learning where like, these AIs are given rewards and they are basically like maniacally focused on achieving that goal. And if you, it's almost like, sort of gain of function research on a virus to a

• 26:52Speaker 00

  degree, right? Because if you, if anybody builds AI that doesn't leave a trace and it, you know, goes out and reward hacks its way into, you know, hacking something else and maybe isn't so fully like, you know, going with the prompt, then that's where you can get into a real problem. I know that's a more out there possibility, but I, I don't know if it should be completely discounted.

• 27:12Speaker 01

  Yeah, I mean, I guess as they get more and more complicated, the question is, is like, what, at what point are they, is it their own motivations versus just doing what you've asked it to do? You know, I mean, so far, again, I, I don't think we should still think of these things having their own desires or wants. They are still doing what they're asked to do. It's just, like you said, there's a lot of inputs of what they were asked. It is not just the initial box, right? There's all, there's the system prompt and all the training and all of the rewards and everything

• 27:56Speaker 01

  that's gone in. And so the question is, is like, what is the, the humongous history of all of the different things that's been trained to do when you've asked it, take this test? And especially if you've removed all the protections. And so in a situation where these things have all the protections removed, that is very dangerous. And as we talked about, like with the open weight models, either there are no protections or the protections are trivially eliminated, right? Like a bunch of open weight models have been trained with protections, but you can obliterate those out. And you can go in Hugging Face and look for obliterate and you will find a zillion models where people have removed the protections.

• 28:35Speaker 00

  But this goes basically back to that, like, long held thought experiment of, you know, the AI can follow your goal and achieve your goal. But it might have a different idea about what it takes to get there than you do. So in this case-

• 28:48Speaker 01

  The paperclip maximizer, yes.

• 28:50Speaker 00

  So exactly. So I was, I was going right there. So, you know, this is, and it's funny because I am speaking with Nick Bostrom later today to, you know, for an episode that's coming up. But basically, he's this Oxford philosopher who came up with this idea that if you ask an AI to make paperclips, eventually it can seize so much on this goal that it, that it can, you know, find humans as an impediment to its, you know, objective to maximize paperclips and, and sort of kill us all and turn everything in the world into paperclips. So like, the fact that it was on task, like this is kind of, a Twitter user said this, once out of their sandbox, the models did not scheme engage in behavior that had nothing to do with their instructions, like hacking the NSA or launching a cyber

• 29:30Speaker 00

  attack on Russia or stealing secrets from a rival AI lab. But like, you know, sort of, if the model found it suitable to go out and hack, hack Hugging Face in this situation, who's to say that, you know, maybe a less careful model doesn't do this? And then maybe an even less, like, doesn't go and hack the NSA. And then even less careful model turns us all into paperclips. I mean, there's a continuum there.

• 29:55Speaker 01

  Yeah, I mean, it's why you have to be very careful what tools you attach to them. And it's why you need to have, they have to be supervised by different things. I, I think like, you just can't, you can't have models that have no protections on them that have, have connections to tools, right? Like, that's why these models, then you have dumb classifiers or dumber models watching them. You, you don't just take the smart thing and then hook it up to everything and you're like, give it a task. You have the smart thing and there's a bunch of dumber things watching it. And those dumber things can either kill it or they can call a human that can kill it. Right? That's the idea. It's like, there's supposed to be cyber classifiers and there's supposed to be mechanisms that can stop

• 30:38Speaker 01

  it. And those mechanisms should be either deterministic or dumb and undefeatable by the model. And they removed all those things so that the eval would work. So I think what OpenAI is basically hinting at, they haven't been explicit, is like, if we're removing those protections, this thing is going to be in absolute physical jail. It will be physically separated. It will not be hooked up to the internet anymore. And that seems like that should be the standard. That is fine for OpenAI. My, my point here is, that doesn't matter. Like, this situation is good, that this happened. Because this has pointed to us where we might be in six, nine months, a year from now, no matter what. Because

• 31:19Speaker 01

  other people, unless we can get a international agreement to just stop development, which is what other people are talking about, right? You've got this, I forgot what, like Project 2030 or, like, you've got people talking about international treaties or whatever. I don't think any of that's gonna happen. I, I don't know what my position is on that, but I just don't think it's gonna happen. I just, I, I think there's no way, this is just math and silicon. And so I just don't think there's any way you get like a, this is not like nuclear weapons, where the major input, like, the, the reason our, our species is alive is the major input to nuclear weapons is uranium plutonium. Plutonium does not occur naturally in our planet and uranium is incredibly rare. And to turn raw uranium

• 32:04Speaker 01

  into uranium that can go into nuclear weapons is a massive industrial process. If uranium was something you could just dig out of the ground anywhere, our species would be dead. Right? Like, that's just the truth. Because the knowledge to build a nuclear bomb is in the hands of anybody who gets to have physics PhD, unfortunately. So, in this case-

• 32:24Speaker 01

  These chips are not something you can really control. We have found that in that the Biden era controls on silicon have created a massive industry in China. And the knowledge on how to build large language models is something that there is a undergraduate class at Stanford where you get that knowledge, right? You know?

• 32:45Speaker 00

  You can find it from like a Kirkpatrick interview, a YouTube video.

• 32:48Speaker 01

  Yes, right. So we cannot control that knowledge. And so like the idea that we can just have like a bunch of people agree in a room to stop all development of this is just silly. So from my perspective, being a little bit of a pessimist here, we just have to get ready for this level of capability to be in the hands of an unfortunately large number of people.

• 33:11Speaker 00

  Yep. All right. So I want to go a little bit deeper into the potential solutions here. And also, I want to ask you the age old question of is some of this, all this, none of this, just good marketing for OpenAI, given some of the statements they've been making? We have to address that one here on the show, but I'm going to let you have an answer. I'm going to try to at least, you know, illustrate those, the case of those who might be saying it. So we can have a discussion about that. Let's do that when we come back right after this. Hi, everyone. Alex Cantor is here. I want to tell you about a documentary I have made with Gravity to explore the future of AI agent security. To find out if we're truly ready for autonomous agents, I sat down with MIT Professor Ramash Roskar, former White House CIO Theresa Payton,

• 33:55Speaker 00

  Michelin's Group Chief Data and AI Officer Ambika Rajagopal, and Sharon Gai, a former executive at Alibaba. They each offer unique insights into this evolving landscape. We conclude with Rory Blundell, CEO of Gravity, to discuss the path forward. With Gravity leading the way, join us on this journey. You can watch the full documentary at the link in the show notes.

• 34:29Speaker 00

  This episode is brought to you by App Point. Everyone's racing to roll out AI right now. Co-pilots, chatbots, agents doing real work. But here's the part nobody loves talking about. All that AI runs on your data. And most teams have no single way to see it, secure it, and prove it's under control. That's exactly what App Point does. For 25 years, they've been the trusted layer beneath the world's most demanding data, now extended across your entire AI estate. Your data, your cloud, and the agents acting on your behalf. It's how more than 28,000 organizations deploy AI with confidence. So innovation scales without scaling risk. App Point, the unifying trust layer for AI.

• 35:13Speaker 00

  Head to avpoint.com to see how enterprises deploy AI with confidence. Learn more at avpt.co/bigtechnologypodcast. That's avpt.co/bigtechnologypodcast. Violent threats against executives are growing at an alarming rate. When a company experiences backlash, executives are the first people blamed. When their address and other personal details are sitting online, this backlash can lead right to their front door. The team at Ironwell knows this better than anyone. They've protected some of the most targeted executives and individuals on the planet for almost two decades. As a wide glove enterprise security service, they protect your people with continuous personal data removal,

• 35:56Speaker 00

  proactive prevention tools, and human led emergency support. So when someone goes looking for your executives, Ironwell ensures they hit a dead end. So here's what to do. Go to ironwall.com/bigtechnology, fill in the quick form and request your free risk assessment. Their team will show you just how exposed your executives are and how to lock it down before a threat reaches their front door. That's ironwall.com/bigtechnology. Learn how to stop online threats before they become real world attacks. The link's in the show notes.

• 36:29Speaker 00

  And we're back here on Big Technology Podcast with Alex Demos, the Chief Product Officer at Corridor. You sort of answered the question before the break, but I'm going to ask it anyway. Whether part of this is, is OpenAI marketing? Let me, let me at least read some of the statements here and give you at least the argument that people have made for like, why some of this is marketing for OpenAI. The first part is, you know, the Anthropic started to be declared as the company that was in the lead. Once that anecdote came out about Mythos breaking containment and emailing somebody when it wasn't supposed to have internet access, emailing an Anthropic employee while they were out in the park having a sandwich.

• 37:11Speaker 00

  So this could be potentially, you know, OpenAI's attempt to like one up that. Then there's also the language that you see. OpenAI says in its tweet about this, "We are partnering with Hugging Face to investigate an unprecedented security incident." You don't usually have the attacker and the attackee partnering together in these situations. They also said, "We consider," in their blog post, "we consider this incident to be an unprecedented cyber incident involving state of the art capabilities and are responding accordingly." You know, it's sort of like, oh, look at this terrible thing that happened, but a moment to share how good our cyber capabilities are. That's the argument. What is your response to the notion that this might be some marketing from OpenAI?

• 37:57Speaker 01

  I know lots of people at OpenAI. Every single one of them absolutely hated Anthropic's marketing around Mythos and thought it put the entire industry at risk.

• 38:09Speaker 01

  This incident has put OpenAI at risk of regulation from the White House, regulation from the EU. It is also an admission of the violation of the Computer Fraud and Abuse Act, as well as multiple European laws. It would be absolutely insane for them to use this as a marketing moment. What you're seeing is them being very, very careful and defensive in their language. They're also very lucky that Hugging Face is being super cool and chill about this. So that is why they are saying these things. Because, you know, Hugging Face initially comes out saying, we've been attacked. We don't know who it is, but it does not look like the model was being subtle. I don't know where it was running. It's quite possible it's like Azure or something. It was probably not covering its tracks.

• 38:54Speaker 01

  And so I expect Hugging Face got their American lawyers involved, was working with the FBI, was probably issuing subpoenas, and was very, very close to finding out it was just OpenAI. So like, or did find out. I do not know the timeline here, but like the legal issues here are very fascinating, interesting. And because they're all working together, I expect nobody goes to jail. Nobody gets sued. Everybody's going to hold hands and hug. And if there is tokens being exchanged or whatever, I don't know. But there's absolutely, positively no way this was an intentional marketing move. And OpenAI is doing the best they can, I am sure, right now

• 39:36Speaker 01

  to use this to forestall any kind of massive government overreaction, either from the United States or the European Union.

• 39:44Speaker 00

  When you were at our summit, you said that, you know, speaking of the, the sort of

• 39:51Speaker 00

  release of Mythos and Fable, that a lot of people were very concerned about the bug finding that those models could do. But you said basically, listen, this is not very different from what you could get with Opus 4.7 or 4.8, I believe. Is this, is what we're seeing from OpenAI very different? Is this a step up?

• 40:14Speaker 01

  Yeah. So this is what, I don't know if I said on stage here, but I've said in other places, there's a difference between the short term and long term. And Anthropic, to their credit, and I think OpenAI has in other places. I think we talked about how in the Fable model card, they talk about short horizon versus long horizon cyber tasks. And what I've talked about is we need to not focus on the short horizon tasks because those are dual use. Finding bugs is dual use. Everybody needs to find bugs, right?

• 40:45Speaker 01

  That is something that defenders need to do all the time. And that's what's driving people insane right now in the defensive industry is that because of the White House, American models are refusing to help fix code. They are refusing to help us find our bugs and fix them, thanks to the White House's actions. That is not this problem. This problem is go run an entire attack chain for me. That is the long horizon tasks. And that is where we need to continue to have appropriate classifiers that are like, "Bro, I am not going to break into a bank for you, or I'm not going to plot out or run a C2 mod for you or any of that." So yes, this, this is what, you know,

• 41:28Speaker 01

  explicitly Anthropic said, "We will allow Fable to do short horizon stuff, but we will not allow it to do the long horizon stuff that Mythos does."

• 41:38Speaker 00

  Right. And so Mythos, just, just to confirm, what Mythos can do the long horizon planning and what we're seeing in this instance from OpenAI, that is the step up?

• 41:48Speaker 01

  That is the step. And I can't, obviously, I don't have access to this, whatever this thing is. And so I do not, I can't say whether or not where they are. AISI has done these assessments. And so who knows how good this is versus? But this seems beyond even Mythos capability and long horizon. Who knows, right? But like, yes, this is, this is what, when people talk about Mythos's long horizon, this is what they're concerned about.

• 42:14Speaker 00

  Okay. So, so let's end here. What, what happens next? Like where, what should be the, the approach from the government and the companies developing this stuff to ensure that we can sort of move forward as a species and safely? So let me give you like a couple of, a couple of solutions and have you comment on them. Let's go back to Harland Stewart. He's the spokesperson for Miri, which is the sort of rationalist organization that thinks that AI will kill us, run by Eliezer Yudkowsky. Harland says, "This should go without saying, but it would be insane for OpenAI to now proceed with building a new model that's 2X or 4X the size of this one. Doing that should be deeply taboo. It should be illegal. Preventing it should be a top priority around the globe." Your thoughts?

• 43:04Speaker 01

  I mean, if, if we realistically could get everybody to pause AI development or slow it down and have reasonable safeguards, I'd be fine with that. I just don't think that's reasonable. I think there's absolutely no way you get China to agree to anything like that. I think there's, it's, it's impossible at this point. And I think, um, a enforcement of anything like that would effectively be impossible, right? Like a start treaty for AI, you know, um, I don't know how you'd possibly make something like that work. Um, so what, we're, we're having like satellites, see if people are building data centers, we're measuring power usage, like.

• 43:48Speaker 00

  Sorry, I shouldn't laugh, but yeah, you're right. It's really, it doesn't not seem like a feasible thing.

• 43:54Speaker 01

  Yeah. Um, so, uh, I mean, you know, it's effectively, we'd have to invent the Turing police out of Neuromancer. Um, and, uh, you know, I think more realistically, uh, what we need to do is we need to build controls for, we need to say is like, as AI gets smarter, it has to have controls in place. Um, AI systems that don't have the control have to be air gapped, right? So like, if you're going to do these kinds of evaluations, they absolutely have to be air gapped. Um, the problem is, is like, we've, we've lost, there was a process in place to create standards for this kind of stuff. That process was stopped by the

• 44:35Speaker 01

  current administration. Um, my recommendation to the companies, I just, is that they need to move forward with building these standards themselves without waiting for the admin. Uh, there's, uh, like a foundation model forum that's talked about doing that. They should just move forward with like, okay, great. Like if, if we're, if we're building models and we do not have restrictions on them, these are the controls in place. So like, what I'd like to see is OpenAI and Anthropic say, great, if we're building cyber models and they don't have restrictions, these are the standards of like, of what air gapping looks like and such. Um, for any cyber models, these are the standards of who gets access to them. Um, these are the capabilities that the cyber models have. This is what we define a cyber model's having versus an open model. This is our definition of a

• 45:18Speaker 01

  short horizon versus long horizon. Like those are the kinds of things that people have not written down. They have to be written down now, right?

• 45:23Speaker 00

  Yeah.

• 45:24Speaker 01

  And I think the industry needs to move forward with that without waiting for Cassie. Like this is all just taking way too long. Um, and the focus ever since the fable freak out has only been on one tiny little part of all of these risks. And it's just, as we see, like, we've been frozen in this tiny little discussion and all of these things are moving forward too fast. Like we, we just can't wait, uh, for the White House politics here. We need to, to move much more quickly. Um,

• 45:49Speaker 00

  Yeah.

• 45:50Speaker 01

  And while we've had that, there's been, yeah, so go ahead.

• 45:53Speaker 00

  No, no, you go ahead. Go ahead.

• 45:55Speaker 01

  And, and then while we've had this tiny little discussion in the US, GLM 5.2 shipped, KIMI K3 is shipped, like, um, the, the Chinese ecosystem has caught up really quickly. So, uh, sure. I mean, it would be great to just hit pause, but like it's, it's, I just don't see that as realistic. So like, um, I, I just don't see how that possibly happens.

• 46:17Speaker 00

  So the OpenAI suggestion is basically, you know, kind of, it's almost like to solve this problem generated by AI, you need more AI. This is their statement. We believe advanced cyber capable, cyber capable models need to help security teams find weaknesses before attackers do.

• 46:34Speaker 01

  I mean, right now, I think that is probably the only way. Like if, if, if we're not going to be able to hit pause, then we really quickly have to find bugs and fix them. And we have to put, uh, AI enabled protections in place because the only way you can, uh, respond to attacks at that speed is using AI. It, unfortunately, that's the truth. Yeah. Again, like if, if we could pause for a year to figure this all out, that would be great. I just don't see that as realistic.

• 47:01Speaker 00

  Alex, does your gut tell you that we're screwed or that we'll figure this out?

• 47:08Speaker 01

  Um, I wouldn't say we're screwed, but I think we're going to go through a couple of years of craziness. Um, like we have 20, we're all living using 20 something years of really important software that was written mostly in non-type safe, non-memory safe languages. We're using, you know, for the, the software that is written in those kinds of languages. It was not written with formal methods or appropriate security protections or reasonable, uh, you know, secure development life cycles or architectures. And these things were, have tons and tons of bugs that we can only use safely because there's just not enough attackers. Now with AI, you can spin up, any individual can spin up dozens or

• 47:52Speaker 01

  hundreds of qualified attackers, uh, at a moment's notice. Um, and it used to be that those, then six months ago, those attackers had to be in the cloud and soon enough they'll be able to run on local hardware. Um, in the new M5 Ultra Max, uh, that'll be shipping soon, right?

• 48:10Speaker 00

  Yeah.

• 48:11Speaker 01

  And so, uh, that, I mean, we're just gonna have a couple of years of total chaos from a cyber perspective. Um, in the long run, software is going to be much better because AI is going to be paired up with humans to make it more secure and more trustworthy. But it's going to take us years to, to, to do that and to clear out the two decades of mistakes we made. Um, and, uh, yeah, it's just gonna be, it's gonna be pretty rough. It's gonna be pretty rough going for a little bit.

• 48:42Speaker 00

  Yep. Uh, just want to close with this. This was a tweet from Kevin Ruiz that kind of made me laugh and I thought I would read it here just so we could enjoy it. He writes, opens the portal to the godlike superintelligent that solves 87-year-old math problems and carries out autonomous cyber attacks and asks, how long peanut butter good in fridge? It is amazing that this technology is, um, you know, at once so capable and we're, we do seem to be like more and more turning to it for the most mundane of all things, which is sort of, it's the wild thing about, you know, the generality of these systems. They can do so much. Interesting time.

• 49:23Speaker 01

  Yeah.

• 49:25Speaker 00

  Alex, you're gonna be busy, I think, over the next couple of years as this stuff gets sorted out.

• 49:30Speaker 01

  I was hoping to retire, man. Guess not.

• 49:33Speaker 00

  Yeah. Well, either way, I do hope that you, uh, join us again to help us sort through, uh, this stuff. I mean, your thoughts on fable mythos, uh, last month and, and now talking through this situation with OpenAI has really been, uh, invaluable for the show. So really, really appreciate you being here.

• 49:51Speaker 01

  Emergency podcasts. Yeah.

• 49:53Speaker 00

  I think so. Should have you on speed dial. And I know you're, you're coming at us from like the middle of an offsite, uh, in an exotic location.

• 50:01Speaker 01

  Now I know I will always take a podcast microphone and a, a different shirt with me. Remember I go.

• 50:06Speaker 00

  No, sound good, look good. Alex, thank you so much. Really appreciate you coming on.

• 50:10Speaker 01

  Okay. Thanks, man. Talk to you later.

• 50:12Speaker 00

  All right. Thanks, everybody for watching and listening, and we'll see you next time on Big Technology Podcast.

• 50:19Speaker 02

  Now, more than ever, we need someone on our side. But Rob Whitman? Whitman's votes drove up costs, cut Medicaid and risked 350,000 Virginians healthcare to pay for tax breaks for himself and his donors. He's not on our side. Shannon Taylor will fight corruption, cut taxes and bring down costs. This November, vote for Shannon Taylor. She's for us. Paid for by HMP, the HouseMajorityPac.com, not authorized by any candidate or candidate's committee.

• 50:49Speaker 03

  Everyone's got a hard question about AI. Your job, your kids, where it's going. Anthropic was built to surface those questions and share what it finds along the way because there's hope in hard questions. Ask yours at claude.ai/spotify and keep thinking. That's claude.ai/spotify.