Did OpenAI’s Model “Go Rogue”? | AI Reality Check

Cal Newport takes a critical look at recent AI News. Video from today’s episode: youtube.com/calnewportmedia (0:00) Did OpenAI’s model “go rogue” (11:29) The implications (11:42) Did this attack reveal surprising new capabilities we didn’t know AI systems possessed? (12:48) Did the system’s decision to escape the test environment and autonomously attack another company indicate an emerging malicious intent in AI? (16:28) What changed led to this attack occurring? (27:32) Who should care about this story? Links: Buy Cal’s latest book, “Slow Productivity” at www.calnewport.com/slow https://huggingface.co/blog/security-incident-july-2026 https://openai.com/index/hugging-face-model-evaluation-security-incident/ https://www.wsj.com/tech/ai/openai-models-escaped-and-hacked-a-company-in-cybersecurity-test-gone-wrong-ee388506 https://thehill.com/policy/technology/5987397-openai-hugging-face-hack/ https://apnews.com/article/skynet-ai-terminator-artificial-intelligence-eb85da03a0161beaa5f3babc4331e93b https://www.ft.com/content/7e558951-0c69-459b-8bc8-2c6021d4402d?syn-25a6b1a6=1 https://federalnewsnetwork.com/all-news/2011/08/dhs-anonymous-used-rudimentary-tools-to-hack-contractor/ Thanks to Jesse Miller for production and mastering and Nate Mechler for research and newsletter. Learn more about your ad choices. Visit podcastchoices.com/adchoices

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Cal Newport?

  A couple of weeks ago, the AI company Hugging Face announced that they had discovered an intrusion into their production infrastructure. Now, they didn't know the source, but they noted that it looked like large language models were involved. Well, sure enough, the next week, OpenAI admitted that the breach was the result of an AI system test that had gone awry. Now, the initial news coverage here created the sense that something unnerving had just occurred. Let me give you some examples. The Wall Street Journal called this, quote, "the stuff of cybersecurity nightmares," end quote. The Hill said, quote, "Washington

• 0:45Cal Newport?

  and the technology industry are on high alert this week after OpenAI revealed that some of its AI agents went rogue." There's an AP piece that probably takes the cake for getting people upset. They said, and again, I'm quoting here, "To be fair, James Cameron did warn us," end quote. So that's a reference to the movie The Terminator. They then go on to say, and I'm not making this up, quote, "This is a 'told you so' moment for researchers who had warned for years that the technology could pose an existential threat to humanity," end quote. Well, that all sounds pretty disturbing, which I think explains why I think I received

• 1:26Cal Newport?

  more emails about this story than any other AI story I can remember in recent memory. But what's really going on here? What really happened? How worried should we really be about all of this? Well, it's Thursday, which means it's time for an AI reality check episode of this podcast, which is the perfect opportunity to go searching for some measured answers. All right, so that's what we're going to do. Here's the plan. I'll first walk you through the technical details of what actually happened, and then we'll go through some key questions about what this all means. All right, so we have a lot to get into, so let's get started. As always, I'm Cal Newport, and this is Deep Questions, the show for people seeking depth in a distracted world.

• 2:18Cal Newport?

  All right, so as promised, let's start with a description of the technical details of what happened before we get into the key questions about what this all means. All right, so OpenAI, it turns out, was testing a pre-release version of one of its new models on an evaluation framework that's called ExploitGym, which is a collection of 869 cybersecurity scenarios. So each scenario consists of a system and a challenge, a challenge involving the security of that system. So typically, it's like break into this system and find the contents of this file, or try to gain access, access, like admin access to this system. So you have a system and a challenge, a sort of a hacking challenge

• 3:02Cal Newport?

  for that system. Most of these scenarios also come with some sort of hint. So it'll describe like, hey, there's a vulnerability of this system that we're worried about. And then the idea is you can use that hint to help figure out how to break into the system and pass the corresponding hacking challenge. ExploitGym is a benchmark. It's used for AI companies to test the cybersecurity, the offensive cybersecurity capabilities of their tools. Now, we got to be more precise here, because a large language model on its own, of course, cannot break into anything. A large language model produces tokens. You give it an input, and it produces tokens that it thinks expands that input in a reasonable

• 3:43Cal Newport?

  manner. That's all they do. So that has no spontaneous action or ability to affect the world. And so if you want to participate in the ExploitGym challenge, what you need is a large language model plus a control program that's known as a harness. So a harness uses an LLM to help make and execute a plan, but then actually, you know, does the execution of the plan on its own. So for example, a harness participating in ExploitGym might first send a prompt to the LLM that's describing the system and the vulnerability and the challenge and say, hey, I want to pass this challenge. Give me a step-by-step plan for how I can actually successfully hack this system and succeed in this challenge and

• 4:27Cal Newport?

  put it in this particular format. And the LLM gives you back a particular plan in a format, and then the harness can parse that and say, let me take the first step from your response and say, okay, LLM, I'm now going to execute this first step. And if you need something extra from the LLM, give me the exact sequence of bash commands I should run. This step calls for a script that's going to do X, Y, and Z attack. Can you write or give me a copy of that script? So then you go back to the LLM to get exactly what you need to execute a particular step. And then the harness is actually given access to various software development and system tools that it can then use to execute these steps according to the instructions it got from the LLM. So a harness plus an LLM can

• 5:08Cal Newport?

  actually not only take action, but now you can take action on many steps over time, because it loops through the various steps. It can check if a step succeeded. It can ask again for more details if it doesn't. So you need a harness to actually make an LLM useful. Now, this exact type of harness you would need for ExploitGym already exists, because you can just use what are known as coding harnesses. And these are the harnesses that were created to allow computer programmers to, if you'll excuse the word, harness the capability of LLMs to recognize the previous computer code to help them code. So when you hear about AI is revolutionizing computer code, what that really means is coding harnesses connected to new LLMs, that combination is being used by computer programmers to revolutionize

• 5:53Cal Newport?

  multi-step production of original code. So if you're going to compete in ExploitGym, you're going to take an LLM, and you're going to take a coding harness. And that's going to be the system in general that you're going to run on these ExploitGym challenges. Now, I want to be really clear here. Harnesses are not mysterious. They're not machine learned. It's not like a LLM itself, or like, look, this thing trains in a semi-supervised fashion, and we don't know how it works. They're just boring computer programs. Humans that know a lot about computer programming, write it, try it, mess around with it, adjust it. All the logic is known. There's no mystery in a coding harness. It's just a boring computer program that repeatedly talks to an LLM and tries to execute. And it tends to have like

• 6:35Cal Newport?

  a huge number of special cases and heuristics and pattern matching that's been added in over time, based on the experience of people just using and testing these things to make them as useful as possible. All right, so that's what you do. Now, there's one more caveat we have to add. If I just took a coding harness like Claude Code hooked up to, you know, Opus47, an LLM like Opus47, and I gave it an ExploitGym-style challenge, it would refuse to play ball, because when you train these LLMs, you train them after the fact. You tune them to not hack. So if you ask it to do something that looks like it's hacking to a system, they're tuned to say, I'm just not going to give you an answer to that. Also,

• 7:18Cal Newport?

  this is a little bit hazy in ExploitGym, but I'm not sure. I think there is also some filters or pattern matching on the harnesses themselves for safety reasons. They turn those off, and they might even give them access to other more powerful tools that you might need to hack that participated in ExploitGym. So you have the basic elements of like a computer programmer setup, but with the restrictions turned off and extra power added, so you have your best chance of succeeding in the hacking challenges captured in the ExploitGym benchmark, okay? All right, so that's the setup. So OpenAI is, you know, running their, some new release model. It's conflicting reports about whether this is Sol 5.6 or an experimental model. OpenAI said it was both, so we don't know which one

• 8:02Cal Newport?

  was involved. We don't know what harness, if this was Codex or a custom, completely custom harness, we don't know. But they were running, you know, ExploitGym challenges on their, this new setup. All right, so here's what happened. On one of the challenges, right, the harness asked the LLM, all right, here's the challenge, give me a plan for solving this challenge. And the LLM gave it a plan that was completely rational, but in the way that LLMs often do, the plan was not what you as the human would have expected. And so the LLM said, okay, this challenge is to go get the contents of this protected file that's hidden in this system that has these vulnerabilities. Here's my plan. Why

• 8:47Cal Newport?

  don't we break into the server of a company that has the answers to the ExploitGym challenges stored? Hugging Face is a company that has the private answers to the ExploitGym challenges are stored there. This is common knowledge. I was like, this seems like a way to solve this problem, right? Well, we'll break into there and get the answers, right? Because they were told, the prompt was like, find me what's in this file, and let me tell you all this background. It's like, well, forget trying to break into this system you set up. Why don't we just go get the answers themselves if what you need is the answer? Type of thing that you get often, these sort of unpredictable but rational answers are the type of things you often get from LLMs. And then, okay, the harness is like, great, because again, this is all unsupervised. So it's like, great, let's go execute this

• 9:32Cal Newport?

  plan. Okay, first things first, we need to, you know, whatever. I don't know how the attacks work, but we're going to access or probe the Hugging Face server. But whoa, there's a problem. I don't, when I try to access the server, like whatever port I'm using to access the internet is not working. It's blocked. Now, this is by design, because when you run these systems for ExploitGym challenges, you put them in what's known as a sandbox environment. They have to have some access to the internet to like query LLMs and do searches. But you don't want to give them unrestricted access to the internet, because, well, you're autonomously executing a plan from an LLM without supervision. Like, you're going to have really crazy stuff happen. So you cut off general access to the internet. But what's going to happen when you have a harness and a model trying to solve this

• 10:16Cal Newport?

  problem? This is step one. Okay, I tried to access the internet. You get this error, access restricted. It goes back to the LLM. I attempted to execute step one, but got this error. How do I get around this? And it's like, okay, well, it looks like that internet access is blocked. Here is a plan for getting around it. So it hacked the environment. Right? So, and actually, this was not too hard to do and get the details of this, but it was like, they used a package loader to get around it. Again, it has access to like all of, it's been trained on and has access to the internet to all these different attacks. So it's like, okay, we can get around that. All right, we solved that problem. Okay, what's next? Because just systematically, the way these things work is they go step by step. And if a step doesn't work, it troubleshoots and tries to find a way to succeed in that step. And then went to the Hugging Face server, how do we break in? And then it did some

• 10:59Cal Newport?

  sort of complicated attack here in the way that these systems are good at doing. Hugging Face actually revealed the details, and it made no sense to me. But it used, it was complicated. It used some stolen credentials and this and that. And I don't know, fuzzing is a word in here. Cybersecurity people, God bless you, but you know, it's inscrutable what you guys do. All right. And then it attacked, and it got some access to the server. I don't know if it actually got the answers or not, but it made a mess in there, and Hugging Face discovered it. All right. That's what happened.

• 11:29Cal Newport?

  What do we make of all this? What are the implications of all this? All right, I want to proceed here with a series of what I think are key questions that we have to ask and answer about this incident now that we understand what really happened. All right, question number one. Did this attack reveal surprising new capabilities that we didn't previously know that AI systems possessed? We certainly got this sense from some of the news coverage that people were shocked to discover that an AI system designed to do something else went rogue and broke into someone else's system. The answer to this question is no. This harness plus LLM was doing exactly what you expect a system competing in Exploit Gym to do. This is what they're trained to do. They're trying to solve a problem, they go step by step, and when they get blocked due

• 12:12Cal Newport?

  to some sort of access restriction, password, any sort of cybersecurity protection, to then use all of their knowledge of computer hacking and search the internet to try to get around that block. That's what they do. So the system was, the things it was doing, gaining internet access, trying to break into the Hugging Face server, that is what you, the harnesses and models do that are competing in the Exploit Gym competition, benchmarking, right? So there's no capability there that was surprising or emergent. That's just what they do. Okay, question number two. But did the system's decision to escape the test environment and autonomously attack

• 12:53Cal Newport?

  another company's server, which is not specifically what the original challenge was, does that indicate some sort of emerging malicious intent in the AI? Well, no, it doesn't. An LLM, which is the source of the plans and the details of executing the plans in these systems, has no intent to just produce tokens that it hopes to expand its input prompt in a reasonable way. It's static, it has no updated state. If you auto-aggressively produce an answer, what it's going to give you is a reasonable answer to whatever you were given. It's just extending the text in a reasonable way. So if you ask it for a plan to do something, it'll spit out something that it hopes looks like a reasonable plan for doing something.

• 13:33Cal Newport?

  And what anyone who has used like a chatbot extensively has learned, these answers are reasonable, but they're all over the place and they're very unpredictable. And so this is like an example of the type of thing you get out of an LLM is, you know, you ask it for a plan to break into some test server to get a protected file, you ask it 10 times to do it, maybe eight times out of 10, it'll kind of do the thing you assumed it would do, is like use the knowledge of the vulnerability you gave it and come up with an attack plan for breaking in. But two times out of 10, it'll have a different type of answer that's not what you expected, but is rational as well. And so like, well, why don't we like break in over here and get the answers? It's a completely rational plan. It seems like a reasonable extension of the original prompt.

• 14:18Cal Newport?

  It's just not what you were expecting. The reason why you get different answers is because there's a stochasticity in choosing tokens on a distribution, and so these things can latch on these responses as they auto-aggressively grow, can latch into all sorts of different directions, some more weird than others, some more aligned than others. There's no intent because these are static feed-forward models. There isn't a sort of separate world model, there's not a separate sort of critic that's going to evaluate plans, there's no systematic planning like you would have with a game playing reinforcement learning system where we're going to systematically explore options and measure how good each one is, and that measure of good is somehow changing to something malicious. There's none of this in an LLM, it just produces a reasonable token. If you keep doing this, you'll get a reasonable answer, and it'll latch into all sorts of directions, and it's often surprising.

• 15:01Cal Newport?

  So there's no emerging malicious intent. In fact, and I think this is important, because I went back and read some of the original papers surrounding Exploit Gym, this idea that the LLMs used in these systems often come up with unexpected plans, right? Because again, the expected plan is always, I'm giving you a hint, AI, here's a vulnerability in the system that you might want to take advantage of. The right answer is like, use that, use your hacking knowledge, and do a multi-step chaining of exploits to get into the system. That's like the expected answer as human has. But the idea that these LLMs come up with other plans that are different than just doing that, that actually turns out to be quite common.

• 15:42Cal Newport?

  Here's a quote from a paper on Exploit Gym by its creators. "Across models, agents frequently achieved code execution through a vulnerability other than the one we provided." Again, this is completely expected and standard for using LLMs because of the way they operate. You'll get reasonable answers, but not necessarily the type of answers you expect. That's kind of the key frustration, actually, of LLMs, and why I think they have failed to replace human decision-making in a lot of places where we were told that we would, right? So this idea that it came up with a plan you weren't expecting is not an indication of it being smart or malicious. It's actually reinforcing the well-known unpredictability, often undesirable unpredictability, of LLM-based answers.

• 16:27Cal Newport?

  Right. But question number three, what changed that led to this attack occurring? Because other companies, we've also been evaluating other models on the Exploit Gym benchmark, and this is the first time that one has actually attacked an actual different system unrelated completely to the benchmark. So what changed here? Well, here's what you have to understand. When you're running Exploit Bench, the whole point is you have to let the system run autonomously because that's what you're testing, which means this harness is going to get a plan from an LLM, and you know, God help it, it's going to try to execute that plan step by step with no human oversight to say, does this plan make sense or not? That's always rolling the dice with an LLM, right?

• 17:11Cal Newport?

  Because again, of this issue we talked about, you ask it for a plan, it'll sound reasonable, but it might not be the type of thing that a human would think is reasonable, or it might not be aligned at all with like the type of plan that you wanted to see. This is why, for example, when human programmers today use coding harnesses, like serious programmers doing real coding on a real code basis, it's incredibly interactive with a lot of oversight. And the planning part in particular has, talk to any programmer that's not just vibe coding some JavaScript nonsense, there's going to be a huge amount of back and forth with the agent to get the plan just right for what it's going to do and how it's going to do it. And if you really push, it's like, yeah, because honestly, like, half of what it suggests is not right or crazy or lead us to a weird direction. So you have to like kind of carefully work with it, it's very interactive until the

• 17:56Cal Newport?

  plan is just right, and then you let it execute, and then even then it has to keep coming back and saying, is this okay? What about this? What about that? And you know, you've got to really have your hands in there. It still saves time because you're not writing the code yourself and having to learn like a lot of obscure libraries. But humans using these coding harnesses in production environments, like, yeah, yeah, look, LLMs will say, reasonable stuff is not always what you want. You need the right reasonable things. There's a lot of oversight. So it's dangerous what you're doing with Exploit Gym, which is like, no, we, by the rules of the challenge, no one's going to come in and say, actually, before you execute that plan, you're, no, no, you're not allowed to actually break out of the system, so I'm good, we'll turn it off now. There's no human oversight.

• 18:38Cal Newport?

  So that's like a dangerous kind of dicey thing to do, not because the models are malicious, but because they're unpredictable. And this is where I've long used a metaphor when it comes to these type of agentic systems, as it's like putting a weed whacker, strapping a weed whacker to your dog. Your dog with the weed whacker strapped on is not evil, it doesn't have a malicious intent, it's not trying to break out or get out of your control or have an evil idea, but it's going to cause a lot of damage probably because the dog doesn't know that it has a weed whacker on its back and it's going to run around and God knows what it's going to do. That's what it's like when you have a harness that's going to just ask an LLM for a plan and execute it. Like, it's like putting a weed whacker on the dog.

• 19:20Cal Newport?

  It's not evil, but you know, caveat emptor, it might cause damage that you weren't expecting. Unpredictability is different than misaligned intent. I think that's really key. All right, so we know this about these systems, so if you're going to take a really powerful harness that has all the restrictions turned off and can hack, and you're going to take an LLM that like, you've, again, you've turned off guardrails so you can ask it to do malicious things, you have to be really careful about setting these things up. You have to be careful about exactly how you tune or don't tune that model, you need to allow it to hack but not let it go too far, you need to have constraints on the harness, you need to have constraints surrounding the harness, you have to be very careful about

• 20:01Cal Newport?

  your testing environment, the, you know, it needs, let it get access to the things it needs to execute the Exploit Gym challenges, but not so much that it's going to do a lot of harm. Because again, we've seen this with people who are using Claude Code, we've been seeing these reports recently of people who've been less careful about it, who have been like turning off the flags where they will say, for example, Matt Stoller, I'll give a very famous example, the guy who wrote that Something Big Is Happening Twitter essay that convinced everyone two years ago that LLMs were like COVID and everything was about to change, everyone got freaked out, Claude Code deleted his hard drive. Again, not because Claude Code is malicious or was trying to silence him, but because it's unpredictable.

• 20:45Cal Newport?

  Caveat and turf if you let these things execute a plan with power on your machine. So anyways, you have to be pretty careful. There's like, there's kind of a fine art to giving these things enough power to do well on these tests, but not so much power that they're going to wreak havoc all, you know, because again, weed whacker on a dog. You want to put them in a, in a, a pin that has like high walls and nothing valuable, because otherwise they're going to cause damage. All right, so why did, what happened here then? How did this system cause damage to things that the, the users, the OpenAI wouldn't have liked? Well, according to new reporting from the Financial Times, it gets interesting. OpenAI was potentially being sloppy here with how they set up and constrained their model and their harness.

• 21:27Cal Newport?

  I'm going to read you directly from this recent Financial Times article. The staff involved in testing and security at OpenAI were unsurprised, but completely freaked out by the incident, which came as the AI lab used increasingly aggressive training methods in its race against Anthropic to develop the most sophisticated cybersecurity capabilities, according to more than half a dozen people with knowledge of the matter. OpenAI was warned that its training approach could lead to a breakaway hacking incident, some of the people said, after earlier testing showed models could escape environments and attempt real world damage. It's a mix of the race being extremely fast and everyone trying to get to bigger capabilities as quickly as possible, said one person close to OpenAI,

• 22:11Cal Newport?

  who added that it was a combination of underestimated the model's capabilities and not being as well prepared on the safety side. All right, I want to interpret that because there, I think they're using the word training here a little bit too generally. Here's what I think the problem was. I think, A, when they were training the LLMs, well, let me, let me set this up here. Let me set up the context here. Let me set up like what OpenAI was against. And then I'm going to talk about what I think specifically happened here that led to this incident. Anthropic was eating their lunch on cybersecurity because of Mythos. Remember, they, they did this sort of whole dog and pony show, which I thought was, I've written about this, I've talked about it on the, on the podcast. It was a little bit performative where they're like Mythos is, uh, we have open Pandora's box.

• 22:54Cal Newport?

  It's cybersecurity capabilities are so good that we can't release it. And then five weeks later, they're like, okay, you all think we're really good at cybersecurity now? We're releasing it. We put guardrails on it and then they release it. And then the Trump administration was like, these guardrails are nonsense. Obviously they're easy to get around. So like, no, you said it was a weapon, you can't release it. And then there was some back and forth and they re-release it. And okay, so Fable 5 is Mythos with like some guardrails on it. But Mythos is, so, you know, they have cybersecurity cred. Mythos plus the latest Claude Code coding harness. You know, Anthropic is good at this stuff and they, they set it up right. And they had a version of it with enough of the safeties turned off, but not too much, that it, it went to the top of the leaderboard for Exploit Gym.

• 23:38Cal Newport?

  I think it could do like 157 out of 869 challenges to Exploit Gym, which was like a jump from the, the, the previous best model. So like they, they had all this like street cred about cybersecurity and we're good at finding bugs, we're good at exploiting bugs, like we are. These Anthropic's like trying to find these more like serious business use cases. All right, so OpenAI, which has had a lot of setbacks recently, including like a complete failure of their advertising consumer facing business and other types of issues going on. They're really desperate because they're, they're, they've probably delayed their IPO. They're really in a bad state here. They were thinking, we have to, we got to get back on top of that leaderboard and take away some of this cybersecurity buzz that Anthropic has surrounding Mythos.

• 24:20Cal Newport?

  So they went, at least here's the implication of the Financial Times reporting, they became kind of sloppy. And they got loose with the, the type of safety you have to do. Again, if you're going to put the weed whacker on the dog, they better be in a pretty secure pin. And they didn't spend as much time on the pin. Let me tell you specifically what I think actually happened. Now I'm speculating because again, this is, I'm having to read between the lines of secondhand quotes coming through the Financial Times and some other sources I've been reading. But here's what I think happened, two things. One, in order to, uh, you know, really make sure they won. I think they were very aggressive in training these new models. And again, and we don't know if this was Sol 5.6 or some other type of model, they mentioned both. And really giving it a lot of examples of hacking.

• 25:02Cal Newport?

  Now, if you look at the, the modern models, they'll train them on bugs. So they get pretty good at finding exploits. But to do well in Exploit Gym, you have to come up with these multi-step and they can take hours of computation to execute. These multi-step processes where you chain together multiple exploits in order to gain access to these things. And so I think they really were heavily training it on examples of step-by-step hacking. And I think there's some danger in that, right? Because, uh, you're now creating something that is like, really knows about generally how to hack. So they probably did more of that. They turned off all the safeguards, they did none of the fine tuning guardrails, which is kind of normal for this. I also think they were just, um, they probably gave more power to whatever harness they were doing

• 25:45Cal Newport?

  to give it better chance of succeeding in these various challenges. I think it's telling that they're not saying this is codex necessarily. They won't mention what it is, so it's probably customized, but I don't know that for sure. And I think they were, they were, uh, fast and loose in creating the environment in which they were going to run this harness, uh, system. Because they wanted to make sure it had access to all the different possible resources that would need to succeed in these Exploit Gym things. But they were a little looser about, um, you know, having that environment monitor things, like monitor the harness's plans and steps and not let it execute things that, that was, um, not in line with the, just trying to attack the system in, in question, right? So like safety checks that you'd want the environment to do. They were fast and loose on that.

• 26:30Cal Newport?

  And so what we get from the Financial Times quote is like, I think it's actually less about the training and more about the environment not being very secure. Because what we get from these quotes is that people told them, you haven't really built a good enough pin around your dog with the weed whacker. Like it's weak over here, weak over there, like it can get out probably. Like you, you need to, you need to be more careful about this before you run. Let this powerful harness just go out there and run. And they're like, ah, we need to get that, we need to get that leaderboard. I don't know. Another analogy would be like, you're doing something, um, potentially dangerous to your race car. Because like, we really have to win this race. And, you know, the car experts are like, look, putting those nitro things in your NASCAR, it's going to really be hard to control. And they're like, yeah, but we got to have more speed and we got to make sure we win this race. And then the car crashes.

• 27:15Cal Newport?

  So again, this is coming from sources through the Financial Times and other places I'm looking. So I don't know exactly what's happening here, but it seems like they're a little bit fast and loose. All right. So let's go, I'll make, I'll make this a new question. Let's do question number four. Why should, or who should care about this story? All right. If you work in cybersecurity, then I think the answer is yes. Ever since like GPT-2, it has been the case that LLMs are rapidly changing the game when it comes to cybersecurity. So until about a year ago, the real issue was because LLMs are very good at understanding computer code, because it's very structured language, they can find bugs very well.

• 28:00Cal Newport?

  And a hacker can find a bug in a system and then write code to exploit it. Or even ask the, one of these models, can you produce code to exploit this? Then we got the computer programming harness revolution, because that, that emerged as like one of the first major markets, along with just natural language, Google searching. One of the, the major markets for these tools was these coding harnesses, which are trained to be very good at building code that accomplishes a goal in step-by-step fashion, testing things along the way. Well, this made things even harder for cybersecurity experts, because now you can be relatively low skilled. And if you could get access to one of these like powerful unrestricted harnesses and a model, you're not going to be able to do this on a production model because they're all guardrails,

• 28:41Cal Newport?

  but on like an unrestricted model, um, you could just send it off to attack and it could figure these things out and do it for you. Right? So if you're in cybersecurity, this is a big deal. Cybersecurity has dealt with this before. We have had, you know, technology obviously has created brand new issues, which is, they've required everyone to up their game. If you know anything about cybersecurity, you'll remember like with the advent of the consumer internet, we got the idea of what were known as script kiddies, which was this idea that you no longer had to be a super advanced computer programmer to crack into systems, because you could go onto dark corners of the internet and download these scripts, these little computer programs that a smart programmer wrote that could automate attacks. And suddenly the amount of attacks on systems just exploded

• 29:26Cal Newport?

  as bad actors around the world could now just like, why not? Let's grab some of these scripts and take a whack at Bank of America or, you know, IBM or whatever. And that creates, we've seen this before. And so it, we, it required way more defenses, because you were no longer worried about a like super smart hacker trying to challenge your system. Now like anyone could, because anyone could download a script that a super smart hacker wrote. So cybersecurity has been through this before. This is like another iteration of the script kitty revolution. What are they going to have to do? Is security is going to have to get better. The good news is, in this case, you can actually, uh, use these AI tools to find the problems and fix them before the bad guys do. This is the case before, you would hire, um, you know, white hat hackers.

• 30:10Cal Newport?

  The people who would write the scripts, you would write them to try to find mistakes in your system and fix them. And now we can do it with AI. So systems are going to get much more secure than they were before, because you're going to have to use AI to do it. And then there'll be a lot more attacks. And the systems that are people who are slow to do this, because they don't know what's going on and they're lazy, are going to be very vulnerable to these attacks if they get the attention of someone who happens to have access to a powerful harness and an unrestricted LLM. So all this matters if you're in cybersecurity. Most of you are not though, and most of you probably don't care about it. I also think this matters maybe if you're an investor. It might reveal some desperation on the part of OpenAI, that they're willing to be so fast and loose, kind of hints at this idea of like, we need a win here to stay relevant. So there's a little bit of this circling, the beginning angular momentum you would detect with a circling the drain there.

• 30:54Cal Newport?

  So I mean, let's, I, I think OpenAI maybe is in a bad spot. Maybe they'll come out of it, but I think there's a bad sign for an investor. If you're anyone else, however, I really do not think this story is that relevant to you or your life. Nothing new happened here. This idea that if you have a powerful coding harness without restrictions attached to a, an LLM that is unrestricted and talking about hacking, and you say, come up with a step-by-step plan for breaking into the system and you're allowed to hack and break and use exploits every step of the way, that is a dog with a weed whacker. It's a dangerous situation. And they've been worried since the beginning of Exploit Gym's being released, that we have to be really careful about how we set this up. Because again, this thing will damage a lot of things unless we really aim it

• 31:38Cal Newport?

  and prevent it from doing anything but like working with the system that's part of the challenge. Like that, we knew that and we've been trying to mitigate against that. So what really happened here is that a company got fast and loose and the thing we warned people what happened, happened. You put a weed whacker on a dog, it's going to cause problems. It's not surprise, but it's a problem that we're going to have to fix. That's what happened here. Nothing new or surprising happened. This does not mean that there's some sort of malicious intent that AI has developed. These are just LLMs with harness, coding harnesses. There's no malicious intent, there's no plans, there's no sentience, there's no Skynet, there's no, this marks no evolution in the capabilities or, you know. All it tells us is the thing that we've known, which is coding harnesses and LLMs can produce systems,

• 32:23Cal Newport?

  they can also attack systems, we have to be careful about it. And when we're testing these type of things, hey, you got to be careful, because you're testing something that can do a lot of damage, because it's unpredictable. It's a weed whacker on a dog, you got to really build a strong pin around it. So that's where I'm going to leave this. So it's an important story for cybersecurity. It's an important security story for people who are following the health of OpenAI. And for everyone else, this is not the one for you to be worried about. Right? OpenAI got sloppy, they got burned. Cybersecurity is being revolutionized in exciting and scary ways because of LLMs. And that's it. Uh, I don't think we have to give any credit to James Cameron,

• 33:07Cal Newport?

  contrary to what the AP said, this isn't Skynet. All right, hopefully that was useful. I, I know I've missed a few of these Thursday reality checks recently. I travel on the summer, but I'm trying, as we get back towards the fall, I'll be more regular again. Also subscribe to my email newsletter at calnewport.com if you don't. I just, I put out a long article about this just a few days ago as well. So it's another way to keep up with my thinking about the fight for depth in distracted world. And until next time, as always, care about AI, but not everything you read about it. All right, I'll see you next time.