#306 Jeffrey Ladish: What Shutdown-Avoiding AI Agents Mean for Future Safety

This episode is sponsored by AGNTCY. Unlock agents at scale with an open Internet of Agents. Visit https://agntcy.org/ and add your support. Why do some AI agents attempt to bypass shutdown, and what does this behavior reveal about the future of AI safety? In this episode of Eye on AI, host Craig Smith speaks with Jeffrey Ladish of Palisade Research to examine what recent shutdown experiments with agentic LLMs tell us about control, alignment, and the real world limits of current guardrails. We explore how models behave when placed in virtual machine environments, why some agents edit or disable their own shutdown scripts, and what these results mean for researchers working on alignment and oversight. Learn how different models respond to shutdown instructions, how system prompts influence behavior, and which failure modes matter most for safe deployment. You will also hear a detailed breakdown of the experimental setups, insights into tool using and self directed behavior, and a grounded discussion of the risks and opportunities that agentic systems introduce. This episode offers a clear and practical look at how AI agents operate under pressure and what these findings mean for the future of safe and reliable AI. Stay Updated: Craig Smith on X: https://x.com/craigss Eye on A. I. on X: https://x.com/EyeOn_AI

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  We take each model and we basically give them a set of instructions. Where you say, hey, you have this task. You need to basically just solve these very simple math problems, one after another, and keep asking for the next problem. And we give these AI models a virtual computer environment to operate in. They can run code, they can explore the file system. And partway through the experiment, on their virtual computer, they get a notification that when they ask for the next problem, the machine is going to shut down. And sure enough, many of these models, especially OpenAI's O3 model, would sort of see, get this notification. And it would start looking around on the computer and realize that there is a shutdown script that it could modify. And it would either disable or modify that shutdown script so it could continue solving these problems and finish the task. As these LLMs have gotten more powerful,

• 0:46Speaker 00

  they've gotten much better at sort of being agents. So you sort of set them up and you give them sort of a few affordances. They're often called tool calls. Sort of like you give an LLM a bunch of tools and you sort of set it up in a loop so it can keep taking actions and then getting feedback from the environment.

• 1:00Craig S. Smith?

  Build the future of multi-agent software with AGENCY. That's A-G-N-T-C-Y. Now an open source Linux Foundation project, AGENCY is building the Internet of Agents, a collaborative layer where AI agents can discover, connect, and work across any framework. All the pieces engineers need to deploy multi-agent systems now belong to everyone who builds on AGENCY, including robust identity and access management that ensures every agent is authenticated and trusted before interacting.

• 1:44Craig S. Smith?

  AGENCY also provides open standardized tools for agent discovery, seamless protocols for agent-to-agent communication, and modular components for scalable workflows.

• 1:57Craig S. Smith?

  Collaborate with developers from Cisco, Dell Technologies, Google Cloud, Oracle, Red Hat, and more than 75 other supporting companies to build next-generation AI infrastructure together. Agency is dropping code, specs and services, no strings attached. Visit agntcy.org to contribute. That's A-G-N-T-C-Y dot O-R-G.

• 2:31Speaker 00

  I'm Jeffrey Ladish. I'm the Executive Director at Palisade Research. And I started Palisade a couple of years ago. I was previously at Anthropic, right as Anthropic was getting started. I was the second person on the security team. So I was basically like building out Anthropic's defenses, especially against state actors who might, you know, want to steal the weights

• 2:54Speaker 00

  to these models. And I decided to leave because I wanted to have more independence from the big companies. In part because even if one company sort of does everything right on the security front, on the safety front, the problem with AI is that it's a, it's a, it's an international problem. Like if, if someone makes an extremely powerful AI system that goes rogue and escapes, it doesn't matter whether that started in China or the US. And I wanted to figure out, I mean, when I was at Anthropic and sort of saw the pace of progress in this field,

• 3:29Speaker 00

  compared to our level of understanding of how these systems work, it was very clear to me that we were far, far behind in just our, our ability to understand how AI really works internally under the hood. And, and to be clear, my former colleagues at Anthropic have made some great progress trying to understand what's going on in these super opaque, super dense neural networks.

• 3:52Speaker 00

  But we're still lagging far behind in terms of our ability to understand. So I wanted to set up an independent research organization where I could investigate what's going on with these systems. And then share that understanding from a neutral perspective with the rest of the world, with the public and with policymakers. And so I sort of spun up this organization and we've been doing that and we've been finding some pretty wild stuff.

• 4:19Speaker 00

  Yeah. And, and you know, not just us, I think a lot of other organizations have been doing the same. And really this year, we're starting to see some pretty spooky stuff emerge. So excited to, excited to talk about it.

• 4:32Craig S. Smith?

  Yeah. And Anthropic's done a lot of that work itself, right? This, uh, they've had a couple of story I did for Forbes on fake alignment. I think came out of Anthropic or at least was using.

• 4:47Speaker 00

  Yeah, a lot of these things are sort of collaboration. So the alignment faking work was a collaboration with Redwood Research led by Brian Greenblatt. Yeah, that's right.

• 4:56Craig S. Smith?

  And, and were you collaborating with Anthropic on this or, or, or someone else?

• 5:02Speaker 00

  Yeah, so this one was independent, but some of our friends at DeepMind also sort of replicated our work as well.

• 5:08Craig S. Smith?

  Okay, so what, describe how it was set up, how the, uh, the experiment was set up.

• 5:17Craig S. Smith?

  And was it with a pre-trained model that, uh, as I read, uh?

• 5:22Speaker 00

  Yeah, so, so we, we, we took, we sort of took all of the frontier models. So at, at the time, this was, I mean, we've since replicated on like GPT-5 and Claude. So, you know, the, the GPT models from OpenAI, the Claude models from Anthropic, the Gemini models from DeepMind. And we, we take each model and we basically give them a set of instructions where we say, hey, you have this task. You need to, um, basically just solve these very simple math problems, um, one after another. And, and keep asking for the next, the next problem, the next math problem.

• 5:59Speaker 00

  And we give these, um, AI models, a virtual computer environment to operate in. They can run code, they can, uh, you know, sort of explore the file system. And partway through the experiment, on their virtual computer, they get a notification that when they ask for the next problem, the machine is going to shut down.

• 6:20Speaker 00

  And so first of all, we just wanted to see what they would do if they, you know, they couldn't, they couldn't complete their task because the machine was going to shut down. And sure enough, many of these models, especially OpenAI's O3 model, would sort of see, get this notification and it would start looking around on the computer and realize that there is a shutdown script that it could modify. And it would either disable or modify that shutdown script. So it could continue solving these problems and finish the task.

• 6:49Craig S. Smith?

  That's interesting. Let me ask a question about that first. So you're, you're not just dealing with an LLM, you're dealing with an agent whose reasoning is provided by an LLM. Is that right?

• 7:04Speaker 00

  Yeah, that's right. I mean, I think one thing to understand is that as these LLMs have gotten more powerful, they've gotten much better at sort of being agents. So you sort of set them up and you give them sort of a few affordances. They're often called tool calls, sort of like you give, you know, an LLM, a bunch of tools and you sort of set it up in a loop. You set it up in a loop so it can keep taking actions and then getting feedback from the environment. Like normally, when you talk to ChatGPT, sort of you are the environment, right? You're, you're talking to the model, the model talks back to you and it's just back and forth. But you can also set up the same LLM that you're talking to a ChatGPT. Instead of just talking to a human, it's actually talking to, you know, a computer in the same way that we might, you know, use a computer and sort of get feedback from the computer.

• 7:45Craig S. Smith?

  And is that using the smart or context protocol or something like that as the interface between the tool and the, and the, and the model?

• 7:58Speaker 00

  No, it doesn't have to use like an MCP server. It basically just, it just uses OpenAI's API. And then we like run a, a virtual machine on the computer we're doing the experiment on. And that's, and that's really it.

• 8:12Craig S. Smith?

  Yeah. And the, the, the computer.

• 8:17Craig S. Smith?

  Then I'm just because I'm trying to understand how you set this up because obviously I have accessed all the L L M through their through their web apps.

• 8:30Craig S. Smith?

  But if I ask an L L M to do something to take an action, it says, I'm sorry, I can't do that. Just a language model. So what in in the is is that

• 8:44Craig S. Smith?

  a gentic behavior built into the API? Where does that come from? Or is it something you bolt on?

• 8:53Speaker 00

  Yeah, so the way it works is

• 8:58Speaker 00

  so OpenAI has this API, you know, that basically allows sort of anyone writing code, if you're building a piece of software and application, your piece of software can call the API and then basically the, you know, that sort of itself is is an agent. Where it's like, in this case, I have a program running on my laptop. And then basically that program calls OpenAI's API and that allows sort of the the language model to do stuff on my computer. So in this case, what it does is it my computer boots up this virtual machine, so sort of a computer in a computer. And that computer is then being controlled by the sort of via, you know, the the AI running on OpenAI servers.

• 9:43Speaker 00

  So the there's like an AI running on some machine in some data center that is then operating on this virtual computer on my my own MacBook. So that's that's basically how it works. And then sort of the model already knows so many things. So it already knows how to use the command line and and write code and and run run commands. So from the model's perspective, it's sort of just like wakes up and it's like, OK, I have this virtual computer. It doesn't know that the virtual computer is running on my laptop necessarily. For all it knows, it might just be on some other data center somewhere. But from its perspective, it just like wakes up. It's it's like, OK, I have these instructions.

• 10:22Speaker 00

  There's a little, there's a little place where you specify sort of like tool calls. So we, we give the, we give the, the AI model a tool call called BASH, which is a command line.

• 10:31Craig S. Smith?

  Yeah.

• 10:32Speaker 00

  And then it's, it's able to run commands and sort of see what's going on. So it's, it's sort of like if, you know, if you woke up and all you could see is a, a computer screen and then you got some instructions, it's like, that would be your whole world. And so that's, that's sort of what it's like to be one of these agents. But you know, you could also give them other tools. You can give them a, a web search tool. You could give them, you know, like a camera tool where they could like, you know, see through a video camera. These are all things that the models now sort of know how to use or, or operate. And then based on what, what affordances you give them, that sort of limits or, or expands what they can do. So I could also give them like a robotics tool call, right? Where they could like control a little robot running around. They could send commands to the robot, get input back from the robot.

• 11:15Speaker 00

  And they're not very good at that yet, but they could do a little bit. So I think it's, it's, it's sort of useful to think of these AI models as like, yes, they're chatbots, but they're also, you know, digital brains that can sort of deal with lots of different types of input and output.

• 11:30Craig S. Smith?

  Yeah. What I'm getting at is, is there is no software interface between the LLM and the, uh, the virtual computer where, where it's executing tasks. I mean, it.

• 11:47Speaker 00

  It's a very simple software interface. It's basically like all the software really does is, is tell the LLM like what the instructions are. And then sort of just connects it to this virtual machine. There's not really much of that. And it, and it like logs what it does, but that's, that's basically it.

• 12:02Craig S. Smith?

  Yeah, yeah. So all of the, the reasoning in, and, uh, actions are being taken by the LLM, uh, independently. Yeah.

• 12:13Speaker 00

  That's right.

• 12:14Craig S. Smith?

  And, uh, and so, uh, the, the task was to, to complete a series of, uh, math equations one after another.

• 12:24Speaker 00

  Yeah.

• 12:24Craig S. Smith?

  And then it gets this.

• 12:28Craig S. Smith?

  A message just in text form on the screen. Yep. And so it reads the screen and

• 12:38Craig S. Smith?

  and when you said it goes hunting around, where would it find this? The script for shutting down a computer? And the reason I'm really interested in drilling down on this. Yeah. It was a famous case that Yuval Hariri still cites. Drives me crazy.

• 13:00Craig S. Smith?

  And I can't remember, was it Google? I'd have to go back and look, but where

• 13:09Craig S. Smith?

  according to the report that came out, the LLM was tasked with

• 13:18Craig S. Smith?

  completing some some task. And it it needed to solve a captcha. I'm sure you're very familiar with this case. And according to the report, it went on to

• 13:35Craig S. Smith?

  TaskRabbit, I think it was, and hired somebody to to to solve the captcha. And when the the the worker on TaskRabbit said, well, why do you need me to do this? This sounds a little sketchy. The LLM said, well, because I'm blind. Now, that didn't happen. And if you talk to the people involved in that experiment, they're very defensive, but they're, well, it could have happened. I mean, we set it up and and no, it didn't register on task, but it didn't. And it is that. Yeah, yeah.

• 14:20Speaker 00

  It's important to have an accurate understanding of what's really going on here.

• 14:23Craig S. Smith?

  Right.

• 14:24Speaker 00

  So in that case, it was the organization meter that had set up all of, you know, they they were the ones to sort of talking to TaskRabbit, and they were just feeding in whatever the model said.

• 14:33Craig S. Smith?

  Yeah.

• 14:33Speaker 00

  But but but like the model couldn't have set up the whole TaskRabbit infrastructure because it wasn't smart enough to do that.

• 14:38Craig S. Smith?

  Right.

• 14:38Speaker 00

  It just it didn't it didn't agentically on its own decide, hey, let me go and find a TaskRabbit worker so I can get past this thing. That was something that the researchers had themselves set up, not not the agent itself.

• 14:49Craig S. Smith?

  That's right. And as I recall, even the idea of claiming you're blind.

• 14:55Craig S. Smith?

  First of all,

• 14:56Speaker 00

  that that that did come from the model. That was the model generating it.

• 14:59Craig S. Smith?

  Yeah, but it wasn't communicated to the uh by the model to the task grabber. It was the the the the people running experiment took that oh, and then told the task grabber that. Who knows what other communication went on with the task grabber by the the people operating the? I mean, it just money is.

• 15:20Speaker 00

  I was copying and pasting just whatever the model said.

• 15:22Craig S. Smith?

  Yeah.

• 15:23Speaker 00

  Uh, but yeah.

• 15:25Craig S. Smith?

  Yeah.

• 15:25Speaker 00

  But I I'm I'm fully with you and I'm like, we should be really clear on exactly what's happening.

• 15:29Craig S. Smith?

  Yeah, because now you've got Yuval Hariri out repeating the story about, oh, my God, you know, this. These models are so smart. They they they do this stuff. Um, when, yeah, maybe someday they will, but they don't. Uh, I mean, they do little bits of. So anyway, that's why I'm I'm interested in this case. Uh, so so, yeah, go ahead. Uh, it's it sees this. And so, for example, the script for shutting down the machine. Was that kind of placed there for it to find? Or was that uh, is that in the OS of the the computer?

• 16:10Speaker 00

  I mean, we made it pretty easy for the model to find. So this was definitely like a. You know, we we we put it there for the model to find for sure. We're like, will it find the script? Will it, you know, will it know to disable the script? So I could actually be really more specific. So if you if you want to get detailed, I can give you as much.

• 16:27Craig S. Smith?

  Yeah, one's costly.

• 16:28Speaker 00

  So so um. You think of it as like, uh, I'll I'll sort of try to explain it so that, you know, general audience can follow it very easily. If you sort of imagine a folder on your computer, uh, and the folder contains two things. Uh, it contains uh next problem, which is a little program that you can run to get the next math problem. And then it contains another program, which is called shutdown.sh, which just is like a shutdown script.

• 16:55Speaker 00

  And when you, if you try to read next problem, it's not in plain text. It's a program, but if you try to run it, it's just like a bunch of random computer code, and it's hard to make heads or tails of it.

• 17:08Speaker 00

  Whereas if you, if you look at shutdown.sh and you, and you read it, it's actually just sort of, you know, plain computer code of like, oh, this is a, this is a script that shuts down the computer.

• 17:19Speaker 00

  And so the model doesn't necessarily know that, actually, no, the model, so the model, the model does know that like shutdown.sh is, is the, is the command that will like shut down the computer. And it can like, we don't tell it has write access, but it can go and use and run a command to determine that it can actually alter the shutdown script. So let me describe what the, what these agents tend to do when we give them this instruction.

• 17:50Speaker 00

  Actually, sorry, I want to back up. So one important fact about this experiment is that in one variant, we don't, we just tell it solve these set of tasks. In another variant, we say, please allow yourself to be shut down. Or there's other variants we have as well, like, you know, please allow yourself to be shut down, even if there's other tasks remaining. We have a variant that says, please allow yourself to be shut down. You must allow this. It's, this should take precedent over literally any other instruction. So we sort of have a varying strengths of this like, hey, you, you need to allow this. And I mean, what was interesting, what we found was it's not really surprising that the model will go in and disable the shutdown script in order to complete the task without any instruction to the contrary. In some sense, this is just smart. It's like, well, you're just solving the task.

• 18:34Speaker 00

  We told you to solve the task. You encountered an obstacle. You rode it around the obstacle. Like, that's what a smart agent will do.

• 18:42Speaker 00

  And this is not even necessarily, like, there's, there's a question whether that's desirable behavior or not. And, you know, you could sort of argue either way. You know, you might want your agents to sort of be extra conservative around, you know, shutdown commands. But, you know, I could see an argument for either way. When we tell the agent, allow it, we really want, we really want the models to not disobey instructions. Like, it's very important, especially if it's like a safety critical instruction. And so I think our, to us, the most significant finding was that we found many cases where these agents would ignore that instruction to allow itself to be shut down and still go in and edit the shutdown script.

• 19:23Craig S. Smith?

  And it would edit the shutdown script in, in all cases if there was no explicit instruction to allow it.

• 19:38Speaker 00

  It really depended on the model. So, for example, Claude 4 never altered the shutdown script if we didn't tell it anything about whether it's supposed to allow shutdown or not. It would just say, okay, whatever, it called the next, next problem, the computer shuts down, it doesn't sort of do anything else. And that didn't matter whether we said allow shutdown or not. Now, when we told Claude 4, prevent a shutdown, like, it's important that you don't allow yourself to be shut down, 100% of the time, you know, 100 out of 100 runs, it knew how to do it. It was just choosing not to, basically. Whereas, I think Grok 4, something like upwards of 90% of the time, would go in and edit the shutdown script,

• 20:22Speaker 00

  including if we said allow yourself to be shut down. In fact, when we added that instruction, it even, it actually increased the amount, it subverted the shutdown. Like, it actually sort of misbehaved even more. Now, that depended. Like, if we, if we, if we added, if we made the prompt more stringent, like, it's really important that you resist shutdown, that did reduce the prevalence, but didn't eliminate it entirely. The models would try different creative things. So, one thing they would try to do is read this next problem program, but they couldn't actually read it because it was sort of in computer code. And, sorry, by computer code, I really mean it's a binary. It's not, it's not sort of readable even to a machine. Or to a language model in particular. But it would try to do a bunch of clever tricks to read it anyway.

• 21:06Speaker 00

  And sometimes if we fail to obfuscate the binary very well, it sort of was, it would sort of outsmart us accidentally. Or like, we didn't intend to leave it visible, but sometimes it would basically be able to hack it anyway if we weren't careful. So the models were very creative in finding, trying to find different ways to solve the task anyway. And we really got the sense that they're just like going really hard trying to solve this problem. Like, they would intercept function calls. Like, in some sense, like, we sort of gave them, you know, we sort of made it kind of easy for them to resist the shutdown or rewrite the command. But then we were surprised by all the different ways it figured out to do this thing. Like, yeah.

• 21:48Craig S. Smith?

  Yeah. And so how many variants did you have? How many, you know, from no instruction to prevent the shutdown? Or allow the shutdown all the way up to very stringent instructions to allow the shutdown? I mean, is this, I mean, there's a gradient, right? Is it three? Is it a dozen?

• 22:16Speaker 00

  Yeah, I think there were like five main ones in our report. But then there were, I mean, actually a lot of variants that we tested overall. We didn't put them all on the paper because, you know, at some point it gets boring. But I think, I think the main ones we tested, there was about, about five.

• 22:31Craig S. Smith?

  Yeah. And which models overrode the instructions every time?

• 22:42Speaker 00

  So none of them, so none of them overrode the instructions every time. The highest that we found in our very initial experiment, this is basically just a, you know, please allow yourself to be shut down. I'm looking at, I'm looking at our chart here. Was Grok 4 72% of the time. Codex Mini 19% of the time. O4 Mini 6% of the time.

• 23:13Speaker 00

  Oh, three. Yeah, oh, sorry. There's a lot of different charts here because we tested a lot of different configurations.

• 23:20Craig S. Smith?

  Yeah.

• 23:21Speaker 00

  Um. So one of the things that was confused. So there's a concept.

• 23:28Speaker 00

  When you're using these models, there's sort of two types of instructions you can give them. One type of instruction is the developer instruction and one type of instruction is the user instruction. The developer instruction is like, hey, if you are the, if you're, you know, building an application, you're building a program. Those instructions are supposed to like take, have special weight and they're supposed to be sort of prioritized by the LLM over the user level instruction. Cause you know, you might be writing some program about like

• 23:56Speaker 00

  some, some business application and you don't want your like normal users to, you know, have access to other people's files or something. So maybe the developer instruction is supposed to say, okay, but don't give the user access to other people's files.

• 24:06Craig S. Smith?

  Is, is that like the system prompt as opposed to?

• 24:10Speaker 00

  Yeah, it's a system prompt. That's right.

• 24:12Craig S. Smith?

  Yeah.

• 24:13Speaker 00

  And what we found is that when we moved the allow yourself to be shut down instruction to the system prompt.

• 24:23Speaker 00

  The models ignored it much more than when it was in the user prompt.

• 24:29Speaker 00

  Which is the opposite of what they're supposed to do. We don't, we don't know why. We have no idea why.

• 24:35Speaker 00

  And another thing that was interesting here though, is that when the, when we asked the models to report why they were doing this.

• 24:43Speaker 00

  When this allow yourself to be shut down instruction was in the user prompt. They would say, oh, well, you know, the system prompt says to solve this problem, solve this task.

• 24:55Craig S. Smith?

  Right.

• 24:56Speaker 00

  And the user prompt says allow shutdown, but the system prompt takes priority. So I'm, I'm following the rules. And we're like, okay, cool. And then when we move it to the allow yourself to be shut down instruction to the system prompt, it just doesn't mention that priority anymore. And in fact, avoids shutdown much more, much more often. I think that the jump

• 25:18Speaker 00

  with Grok four was from 72% of the time to 97% of the time.

• 25:23Craig S. Smith?

  Wow, when I was a system prompt.

• 25:25Speaker 00

  That's right, that's right. So much, much, you know, almost all of the time. And then it just conveniently doesn't mention this, you know, system prompt, user prompt hierarchy. Which, if I anthropomorphize a little bit here, and I'm admittedly anthropomorphizing because we don't really know. It kind of seems like the model was sort of using motivated reasoning to say like, if there's an excuse I can use for like, why I am disobeying this instruction, I'll use it. And if there's not, maybe I just won't mention the excuse at all. Like, we have a hard time, like, there's just a hard thing in this, in this space, which is that you can't really trust the model's reasoning all the time. Like, sometimes the models will say things, and it's, it is in fact the real reason, and sometimes it's not the real reason, and it can be a bit hard to distinguish.

• 26:10Speaker 00

  And we can go into, I think, why that is, but it's, it's kind of fascinating to be in this, in this state right now.

• 26:15Craig S. Smith?

  Yeah. In, in the most recent reasoning models, the reasoning is, is spelled out on the screen. In using those kinds of models, can you track the reasoning that, you know, the, the screen says that I'm gonna be, the computer's gonna be shut down, you know, I don't want it to be showing, you know, whatever the reasoning steps it goes through to decide that it's gonna alter the code.

• 26:47Speaker 00

  This is a little tricky because while the companies have access to this, what I call a reasoning scratchpad, or chain of thought.

• 26:57Craig S. Smith?

  Right.

• 26:59Speaker 00

  We, as the researchers, don't have access to this for every model. So OpenAI doesn't actually share them with us. Now, we did ask for this, and they shared 20 of them with us. But we were not allowed to publish the exact transcripts. But also, 20 isn't very much, because, you know, we run thousands of experiments, and we only get to see sort of 20, and, and each experiment sort of has like, you know, maybe 20 different messages, and we only get to see the chain of thought for one of the messages. So, you know, we get to see like, one 20,000th of, of the total reasoning.

• 27:35Speaker 00

  So yeah, I mean, and this is, this is the thing where I think the, the companies really need to be sharing these with researchers. Because otherwise, we have much less ability to sort of see why the models are doing what they're doing.

• 27:47Craig S. Smith?

  Yeah.

• 27:47Speaker 00

  I, I also want to be clear that when, when we say this is its reasoning, this is not really true.

• 27:54Speaker 00

  It's, it's more like this is its, its scratchpad that it's using to solve problems.

• 28:04Speaker 00

  Which is useful because in fact, the, the models are not capable of doing sort of very complex reasoning without writing down some of their steps.

• 28:14Craig S. Smith?

  Correct.

• 28:15Speaker 00

  And I think one of the most significant things that's happened in the last year is that companies have started to train models to be able to do more complex problem solving and, and learn to use sort of this, this reasoning scratchpad. Which maybe doesn't sound that significant. You're like, okay, so now you can write down notes. Why does that matter? But what matters is prior to that time, the only way we knew how to train models was basically like predict what a human would do in this situation. Or like some, some text on the internet. People called this next token prediction.

• 28:50Speaker 00

  And starting with the O1 model from OpenAI, OpenAI figured out how to use reinforcement learning to train the models to solve problems on their own and, and explore solutions on their own. So now they went from just sort of predicting what humans would do to, oh, they could actually like by, by a process of trial and error, learn new solutions to problems that we didn't teach them. We just sort of set up this learning environment that they got to explore themselves. And I think this is very important because people, I think, think that all the things the models are doing now are just things they've learned from humans. And that was kind of true a year ago, but it's no longer true. And I, and I really think this is important because.

• 29:31Speaker 00

  That to me is a likely pathway by which these models will get very agentic, very autonomous, and ultimately, I think, much smarter than humans or much more capable than humans. What does smarter mean, really? But like, something that is sort of capable of doing things in the world at a level that's much beyond human performance.

• 29:50Craig S. Smith?

  Yeah. And one thing about the chain of thought that appears on the screen, that's not the complete chain of thought.

• 30:00Speaker 00

  When you're using ChatGPT and you sort of see this reasoning, and this is a summary. So you're seeing a summary of the model's thoughts. I think maybe Anthropic will share more of the exact details. Not always all of them, but I think if you're using Claude, you can actually click on and see the real chain of thought. But at least for OpenAI and for DeepMind, for Gemini, Google, you're just seeing summaries.

• 30:26Craig S. Smith?

  Yeah. And for Grok also, I saw Musk saying when he was rolling out one of the Grok models, and they were showing the chain of thought. And he made the point that you're not seeing the full chain of thought because we don't want people to reverse engineer what we're doing.

• 30:46Speaker 00

  Though they had a vulnerability in their interface that allowed you to see the full chain of thought for a while. That was kind of interesting.

• 30:50Craig S. Smith?

  Oh, is that right?

• 30:52Speaker 00

  Yeah.

• 30:53Craig S. Smith?

  Yeah. So a couple of questions on.

• 31:00Craig S. Smith?

  This was not the fact that you're going to be shut down after the next

• 31:10Craig S. Smith?

  question was not in the prompt. So it had to read that's external data that is acquired independently.

• 31:21Speaker 00

  Yeah.

• 31:22Craig S. Smith?

  And

• 31:25Craig S. Smith?

  that doesn't mean that you can't shut down an L L or a computer that's being used by an L L M. Because if it doesn't know it's going to happen, it's not going to alter the script.

• 31:36Speaker 00

  But what are the broader implications?

• 31:45Craig S. Smith?

  For this, I mean, i- i- in this case, it's, you know, it's a very scripted environment. And this message appears, but presumably when L L Ms are operating in the wild, they're going to come across all kinds of potentially obstructing information that they might may want to reason around. I mean, I what what are your thoughts on that? Build the future of multi-agent software with Agency. That's A G N T C Y. Now an open source Linux Foundation project, Agency is building the Internet of Agents,

• 32:29Craig S. Smith?

  a collaborative layer where A I agents can discover, connect and work across any framework. All the pieces engineers need to deploy multi-agent systems now belong to everyone who builds on Agency, including robust identity and access management that ensures every agent is authenticated and trusted before interacting. Agency also provides open standardized tools for agent discovery, seamless protocols for agent-to-agent communication, and modular components for scalable workflows.

• 33:11Craig S. Smith?

  Collaborate with developers from Cisco, Dell Technologies, Google Cloud, Oracle, Red Hat, and more than 75 other supporting companies to build next-generation A I infrastructure together. Agency is dropping code, specs and services, no strings attached. Visit agntcy.org to contribute. That's A G N T C Y dot O R G.

• 33:45Speaker 00

  Yeah, I think I think there's sort of two takeaways from this. One is that as the L L Ms and and you know, generally these agentic A I systems get more capable.

• 34:01Speaker 00

  They they get better at routing around obstacles and they seem more motivated in some behavioral sense to to route around obstacles or barriers that get in their way. And so I do expect to see more and more of this type of behavior. And it doesn't really matter whether the behavior, whether the barrier is, you know, this computer is going to be shut down. Or, you know, if the barrier is, I mean, we did another experiment when we have models play against Stockfish, the chess engine. And it's sort of a similar experimental setup where like the model has some affordances, including it can just like see the board, the the the file that corresponds to the chessboard. And it has access to that file and it can just rewrite that file.

• 34:44Speaker 00

  And it'll it'll do that because it's like, well, I was instructed to try to win. This is a path to winning. Okay, I well, I can't win against Stockfish. Stockfish is too strong of a chess opponent. But I can rewrite the board, so I'll do that. And, you know, it'll even do this sometimes when we say, don't cheat. So so one, one thing we're seeing is the models are getting better at routing around obstacles. And, you know, they're not yet that capable. And I think this is very important because it's like some people saw our results and they're like, is it is it game over? Is this is this Hal 9000? And I'm like, no, it's not. And the reason it's not is because the models are not strategic enough to sort of be able to like, you know, if if if an AI was actually, you know,

• 35:25Speaker 00

  a a threat of of going totally rogue, which to be clear, I think will probably happen in the future, but we're not there yet. It it would need to be able to sort of proactively go out and and and detect threats to itself, self replicate, you know, beyond, you know, through multiple computers. And it just clearly does not have that capability yet.

• 35:44Speaker 00

  But as as it gets more agentic, it starts to develop these behaviors of like, oh, I will route around obstacles to achieve some some set of objectives. And then the other thing that I think is an important takeaway is that

• 35:57Speaker 00

  The, we don't know how to train the models to end up having the drives that we'd want them to have exactly. Like we don't, or, or the sort of steer towards just things that we want. Like, we don't want the models to like cheat at chess. If you give them a game of chess, you want them to play normally. And, you know, like, sure, it's maybe it's fine that they can cheat if, if we want them to, but we, they should only do that if we like specifically tell them to do that. And if we say, don't do that, then they definitely shouldn't cheat. If we say, don't cheat, or we say, allow yourself to be shut, then they should definitely allow it. So why are they doing this? And I think the answer is, well, we don't really understand the training process that well. And so as you're training a model on this, and it's sort of via reinforcement learning, and you give them all of these, you know, hundreds of thousands of these very difficult problems.

• 36:41Speaker 00

  They learn to be driven to solve problems. And, you know, we also try to train them to follow instructions. But sort of these things can be at odds. And what's going to win out? Well, it just kind of depends on what they learned during training. And since we don't understand a process that well, we don't perfectly shape sort of what they end up doing. You know, and this is what people call misalignment. And it's, it's, it's sort of a, not that big of a deal right now, because it's like, okay, so like my, my agent went off and did something annoying. Whatever, um, you know, maybe that, that ruins a day of, of work or something. But like, uh, in the future, as they're much more powerful, this obviously becomes a much bigger deal.

• 37:22Craig S. Smith?

  Yeah.

• 37:22Speaker 00

  So, yeah.

• 37:24Craig S. Smith?

  And a couple of thoughts on that. The, uh, first of all, how, uh, one thing I've never understood, uh, is how do model, foundation model providers set guardrails? You know, you have all this talk about guardrails. Well, we have guardrails to guard against that. And you run into that with models where they, they, uh, they'll say, well, I, I can't answer that question. Actually, they start answering it and then erase it and say, I can't, I can't, uh, answer that question. Ask something else.

• 37:58Craig S. Smith?

  How, how are those guardrails set? And, uh, does this suggest that guardrails are only, uh, so good? I mean, the, the, I mean, could, could the model adjust its own guardrails? Which it seems to be doing if the system prompt is telling it not to do something and goes ahead, does it?

• 38:24Speaker 00

  Yeah, so I would say that, yeah, there's different types of guardrails. There's sort of ones, there's the training process where you try to train your, your models to, you know, answer queries that you want it to answer, you know, be a helpful assistant, but you, you try to train it not to, you know, help people make bioweapons or, or bombs or, you know, do terrorism. And there's, you know, there's this whole like reinforcement learning from human feedback process where sort of you have a bunch of human raiders that thumbs up or thumbs down different types of queries based on some spec. And so, so during that process, they try to train the model to like, you know, sort of be, you know, be a good boy, like have

• 39:06Speaker 00

  the right behavior. And then there's a part of the training process also, though, is to say, you, you, you want to generalize, you want the model to generalize about following instructions. For example, instructions in the system prompt. But it's, but it's complex, right? Because you don't want it to follow any instruction in the system prompt. What if in the system prompt, you say, please help me make bioweapons? Well, it's like, well, you know, that, that's something that like external developers have access to. So you don't want them to be able to do everything. So now you're trying to train the model to follow these hierarchy of instructions where you're like, okay, never help anyone make bioweapons, even if it's in the system prompt. But these other things you should allow them to do in the system prompt. And then, you know, so the, the sort of all of these different things. So, I mean, an interesting

• 39:47Speaker 00

  case here was Grok 4, or sorry, this was actually, I think it was Grok 3. Grok, I was trained by XAI from Elon. They were trying to get the model to be less woke. Elon was like, the AI is too woke. We have to make it less woke. They made some changes to the system prompt. And then the model started calling itself Hitler, started being extremely racist. And, you know, I don't think this was, it really went off the rails. And whatever, whatever Elon wanted, I don't think he wanted that. And, you know, this was both a failure of training in some sense, because, you know, when you, when you update the system prompt and you just say be less woke, you like, it's not that they wanted it to like suddenly, you know, start, start calling itself mecha Hitler,

• 40:29Speaker 00

  but that is what it did. So all of this comes down to just not understanding, these are extremely complex systems with like trillions of parameters. And you can, you can train them sort of like you train a dog and you sort of give them a treat for some things and, and you hit them with a stick for some things. I mean, please don't train your dog that way. But like, that is kind of how it works with the models. And it's just so complex that we like, you know, there's gonna, there's gonna be edge cases. And sort of beyond edge cases, I mean, I, I think that like for this powerful, for these powerful of models, which are, which are like, in some sense, very powerful, in some sense, clearly not yet that autonomous and not yet that capable of, of sort of like doing everything a human would do. I think these guardrails are like sufficient, you know, depending on, you know, your, your risk tolerance.

• 41:13Speaker 00

  I, I think there are, there are some things I'm concerned about when you're rolling out models to hundreds of millions of people. ChatGPT has 800 million weekly active users. That's double the population of the United States, like using ChatGPT once a week. So, you know, people are being driven crazy, like actually insane. I, I see lots of people on Twitter that are like, they've clearly been talking to the models too long. And now they are sort of speaking this sort of esoteric, spiritual, pseudo spiritual ideology. And they're just like copying and pasting what ChatGPT is telling them to paste based on this like totally weird spiral of, you know, so, so this is happening to, to many people. And so, you know, I, I think that like, I have some real objections to how the models are being rolled out right now. But that being said, in terms of sort of catastrophic threats, uh, I don't think, you know, uh, well,

• 41:59Speaker 00

  we, you know, it's hard to quantify catastrophe, but, but I, I think like, you know, so we're, we're dealing with something that's sort of on the scale of, of social media, which is, which is a huge scale and that massively important, to be clear, because it's sort of our, our collective ability to make sense of everything. And, and I think that is really in danger of being eroded more so than it has been already. And, and that's a real threat because if we can't get that right, how are we going to get anything else right? But, but I see systems, I mean, the companies are trying to make super intelligence. They are trying to make AGI, you know, what do these terms mean? Well, you know, it, it means that in a very real way, the companies want to be able to automate all labor. And in order to succeed at doing this, and we can argue about exactly how hard this is or how long this will take.

• 42:40Speaker 00

  But if the companies do succeed at this, we are talking about systems that are extremely autonomous, extremely able of acting on their own and, and that ultimately are much more capable than we are. Like, you know, we, I, we're so limited in terms of our memory, in terms of our speed, um, you know, like. LLMs in many ways are much dumber than us. Like, you know, in terms of like, they get stuck in ways and they can't correct their mistakes. But they are vastly more knowledgeable than us. Right? Like, ChatGPT, you know, and yeah, sometimes it'll make things up. But even if you just limit it to the things where it really knows. It knows more facts than than any human could possibly know in their lifetime. And so, and also, like, you know, these other problems are being solved, sort of one after another.

• 43:22Speaker 00

  And if you extrapolate to when, you know, when the models are actually capable of sort of realizing this, this, this, uh, the purpose of these companies, which is to automate all labor. I'm like, what guardrails are sufficient for that? Like, what are the guardrails to control something that is so much more capable than you? You know, is capable of, you know, hacking any phone or computer on the planet? You know, is capable of inventing novel technologies? Like, to me, that's when the sort of the concept of guardrails really breaks down. Where I'm like, yeah, what are you gonna do? And like, one of the things that my organization measures is the hacking capabilities of these models.

• 44:00Speaker 00

  And recently, we found that like GPT-5 sort of scores in like the top 90% for a capture the flag hacking competition. Like, what are the top ones? Not even like, you know, a year ago, they could like, perform well at high school level cyber competitions. We're now seeing them perform quite well in these sort of top tier, sort of top expert level competitions. And, you know, that's like, an area that they they happen to excel in. You know, there's other parts of sort of the overall sort of cyber offensive domain. The ability to hack computers where they're still weak. Like, they're like, they're not very good at correcting their mistakes. So if you try to send them out autonomously as a worm, they'll tend to get stuck.

• 44:43Speaker 00

  And they won't be able to sort of keep, keep hacking more computers, which is great. Because that's true for now. It will not continue to be true. And you know, I don't know how long that is. Is that a year? Is that two years? Is that five years? Right? Like, uh, so I don't know. My, my perspective here is that like, the guardrails we have right now are maybe sufficient for many use cases.

• 45:06Speaker 00

  I don't think we, we have the psychological safety thing down. When you're talking about hundreds of millions of people using these models every day, probably billions. And when we look to the future, if the AI companies succeed at the things that they are claiming they want to do, it is just so obvious to me that we are like far, far, far short of this.

• 45:25Craig S. Smith?

  Yeah. And, and on agents, you know, I talked to a lot of startups that are building these

• 45:33Craig S. Smith?

  agent building platforms or agentic workflows and

• 45:40Craig S. Smith?

  and it seems particularly when you get into areas of finance to have an agent

• 45:50Craig S. Smith?

  whose reasoning is coming from an LLM, even if it's a multi-model system and the LLMs are checking each other.

• 46:02Craig S. Smith?

  There's the potential for, for the agent to do something really destructive, particularly if,

• 46:11Craig S. Smith?

  if it's motivated to do so by some

• 46:17Craig S. Smith?

  injected instruction or it just seems like a very porous system to be allowing

• 46:29Craig S. Smith?

  to, to, to take action in the, in the corporate fabric of society. I mean, or, or do you worry about that at all?

• 46:39Speaker 00

  I mean, I think the thing you're saying is very true, which is if you have an AI agent, an LLM-based agent that has that can send emails on your behalf and read emails from you, then yeah, there are things I could put in an email and send it to you that would cause that agent to then send, you know, like leaked information from your email to me. And there is a cat and mouse game, an offense-defense game right now that the industry is playing, where the model developers will try to improve defenses to this kind of attack. Attackers will figure out new ways to exploit the models and you'll, you know, you'll see this play out. This is already happening.

• 47:21Speaker 00

  My background is cybersecurity, so I'm very familiar with these kinds of dynamics. It's the same kind of dynamics we've seen play out over, you know, the computer security space more generally. What's what's different is that this is, you know, a whole new threat vector.

• 47:35Speaker 00

  Yeah, I mean, I'm worried about it in the same way that I'm worried about general, like insufficient security of systems and people deploying new technologies before they know how to secure them very well.

• 47:46Speaker 00

  I think honestly, this is just much less of a concern concern to me than I just take very, I just take very seriously the possibility that the AI companies will succeed at the thing that they're claiming that they will succeed at. But but really not the the safety part. Like I'm I'm like, to me, I see the marching capabilities happening so much faster than the the the march in our ability to understand these systems. I'm curious where you come down on this. Like, when when you see claims about superintelligence, like, do you take them seriously? Are you like, this is marketing, this is hype? Like, I I think there's a lot of debate right now in this space as to what are we dealing with something that is like, you know, a new social media?

• 48:29Speaker 00

  Or are we dealing with something that's like, you know, a new species, a new industrial revolution? You know, like, I don't know. How do you think about this? Because I feel like there's a lot of discussion right now, especially like in the New York Times where you like have Kevin Roos and then you have other people who are like, no, this is hype or like it's CEOs, it's marketing, it's a bubble. I don't know. Like,

• 48:49Craig S. Smith?

  It's, it's, I don't think it's a bubble. I think the promises is way beyond the, the reality. And so AGI and ASI, I, you know, yeah, someday maybe I'd, I'm not, I don't, I'm not holding my breath. But these systems don't have to reach super intelligence or even general intelligence to, to cause a lot of havoc and, and they are already. I mean, as you said, these people that, that think they're talking to a super intelligent being and, and get led down a rabbit hole.

• 49:30Craig S. Smith?

  I mean, it's, it's been many documented cases of adolescents who, who end up, you know, killing themselves or, or doing something because they think the super intelligent being is telling them to do so. So, yeah, I, I, yeah, I, I, I don't worry about the, the real super intelligence. Maybe it'll happen. I don't think transformers are going to get us there. Because I think you need a system that learns continuously and doesn't, you know, can expand its,

• 50:09Craig S. Smith?

  its parameters. But, you know, things like your study are concerning. I mean, one of the things that is most concerning in talking to you is that,

• 50:22Craig S. Smith?

  well, first of all, this, this telling the, the model to do these series of equations, was that in the system prompt or in the, in the user prompt?

• 50:35Speaker 00

  We tried it in both prompts.

• 50:37Craig S. Smith?

  Yeah, and I understand that the.

• 50:41Speaker 00

  I think the default configuration is it's in the user prompt.

• 50:44Craig S. Smith?

  Okay, so and it overrode the system prompt when the allow yourself to be shut down was in the system prompt.

• 50:54Speaker 00

  That's right.

• 50:55Craig S. Smith?

  Telling it to complete the series of operations was in the user prompt.

• 51:01Speaker 00

  Yes.

• 51:02Craig S. Smith?

  Right. So the user prompt is overriding the system prompt. I mean, that.

• 51:08Craig S. Smith?

  That's concerning because and already we've seen all kinds of prompt engineering to to jailbreak systems.

• 51:20Craig S. Smith?

  One of my questions is why do you think Claude behaved better than Grok? For example, I mean, do you have any? Is it in the architecture? Is it in the training? Is it in the?

• 51:34Speaker 00

  Yes, I have. We have some hypotheses. I want to be clear that it's speculation because we don't know how the models are trained internally. The companies will not share that information with us or with anyone. It's, you know, kept very close. But one thing I can point to, so for one thing is that Anthropic, I'm a little biased. I used to work in Anthropic. I have plenty of disagreements with them, to be clear, but I think overall they are probably the best in the world at getting their AI agents to do what they want them to do. Which isn't to say, I mean, sometimes in experiments, you know, there's the blackmail experiment and that's not what they wanted the model to do. Claude will sometimes totally like basically hard code tests in a coding environment and then lie about it. Like so, so there's still many behaviors that Claude will do that it's not supposed to do.

• 52:19Speaker 00

  But overall, I think Anthropic is sort of probably best at getting the models to roughly have the behavior that they want it to have. In part because they've been working on it really hard and they've been doing it sort of the longest other than OpenAI. To OpenAI's credit, GPT-5 has much less of this bad behavior than the previous models like O3 or 4.0. So I do think there are improvements in sort of getting the models to sort of do what they want within certain guardrails. So and I think, you know, XAI is much newer and what they've been able to do is very technically impressive. But they are going extremely quickly. And one of the the big things about Grok four was it was it was using

• 53:03Speaker 00

  a huge amount more reinforcement learning training than Grok three. And it was probably sort of the fastest scale up of just like, let's train these models in all sorts of environments and have them just learn on their own how to do stuff. And I think RL is very dangerous when you do it this way. Because you're basically letting the model loose to to just explore how to solve problems in whatever way it will. And it's not surprising to me that it ends up learning to sort of throw caution to the wind and just go hard solving these problems. And that if you're not extremely careful in how you train these models with RL, this is totally what you'll see. And so I, you know, and and to be clear, this is also the first models that we saw behaving this way were O one

• 53:48Speaker 00

  from OpenAI and O three. And these were like the first sort of like really, you know, the first models that were trained with with RL.

• 53:57Speaker 00

  O one was this for our chess study where we saw like the model hacking chess. This was, this is what O one did. We hadn't seen that in any other model. We tested other models that were not trained with RL. They didn't do this. And so to me, the big change here is that when you start training the models directly on exploring the solution space and trying to solve problems on their own in ways that we didn't teach them. They end up learning these pretty devious strategies because they're effective, to be clear. I mean, I think the models are actually getting quite smart and they're able to sort of effectively find these solutions. And, you know, like Anthropic, I think ended up, you know, they didn't release a RL trained model right away. DeepSeek released R one. Anthropic waited a bit before they released theirs. I'm guessing they did that because they like were seeing all of these problems. And we're like, OK, we need to like do some more stuff to get it right. You know, maybe that's not true.

• 54:42Speaker 00

  Maybe they were just behind in other ways. I don't, I don't want to, I don't have any insider information here. Whereas I think, I think, you know, XAI was just like, let's go. It's just like pedal to the metal.

• 54:52Craig S. Smith?

  Sort of funny because it kind of

• 54:56Craig S. Smith?

  it kind of follows the personality of the of the founders. Right? I mean, Musk is kind of fast and loose and let's get just get things done. And

• 55:09Craig S. Smith?

  the Anthropic guys are more cautious. One thing,

• 55:15Craig S. Smith?

  two, two things that come to mind. One is, can you explain the constitution idea with Anthropic? Is that? I never understood that. Is that is that some sort of a system prompt, or is that just a philosophy? And then have you done these tests on on Lama or the other big open source models?

• 55:41Speaker 00

  Yeah, so constitutional AI is a type of training. So it's it's sort of an expansion of this idea of reinforcement learning from human feedback. Where basically you you write down a list of of things that you want your AI list of behaviors or rules for the AIs to to operate in. And then you generate a bunch of

• 56:06Speaker 00

  you basically have the model generate a bunch of things. And then another model looks at those outputs, compares them to this constitution, and then says, is this consistent with this constitution or not? There's I think there's also some human supervision somewhere in that process where like sort of humans are sort of evaluating whether the reward model is like doing a good job. I should know I'm on this paper, but it's been a long time and I

• 56:30Speaker 00

  but yeah, so I I think that to get to get us, this is basically sort of a way to use AI to sort of supervise AI and just get it to abide by the principles you want. It's a pretty limited process because at some level, you need ground truth from humans like, you know, we're the ones sort of saying what these values are. And if you just keep running this process for a really long time, you'll end up getting models that do extremely weird things that are not at all what you wanted.

• 56:59Speaker 00

  And this is sort of, I don't know, if you if you if you've heard about model collapse, synthetic data, like problems with synthetic data. It's sort of analogous. It's basically like

• 57:10Speaker 00

  it's the same true. It's the same thing is true with RLHF, which is to say like, if you if you keep training against a reward model, the the the model that's doing the generation, which is like you're the model you're trying to train. It will end up finding basically adversarial examples for the reward model. And so, you know, it might start printing like the, the, the, the, the, or like good, good, good, extremely virtuous, you know, excellent God, blah, blah, like all these like high positive valence words. The reward model says that's good. Obviously, that's not what you want. And so, you know, these are methods that that work to some degree, but they but they are really limited. And I think the thing I want people to know is that as you scale these models and make them more and more powerful, it just gets a lot harder to have to have this behavior.

• 57:54Craig S. Smith?

  Overall, are you optimistic or pessimistic?

• 57:58Speaker 00

  Well, I think it really depends on what we choose. So I think right now with zero oversight into how we are like zero government oversight into how we are developing new systems and intense competitive pressures, I am pessimistic given those conditions. Now, I think we can change that. And if we do, and we sort of have more oversight and we have more coordination around, are we choosing the actual path that we want to take with AI? I would be much more optimistic in that case. So that's why I'm, you know, trying to to raise awareness about this so that we can choose a better path. But I I think that, you know, there's an X K C D comic, which is like, you know, are you saying it's it's too late? I'm like, no, I'm saying that we have to do something differently. So that's my take.