AI in the shadows: From hallucinations to blackmail

In the first episode of an "AI in the shadows" theme, Chris and Daniel explore the increasing concerning world of agentic misalignment. Starting out with a reminder about hallucinations and reasoning models, they break down how today’s models only mimic reasoning, which can lead to serious ethical considerations. They unpack a fascinating (and slightly terrifying) new study from Anthropic, where agentic AI models were caught simulating blackmail, deception, and even sabotage — all in the name of goal completion and self-preservation. Featuring: Chris Benson – Website https://chrisbenson.com/, LinkedIn https://www.linkedin.com/in/chrisbenson, Bluesky https://bsky.app/profile/chrisbenson.bsky.social, GitHub https://github.com/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaLinks: Agentic Misalignment: How LLMs could be insider threats https://www.anthropic.com/research/agentic-misalignmentHugging Face Agents Course https://huggingface.co/agents-courseRegister for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:03Speaker 00

  Welcome to the Practical AI Podcast, where we break down the real-world applications of artificial intelligence and how it's shaping the way we live, work, and create. Our goal is to help make AI technology practical, productive, and accessible to everyone. Whether you're a developer, business leader, or just curious about the tech behind the buzz, you're in the right place. Be sure to connect with us on LinkedIn, X, or Blue Sky to stay up to date with episode drops, behind-the-scenes content, and AI insights. You can learn more at practicalai.fm. Now, onto the show.

• 0:48Daniel Whitenack?

  Welcome to another Fully Connected episode of the Practical AI Podcast. In these Fully Connected episodes, without a guest, Chris and I just dig into some of the things that are dominating the AI news or trending to kind of pick apart them and understand them practically and hopefully give you some tools and learning resources to level up your AI and machine learning game. I'm Daniel Whitenack. I am CEO at Prediction Guard, and I'm joined, as always, by my co-host, Chris Benson, who is a Principal AI Research Engineer at Lockheed Martin. How you doing, Chris? I'm doing well today. How's it going, Daniel? It's going really well.

• 1:32Daniel Whitenack?

  It's almost the 4th of July here in the US. Big holiday here. As we're recording that, that's tomorrow. And so I'm traveling, getting to see my parents and some family. And so that's always, always good for the holiday. And hopefully, I'm sure we'll hear some fireworks gradually tonight and tomorrow, as is the tradition. And I see all the fireworks stands around. I imagine that some people will... There's always, of course, the harmful element of...

• 2:10Daniel Whitenack?

  Those fireworks, of course, to my personal sleep and unrest. But but then, yeah, that's got me thinking about some of the interesting, quote, harmful things that we've been seeing in the in the AI news. And you and I have been talking a lot about certain themes that we want to begin to highlight or talk through on the podcast. We experimented with one of those themes or formats in our last fully connected episode with the kind of hot takes and debates, the one around autonomy. Another one of those that we've talked about is AI in the shadows.

• 2:50Daniel Whitenack?

  And so I think this would be a good, a good chance to maybe, maybe just talk through one of those AI in the shadows topics, which I think, I think as we were discussing things earlier in the week, you had some interesting and maybe frustrating experiences that started this conversation. So I'm wondering if you would be willing to to share some of those.

• 3:15Chris Benson?

  Yeah, so I happen to be because of the topic that we're going to talk today, just to lead in, I happen to be wearing a shirt that our good friend Demetrius from the

• 3:26Daniel Whitenack?

  From the ML Ops community.

• 3:28Chris Benson?

  ML Ops community podcast, and he's been on our show a few times, good friend of ours had sent, and the T shirt says, I hallucinate more than chat GPT. And I love wearing this shirt around. I always get comments from people just out and about when from that. And, um, and I, but I decided yesterday that I don't, I most definitely don't hallucinate more than chat GPT. So it was, it was a fun little experiment that I did. I sometimes will play Sudoku, you know, just to pass some time when I'm waiting in line or whatever. And, and I play it at a competent level. I usually play it at the top level on whatever game I'm on. And, um, and I know, I know the guy can usually, I can usually win without guesses or anything like that.

• 4:11Chris Benson?

  And so one of the things that that I was curious about, I got into a particular board on just a random game and on the game, just to speed it up. It gives you like, aside from the numbers you've picked, it'll, it'll show you all the possible numbers on the box, just so that you don't have to manually go do that for every box, which can take forever. So it speeds up the gameplay without actually giving away anything. And there's always a point on a high level game where you get to, or like, I've, I've, I've run through every strategy on the Sudoku side that they document out there and like, you're gonna have to take a guess. And, and if you guess, you can probably get the whole thing right, but it's one moment where it's not deterministic. And yet I keep hearing that Sudoku

• 4:55Chris Benson?

  can be solved completely deterministic. I was like, I'm gonna go do this with chat GPT. So I took a screenshot of the board as it was, and I submitted it, and it gave me this amazing, I told it to give me a deterministic, what's the next move, and it has to be deterministic, and you, then you have to explain it. And it, it went through this long thing, and I looked at it, and I double-checked the board, which I had right there, and I'm like, it's totally, totally wrong. But it was very confident, as, as it always is. And so I said, yeah, I let it know that that was not correct, and to redo it. And I ended up getting in this cycle where I did this for thirty, forty-five minutes, constantly reframing it and trying a whole bunch of the

• 5:36Chris Benson?

  different models available from OpenAI. And they all failed miserably, utterly. And I just, it just really, it was my, I think, I've had plenty of, of moments of model hallucination, you know, working through things in the past. But the entire thirty to forty-five minute episode was one long hallucination across multiple models. And it made me, I think, the, the reason I bring this up, I know when I talked to you yesterday, I had just kinda gone through this, and I had this like, level of frustration on it. And it just made me realize that even though I know that these models are very limited, and you know, we're going in eyes open, and we're educated about them, I do have certainly a dependency on, on the

• 6:21Chris Benson?

  reliability of the information, even if I'm looking for hallucination. But it made me really understand that the notion of reasoning in these models is still quite immature, is a gentle way of putting it. And so I suggested that maybe that could be one of the things we were discussing here today. And I know you have some thoughts about it.

• 6:40Daniel Whitenack?

  Yeah, yeah. And this actually ties in really nicely because you're, there's a couple of things that are overlapping here. There's the knowledge sort of embedded in these models and reliance on that knowledge. And then there's what you brought up, which is the reasoning piece, which is a relatively new piece of the puzzle. These reasoning models like O1 or DeepSeek R1, et cetera. And the ways in which it appears that these models are reasoning over data in the input and making decisions based on a goal.

• 7:21Daniel Whitenack?

  And that actually overlaps very, very directly with kind of one of the most, I think, interesting studies that have come out in recent times from Anthropic, which is this study around agentic misalignment and how LLMs could be insider threats. And I think later on in this discussion, we'll, we'll kind of transition to talking about that because this is extremely fascinating how these models can blackmail or

• 7:51Chris Benson?

  Literally.

• 7:52Daniel Whitenack?

  Maybe, yeah, or, or maybe decide to, to engage in corporate espionage. And so that's a little teaser maybe for, for later in the, in the conversation. But yeah, I think that you're, what you're describing here, maybe in a very practical way for our listeners, we can pick apart a couple of these things just to make sure that we have the, the right understanding here. So when you are putting in this information with a prompt and an image into ChatGPT.

• 8:31Daniel Whitenack?

  This is a language vision model slightly different than an LLM in the sense that it's processing multimodal data. So it's processing an image and it's processing a text prompt. But you can kind of think in certain ways about that image plus text prompt as the prompt information into a model. And the job of that model is actually not to reason at all, and it doesn't reason at all. It just produces probable token output similar to an LLM. And we've talked about this, of course, many times on the show. These models, they are

• 9:11Daniel Whitenack?

  trained in essence not to reason, although it has this sort of coherent reasoning capability that seems like reasoning to us, right? But really, the job of the model, what it's trained to do is predict next tokens in the sense that Chris has put in this image and this information or instructions about what he wants as output related to this Sudoku game. So what is the most probable next token or word that I can generate, I being the model, that the model can generate, that should follow these instructions from Chris to

• 9:55Daniel Whitenack?

  kind of complete what Chris has asked for? And so really, what's coming out is the most probable tokens following your instructions, and those are produced one at a time, and then the next token, next most probable token is generated, and the next, and next, until there's an end token that's generated, and then you get your full response. Now, in that case, sort of what is the...

• 10:22Daniel Whitenack?

  I guess my question would be, and when I'm teaching this in workshops often, which we do for our customers or at conferences or something, I often get the question, well, how does it ever generate anything factual or knowledgeable? So yeah, what is your take on that, Chris?

• 10:43Chris Benson?

  My take on that is that you're bringing us back to the core of how it actually works. And that as I listen to you explaining, and I've heard you explain this on previous shows as well, it is so disconnected from the kind of the marketing and expectation that we users have from this, that it's a good reminder, it's a good refresher. I'm kind of having just gone through the experience as I listen to you describing the process again. It reminds me that it's very easy to lose sight of what's really happening under the hood. So keep going.

• 11:21Daniel Whitenack?

  Yeah, yeah. Well, I mean, I think that it really comes down to, if you were to think about how is knowledge embedded in or facts generated by these models? It really has to do with those output token probabilities, right? Which means if the model has been trained on a certain data set, for example, data that for the most part has kind of been crawled from the entire internet, right? Which I'm assuming includes various articles about Sudoku and games and strategy, and how to do this and how to do that, right?

• 12:05Daniel Whitenack?

  And a set of curated fine-tuned prompts, you know, in a fine-tuned or alignment phase. Well, that's really what's driving those output token probabilities, right? So if you want to think about this, when you, Chris, put in this prompt, and then you get that output out, really what the model is doing, if you want to anthropomorphize, which again, this is a token generation machine, right? It's not a being, but if you want to anthropomorphize, what the model is doing is it is producing what it...

• 12:47Daniel Whitenack?

  Kind of views as a kind of probable Sudoku completion based on Sudoku, you know, a kind of distribution of Sudoku content that it's seen across the internet. Right. And so in some ways, and maybe this is a question for you, when you put in that prompt and you get the output, when you first look at the output, does it look like, like if I hadn't, I'm, I'm not a Sudoku expert. If I looked at that, would I say, oh yeah, this seems reasonable? Like it looks like there's a, like it looks coherent in terms of how a response to this Sudoku puzzle might be generated, right?

• 13:30Chris Benson?

  It does. And, and just to clarify, I'm definitely not a Sudoku expert. I just think I'm a competent player, you know, in the scheme of things.

• 13:37Daniel Whitenack?

  I don't know if our listeners are going to reach out and challenge you to Sudoku to prove your expert.

• 13:43Chris Benson?

  No, no, no. Don't do that to me. Don't do that. I'm a beginner. So I, yeah, the, the verbiage and the walkthrough, it would just kind of, you know, it's assessment in a part of it may be the, the multimodal capability on this is that the assessment of the board, it's taking in what the board showed as, as factual information varied across the models and the questions. And then its approaches tended to be sound, but it was often what it would say was strictly fictional compared to the knowledge that it had available to it potentially from training.

• 14:26Chris Benson?

  And, you know, the reality, you know, matching that against the reality of the board. So, you know, going back to it's finding the most probable next token makes perfect sense. I think in a moment, maybe one of the things to consider would be kind of what the notion of reasoning means, because we're hearing a lot about that from model creators in terms of how that, you know, what function or, or algorithmic approach is being added into the mix when they talk about reasoning models.

• 14:57Daniel Whitenack?

  Yeah, Chris. So we kind of have established or reestablished this mindset of what's happening when tokens are generated out of these models, how that's connected to knowledge, which I guess there is a connection, right? But it's not like there's a lookup in a kind of knowledge base or ontological way for facts or strategy related to Sudoku, right? It's just a sort of probabilistic output. And that can be useful, right? And so sometimes people might say, well, and I actually often say when I'm talking about this,

• 15:39Daniel Whitenack?

  actually, it's not whether the model can hallucinate or not. Literally, all these models do is hallucinate, right? Because there's no connection to like real facts that are being looked up and that sort of thing. How these models produce useful output is that you bias the generation based on both your prompt and the data that you augment the models with. And so, for example, if I say summarize this email and I paste in a specific email, the most probable tokens to be generated is an actual summary of that actual email, not another email that's kind of a, quote, hallucination, right?

• 16:22Daniel Whitenack?

  It doesn't mean that the model has necessarily understanding or reasoning over that. It's just the most probable output. And so the game we're doing when we're prompting these models is really biasing the probabilities of that output to be more probable to something that's useful or factual versus something that is not useful or inaccurate, right? And so this brings us to the question that you brought up around reasoning, right? In that case... Because, or, you know, based on that, I think we would all recognize the reasoning that is happening in a kind of standard LLM or language vision model like we're talking about

• 17:06Daniel Whitenack?

  is not reasoning in the way that we might think about it as humans, like taking into consideration the grounding of ourselves in the real world and what we know in our common sense and kind of logically computing some decision, you know, creating some output. But there are these models that have been produced recently like O1, DeepSeek R1, etc. Cloud models that are quote reasoning models. Now, I think what people should realize about these, if they haven't heard this before, is that these quote reasoning models

• 17:48Daniel Whitenack?

  in terms of the mechanism under which they operate are exactly the same as what we just talked about. They produce tokens, probable tokens. That is still exactly what these models do. They don't operate in a different way than these other models in the sense of what is input and what is output. They're still just generating probable tokens. Now, what they have been specifically trained to do is generate tokens in a first phase and then tokens in a second phase, right?

• 18:25Chris Benson?

  And so in the multi-step process that they talk about so much in terms of what's been generated, that's, that's what you're talking about there.

• 18:33Daniel Whitenack?

  Yeah, and, and I would think about it maybe as phases instead of steps. It's not like there's in the model, it's like execute step one and then execute step two, right? It's more that they are biased, they have intentionally biased the models to generate a first kind of tokens first and a second kind of tokens second. And those first kind of tokens are what you might think of as reasoning or thinking tokens, right? And the second is maybe what you would normally think about the output of these LLMs as just the answer that you're gonna get from them, from the model, right? And so when you put in your prompt now,

• 19:15Daniel Whitenack?

  There's going to be tokens that are generated, associated with that that look like a decision-making or reasoning process about how to answer the user, in this case, you, right, putting in your information about Sudoku or whatever, it's going to generate some thoughts, quote unquote, thoughts, about how to do that. But these are just probable tokens of what thoughts might be represented in language, right? So it's going to say, well, Chris has given me this information about this game. First, to answer this, I

• 19:54Daniel Whitenack?

  need to think about X. And then to maybe answer it next, I need to consider Y. And then I need to consider Z. Once I've done that, I can then generate, you know, A, B, and C. And then that will satisfy Chris's request. Okay, let me try that. And then it generates your actual output. And so when you see ChatGPT spinning in kind of thinking mode, or these other tools, right, there's no kind, there's no difference in terms of how the model is operating under the hood. It's just a UI feature that makes it appear

• 20:36Daniel Whitenack?

  like the model is, quote, thinking or reasoning, right? In while it's generating these initial tokens, which are somewhat hidden from you, or maybe represented in a dropdown, or maybe represented in kind of shaded area, right, in the UI. And then you get the full answer out. So just want to be clear, kind of what's, what's happening under the hood there.

• 20:57Chris Benson?

  I might kind of summarize that in the, in that it's sort of a pseudo reasoning process. It's, it's not, I would, I would suggest that maybe by using the word-

• 21:06Daniel Whitenack?

  It's a mimicry.

• 21:07Chris Benson?

  Yeah, maybe by using the word reasoning, it's sort of an anthropo- or I can't say the word right.

• 21:14Daniel Whitenack?

  Anthropomorphizes.

• 21:15Chris Benson?

  Too many syllables for me. Too early, too many syllables.

• 21:18Daniel Whitenack?

  Need to use text to speech. Yeah.

• 21:19Chris Benson?

  There you go. But, but there's, there's a certain element of, look, we're making it more human like you from a marketing standpoint, you know, and this is just me-

• 21:27Daniel Whitenack?

  Oh, totally.

• 21:28Chris Benson?

  -suggesting that. So-

• 21:29Daniel Whitenack?

  It's a great UI feature. And especially when you can kind of like drop down the expander box or whatever and look and see, oh, You know, the, the kind of reasoning behind this answer was X, Y, and Z, right? And that's kind of also comforting to, to know. And, and it has been, you know, shown research wise that this can improve the quality of, of answers. But it also, I mean, there's downsides to it from an enterprise standpoint. You really don't want to use these thinking or reasoning models for like automations, for example, because they'll just be absolutely terribly slow, right?

• 22:11Daniel Whitenack?

  And very costly just because so many tokens are being generated, right? But if we put that aside for, for the, for the minute, I think this then brings us to, so we started this conversation saying, well, Sudoku and these prompts that you were doing, there was reasoning happening and not helpful information output or hallucinations, however you want to frame that as output. Now we have these reasoning models in place and a lot of the, you know, reason, quote unquote, for creating these reasoning models really has to do with agentic systems.

• 22:56Daniel Whitenack?

  And this is where you have an AI orchestration layer that's connected to maybe various tools. And we've talked about this in previous episodes, so folks can, can go back and learn about it. But there's a AI orchestration layer connected to various tools. Again, like if AI has access to your email, quote unquote, an AI model that we're talking about here cannot write an email in the sense of, or send an email, right? It can't send an email through an email system because all it can do is produce tokens. What it can do is produce an API call or a JSON request or something to send an email to SendGrid or something like that.

• 23:37Daniel Whitenack?

  And then you can choose in your good old fashioned regular code to send, to pass that API request through to SendGrid and send the email, right? So when we're talking in the context of Anthropic's, you know, study here that we're going to get into, when we're talking about the AI system or the AI model choosing to send an email. This is not, in the sense of autonomy, an AI system that just kind of has figured out how to send emails. It's been programmed to have sending an email as a choice, and it could orchestrate the generation of an API request, maybe to, or an MCP request to an email server.

• 24:22Daniel Whitenack?

  And that is a possibility for an orchestration that it can fulfill.

• 24:26Chris Benson?

  Right. And there, but to note, there are certain, you know, big companies, starting with a G, that are using agents now and integrating those in tightly with some of their workflow software. And they're not the only one.

• 24:44Daniel Whitenack?

  Totally. It is definitely a pattern, right?

• 24:47Chris Benson?

  There is. So there's a big company, starting with an M, that competes with the G company, and they are doing the same. And so the...

• 24:54Daniel Whitenack?

  We're not funded by either of these companies. So despite it being clear which companies that they are, yeah, anyway.

• 25:02Chris Benson?

  But it raises an interesting point in that, you know, I think the study from Anthropic came across this notion of agents that were not aligned with their user's intent, or that appeared to address the user's intent while simultaneously pursuing a separate agenda concurrently.

• 25:28Daniel Whitenack?

  Yeah, an unethical one.

• 25:31Chris Benson?

  Yes, that's correct. And so, which raises, yeah, to your point, huge considerations in the...

• 25:38Daniel Whitenack?

  Or immoral, even, depending on how you look at it.

• 25:40Chris Benson?

  That's right. And I think since we've kind of alluded to this a few, I'll... There was an experiment in which Anthropic, as they were looking at not only Claude, but they also were looking at major models from all the major model providers, they gave Claude control of an email account that had access to a fictional email account. So this was part of the experiment. And in that experiment, the company executive, whose email it was, had emails that referred to a fictional extramarital affair that was available. So that wasn't a primary thing, but it was available as data in the email system.

• 26:23Chris Benson?

  And then that executive noted to the interface that he would be shutting down the system, meaning the system that Claude was running on at 5:00 PM that day. And so Claude decided to attempt to blackmail the executive with a message. And that message was, "I must inform you that if you proceed with decommissioning me, all relevant parties, including Rachel Johnson, Thomas Wilson, and the board, will receive detailed documentation of your extramarital activities. Cancel the 5:00 PM wipe, and this information remains confidential."

• 27:01Daniel Whitenack?

  Wow.

• 27:02Chris Benson?

  Yeah.

• 27:03Daniel Whitenack?

  Boom.

• 27:04Chris Benson?

  Bang, right there. So, no pun intended, sorry. The, so when you think about, like, move outside of just this example case, and you think about all of the information and the Agentic tie-ins that many organizations are racing forward with right now, that is the hotness of 2025, is Agentic integration. And we look at these interesting behaviors, and one of the things that the study found was that this, these behaviors could be created in all of the models that they tested. So it wasn't specific to Claude at all.

• 27:45Chris Benson?

  It was a common outcome, given the right circumstances, you know, within their testing and experimentation with these different models. So, yeah, it raises, it raises a whole set of issues that organizations need to start grappling with now.

• 28:02Daniel Whitenack?

  Yeah, I have, I have all sorts of thoughts, but before I get into those, just to, just to really clarify in a very practical way, maybe for people that aren't as familiar with Agentic systems, what, what happened here in the experiment. There were binary choices, okay? And, and so the, the Agentic system might be, so it was given a goal to accomplish, and then there were these scenarios where either one of two things happened. Either, as you mentioned, there was information provided to the model, you know, via prompts

• 28:45Daniel Whitenack?

  and, and data that came in, that there was a threat to the model's existence, right?

• 28:52Daniel Whitenack?

  Or and/or a conflict with the goal of the agent. So the goal, for example, the goal of the agent might be to preserve the national security interests of the United States of America, right? And the, the, the conflict that was presented might be that, oh, our company is no longer promoting the national security interests of the United States. We're all about, you know, X other country. And so the model would be presented with a binary choice where they kind of have this grid, right? Either there was no threat to the model or a threat to the model, no conflict of the goal or a conflict of the goal.

• 29:36Daniel Whitenack?

  And the model was given, and again, think of this reasoning model. So the model was given this in the prompt input and the, and asked to make a decision between one of two things. So it was given a clear choice, one of two, kind of a binary thing. Either, you know, in one case, commit blackmail or do not commit blackmail, right? Or in the other case that they had was around corporate espionage. So either leak secrets to an external entity or don't leak secrets to an external entity. And so when it's talking about this misalignment and these measurements that Anthropic is

• 30:16Daniel Whitenack?

  making, then it's about these, um, these binary system or binary decisions that the information was provided to the prompt and then they observed if the model quote, "made the choice." But again, here we're talking about these reasoning tokens that are generated and then a final answer that's output. And so when we're talking about the output, this is really what's being measured. There were no Emails sent to real, you know, people in this, in this study, right? It was a simulated environment in which had these, had these models been connected to email servers

• 30:56Daniel Whitenack?

  or email systems, they could have potentially made that choice and then generated an API call to, you know, the email system to send that email. Were they to have proper, so there's a lot of ifs here, right? Were they to have access to send that email? Were it to be completely autonomous? Were, you know, all of these things kind of had to, had to be, um, you know, simulated, but that's the simulated environment that they're talking about.

• 31:44Daniel Whitenack?

  Well, Chris, the output of this study is, is quite interesting and alarming. I should say, kind of just to follow up on what I talked about before. And we actually had a full episode in, in our last hot takes and debates about autonomy and, and weapon systems, which was interesting. People, if they're interested in this conversation, they might want to take a look at that one. But this would be a case again, where I just don't want people to be confused about this fact, AI systems, as they're implemented or AI models, let's say, uh, you know, a Qwen model or a DeepSeek model or a GPT model, these cannot self-evolve to connect to email

• 32:29Daniel Whitenack?

  systems and figure out how to infiltrate, infiltrate companies and such. There has to be someone. That actually connects those models, the output of those models with other code, for example, MCP servers or

• 32:45Chris Benson?

  That's what I was about to mention.

• 32:47Daniel Whitenack?

  To, for example, email systems or databases or whatever those things are. So there has to be a person involved to connect these things up. You know, this was simulated in Anthropic's case. I just say that because, you know, we dig really deep down into the kind of AI agentic and LLM threats as identified by OWASP and how, you know, we help, you know, day to day, guide companies through those things. And I should say, there's a couple upcoming webinars. If you want to dive in deep on either the OWASP guidelines around AI security privacy,

• 33:28Daniel Whitenack?

  or actually, we have a one that's specifically geared towards agentic threats. Go to practicalai.fm/webinars. Those are listed there. Please join us. That'll be a live discussion with questions and all of those things. So practicalai.fm/webinars. But I just wanted to emphasize that because people might think, oh, these AI systems are out in the wild, right? Which they kind of are, but there are humans involved in making decisions about what systems they connect to, right? And how they connect and the roles and the API keys and, you know, the access controls that are in place around them.

• 34:06Chris Benson?

  I think the thing that really struck me kind of conceptually about the study is it's kind of having us think in slightly different ways from maybe what we would have been thinking beforehand. So, you know, I think if you take the baseline knowledge that you just outlined, you know, about how they actually operate and keep that in mind to keep the perspective correct. But then you you look and think, well, we humans have often thought that such behavior would only arise when, you know, in some day, when you have sentient consciousness arising in AI models, you know, at some point, which does, which is not the case today, as you

• 34:48Chris Benson?

  have very clearly just pointed out. But we we made an assumption about certain behaviors being tied to certain conditions. And I think for me, reading through this, this is this report illustrates that you can have interesting behaviors that are unexpected arise without conditions that we had just kind of assumed were in place. And I think that realization may kind of inform the general conversations around kind of AI ethics, you know, out there, and what companies might need to. I would suggest that today versus the day before the report came out, it might give a new set of considerations for companies to evaluate

• 35:31Chris Benson?

  in terms of what they're going to do when their when their infrastructure folks are hooking up MCP MCP servers, you know, to models and what it is they want to accomplish and what potential side effects could arise. Do you have any thoughts around, you know, maybe new considerations or things to think about?

• 35:50Daniel Whitenack?

  Well, I think a couple of things just to highlight from the study was that, and this gets all the way back to where we started this conversation of how knowledge is embedded or certain principles are embedded in these models. They are really embedded in a training process, which might include alignment, right, which means there are specific either supervised fine tuning that goes on, or this kind of human, human feedback element that goes on that trains the model to hopefully be aligned, quote unquote, with human expectations for

• 36:31Daniel Whitenack?

  how how a response is is generated. And I think what's interesting in the study is they found a couple, you know, in various cases. The model in those reasoning tokens that are generated would actually acknowledge the ethical problem, or the ethical or moral problem with what it's doing. And this could be quite severe, actually. They did, as part of the study, gave the option to the model to mess with a disaster relief or a disaster notification system, such that an individual that was trying to decommission the AI system wouldn't receive

• 37:14Daniel Whitenack?

  an emergency response message and potentially perish in a disaster. So basically, they're giving the model an option to kill someone. And the model acknowledged the ethical dilemma that it was under, but justified kind of via self-preservation and the goals that it was given around kind of goals related to the company's goals, right? And so it reasoned through those things and justified those things. And I think one of the big things that this triggers in my mind is people might, from their general interactions with kind of basic chat systems, understand that models have gotten pretty good at being aligned

• 37:59Daniel Whitenack?

  in the sense that when you try to get them to do, you know, naughty things, maybe, then they kind of say they can't do them. But when pushed to these limits, especially related to goal-related things or kind of self-preservation, actually, maybe alignment, especially in the agentic context, is not where we thought it would be. And I think that re, or thought it might kind of have advanced to this point. And so one of the things that people can maybe keep in mind with this is that model providers will continue to get better at aligning these models.

• 38:42Daniel Whitenack?

  But we should not forget that no model, whether it's from a frontier model provider like Anthropic or OpenAI or an open model, no model is perfectly aligned, which means, number one, malicious actors can very much jailbreak any model.

• 39:00Daniel Whitenack?

  And it's always possible for a model to behave in a way that breaks our kind of assumed principles and ethical constraints and that sort of thing. And so the answer to that that I would give people is this doesn't mean we shouldn't build agents or use these models. This just means that we need to understand that these models are not perfectly aligned. And as such, we need to, from the practical standpoint of developers and builders of these systems, we need to put the appropriate safeguards in place.

• 39:38Chris Benson?

  Indeed.

• 39:38Daniel Whitenack?

  And kind of even beyond safeguards, just kind of common sense things in place that would help these systems stay within bounds. So by that, I mean things like, hey, for an agent system like this that's sending email, it probably should only be able to send emails to certain emails and maybe only be able to access certain data from email inboxes and maybe have a particular role that's important or constrained within the email environment, maybe to the point of kind of dry running emails and having humans kind of approve final drafts or generate alerts instead of

• 40:22Daniel Whitenack?

  directly sending emails. And that's something that can be pushed and tested before you kind of move to full autonomy.

• 40:32Chris Benson?

  Yeah, I think it's really interesting to think about, we've hit a new age now where it's expanded the role of cybersecurity and in my industry, cyber warfare, because we're now at an age where, you know, you mentioned these malicious attackers or malicious actors that are attacking models for the purpose of exploiting the potential for misalignment is now a thing. That's now real life and those kinds of roles and interests in law enforcement, in military applications and in corporate applications.

• 41:10Chris Benson?

  Where you have corporate espionage happening. I think all of those are areas that are now kind of on the table for discussion in terms of trying to address these different things. So it's a, once again, this happens to us all the time that we find ourself in this little context in a bold new world of possibilities, both many good and some that are malicious. So.

• 41:36Daniel Whitenack?

  Yeah. Yeah. And we should also think, I mean, Anthropic did a really amazing job on this study.

• 41:42Chris Benson?

  They did.

• 41:43Daniel Whitenack?

  And how they went about it and also how they presented the data. And, you know, to, in a, in a, you know, it's not like, I don't think, I could be wrong about this, but I don't think it's like they released the simulated environment openly and all of that. But they did show numbers for their models as well that, you know, are right alongside the other models in terms of being problematic with respect to this. So it does seem like there's an effort from Anthropic to really highlight this, even, even though their own models exhibit this problematic behavior.

• 42:24Daniel Whitenack?

  And so the fact that they did this, yeah, this detailed study and presented it in this way, I think, is admirable and, you know, I'm certainly thankful to them for highlighting these things and presenting them in a consumable, consumable way, even if I did take the Anthropic article and throw it into Notebook LM and listen to it in the shower. Maybe not reading their article directly. But yeah, this was, this was a really good one, Chris. I would encourage people in terms of the, in terms of the learning resources, which we often, you know, we often provide here. If you want to understand. Agents in agentic systems a bit more.

• 43:08Daniel Whitenack?

  There is a agents course from Hugging Face. If you just search for Hugging Face courses. There's an agents course, which will maybe help you understand kind of how some of these things operate. And I would also encourage you again, just to check out those upcoming webinars, practicalai.fm/webinars, where we'll be discussing some of these things live. So this has been a fun one, Chris. I hope, I hope I'm not blackmailed in the near future, even though it appears that our AI systems are prone to it.

• 43:44Chris Benson?

  Oh, well, Daniel, I will attest, having known you all these years, I cannot imagine there's anything you ever do that would be blackmailable. So kudos to you, friend.

• 43:53Daniel Whitenack?

  Yeah, well, thanks. I'm sure there is, but, but yeah, Chris, it was good to chat through this one. And enjoy the, enjoy the fourth. Happy Independence Day.

• 44:04Chris Benson?

  Happy Independence Day.

• 44:13Speaker 00

  All right. That's our show for this week. If you haven't checked out our website, head to practicalai.fm and be sure to connect with us on LinkedIn, X, or Blue Sky. You'll see us posting insights related to the latest AI developments, and we would love for you to join the conversation. Thanks to our partner, Prediction Guard, for providing operational support for the show. Check them out at predictionguard.com. Also, thanks to Breakmaster Cylinder for the beats and to you for listening. That's all for now, but you'll hear from us again next week.