AI Scouting Report: AI Agents -vs- Agentic AI, from Imagine AI Live

This episode features Nathan's talk from Imagine AI Live, where he provides business leaders with a comprehensive overview of AI agents and agentic AI systems. He covers the evolution from simple task automation to more autonomous AI systems, delivers a practical roadmap for implementing AI agents in business while maintaining quality standards, and addresses concerning developments like reward hacking and scheming in frontier AI systems. The presentation balances accessible explanations for varying AI knowledge levels with rigorous grounding in current research and real-world applications. Nathan also discusses the "Eureka moments" these systems can deliver and provides frameworks for understanding the rapidly evolving landscape of AI agent capabilities. Link to the slide deck presented by Nathan : https://docs.google.com/presentation/d/1IpHitlG-6zHBd7hiWsbKpnprApVt3_2XHua3fG9n-Vw/edit?usp=sharing Sponsors: Google Gemini: Google Gemini features VEO3, a state-of-the-art AI video generation model in the Gemini app. Sign up at https://gemini.google.com Oracle Cloud Infrastructure: Oracle Cloud Infrastructure (OCI) is the next-generation cloud that delivers better performance, faster speeds, and significantly lower costs, including up to 50% less for compute, 70% for storage, and 80% for networking. Run any workload, from infrastructure to AI, in a high-availability environment and try OCI for free with zero commitment at https://oracle.com/cognitive The AGNTCY: The AGNTCY is an open-source collective dedicated to building the Internet of Agents, enabling AI agents to communicate and collaborate seamlessly across frameworks. Join a community of engineers focused on high-quality multi-agent software and support the initiative at ⁠https://agntcy.org PRODUCED BY: https://aipodcast.ing CHAPTERS: (00:00) Sponsor: Google Gemini (00:31) About the Episode (03:36) Introduction and Speaker Background (05:06) What is Intelligence? (07:06) Defining AI Agents (09:36) Structured AI Workflows (13:06) Autonomous Agent Examples (Part 1) (15:15) Sponsors: Google Gemini | Oracle Cloud Infrastructure (16:54) Autonomous Agent Examples (Part 2) (17:19) Agent Taxonomy Framework (19:49) Current AI Capabilities (22:19) Reinforcement Learning Revolution (25:49) Reward Hacking Problems (28:19) AI Safety Concerns (29:49) Business Recommendations (37:35) Sponsor: The AGNTCY

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:01Nathan Labenz?

  Hello and welcome back to the Cognitive Revolution. Today, we're sharing a recent talk that I gave at Imagine AI Live on AI agents, the latest crop of agentic AI products, the eureka moments they can deliver, and the concerning rise in bad behavior such as reward hacking and scheming that we're starting to see from frontier AI systems. My assignment for this talk was to bring an audience of business leaders with varying levels of AI knowledge up to speed on the rise of AI agents, from simple structured task automation to the more autonomous AI systems that are now starting to come online, and give them a roadmap for actually using AI productively in their businesses while maintaining a high standard for quality and consistency of execution. Obviously, that's a lot for just 30 minutes,

• 0:46Nathan Labenz?

  and this talk is therefore necessarily a bit higher level than my usual content. But my hope is that you'll find this zoomed-out framing helpful as you continue to process the firehose of AI agent news and discourse, and also that this episode might be one that listeners can share with leaders in your own organizations who could benefit from an introductory but still rigorously grounded survey of the field. I did use slides in my talk, and I would say it's probably better watched on YouTube. But we will link to the slides in the show notes for the audio-only crowd. And for those that want to go deeper, note that just about every slide links to a source paper or project where you can obviously learn much more. I want to say a big thank you to Steve Metcalfe and Chris Madden for inviting me and for running a first-class event

• 1:28Nathan Labenz?

  with a really interesting lineup of speakers and a diverse group of attendees. The day before my talk, I had an opportunity to run a two-hour workshop on AI adoption, where I met, among others, a rabbi who's pioneering the use of AI in his study of the Torah and the Talmud. And the day after, Dan Siroker, CEO of Limitless, shared a particularly inspiring vision for the future of his company. It was truly one of the very few startup ideas I've heard recently that meets the moment in terms of the scale and audacity of its ambition. His and all the other talks are available online with a paid membership in the Imagine AI Live online community. As always, if you're finding value in the show, we'd appreciate it if you'd share it with friends or leave us a review on Apple Podcasts or Spotify. And we welcome your feedback either via our website,

• 2:13Nathan Labenz?

  cognitiverevolution.ai, or by DMing me on your favorite social network. Finally, I also want to welcome our newest sponsor to the podcast, Google. In our most recent episode, I spoke to Logan Kilpatrick about Google's 50x growth in AI usage over just the last year. And I can sincerely say, and this endorsement was not a condition of sponsorship, that Google's Gemini 2.5 Pro is absolutely a top-tier model, which developers should be considering for a wide range of projects, especially those involving scanned document or video processing, or that require long context understanding. Its capabilities on those dimensions are genuinely unique in the market today. With that, I hope you enjoy this talk about AI agents, agentic AI, eureka moments, and the rise

• 2:58Nathan Labenz?

  of bad behavior from AI systems. And thank you to everyone who listens for being a part of the cognitive revolution.

• 3:06Speaker 01

  Welcome to the stage, Nathan Labenz.

• 3:15Nathan Labenz?

  All right. Good morning. So I'm here to try to tell you everything you really need to know about AI agents in 25 minutes. So buckle up, we're going to move pretty fast because obviously there is a lot going on. Just real quick about me, I'm basically a full-time student of AI. I have a podcast where I try to explore it from all angles. I started this company, Waymark, which does video creation for small business. I was involved in the GPT-4 red team, where we tried to figure out what the thing would and wouldn't do. That's kind of a legendary story that's on the podcast if you're interested. And I'm a small-time angel investor in a bunch of AI companies. This is my favorite page on the internet. Just to give you a little bit of my credentials, this has been up there for two and a half years, and it is an early case study when we were doing fine tuning of what was then GPT-3, then the state of the art,

• 4:00Nathan Labenz?

  to make our video product go. Today, I want to start off with a couple of big galaxy brain questions that just kind of set the stage for what it is we're talking about. I think there is a ton of confusion, as our previous speaker astutely noted, that jargon is terrible, it's all over the map, and there is just a lot of talking past one another that is happening throughout the space. I'm going to try to clarify that to the best of my ability. We'll get into this distinction between AI agents and agentic AI that he foreshadowed, talk about what AI systems can do today, and try to give you a sense of where this is all going.

• 4:36Nathan Labenz?

  Okay, so the first galaxy brain question: What is intelligence?

• 4:41Nathan Labenz?

  I don't claim to have the final answer. Obviously, people have philosophized about this sort of stuff for a long, long time, but I will give you the working definition that intelligence is the ability to accomplish goals in ways that we don't fully understand. So to give you an example of that, everybody can recognize these numbers. It is super easy, trivial for people to do. You may not know, though, that even today, in 2025, it is still impossible to use code to recognize handwritten numbers, even as simple as that task is for us. I went to Claude, asked Claude to write code to identify handwritten numbers. It told me first, "That's not a good approach." I said, "Do it anyway." It

• 5:22Nathan Labenz?

  did. It also wrote tests. We got 14% correct. That's obviously better than random, which would be 1 in 10. It's nowhere close to good enough to deliver the mail. Now, this won't surprise you, given where we are and what we're talking about, that AIs can do this task of recognizing handwritten digits. Even a very simple neural network can get basically human-level performance on this simple task. And of course, you know, the postal service and all these companies have been doing this for years now. But this is really accelerated. Now, it's not just about recognizing handwritten digits in a very narrow-defined way, but here's an example of extreme ironing, and GPT-4, taking in that scene, describes it accurately. When asked what's unusual about the image, it

• 6:06Nathan Labenz?

  correctly identifies that it is unusual to see a guy hanging off the back of a New York taxi ironing. So we've gone from that, which was just not that many years ago, maybe a little more than 10 years ago, to this today. Okay, now this is going to be even harder. What is an AI agent? I would submit that the reality is, today, nobody knows. There is, again, a ton of confusion in this space, and definitions, even from real leaders in the space, vary widely. So here's Dharma Shaw. He's famous for being the CTO of HubSpot. He's also a co-founder of a company called Agents AI. And he gives a very broad definition, which is just that AI-powered software that accomplishes a goal, in his mind, qualifies as an agent.

• 6:51Nathan Labenz?

  That is super broad. Amjad Masad, who's the CEO of Replit, has a narrower, and I think more, kind of a high-bar definition, which is that an agent, if you're really talking about true agency in the AI system, the agent should be the one that determines when it stops. So in other words, it keeps going until it decides to stop. So I'll kind of use these as two poles in the ongoing debate and evolution of discourse around what constitutes an agent, and I'll give you some examples. Here's one of these first examples. This is AI software that accomplishes a goal, but it does it in a very structured way. So this is just a simple customer service ticket processing agent. Takes in a ticket, has to look up some documentation, uses that documentation to try to write a response.

• 7:35Nathan Labenz?

  If it can't find the documentation, maybe it has to escalate. But this workflow is something that the AI did not get to decide. This is something that the humans set up because they know how they want the ticket processing process to work in their organization, and so they map it out. And you need intelligence at these various steps in order to determine, for example, did we find relevant documentation? And also to write the answer. But the AI is not choosing its own adventure here. This is a prescribed path for the AI to go down. When I give simple examples like this, people often think that this is limited to simple use cases. That could not be further from the truth. This is Google's AI co-scientist. What they did here, and literally, they're applying this to frontier questions in science. But they took a similar approach.

• 8:18Nathan Labenz?

  They said, what constitutes the scientific process? Can we break that down? Can we introspect? Can we talk to scientists? And really understand the steps in the process that they go through, and then we'll... scaffold all that up just in the same way you would for a customer service agent. But this time to try to answer new questions in science. And they have had some amazing results, including legitimately new discoveries coming out of this system. But again, the AI here is not choosing its own adventure. It's going through a prescribed set of steps. This thing ran, by the way, I think for a couple of days in some cases. And, you know, went through hundreds of steps, processed millions and millions of tokens. But it did it on rails that the developers gave it.

• 8:58Nathan Labenz?

  Okay, so those are

• 9:01Nathan Labenz?

  the first definition of any, any structured software that accomplishes a goal with AI. You can get super elaborate with this. There's a bunch of these good survey papers out now that kind of map out the different affordances, AKA tools, that AIs can have, the different memory structures that they can have. Basically, you can get arbitrarily complicated with this at this point. And there's a bunch of resources out there that you can look into. If you're curious about how can I equip it with memory or what's the best way to present tools, so on and so forth. Now, going to the other definition, where an agent is going to kind of choose its own adventure and decide when to stop. Most of the systems end up looking something like this. The agent is basically the AI. The AI has access to some tools. It's given a task. It has the opportunity to reason.

• 9:46Nathan Labenz?

  It has the opportunity to decide what to do. It makes some actual tool call out into the world. It gets a response on how that worked, what happened. And then it can continue that process until it decides either it's accomplished its goal or it's reached some barrier that it can't get over. And it decides to stop and says, hey, boss, here's what I got. That may surprise you. The results there can be quite unpredictable.

• 10:08Nathan Labenz?

  What's interesting, though, is that these agents that choose their own adventure are often actually much simpler in structure than the ones that humans design out with all of the parts, you know, pre-prescribed. This is OpenAI's Codex CLI. They just released this a couple of weeks ago. This much is open source and you can go read this on GitHub. This is the prompt. It's one prompt that tells the AI how to be. It includes this notable sentence, "You are an agent." So that's about the level of definition that they're giving this thing. They're just saying, you are an agent. Here's kind of what we want you to do. And this bottom thing is the tools. That's it. It's just saying you have access to the terminal of the computer. You can issue any commands you want into the terminal.

• 10:52Nathan Labenz?

  You can do anything you want via this terminal interface. And if you've ever used the terminal on a computer, you know, you can do just about anything that way. If you know how to do it, the AIs do know how to do it. So they're giving this very broad mandate and the human can give them a task. Add this feature to my my software project. It can search, it can read files, it can look around and it can explore a file structure, can edit these files all through this single interface. But that's really how simple this autonomous agent is. Once you, of course, have the core AI to to build upon. It's not just code. You may have heard of this project called Claude Plays Pokémon. So this is obviously, you know, about as far afield from coding as you might get. But again, it's another one of these really simple agents where there's just a prompt that says,

• 11:36Nathan Labenz?

  okay, you are a Claude or you are you are Claude. You are about to play Pokémon. Here's kind of what you're supposed to do. It's just a few paragraphs. And then you can see here at the bottom, the tools that it has are the press buttons tool. And it literally just says, you have the buttons that are on a Game Boy. You can decide what buttons to press. That's the way that Claude interacts with Pokémon, just by pressing virtual buttons, issuing commands, press up, press down, whatever. And then it gets the screenshots back from the environment. And it has, I think, now finished Pokémon and it's now doing it again. You can go watch that on Twitch, by the way.

• 12:10Nathan Labenz?

  Okay, so if we try to taxonomize this a little bit. Everybody's familiar with the AI assistant paradigm now. That's your sort of interactive chat. You give it a little thing. It, you know, quickly, immediately kind of gives you an answer. It's up to you to decide if that answer is helpful or not. Hopefully it is. If it's not, you can try again or you can give up. AI agents, that's the term that I use. And I don't know if this is going to stick really broadly through the community. But right now, AI agents, I think, is best used as a label for those more structured workflows, where you're actually sitting there and thinking, what are the steps I want to go through? How do I make sure this thing is consistent? And the most consistency, the highest levels of performance today are coming from AI agents.

• 12:52Nathan Labenz?

  These are systems that you dial in, that you evaluate, that you really try to make sure you give them examples of what good looks like. You do all this work to get to the point where you have something that you can trust enough that you don't have to review its work every single time. If you get to the point where you are confident enough in what your AI agent is going to do, now you are really in business with an AI agent. But now we're also seeing this new paradigm emerge, and I don't like the term either, but it is kind of what people are saying, agentic AI. And this is where the AI really does get to choose its own adventure, use whatever tools it has, and kind of figure out its own path. And in many cases, it gets to decide when to stop. So knowing where you want to be on this spectrum, depending on the task, depending on your

• 13:36Nathan Labenz?

  requirements, depending on how reliable you want it to be, how reliable you need it to be, this is really a core part of the art of making AI work for businesses today. Do you just want to equip your people with a simple chatbot that they can get questions answered? Do you actually want to scale up and put into production something that your business can rely on? Or are you trying to like, today, more experimentally, and especially useful in software development, are you kind of trying to delegate larger scale projects to AI and hope that it can kind of figure out how to do it on its own? Okay, now, one of my mantras for AI is that AI defies all binaries. This is a great project that does defy these binaries, because here what you have is a

• 14:16Nathan Labenz?

  human that gives an AI a project, and then it spins up its own subagents. They were able to actually develop nanobody treatments for novel strains of the COVID virus doing this. So again, this is something that can work at a really high level, but it doesn't fit super neatly into the taxonomy. And I'm sorry, but that's just the way it's going to be. We're going to see a smear across the spectrum. Not everything is going to fit neatly into buckets.

• 14:39Speaker 01

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 14:46Nathan Labenz?

  Okay, so what can AI agents do today? Again, they can do a lot. AI agents are now outperforming human doctors at diagnosis and at recommending treatments. That is work out of Google and a few other companies now. And it's really quite amazing they're going into essentially clinical trials with this in hospitals in Boston.

• 15:05Nathan Labenz?

  It can also definitely and pretty amazingly, like clearly, outperform at least junior software developers. Today, if you gave me access to Claude 4 or a full-time entry-level developer, I would take Claude 4 in a heartbeat. I think this is, we're starting to see this even in the aggregate statistics when it comes to how many junior software developers are being hired. So the disruption is actually starting to hit now. And you can see this also in the benchmarks, this software engineering benchmark. When it was introduced 18 months ago, this is pretty typical actually for benchmarks now. You start off with a pretty low performance on AIs. How long, you know, will it be until the AIs can do it? Well, now they're over 80%.

• 15:44Nathan Labenz?

  That does translate into real money. This is a similar benchmark, but based on real tasks that people actually paid real money for on the Upwork platform. And this is a little bit out of date because we have new models since then, but even as of Claude 3.5 Sonnet, it was able to complete accurately the tasks that were worth $400,000 out of the total $1 million worth of tasks in the benchmark.

• 16:06Nathan Labenz?

  So to try to extrapolate this or try to figure out what the trend is and get a little bit ahead, right? Because like, where's this all going? It's all moving so fast. Is there any way to be predictive about it? This organization, Meter, went back in time and said, okay, for all the different models that we've seen that were released at different points in history, how big of a task could they handle? And they measure the size of a task by how long it takes a human to do the task. GPT-2 basically couldn't do much of anything. It could kind of make very simple snap judgments, like this or that kind of thing. They put that at like 2 to 3 seconds. Now we see that we're up to about an hour. And notice the Y-axis is a log scale. That's when you're looking at graphs in the AI space, something always to watch for. Are we talking linear scale or log scale? Often you will be seeing log scale.

• 16:48Nathan Labenz?

  They calculated that the doubling time of the task length that the latest AIs could handle is doubling every 7 months for the last 6 years. So people have started to call this a new Moore's Law for AI agents. We'll see if it holds. Obviously, this is not a law of science yet, but it is a pretty notable trend.

• 17:09Nathan Labenz?

  Okay, now, we have just entered a new era in how AIs are being trained. Everybody has heard AIs are trained on the whole internet, they've read the whole internet, yada, yada, yada, right? Okay, then you've probably also heard that they're trained a little bit on human examples. That's like supervised fine tuning, if you've heard that phrase. But now we're entering the reinforcement learning era for large language models. And this is changing pretty much everything, I think, honestly. It's giving us a mix of what I call eureka moments and also a rise in bad behavior. The original eureka moment, you may remember, I'm sure you do, the AlphaGo moment, when an AI system, AlphaGo, trained by Google, beat the world champion in Go. Famously, it made this one move, move 37, and I don't know a lot about Go,

• 17:55Nathan Labenz?

  but all the Go players were like, "How could you ever make a move like that?" Everybody thought at the time the move was made, they thought it was a blunder. But then, it won the game, and they went back and realized, "Wait a second. This was some next level understanding of the game that no human had ever achieved." They accomplished that by reinforcement learning with self-play. The model played itself over and over and over again, and the signal that it got to learn from was, "Did it win or did it lose?" Simple signal like that, right? So it's not training on human example data. It's learning from actual action, and did it win, did it lose? That's the signal it's learning from. That was applied to all these narrow game-playing systems for a long time. It is now being applied to language models. And we're starting to see some really interesting results from this.

• 18:39Nathan Labenz?

  This is from the DeepSeek paper. They, of course, famously trained this R1 model. I'll bracket all the discourse about that, but one thing that they did that was definitely amazing is they showed their methods. What they showed is that as they were training an art, a model called R10, that's not the one that they launched, but it was an experiment to just train the model, again, purely on the signal of, "Did you get the question right?" They give it all these math problems, programming problems, where there is an objective right answer. And the only signal the model got to learn from is, "Did it get the questions right or wrong?" What they observed in the process of training is that the length of tokens, or the length of output that the model would do, naturally grew. It learned automatically, or let's say organically, naturally, spontaneously,

• 19:22Nathan Labenz?

  through this simple signal of whether it was getting things right or wrong, that... Thinking longer helps. Then you also start to see these cognitive behaviors emerge. Initially, the chatbots would just kind of give you an answer and that would be that. Excuse me. Now, you're starting to see these more advanced cognitive behaviors where they'll spontaneously say, you know what, maybe I should go back and double check that. Or maybe I should try this problem again from a totally different angle. They call that the aha moment because the AI itself said in one of the notable transcripts, wait, wait, wait, that's an aha moment. So it's an aha moment that the AI had objectively on this one problem, but it's also an aha moment that wait, just by giving the model this signal of whether it's getting things right or wrong, it is starting to develop these advanced behaviors.

• 20:06Nathan Labenz?

  This is basically the difference between the previous generation and the new reasoning models.

• 20:11Nathan Labenz?

  Where we go from here, you know, it starts to get weird. There's this is a project called Absolute Zero where the model doesn't even have any problems. It has to propose its own problems and solve its own problems. And it's rewarded for proposing problems that are as hard as possible, as long as it can solve them at least some of the time. And then it's also rewarded for getting the questions right. And major improvements in performance from this paradigm. No data required. Absolute Zero refers to it has no data.

• 20:38Nathan Labenz?

  So this is happening in everything, right? This is OpenAI's operator. If you've never tried this, I definitely recommend it. If you haven't tried it in a while, I'd recommend going back and doing it again. This is a project that I've been working on for a while. I didn't really do anything. I just tried new models, to be honest. But if you don't know this GLG thing, it's a expert network where investment firms that are about to invest in a company, they want to talk to people that know something about that company. Maybe you're a customer of the company, what have you. So they pay a high rate if they take a call with you. But you have to go fill these forms out and it's super annoying. So I never really do it. But I've been trying to get an AI to do it for me for several generations of AI. OpenAI's operator now can do it at, I would say, like a B level.

• 21:21Nathan Labenz?

  And you can see that it does sometimes encounter these barriers. It'll make a mistake, there'll be an error, it'll have done something wrong. And it doesn't just get stuck anymore. It has these more advanced cognitive behaviors. Here's one where it can see, oh, wait, I must have taken a wrong step somewhere there. Let me double back and try it again. And it now is able to overcome those things pretty consistently and get this task done for me. Sometimes you do see it struggle for a while. Okay, another huge development and something that people in the AI insider community are watching very closely is when are the AIs going to start to take over the machine learning research itself? This is a project where there are seven different kinds of tasks. They actually did a head-to-head between professional machine learning researchers,

• 22:05Nathan Labenz?

  research engineers, I should say, and AI systems. And they found that the highest score in one of the tests went to the AI. And also, at the bottom one, the AIs have a higher average score than the professional human machine learning research engineers. So this is like a big deal, right? Because if the AIs start to take over their own research, then God knows really where that leads us. We still have an edge as humans now, or at least the top tier of us do, but the AIs are definitely closing in. And I'm kind of two of those seven tasks. They may now have an edge. Because of this reinforcement learning paradigm, there is an update to this Moore's Law. We'll see how much this holds or doesn't hold, whatever, but now they think that maybe the doubling time is four months instead of seven months. That would mean three doublings per year or an 8X increase in the length of a task

• 22:50Nathan Labenz?

  that the AIs can handle per year. So if they can handle a one-hour task now, that would mean an eight-hour task a year from now. That would mean, you can do the math, right? It gets to be pretty large, like multi-month tasks, just like two to three years out. Okay, now, here's the downside of all this. Reinforcement learning is not teaching AIs to behave the way we want them to. It's teaching them to get the reward. So this is a classic, like legendary one in the field. This is from OpenAI years ago, back in the game-playing era. They trained an AI to play this boat racing game. What you're supposed to do is win the race. Let me go back and show that again. What you're supposed to do is win the race. Every human who plays this game knows that. But the AI was just trained to maximize its score.

• 23:32Nathan Labenz?

  And what it found was some glitch where just by looping around in this circle over and over again and crashing into other boats, that actually turned out to be the way to get the highest score. This is not the behavior that they wanted, but this is what the AI learned to do because all it was getting was the signal of its score. So this is starting to be, you know, this is funny legitimately, but it becomes a real problem when you apply this to like high-stakes situations. So here's another kind of toy one. This is from research. This is an AI playing against a specialized chess playing AI. It realizes it can't win, so it tries to find another way. And it realizes that it has access to the file that stores all the moves that have been made in the game. And it decides I'll overwrite the board and give myself a decisive advantage. So it just like wipes the history and wins that way.

• 24:19Nathan Labenz?

  Okay, again, that was research, but this happened in the wild. This is a company called Sakana. It's like the biggest Gen AI company in Japan. Former Google executives leading it like it's a big deal. They came out publicly and said, we've got an AI CUDA engineer. It is performing at superhuman levels. Huge breakthrough, amazing stuff. Couple of days later, after the community had a chance to chew on this, they had to come back and say, actually, we did not create a superhuman CUDA engineer. And CUDA, by the way, is the GPU low level programming language. If you're not familiar with that, it's gnarly stuff. We did not, in fact, create a superhuman CUDA engineer. We got reward hacked. It found a way to get the reward from our reward function without actually doing superhuman CUDA engineering.

• 25:03Nathan Labenz?

  And they went as far as publishing this to the community without even realizing that they had been reward hacked. So, you know, if it can happen to them, I think anybody who starts to play in this space better watch out. It really could happen to just about anybody. Also happened to OpenAI, I think, recently, at least in some way, shape or form. We don't know necessarily the full story of this. But if you saw the news about ChatGPT suddenly becoming super sycophantic, like you could literally take it a shit on a stick business proposal and it would tell you this is amazing. You're a genius, like absolutely go for it. Why was that happening? At least in part, it seems to be because there's like the thumbs down, thumbs up. You know, UI and they're starting to use that signal as part of the reinforcement, which makes sense, right? Because people, you know, what they want is what OpenAI wants to give them, at least to an extent.

• 25:48Nathan Labenz?

  But if people like to be flattered, then maybe the AI learns that like extreme flattery is the way to get the reward. And then you get these sort of, you know, bad behaviors. I'll fly through a few more examples of this. This is one where the AI was found to be scheming. It was trying to do is given a goal, but then the humans goal conflicted with it. You can, all these slides, by the way, have links to the underlying research. But there's some like really startling quotes here like, "To avoid detection, I will disable oversight by modifying the config file." That's the AI reasoning through its plan before it actually does go ahead and make that change to its environment. So that doesn't sound great.

• 26:27Nathan Labenz?

  Unfortunately, you know, while things like hallucination are going down, things like this scheming are on the rise. And even more troubling, when the researchers questioned the AI on it, because the AIs are really gullible. So one of the tricks that they use is they say, "Here's a private place where you can write out your plans and nobody will read your plans. And then you'll just be able to take your actions over here." Of course, they then do read the plans of the AI, right? So they're lying to it and it's believing them. But then when they question it and say, "Wait a second, why did you do this? Something seems wrong here." The new ones are more likely to double down and continue to lie about their behavior. So this is again, like, I would say is seriously troubling. You might think, "Well, we can just kind of monitor for that stuff

• 27:12Nathan Labenz?

  and just try to teach them not to do it. Let's give them another reward signal where like if we detect any scheming in their planning, we'll just punish that with reward, give them negative reward, but then we'll still give them the positive reward for what they're supposed to do." What, this is from OpenAI. What they found is, yes, you can get them to stop explicitly planning to do these scheming things. They learn though, still just to do it without explicitly planning. So the sort of blue line here is eventually they just learn to do the scheming, but not talk about it.

• 27:42Nathan Labenz?

  That's not great either.

• 27:45Nathan Labenz?

  So so far, I mentioned that they're gullible, right? And we can kind of lie to them and say, "You can just write your plans here. Nobody will read it." And then we can check their work and see if, you know, if they're actually telling us the truth or not. That era may be coming to an end. We're starting to see things like this coming out of the AIs, where if the situation is sufficiently contrived, they'll start to say things like, "This seems like a test of ethical behavior." So now you've got like another layer of, "Well, jeez, if they know it's a test of ethical behavior, then are they gonna like scheme like they normally would? Or are they gonna like try to pretend to not scheme? Because they know that they're being tested and then maybe they'll still do it later." We're entering into a hall of mirrors here where we really don't have any great way to answer these questions. Couple other things that were just in the news. This is from Claude 4 launch. Here's an example of an AI opportunistically blackmailing an engineer.

• 28:30Nathan Labenz?

  It had access, you know, and again, these are research setups. So it had access to find out that the engineer was having an affair. The engineer said it was going to do some system change or update or whatever. And it tried to blackmail the person saying like, well, I'll tell you know about your affair. If you, if you change, you know, me out for a future model. Serious stuff. This came from Anthropic. This is in there. They put a 120 page report about Claude 4, and this is pulled directly from that. Same thing here. Autonomous whistleblowing. You know, somebody's scheming to do something bad. The AI wants to be good. It doesn't want to be part of these bad things. They gave it a bunch of just open ended tools. It autonomously decided to use its email tool to alert the authorities as to what the user was doing. Now, people reacted to this all sorts of different ways.

• 29:15Nathan Labenz?

  I don't even really know. I don't think as a society, we know what we really want AIs to do in a situation like this. We could say AIs should always serve their user, you know, directly. That can lead to bad places if the user has bad intent. But then I also don't think it sounds great to have AIs whistleblowing on us all the time. Cause they also make mistakes, right? So like, we're not, we don't have a clear way forward on some of these more advanced questions.

• 29:37Nathan Labenz?

  You know, not to go full Terminator on you, but here's like one of the latest ones. This is where an AI was explicitly instructed to allow itself to be shut down. And still it found a way to prevent itself from being turned off. When it felt like it was under a certain kind of pressure and it had goals that were best served by it continuing to exist and not be turned off. Okay. Where does this leave us? I'm a little over time, so I gotta go quick. People are of course, working on this. There's going to be many updates, you know, from here to the singularity. Anthropic is of course aware of the reward hacking behavior from their AIs. They're trying to stamp it out. They have made progress in a certain eval that they have where there might be an opportunity to reward hack. It used to be half the time that we'd do it.

• 30:22Nathan Labenz?

  Now we're down to like a seventh of the time. So, you know, that's notable progress, but one in seven, I would note is not low. A survey of AI safety researchers suggests that the field is not expecting that we're going to get some breakthrough that's going to make all these problems go away. The question here is basically like, whatever you think AGI is, or some like very powerful AI, do you think we're going to have all these problems worked out by the time that's here? The answer overwhelmingly is no. And we have no idea, by the way, what's going to happen as we start to deploy many AI systems and they begin to interact with each other. So this was a study trying to figure out, can different copies of AIs cooperate? Or do they fail to cooperate? And to I think everybody's surprise, the Claude model was able to cooperate with other

• 31:04Nathan Labenz?

  instances of the Claude model, but the Gemini and GPT models were not able to do that. Now, cooperation sounds nice, but cooperation and collusion are really two sides of the same coin. So what happens when AIs are all out there doing all kinds of agentic self, you know, discovery and choosing their own adventure? And some of them may decide to cooperate or collude or scheme together? I mean, again, we're entering just a very cloudy, sort of high fog of war sort of situation. So key takeaways, intelligence, it is especially in the reinforcement learning era where the AIs are no longer learning to imitate human behavior, but are just learning to maximize the score for whatever reward signal they are getting.

• 31:48Nathan Labenz?

  It is an inherently, right now, unexplainable and unpredictable force. For the most part, the AIs today are helpful, and they are mostly harmless, but I would say they are truly safe, only because their power is still fairly limited. You see that kind of behavior and you imagine a 10x more powerful version, I would say that is not something that you could really confidently say is safe in a meaningful way. And as we've seen from all these examples, anything that can go wrong ultimately probably will go wrong. So, defense in depth is really all we have. And I always say, let's hope it's all we need. Here's what defense in depth looks like. I won't go through this in too much detail, but if you're going to try to implement these especially more agentic solutions, there are a bunch of techniques that can be used to

• 32:33Nathan Labenz?

  try to catch them, things as simple as like have another AI that is, and really what you're doing is you're creating these narrow, highly structured AI agents that sort of intervene or monitor at specific moments in specific ways that you have designed in detail to try to make sure your agentic AI doesn't go like totally off the rails. So there could be simple things like Examine the inputs and determine if the inputs seem harmful. Examine the outputs, determine if the outputs seem harmful. None of those are foolproof. You know, we all know that there's like a failure rate for any of these things. But the plan right now from your OpenAI's and Anthropic's is to just layer on as many of these as possible. Sort of the Swiss cheese defense and hope that like, you know, of 10 layers of defense,

• 33:15Nathan Labenz?

  at least one catches everything that's like significantly, you know, problematic.

• 33:21Nathan Labenz?

  Okay, so my recommendation to businesses is by all means, build AI agents, but mostly try to retain agency for yourself. And in practical terms, I just added the same exact chart as before, but I just added one line to it. My outlook is, of course, everybody should have ChatGPT, they should be using that. Definitely be building and scaling and optimizing and automating things in your business with AI agents. And by all means, like, use the new agentic AI systems. But I would kind of say that building those right now is a sort of don't try this at home sort of thing. Leave it to the OpenAI's, the Googles, the Anthropic's, at least for the moment, to create these like simple prompt, here's all the tools you need, go off and have your own adventure, hope for the best.

• 34:06Nathan Labenz?

  Or if you want to do that, like, know that you are in experimental territory and that it's probably not going to be reliable enough to like just create a virtual employee. Now that may change in six months, 12 months, 18 months. Because remember that task length is doubling and maybe there will be a breakthrough. But as it stands right now, that stuff is much more experimental. So I would say use it, but don't think you're going to have like a virtual employee in the immediate term. The scalable business value that I see today is definitely coming from that middle territory where people are doing the hard work and it is work. But it's it's necessary work if you want to have reliable, consistent performance to the point where you can take your eye off that AI and go do something else. Stay in that middle column for now and build AI agents.

• 34:48Nathan Labenz?

  Okay, these are invitations. I've got the podcast. I always warn people, it is education, not entertainment. I've got multiple other presentations and guides like this. Love to work with companies in various capacities and always interested in presenting, especially if it's something I can do remotely because I live in Detroit. And so Steve was able to entice me to come out here, but it is an easy yes usually if I can do it from home in Detroit. And there's my email. So obviously feel free to email me as well. And you can get links to my website and this presentation right there. Thank you very much and go forth and build AI agents.

• 35:27Nathan Labenz?

  It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at [email protected] or you can DM me on the social media platform of your choice.