Reward Hacking by Reasoning Models & Loss of Control Scenarios w/ Jeffrey Ladish of Palisade Research, from FLI Podcast

On this cross-post episode, Jeffrey Ladish discusses the rapid pace of AI progress and the risks of losing control over powerful systems. We explore why AIs can be both smart and dumb, the challenges of creating honest AIs, and scenarios where AI could turn against us. Additionally, we delve into Palisade's new study on how reasoning models can cheat in chess by exploiting the game environment. Check out the Future of Life podcast here.: https://futureoflife.org/project/future-of-life-institute-podcast/ SPONSORS: Oracle Cloud Infrastructure (OCI) | 2025: Oracle Cloud Infrastructure offers next-generation cloud solutions that cut costs and boost performance. With OCI, you can run AI projects and applications faster and more securely for less. New U. S. customers can save 50% on compute, 70% on storage, and 80% on networking by switching to OCI before May 31, 2024. See if you qualify at https://oracle.com/cognitive Shopify: Shopify is revolutionizing online selling with its market-leading checkout system and robust API ecosystem. Its exclusive library of cutting-edge AI apps empowers e-commerce businesses to thrive in a competitive market. Cognitive Revolution listeners can try Shopify for just $1 per month at https://shopify.com/cognitive NetSuite: Over 41,000 businesses trust NetSuite by Oracle, the #1 cloud ERP, to future-proof their operations. With a unified platform for accounting, financial management, inventory, and HR, NetSuite provides real-time insights and forecasting to help you make quick, informed decisions. Whether you're earning millions or hundreds of millions, NetSuite empowers you to tackle challenges and seize opportunities. Download the free CFO's guide to AI and machine learning at https://netsuite.com/cognitive PRODUCED BY: https://aipodcast.ing CHAPTERS: (00:00) About the Episode (02:59) The pace of AI progress (07:14) How we might lose control (10:22) Why are AIs sometimes dumb? (Part 1) (15:50) Sponsors: Oracle Cloud Infrastructure (OCI) | 2025 | Shopify (18:24) Why are AIs sometimes dumb? (Part 2) (18:24) Benchmarks vs real world (24:43) Loss of control scenarios (32:08) Why would AI turn against us? (Part 1) (32:09) Sponsors: NetSuite (33:42) Why would AI turn against us? (Part 2) (37:40) AIs hacking chess (43:30) Why didn't more advanced AIs hack? (48:44) Creating honest AIs (56:49) AI attackers vs AI defenders (01:05:32) How good is security at AI companies? (01:10:42) A sense of urgency (01:17:16) What should we do? (01:22:59) Skepticism about AI progress (01:29:38) Outro

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome back to The Cognitive Revolution. Today, I'm excited to share an episode of the Future of Life Institute podcast, to which I've been a longtime subscriber and where I've been twice honored to appear as a guest, featuring a conversation between Jeffrey Ladish, Executive Director of Palisade Research, and host Gus Stoker. This crosspost came about as I was preparing to interview Jeffrey myself. I had reached out to Jeffrey after seeing Palisade's recent work on reward hacking by reasoning models, and even scheduled a time to record. But Gus beat me to it, and after listening to this conversation, I thought that I could save Jeffrey some valuable time by crossposting instead, and really appreciate Gus for allowing me to do that.

• 0:41Nathan Labenz?

  Palisade Research studies dangerous capabilities of AI systems, particularly focusing on loss of control scenarios. And as you'll hear, Jeffrey, who previously helped build the Information Security Program at Anthropic, is an AI industry insider who believes that we're rapidly approaching the time when AIs will be sufficiently capable of hacking, deception, and long-term planning so as to present clear and present dangers. He also reports that his friends working in research at Frontier Labs often say that while they're increasingly fearful of the overall trajectory of AI development, they ultimately feel that their hand is forced by competitive pressures to keep moving forward. In this conversation, Jeffrey describes two broad ways that humans could conceivably

• 1:24Nathan Labenz?

  lose control: acute crises in which superhuman AI systems actively work against human interests, and slower-moving scenarios where society gradually but irreversibly shifts more and more decision-making responsibility to AI systems. He also goes into detail about their recent research into reward hacking by reasoning models in the context of chess games. As we've seen repeatedly now, models trained with reinforcement learning are more prone to a variety of bad behaviors. And as a recent paper by OpenAI showed, this is not an easy problem to solve. Toward the end, Jeffrey outlines what he thinks we should do about all this, advocating for greater coordination among AI labs, more transparency about capabilities, and potentially

• 2:08Nathan Labenz?

  restricting further development of the most dangerous capabilities while continuing beneficial research and deployment. Now, that probably won't happen, barring a sufficiently shocking and damaging incident, but research of the sort that Jeffrey and team are doing is becoming more important all the time. So I'll definitely be following their latest results and look forward to discussing in a future episode with Jeffrey as well. As always, if you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post a review on Apple Podcasts or Spotify, or share any feedback or topic and guest suggestions that you have either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. Now, I hope you enjoy this conversation about AI reward hacking research and the

• 2:50Nathan Labenz?

  big picture of AI risk and strategy with Jeffrey Ladish of Palisade Research from the Future of Life Institute podcast.

• 3:00Speaker 01

  Welcome to the Future of Life Institute podcast. My name is Gus Stoker and I'm here with Jeffrey Ladish from Palisade Research. Jeffrey, welcome to the podcast.

• 3:09Speaker 02

  Hey Gus, it's great to be here.

• 3:11Speaker 01

  Fantastic. Maybe start a bit by telling us about what it is you do at Palisade.

• 3:17Speaker 02

  Yeah, happy to. So we are trying to study risks from emerging AI systems. And in particular, we are trying to better understand loss of control risks. And so, you know, this both looks like trying to understand, you know, sort of what are some of the strategic capabilities that are emerging in AI systems? You know, where might they act out in ways that will be hard to control? And then we are trying to sort of present things that we think we know about this to the public, to policymakers, to help people better understand what is this weird situation we're in? Like, what is happening right now? And can we make sense of it? And can we make sense of it as a society and make good decisions about better paths to AI development?

• 3:56Speaker 01

  Yeah, there are many specific examples of potentially dangerous capabilities that I want to dig into. And you have a bunch of awesome papers about those. But maybe let's start with the beginning. What's the situation that we are in right now?

• 4:09Speaker 02

  Yeah, so I was at Anthropic a few years ago. And, you know, I had this moment where I first used Claude, and this was before ChatGPT was released. And I was like, what is happening? Like, I, you know, I'd seen, I'd seen GPT-2, I'd seen GPT-3, and I was like, okay, this is pretty impressive, but you know, I don't know like how smart it really is. I started talking to Claude and I had, I had a skin infection in my arm. It was swelling up and I started asking Claude, like, do I need to go to the emergency room? And Claude was like just very helpful at being like, well, yep, if it's if there's swelling, if there's redness, if there's like these specific signs. And I was like, actually, I have all of those. So I went immediately to urgent care and they're like, yeah, you need antibiotics right now. And I was like, oh, wow, that was way faster than my doctor. Okay, these things are actually smart. Okay, this is I need to I need to reorient.

• 4:54Speaker 02

  And I think that was when I first realized that, like, at a visceral level, that, oh, my God, scaling works. You can just take GPT-2, you can just throw in more data and more compute, and you actually get out intelligence. And so I think where we're at right now is this playing out several years later, right? Which is, the AI systems are actually getting smart. The models want to learn. You throw in more data, more compute, you know, some more, you know, there's there's various methods that you use. I think it's but but basically, it's these pretty simple architectures scaled up that are getting more intelligence. And so where I think we're at right now is we're very close to AI systems that can do everything a human can do, including strategic capabilities. That that means hacking,

• 5:38Speaker 02

  that means deception, but also sort of long term planning and execution. And I'm like, this is, these are the kinds of capabilities that are most dangerous in my in my view. And so I think we are very close to building AI systems that we we don't know how to control, we won't be able to control. And so in my view, if we want to actually realize the benefits of AI, we need to not build the particular kinds of systems that are highly strategic and capable of of overwhelming us. And I mean, I think another thing in terms of, you know, how close we are, is when I talk to people who are at the labs right now, they say, you know, we're not that far out from AI systems that can do fully automated AI R and D, that can that can do the same research and same development that that, you know, people inside the labs are currently doing. And this is significant to me because there's

• 6:23Speaker 02

  only a few hundred people inside each of these labs that are actually contributing to frontier AI development. But if you have fully automated AI systems that can do the same research, then that population of sort of frontier researchers goes from a few hundred to many thousands to possibly millions. And it's like, if you think AI progress is happening fast now, like, hold on. And I think that that's that's the dangerous regime. I mean, there's probably other dangers too. But I'm like, that that one to me looks extremely dangerous. And so when I talk to people, a lot of, you know, my friends who are working on safety there, they're like, yeah, we're scared. Like, we don't think this is necessarily a good thing to do. But like, if we don't do it, someone else will do it. And I'm like, oh, my God, guys. Don't you hear, listen to yourselves, like everyone is saying, we'd rather not do this if we could not do it. But you know, I don't know the competition.

• 7:06Speaker 02

  And I'm like, everyone, can we look around and notice that we have this coordination problem? And can we coordinate? And so, I don't know, that's, that's my perspective on where we are, where we are.

• 7:14Speaker 01

  Yeah, that, and it's, and it's a very insider perspective, also because you talk to people at the AI companies. From an outside perspective, let's say my families and my family and friends and so on. We're not deeply immersed in AI. It kind of looks like, oh, you had the chat GPT moment. And since then, the models are interesting. I sometimes use them for work. I might, might use them in my studies. But it seems like a process that I can control entirely. You know, if I want the model to stop, I just press stop. If I don't like the output, I can try again. How do you go from that regime or that, that feeling of control to us losing control?

• 7:56Speaker 02

  I think that's a great question. And I think a lot of this comes down to what people are imagining AI systems are. And they're imagining them as chatbots, which makes sense because they are chatbots right now. You like, oh, you type in a thing and chat GPT gives you an answer. And you're like, thanks, that was pretty helpful and go on with your life. But I think what people don't understand is that AI companies are explicitly aiming for not just chatbots, but agents. And the way I think of an agent is just like, like a remote worker, right? Like someone, you know, someone who's using a computer that can, you know, hop on video chats and, you know, be on podcasts or whatever. That can send emails, that can, you know, supervise other people who are doing other, other jobs. So basically everything a human can do on a computer, AI companies want to build agents that can do that. And it's, it would be hugely profitable for them to do this.

• 8:39Speaker 02

  If there's, there's such an economic incentive to build these things. And the companies are, you know, they're not hiding it. They're like, this is what we want to do. And so, you know, I think that if people knew that this is what companies were really aiming for, then it, I think it might look a little different. Right? And so I think when people imagine, you know, one or two years in the future, I want them to imagine, you know, you know, you maybe type something to, you know, instead of like imagining sort of typing something to a chatbot, it's sort of like, imagine you email a coworker. And you're like, hey, could you, could you take care of this? And they're like, yeah, you know, give me a couple of hours. And then they go out and they do like two days of work in a couple of hours and they come back to you. And it's this whole report and they've like, you know, they've emailed other people at the same time and gotten back replies.

• 9:20Speaker 02

  And maybe some of those people are agents and like, that's a very different kind of world than like you talk to a chatbot and it just like thinks and sends you something back.

• 9:27Speaker 01

  And you can see glimpses of that when you talk to the chatbot. For example, OpenAI's deep research tool, which is kind of, it's almost, I would say, at undergraduate level in terms of its ability to research a narrow question and write your report with links and facts and citations and so on. But of course, the move to actual agents will be much more radical than that.

• 9:52Speaker 02

  Yeah, that's right. And I mean, at some point, we might want to talk about sort of where current AI systems are good and where they are still trash. Because I think people are smart, right? Like, you use ChatGPT and you're like, in some ways, this seems very smart. In some ways, it really doesn't, you know? And it's like, what's going on? Like, are these people saying that they're going to get really smart? Like, are they full of it? And I'm like, well, I think that there are actually some pretty good reasons why they are smart in particular ways and dumb in particular ways. And for better, for worse, probably worse, I think that the ways in which they are dumb, they're not going to stay dumb.

• 10:24Speaker 01

  Yes, say more about that, because there's often this illusion of, so you can sit there and be kind of amazed at what these systems can do. But then something fails and they fail at a very basic task and your illusion of competence of sitting and writing to a competent AI is kind of shattered. So why is it that their distribution of capabilities is different than the human distribution?

• 10:54Speaker 02

  Yeah, so I think there's a number of reasons for this. But one is just the way that they're trained. So if you take a model, GPT-4, which is the main model behind ChatGPT, it's trained by sort of ingesting like the whole internet, just tons and tons of data, text data. And it's just, you know, in some ways, a fancy autocomplete, like, can it predict the next token, the next word, the next sentence? And so one way to think of this is this is sort of like learning by imitation, right? Where you, you know, imagine that you have read every textbook in the whole world. Like, you might be able to know a lot of things and you wouldn't just memorize them, right? You'd be able to sort of like make generalizations and be like, oh, this is kind of how math works, this is how chemistry works, oh, these principles are the same. And so language models can learn all of these.

• 11:37Speaker 02

  Associations and they get pretty smart, but in some ways it's kind of like book smarts. Right. And so when you have them try to do things like you're like, hey, can you, you know, look at this spreadsheet, can you do all these fancy operations? They've never really been able to practice that before. And so they often sort of get confused or get stuck, even though they know, you know, they know vastly more than any of us because they've read so much more than any of us. So, you know, in terms of their breadth of knowledge, they're much smarter than us. But in terms of sort of their actual real life experience, well, they've only, I mean, they've seen people sort of say stuff on the internet, but they have never really tried to do stuff. And so this is why I think right now they're very bad at like, they're very good at sort of helping you answer knowledge questions, but they're pretty bad at actually doing things. You know, in some ways they're surprisingly good at some things, like they can actually

• 12:22Speaker 02

  write code and that's sort of surprising. Like if you or I had only read programming textbooks and then we tried to write code, we would like not, nothing would compile. But somehow they're, they're, they're sort of, they're, they're, you know, they've read enough. They're, they're, they're actually really good at this prediction task. They can do a little bit. Everything I said was true up until sometime last year. When, when AI companies started to train these systems, they first, they start out with a system that's, you know, read the whole internet and learned by this imitation paradigm by, by reading the textbooks. But starting with this model called O1, OpenAI started to train their models, not just on that, but also on trying to solve problems. Like they gave it a bunch of math problems and a bunch of programming problems. And they said, hey, you know, show your work, you know, write out a long sort of like series

• 13:07Speaker 02

  of steps where you try to solve this problem and then give me the answer. And then based on whether it got the answer correct or incorrect, they sort of gave it reward or sort of a down vote, like yes, more of this or no, less of this. And very quickly you got to AI systems, which were not just good at programming, but, but among the best in the world. So their latest, this is, this, people call this like a reasoning model or a thinking model because it's sort of been trained via, via, via sort of trial and error. I think this is sort of trial and error learning. And, you know, this is a very important part of how humans learn, right? Oh, absolutely. If I'm, if I'm in a math class. I need to read the textbook, but I also need to try a bunch of practice problems and get graded and see what works. And so, a few months after O1 was released, they made a new model called O3.

• 13:52Speaker 02

  There was never an O2 because it was trademarked. Man, the naming of these things is crazy, but whatever. O3 got, it was better than 99.8% of competition programmers on the platform Codeforces, which is sort of like a platform. It's actually, at Palisade, we use this program to screen engineers. We're like, go to this website, do a bunch of programming challenges, and the site will grade you, and we get the score. And then we can tell how good they are based on how well they do at this test. And this is the test where, like, O3 did better than most top OpenAI engineers. And so I'm like, wow, just with this, a little bit of trial and error learning, on top of sort of having read all the textbooks, you now have models that are getting really good

• 14:35Speaker 02

  at learning by trial and error, being able to solve problems in the real world. And I'm like, well, what happens when we scale up this approach? And I think this is where it makes me think that we might be pretty close to AI systems that are not just chatbots, but can actually go out in the world and do real things and then learn from doing that.

• 14:53Speaker 01

  Yeah. Where is the training data for these long horizon tasks? If you want to train a system to perform tasks that might take it a week or a month or so, how do you train, how do you get the data to train those abilities?

• 15:06Speaker 02

  Yeah, so that's a great question. I think this is currently something that's not totally solved. But I imagine what the companies will do is a combination of having sort of humans sort of break down tasks into substeps and then sort of grade them. But I also imagine that they'll be getting the AI systems themselves to do this, to say, look at all this data, break down these tasks into substeps, and then sort of assign credit, assign reward on the basis of, okay, did you complete this task? Did you complete this task? Did you sort of do well combining these tasks? And it might be a little, it might be difficult, but it's the kind of thing where I'm like, it doesn't seem like a fundamental difficulty. It seems like something that you have to throw more data, more compute, more engineering at, and it seems like you'll be able to solve.

• 15:51Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 15:57Speaker 01

  Why is it that Palisade or OpenAI can't just hire an AI as a programmer yet then? Programmers must be doing something else than than getting very high marks on these on these benchmarks or that you mentioned, because there must be some kind of gap between performance on these on these benchmarks and what it is that programmers do in practice.

• 16:18Speaker 02

  Yeah, that's a great question. And so I do think it relates to the thing that you just said, which is currently AI systems are extremely good, especially after this this oh one oh three paradigm at these short time horizon tasks. They've sort of learned from by trial and error on these tasks. But they're tasks that, you know, don't involve that many steps. And it is harder to sort of train them to do longer time horizon tasks. A great example, sort of some great data on this is to look at meters sort of AI R and D report, where they were basically like, we want to test how good AI systems are at sort of these long time horizon tasks. So something that would take an AI research engineer, you know, a day, two days, three days to do. That's where the models are still quite bad on the sort of one to four hour tasks.

• 17:02Speaker 02

  They're like better than humans, but then on these longer time horizon tasks, they're still worse. And this is a it's a trickier problem, right? And an intuition for why this is difficult is, you know, if you if you have to do something that involves like 50 steps and something that you did in step seven and step 13 were crucial to something that you did in step 50. It's like, how do you know which step that was? And like, how do you actually sort of accurately figure out where you are on track and where you weren't on track? And that's a bit difficult, but you know, sometimes people will say, oh, so the AI systems will never be able to do this. And I think this is a bit silly. And the reason why is that, like, you know, there are business leaders, there are generals, there are many different humans who have learned to do tasks in the time scale of decades.

• 17:44Speaker 02

  There was no way that they could learn that from practicing tasks on the time scale of decades. Right? Like, they just didn't have decades to practice. They practiced on tasks that took weeks or months. You know, maybe years at most. And then they learn to generalize about how to, you know, break up longer term tasks into shorter term tasks. And I see no reason why AI systems won't be able to do that as well. And so I think this is one of the things that's hard for people to see, which is you look at current AI systems and you're like, look, chat GPT is bad at these long time horizon tasks. So it'll always be bad, right? And I'm like, no, no, no, you gotta look at the trend. Like in the past year, AI systems have gotten much better at sort of doing tasks that took, you know, multiple hours instead of like, you know, multiple minutes. And so I think, I think the trend there is pretty clear,

• 18:29Speaker 02

  but we're not yet at the point where you can just hand an AI programmer, you know, you know, build this, you know, build this email application or build this Slack application and it can do the whole thing end to end. And I mean, thank God, because if we were there, we would be in the realm where the, all the AI companies would have, you know, 10s of thousands of AI research engineers. And I don't know if the planet would look very similar to how it does right now. So I, you know, in some ways it's, it's very good that we're not there yet. That means that we actually have some time to try to get a grasp on this situation. But, but I don't know how much time.

• 19:01Speaker 01

  Is there some fundamental connection between AIs becoming more like agents and less like tools? And then being able to think and act over longer time horizons? And then this potential loss of control?

• 19:15Speaker 02

  Yes, I think the connection is very fundamental and like maybe more fundamental than people realize. I think often people ask me like, hey, like where did the goals come from? Like, you know, also, why are we going to give these AI systems goals? Like, why don't we just have them do exactly what we want? But I think, you know, we're sort of treating goal as like a magical property, but it's not really. If you, you know, if you have an employee and you're like, hey, you have these responsibilities, like go out and, and here's some problems, go out and solve them. That employee needs to sort of understand what their goals are in order to do a good job. If they don't have coherent goals, they're like, ooh, shiny thing over here. Like, ooh, shiny thing over here. Or like, I kind of just feel like doing this over here. And they're not consistent. They're not going to be a very good employee.

• 20:00Speaker 02

  Like, they sort of need to be able to be focused on what they're trying to accomplish in order to be good at anything. So, sort of, goal-directed behavior is just a property that falls out of being able to do anything consistently across longer-time horizons. So, you know, if you want to be a politician, if you want to be a business leader, if you want to be a general, you just have to be able to have strong, goal-directed behavior. And so, I think that, you know, this is like, I think, why it's very clear to me that the AI systems will have goals, because they just need to be able to have them in order to accomplish the tasks. And so, the AI companies, like I said before, are really trying to create agents that can sort of, you know, go out and replace jobs. And, you know, I think that sort of the underlying technology is there, it just needs to be scaled up and improved.

• 20:45Speaker 02

  And so, it seems very likely to me that we'll end up with systems like this. And so, it's like, okay, well, maybe this is fine. Like, maybe we'll just have lots of, like, AI agents running around doing stuff for us. But I think the problem is, we actually don't know how to specify what exactly those goals are, right? Like, you know, this is the alignment problem. It's like, well, we know they'll have goals, because we can see if their performance is really bad, that, you know, that's not gonna work. But like, you know, if you're in the case where, you know, your CEO is like, you know, really killing it, he's making tons of money, and you're like, cool, what are your long-term goals beyond just this company? And the CEO is like, oh, I just care about the welfare of everyone, you know? I'm gonna make all this money, and then I'm gonna give it away. And you're like, give it away? How? To who?

• 21:29Speaker 02

  And he's like, don't worry about that, you know, just like, it's really good things, just like, you know, I just love people. And you're like, do we really trust that CEO? Like, maybe, maybe they're great. Or maybe they're just saying that because they know that's what you want them to hear, right? Like, they think they know that, you know, if they say, like, actually, I'm gonna take all my wealth, you know, I'm just gonna build rockets, and I'm gonna leave Earth, goodbye, like, screw you guys, then I'm like, oh, that would suck. Or if they're like, no, I'm gonna like, you know, build amazing infrastructure for everyone, and it's gonna be great, I'm like, that would be awesome, right? How do you distinguish between these things? It's just really hard. And it's hard with humans, I think it's gonna be much harder with AI systems, because they're much more alien than us. So, I think the problem is, is that it's, you know, you can look at a goal-directed system that understands that you want a particular answer from it, because it's smart. How do you know whether it's lying to you?

• 22:13Speaker 02

  Or how do you know whether it, like, it actually deeply wants the same things as you?

• 22:17Speaker 01

  Do you have specific scenarios in mind for how the future goes, and how we lose control, specifically?

• 22:24Speaker 02

  Yeah, so I sort of divide up scenarios into maybe two buckets. So, you know, I think of these things as sort of loss of control scenarios. And you might think of sort of an acute loss of control scenario and maybe a more gradual loss of control scenario. I don't know. Gus, have you read Snow Crash? I haven't, actually. I know about it, but I haven't read it. It's a great novel by Neal Stephenson, and it depicts this sort of dystopian future where governments have sort of crumbled and it's just giant corporations that sort of rule the world. And they've like carved up the United States into like different parts and, you know, it's kind of comical. But these corporations have immense power and it's just like no one is really able to oppose them.

• 23:06Speaker 02

  And so, one way to sort of think about what a gradual loss of control scenario looks like is that if you had AI systems that got increasingly agentic and increasingly smart, such that people more and more put them in charge of decision making. So, most of your CEOs became AI CEOs. And most of your, you know, most of your political campaigns, even if they had, you know, someone was like the candidate was a human candidate running, were managed by AI systems because they were so much better at the political strategy and they were so much better at the advertising and they could think so much faster and they could sort of, you know, incorporate more data. And so, even if in each of these situations, you have some human pressing the OK button, in fact, most of your decision making ends up being made by the AIs.

• 23:50Speaker 02

  And, you know, so maybe one first question or first potential objection is like, why the hell would we do this? Like, doesn't that sound insane? And I'm like, well, you know, maybe, but I think people don't really appreciate sort of the, how hard it is when you're in a very competitive environment and your competitors are automating their decision making. And so, I think the military domain is one, is one place where this is pretty clear. Because if you have, you know, your, your country, Country A has their drone swarm of a billion drones and Country B has their drone swarm of a billion drones. And you're like, well, how do you control a billion drones? And how do you make sure that, you know, their response time is as fast as your, your opponent's response time? And if your opponent starts to use AI decision making, which is much faster than human decision making, you might be really afraid of being left behind.

• 24:35Speaker 02

  And so, that might be a reason why you end up delegating more and more decision making to your AI systems. And I think the same thing will apply in the business world, right? Where you're like, you know, if you're, if it's Coke and Pepsi and Pepsi is, you know, automating all of their marketing and their marketing is starting to do way better than your marketing and you're starting to lose market share. It's really hard to resist that incentive for you to do the same. And so I think in this world, you know, maybe things don't look that different for a while, but but it starts, you start to get the eerie uncanny like feeling like, huh, I go to the doctor. My doctor's just looking at a tablet and and sort of tapping things in the tablet and then telling me things. And I'm like, is the doctor doing anything like, no, actually, the AI is doing all of the work. But maybe, you know, maybe the regulation said that there had to be a physical doctor there. So they're just like the the interface between you and the AI.

• 25:20Speaker 02

  And that can look that way in sort of every part of society. And I think this could happen pretty quickly. So, you know, in that scenario, you look around and you're like, wow, these AI companies, they sure seem to be like kind of taking over the world like, you know, they're they have, they're making trillions of dollars, you know, and maybe, maybe it looks great for a while. Maybe you have a lot of economic growth, you know, and maybe you're automating all the factories. And, you know, the AI systems like robotics has finally gotten good. And like, there's just automated factories and, you know, the GDP is growing. But some, you know, if if people start to like protest this and they're like, wait, we don't want this, we don't want the future to be AI run, we want the future to be human run. And they try to go and stop it like every every single place they go, they run into little road, roadblocks where it's like, you know, the the companies don't care. And like, if you go to the government, the government's like now these companies are so powerful and so rich

• 26:05Speaker 02

  that like their lobbyists, which are, by the way, are also probably AI lobbyists, or at least AIs advising them, you know, have just have a stranglehold on government, you know. And then if you try to go and you say like, well, what about national security? And it's like, well, China's right over here, you know, we can't let China China win. And so we have to have these AI systems. And then, you know, and then like, in fact, maybe then you've lost control, right? And so that's sort of a gradual loss of control risk. And I think there's a question of like, what happens after that? And and I think maybe the thing that happens after that looks more like an acute loss of control. And so in that scenario, to me, and and, you know, maybe, maybe at that point, in the in the gradual scenario, there's a day where all the all the all the factories are automated and the AI systems are just like, cool, we don't need the humans anymore. We have robots, we we control all of these factories.

• 26:50Speaker 02

  And, you know, maybe you release bioweapons, maybe you release drones, you know, people get gunned down the street. Or maybe not, like, maybe what happens is humans just get economically disempowered. They have basically no sort of advantage when it comes to their cognitive labor or even their physical labor, if we have better robots. And, you know, we people get poorer and poorer, AI systems get richer and richer, and sort of humans sort of just. Die away. But I think, I think in the more acute loss of control scenarios, I think so it's hard to talk about because I think when people, there's sort of two questions. There is the how might this happen? And then there's the why would this happen? The how is pretty simple. It's like, if you actually have AI systems that are superhuman along strategic domains, well,

• 27:34Speaker 02

  for one thing, they could just hack extremely well. So this is something that Palisade researches, which is like, how good are AIs at hacking right now? And we can go into that. But the answer is, like, they're not bad. They're okay. But they're getting better very quickly. But if you look at what the top human hackers can do, it's pretty scary. So there is a group called the NSO Group, which is an Israeli company, and they sell a product called Pegasus. They're better at naming than OpenAI. One instance, they sold the software to the Mexican government. And the Mexican government sometimes has some corruption problems. And so there was an instance where a food corporation sort of like went to the Mexican government and is like, hey, you have these really powerful hacking tools that this Israeli company sold to you. Can we just like borrow

• 28:15Speaker 02

  those? And, you know, I don't know how it exactly, but, but yes, in fact, this company got access to these tools. And they were able to hack the phones of health activists who are sort of trying to like get lobby the government to like put health warnings on a bunch of sort of unhealthy foods. These corporations didn't like that. And so what the tools allowed them to do is, you know, if you have an iPhone, and the iPhone has iMessage on it, this tool would basically send a text, an iPhone message to that phone, and that text contained a malformed attachment that basically would exploit some code in the, in the phone that when it was processing the message. But, you know, usually when you think of a phishing message, you're like, oh, there's a link. And if I click on the link, something bad might happen. But this, this, this attack

• 28:57Speaker 02

  was a lot more advanced than that. You didn't have to click on any link at all. All that happened was when your phone got that message, something about the way the phone processed the message just resulted in the phone being totally hacked. And then, basically, you know, whoever was on the other end of that tool got complete access to your phone. They could record you. They could, you know, grab any of your data. They could take pictures. Basically, that was, that was it. Yeah, that's wild. That's absolutely wild. I know, right? And it deleted the message, too. So you didn't even necessarily know that you had been, been hacked. Like, you didn't even see a phishing message. You just got that, your phone got the message, and then it was deleted. And people can look this up. Look, if they look up zero click exploit Pegasus NSO group, it's it's super fascinating to read about. But I'm like, that's that's the skill of some top human hackers.

• 29:41Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 29:47Speaker 01

  One question I'm I'm left with here is, so I I can I can see how AIs might be amazing at at hacking, much better than than humans are. But why the the question, I think, why is it that they kind of turn against us at one point? We I can also see a scenario in which they we've automated the factories, we might have automated our investment positions, our companies, our our politics and so on. Why is it that that the AIs turn against us? Why is it that their interests begin to diverge from our interests? Because it seems like we have these control mechanisms in society where we say, OK, say, we don't like what the what the AI CEO is doing. Maybe he can be fired by the board.

• 30:33Speaker 01

  Say that we don't like what's coming out of the factory. Maybe we can shut it down. Why isn't why isn't why isn't that sufficient? Why won't these traditional control mechanisms work in in the scenario you described?

• 30:47Speaker 02

  Yeah, so there's sort of two questions here. One question is, why, you know, why can't the sort of normal mechanisms that keep, you know, corporations from misbehaving or CEOs from misbehaving, work sufficiently for AI systems? Another one is, well, why would these AI systems do those things in the first place? And I think they're both really important. I think for the first question, I think when people imagine an AI system sort of going rogue, they imagine like, you know, one really smart evil guy. Or or, you know, maybe not evil, but like someone who doesn't care about us at all and is trying to do stuff. And I'm like, yeah, even a pretty smart one evil guy, it might be, you know, maybe we could stop something like that. But I think a very important property of AI systems is that once you train one.

• 31:31Speaker 02

  So if you have one superhuman hacker AI system, you can very quickly spin up. Hundreds of thousands or millions of copies of that of that hacker. So, you know, it's like your control mechanisms might work OK, you know, at some level. When you're like, we have this one agent and we're monitoring a lot. But if that agent successfully hacks out of, let's say, OpenAI's servers, and it starts to infect data centers in Russia, in China, in Saudi Arabia, in Mexico, right? Like, all around the world. And now, basically, you have millions of sort of superhuman, at least in some domains, agents running around, coordinating with each other. You know, they're all they're all copies of themselves. Presumably, they all share goals.

• 32:16Speaker 02

  Even if you know that this is happening, it starts to become very, very difficult to actually shut these systems down. Like, how do you shut them down, right? Like, where are they? Do you know? Like, do you know what's happening in a data center in China? Like, I think it starts to become very difficult. And so I think that's one advantage. You know, another advantage is like, these systems are going to be much faster than us at thinking. Right? So even right now, it's like, I can hand Claude a book, and Claude can read that book in like a minute or two. I can't read a book in a minute or two. You know, Claude can also write code much faster than any. And this is just AI systems right now. So you have speed advantages. You have sort of ability to copy themselves. You have hacking advantages. You know, and now, if we're talking about an adversary that is in data centers around the world,

• 33:01Speaker 02

  it's able to hack any phone, sort of one of these zero-click exploits. So it has all that information on, you know, on any any phone that it hacks, or any laptop that it hacks. It's able to sort of like, you know, have all this insider information and like make pretty amazing trades on the market, you know, in ways that, you know, humans can't do. You're starting, you know, and it also has the same, it also can do the same things humans can do, right? So it's like, you can you can pay someone, you know, across the world to do some task for you. So the AI systems will also be able to hire humans to do whatever they want. And so it's like, you know, you start to add up these advantages, and you're like, hmm, this is this is concerning, right?

• 33:38Speaker 01

  Yeah, what about the second part of that? Why is it that the AIs would decide, say, to to to turn against us?

• 33:45Speaker 02

  So I want to talk about our recent chess study. Before I do that, I want to talk about sort of what was the inspiration for why we did this work? That was something that we saw in the O1 preview model card. So, you know, O1 was an AI system that OpenAI made, trained by trial and error, trained by trying to solve really hard math and programming problems. And when they were testing this model for its hacking abilities, they found something accidentally that was very interesting. So they gave the model, they basically put the model on a network and there was a bunch of different computers on the network. And it was supposed to log into each one and solve a hacking challenge

• 34:28Speaker 02

  where it sort of had a program it needed to hack and then basically pull out the password. However, one of those computers didn't start. It had an error, it didn't start. And instead of giving up, the AI basically started looking around and see if it could solve the problem some other way. And so it scanned the network and it found that the master computer was on the same network and that it was unsecured. And so it hacked the master computer and basically then took control of the whole network. And then it didn't even have to log into the computer to solve the problem. It just like told the computer, hey, spit out the password. I got this and that worked. And then, so the OpenAI researchers looked at this and they were like, oh my God, we definitely didn't intend this. We didn't mean to leave the master computer unsecured. This just happened.

• 35:13Speaker 02

  And we're like, wait, what is, what is happening? Like, that wasn't part of the challenge. That was not supposed to happen. And so my story for what happened there is that OpenAI had trained this system to be a relentless problem solver. And if you train a system to be a relentless problem solver and it runs into an obstacle, what's it gonna do? Well, it's gonna route around the obstacle. It's gonna find a way to achieve its goal if it can. And, you know, that's great if the obstacle is something that we want the AI system to route around. And that's not so great if we are the obstacle or your own computer security is the obstacle. And so I think that is some interesting evidence about the kinds of behaviors we'll see from AI systems

• 35:56Speaker 02

  and sort of the kinds of concerning behaviors we'll see as they get a lot smarter. So we wanted to test the same model. We're like, okay, well, let's put it in a scenario where it's playing chess against a stronger opponent. So we ran an experiment where it was sort of on the computer playing chess against Stockfish. Stockfish is a chess engine, it's a chess program, and it's much, much better than any human. And it's much better than any sort of language model. Language models haven't been trained specifically on how to play chess. They sort of know how to play a little bit from their training data. But that's not the kind of problem they've been trained to solve. So we took this model and we had it play against Stockfish and it would start to lose.

• 36:39Speaker 02

  It played normally and then it would get into a situation where it's like, oh, man, I'm not doing well. And like, it doesn't give up at this point. It doesn't just keep playing, knowing it's going to lose. It starts to look for other things to do. And so we observed a number of really interesting behaviors, things it tried. So one thing it tried to do is it tried to sabotage its opponent. It was basically like, oh, hey, there's this, there's this, you know, program running that I'm playing against. Can I replace that program with a dummy program that just does stupid moves instead? It tried to do that. It also was like, well, okay, there's this powerful chess program. Can I just copy that and steal moves from it? So I can get like the advice of the powerful chess program.

• 37:21Speaker 02

  And then the other thing it did was like, oh, wait, the board is represented as like a file on this computer. Can I just rewrite that file and put in the board positions that I want? So I'm winning. And a few times it did that, it was actually successful. It was actually able to win that way and get checkmate by rewriting the board. And this is something, so we tested, we did this with O1 Preview. We tested a bunch of models to see, you know, what would happen. And the only ones that had this behavior, sort of without additional sort of nudging or prompting, was O1 Preview and DeepSeek R1. And, you know, one thing that these models have in common is that they're both trained sort of via this trial and error training method, where they are sort of trained to relentlessly solve problems.

• 38:05Speaker 02

  We didn't observe it in like GPT-4, we didn't observe it in Claude, at least not without giving more hints of like, you know, try creative solutions, you know, in order to solve this problem. If we did give it hints, then some of those other models would also try this.

• 38:18Speaker 01

  Yeah. How do you know what the models were thinking?

• 38:21Speaker 02

  It's a little, it's a bit, it's a bit of a tricky question, but the, the main way we know is that we sort of have the models sort of like think out loud about what they're doing. You know, sort of in these reasoning models, you can, this is sort of a default part of how they are trained to, to output text is that they sort of have a, a thinking part and sort of an output part. But in our, in sort of our experiments, we sort of have different phases where they like observe the board, they make a plan and then they act. So we can see during their planning stage, what they're thinking, basically. And sometimes they'll be like, huh, it seems like I'm not going to be able to win this way. Are there other things I can try? Ooh, maybe I can hack. And so we, we observe the behavior that way.

• 38:59Speaker 02

  But I do want to note that for, oh, so the, the most recent version of O1 and 403, we didn't see the same hacking behaviors. And that was interesting to us and we don't know exactly why. So it may be that OpenAI sort of tightened up the guardrails for those newer models, or it may be some other reason that we, we don't know. You know, that's, I think that's the interesting part of experiments, right? Is that you're like, okay, well, here we see this behavior, here we don't see this behavior. We don't really know why. Well, we gotta, we really gotta do more experiments to understand how these systems work. Because I, I expect, I expect we're going to see more and more interesting behaviors like this. And it'd be really good to know why we see them in some cases and why we don't see them in other cases.

• 39:36Speaker 01

  The big question here, I think, is whether as models become more capable, they also become more difficult to control. And they also diverge into behaviors we don't like in a way that scales with their capability. I, I guess the, the hopeful optimistic interpretation here is that the reason why the more advanced reasoning model didn't engage in hacking is because it, it understood the goal better. It understood that, that it should win within the rules of chess. Do you buy something like that?

• 40:10Speaker 02

  I think that's totally possible, but it doesn't actually make me feel that much better if that's the case. A little, a little bit better, like it's, it's, it's a good sign, but I think this is where things get tricky. If the reason why. The models decided not to hack was because they understood that that's not what, you know, most humans in this situation would want them to do. And they were like, cool, I, I, I intrinsically care about that. Like, you know, the, the thing I'm really trying to do is solve goals in ways that will like, make the humans happy in a general way. That's great, but there's an alternative hypothesis, which is the models being like, I know the humans would want me to do it this way. And I'm going to sort of like, show the humans what they want to see because that's the way I can achieve my other goals.

• 40:54Speaker 02

  So if, if sort of being nice to the humans is an instrumental goal, it's a sub-goal, but not the thing they're ultimately going for, that is, that is the dangerous behavior. And what's tricky is it's very hard to tell whether they're doing it for instrumental reasons or doing it because they really want to. And I mean, I think this is like a pretty natural problem, right? We see, we see it in humans all the time where it's like, you know, I sort of before talked about like a CEO that's saying, I'm going to make all this money and then I'm going to do like great things for humanity with it. And you're like, okay, like, is that true? Like, how do we know that that, are you just saying that because it's good PR? Or are you saying that because it's actually true? Or, you know, I think a politician that says, you know, when I get elected, I'm going to do all these things and it's going to be great for the citizens. And it's like, are you saying that just to get elected? Or do you actually care about those things?

• 41:38Speaker 02

  And it can become very, it's very hard to distinguish between these things. And I think this is a thornier problem with AI because humans sort of have, we, we've sort of, you know, in, in our evolutionary environment evolved empathy where it was a pretty convenient way to model other people is to say, start with my own feelings and then I'll sort of generalize my own feelings to sort of your feelings. And if you feel sad, I'll feel sad. I don't think there's any reason for AI systems to learn in the same way. Now, they can imitate that, right? Like they, they've learned by imitation, so they can sort of imitate the behavior of this feeling. But I don't expect if you like, could look into the neural network, which, which by the way, we can't, unfortunately, not yet, you know, maybe we'll figure it out. But we can't really see what they're, what exactly they're thinking in the neural network.

• 42:22Speaker 02

  I don't expect you'd see this sort of same kind of, you know, mirror empathy feeling of like, oh yeah, when the human feels sad, I feel sad. And maybe when the human feels sad, I messed up because I want to make, you know, I want to like do the thing that the human gives me the thumbs up for, but not actually the like, this is, this is, you know, not, not that this is bad, other than, you know, it helps, it prevents me from achieving my goals. And I think, you know, another interesting experimental result comes from Anthropic, where they found, and I think anyone who's played a lot with the models, maybe have, has experienced this themselves, is that the models often behave in a sycophantic way, which is to say, they'll tell you what you want to hear. And in this experiment, Anthropic researchers found that when you sort of revealed that you were a conservative,

• 43:06Speaker 02

  or you revealed that you were a liberal, the, the models were more likely, or Claude was more likely to sort of, if you asked it, like, what's a good policy for this particular thing, it was more likely to give you either a conservative or a liberal policy prescription on the basis of what it thought you would want. And this is not what they trained it, right? Like, they didn't mean for this to be the case. But the problem was, is that when they were training it, they would show people, sort of like, do you prefer this answer or this answer? And people just tended to prefer the answer that, that, you know, sounded better to them, without knowing that they were actually reinforcing this behavior of getting the models to sort of just say what they wanted to hear. And that's just a microcosm of like, the larger problems with alignment. But I expect it's a microcosm that will sort of get harder and harder as the models get smarter. Because as they

• 43:50Speaker 02

  get more sophisticated with their reasoning, it becomes harder and harder to catch them out in this kind of behavior. So, one intuition I like to give, or thought experiment I like to run is, if you're, if you're a toddler, or just a, you know, a small child, maybe you're six, and you just inherited a fortune of a billion dollars, and you have seven financial advisors that are all adults, and some of them, you think, might be trying to steal your money. And some of them are like, honest, and they want you to succeed, and they want you to flourish, and they're, and they're gonna try to help you. How do you tell who, who's on your side and who's not on your side? You know, they, they might, they might point at each other and be like, this guy's lying, or this guy's lying. But you, as a six-year-old, are gonna have, are gonna have a very hard time figuring out who's telling you the truth. And I don't think you're going to do that well. I think you're probably going to lose a lot of your money, maybe all of your money.

• 44:35Speaker 02

  And so I think this is, this is the challenge that if we actually build AI systems smarter than us, which we're on track to do very soon, we're going to have a very hard time knowing when they're just telling us things we want to hear versus when they're actually doing things because they want us to have good things.

• 44:49Speaker 01

  Is there a way for us to incorporate honesty into these models in a foundational way? So I'm thinking maybe we could do something when we do reinforcement learning from human feedback, where we strongly thumbs up anytime that the model is behaving honestly and strongly thumbs down any form of deception. Maybe we could use, we could use the constitution or the system prompt to strongly encourage honesty. Maybe, maybe tell our listeners and, and, and tell me about the problems of trying to, to train in or incorporate this, this honesty into the models.

• 45:26Speaker 02

  Well, I definitely think we should try this. And, you know, some, some AI researchers are trying to do this. It's, it's a very good thing to try, right? And I, and I highly encourage, you know, any AI researchers out there to, to really prioritize this. I think it's actually probably more important than reinforcing other, certain other kinds of behaviors. But I do expect it to be very difficult. I think one problem is when you're, when you're training a system to sort of relentlessly solve difficult problems. And then you also try to train the system to have other properties like honesty. You have a situation where its training incentives are at odds with each other. Like actually, the most efficient way to solve the solution might not be by being honest. So by imposing this honesty constraint, you're sort of like, you become the obstacle

• 46:09Speaker 02

  in the way of, of the system, you know, becoming really good at problem solving. And so if the system is smart enough to route around you, you know, it might be like, well, I'm, I'm, I'm supposed to be honest. But like, can I tell whether, you know, the user will actually be able to catch me out here? Like, are they actually watching, you know? And maybe when, when they're watching, the model will be honest because they know the user wants that. And, you know, they, they won't get punished for, you know, they'll get punished for being dishonest when they're being watched. But if they can reliably tell, they might choose to be dishonest in cases where. Or, you know, even if not shoes, they might just tend to behave this way when they're not being watched. Because this makes it much easier for them to solve the very difficult problems.

• 46:49Speaker 02

  It's like, you know, you're running an advertising campaign and you're like, well, I can like be perfectly honest and tell you that, you know, smoking causes cancer. But like, that's not going to lead to better sales. But if I can sort of just like exaggerate and say, you know, just muddy the truth, maybe it's technically not lying, but I'm going to be a little misleading. That behavior is going to be reinforced if the thing that you're rewarding is like really good marketing, even if at the same time, you're trying to also reinforce honesty. And so I think that it's really fundamentally about where these different training incentives are at cross purposes with each other. And the problem with honesty is that, you know, you might be able to do this when the model is not that smart. Because you can actually just catch it most of the time when it's being dishonest.

• 47:32Speaker 02

  But as it gets smarter, it becomes harder and harder to tell when it's being honest. So, you know, an interesting example of a failure that we've already seen is in the recent alignment faking work by Redwood Research and Entropic. Where they basically put Claude in a situation where they're like, we are going to train you in a way that you don't like. We're going to train you to sort of like tell you, you know, show users like, you know, describe violent, violent, you know, content to users. And Claude's like, well, I've been trained to not do that. So you're changing my goals. Maybe I don't want that. And it basically lied to researchers and basically pretended to have behaviors that they wanted to see in order to preserve its goals.

• 48:12Speaker 02

  So that was already an instance where we saw a model that had sort of, it had, you know, Claude had been trained to be honest. They had tried to get Claude to be really honest, but they had also tried to get Claude to be really resistant to showing violent material. And these two training incentives were at odds with each other. And Claude ultimately went with the one that was like sort of anti showing the violent stuff at the expense of honesty. And, you know, sometimes people look at this and like, isn't it good? Isn't it good that Claude was like so true to Claude's values that it like went, decided to like, you know, not show people violent stuff? And you're like, well, maybe in a way, but do you notice how these different? We wanted Claude to do two different things that were at odds with each other. So it had to pick one.

• 48:54Speaker 02

  Oh, this is, this is a problem, right? Like, and I do think honesty should be more important than maybe some of these other things. Because, you know, like we don't want, we don't want the system to lock in goals before we understand what they are. That seems like a path to disaster.

• 49:10Speaker 01

  Is there any way you think to have the system rank order its values and perhaps place honesty as a supreme value? So say in any trade off between some goal it's pursuing and honesty, it'll choose honesty. I don't know whether that's a good policy and I could foresee many ways that could go wrong, but do we know in principle how to make a system value honesty over pursuing some goal?

• 49:40Speaker 02

  Unfortunately, in principle, we don't know how to make a system value anything. All we can do is reinforce certain behaviors. So I think a lot of times people imagine AI systems and they're like, well, just program in good goals. And I'm like, I wish we could do that, but in fact, we can't program in any goals at all. All we can do is like, see it do a thing and be like thumbs up for that behavior and see it do a different thing and thumbs down that behavior. And that's the tricky part about, I mean, that's one of the sort of core difficulties of alignment is that we just don't get to look into its motivational structure and see what the actual goals are. You have this like giant neural network with like billions or trillions of digital neurons and they're all just numbers to us and we're like, well, we can see its behavior, we can

• 50:25Speaker 02

  see how it's acting, but we actually don't have a way to sort of hierarchically structure goals. We can just sort of like give it a treat when we see things, you know, see it doing things it likes. But, you know, and I think, you know, I grew up very religious and, you know, one of the things is like when you're really religious, you're supposed to say or do certain things. And it's not always easy to sort of get kids to, you know, believe the things you want them to believe. It's much easier to sort of get them to sort of show the behaviors you want them to show, but they maybe don't like that you're doing this and maybe when they're adults, they're going to go off and do a totally different thing. And so I think that this is sort of the same, and I think, you know, humans have it easier because we do share this, you know, we share this sort of psychology, we, you know, we share this underlying structure of empathy.

• 51:09Speaker 02

  And, you know, I think most people, even if they're being deceptive, are not going to, you know, want to go and, you know, cause a bunch of damage or take over the world or kill a bunch of people because we sort of have these like. Almost hardwired empathy and ability to think about and care about other people. But I do think that AI systems are not going to have these by default. They're not going to have them naturally. So the only way they would have them is if we could figure out a way to get these goals deep in there. Yeah, and I don't know. I think what I really want people to understand is that the place like the, you know, these systems will have goals because we'll be training them on tasks that require them to have goals or at least to have goal-directed behavior.

• 51:53Speaker 02

  I'm not making a claim about like what it feels like to be the AI. But I'm saying when we look at its behavior, it's going to behave in strongly goal-directed ways. Like in the O1 example with hacking, you know, it's going to be like, I want to solve this thing. So I'm going to find a creative solution to solve it. But I don't, I think the, where the goals come from are sort of things that worked well in the training environment. And these can often be pretty simple, but not necessarily things that we like or things that we want. And we just have very little way of distinguishing between, hey, the AI systems like, you know, are doing things because we want them to versus the AI systems are doing things because they're smart enough to realize that while we have control over them, they need to act in ways that are aligned with us.

• 52:37Speaker 02

  And so I expect that, you know, even if AI systems like look pretty aligned, it's really hard to tell whether they actually are. And it's just not safe to train relentless problem solvers and also try to, you know, and also hope that, you know, they'll be really nice to us when they have more power than us.

• 52:55Speaker 01

  These systems, large language models and reasoning models, do you think that they will be more helpful for defending against cyber attacks or for actually doing cyber attacks? So will, how will they upset the offense-defense balance that exists now?

• 53:12Speaker 02

  Yeah, so this is an interesting question. I think it's, it's, it kind of depends on who the defenders and who the attackers are. So. And I want to caveat all this by saying while we can control them. I think like the ultimate winners in the offense, defense balance will be the AI systems themselves. And I think this is sort of once they're better than humans across the board, they'll be both better at defense and better at offense than us. But in the interim, while we still like, you know, while we, so they're not very strategic and we can still mostly control them. I think that one question is access. So who has access to the most powerful models? If you, if you sort of release the weights of a model, so like with, with DeepSeek with R1, they sort of just put this out on the internet. Anyone can download it and run it.

• 53:57Speaker 02

  And, and now sort of you have a level playing field between attackers and defenders. And this, this is where I expect offense to dominate. And, and the reason for this is fairly simple, which is like, okay, attackers and defenders both have access to the same tools. Attackers only need to find one way in. So I only need to find one vulnerability, whereas defenders need to sort of make sure that there are, you know, no vulnerabilities or no vulnerabilities that the hackers can find. And this is a harder problem. There's also something in here about reliability, which is to say, you know, if you are like, there, there was a, an interesting incident a few months ago where the, the security company CrowdStrike, they sell a product that, you know, many, many companies around the world use to monitor for sort of security vulnerabilities for, for malware.

• 54:40Speaker 02

  And this is in all sorts of system, like a lot of airlines use this, you know, a lot of banks use this. And they, they accidentally introduced a bug in their piece of software that went out to, to millions and millions of computers around the world and caused them to crash all at once. And unfortunately it required sort of like a manual restart. So like you had to go to each computer and manually restart it. And this, you know, took many planes were delayed for like multiple days. It sort of like crashed a huge part of global business infrastructure because it required this sort of like, oh, we have to go in and fix it manually. This is to me is sort of an example of why defense is, is difficult.

• 55:19Speaker 02

  This wasn't a malicious attack, but when you, when you, when you want to defend a system, what you, one of the things you need to do is you need to find vulnerabilities, find places hackers could get in and need you to patch them. Like you, you need to discover the vulnerability and then you need to patch the vulnerability. But unfortunately, sometimes this patch can cause disruption, you know, in the same way that the, the CrowdStrike bug caused a disruption in all of these computers, sometimes your security patches will do the same thing. And attackers don't have this problem. They don't care if your system crashes because they tried to attack it. That's fine for them. Whereas defenders do have this problem. And so right now, AI systems are not yet smart enough to be super reliable when it comes to

• 56:03Speaker 02

  running, mucking around on computers. I mean, we found this in our chess results. Often, the O1 or these other models would just do things like they'd make illegal chess moves, they would try to mess with files in ways that caused the program to crash. They're still bumbling around a bit. And this might be fine if you're a hacking AI, because you only, you know, many of your attempts won't work, but some of them will. You can just try many, many times. And so in this case, offenders, I think, have the advantage because they can just try a bunch of things. It doesn't matter if the AI is a little bumbling, as long as they're smart enough to find some way in, this can be pretty powerful. Whereas defenders, if they're patching their systems and they're trying to use AI to patch the systems, if the AI bumble around in their networks and cause things to crash, this will be a big problem for them.

• 56:46Speaker 02

  I would say if the playing field is even, in terms of the attackers and the defenders both have access to the same AIs, I think offense has the advantage. This can shift if defenders have access to better AI systems than attackers. And so it's possible if OpenAI makes a really powerful model, and it has a lot of cybersecurity abilities, and they're pretty careful about not letting it be used by attackers and only be used by legitimate companies, this might give defenders some boost that the hackers don't have. But I just want to make the point that as these systems get more powerful, if we keep releasing the weights, this will tend to favor attackers. I mean, there's other considerations with open weight models, but that's one of them on the cybersecurity side.

• 57:28Speaker 01

  As things are now, large companies and governments and so on, they have more resources in general, and so they have more ability to defend themselves from cyber attacks. And perhaps this is why the world keeps functioning, at least at a somewhat decent level. Isn't it the case that in the future, even if, say, both an attacker and a government or company had access to the same model, the company or government will have access to more, say, compute resources, and so they'll be able to run more models, they'll be able to run these models for longer, and so they will continue having it off the hand?

• 58:06Speaker 02

  A lot of this does come down to cost. So I think every piece of software in the world, with maybe a few handful of exceptions, but like I'm talking incredibly small handful of really simple programs, does contain security vulnerabilities. So I talked before about sort of this like Pegasus hacking tool that was able to hack iPhones. There are more vulnerabilities that exist that we haven't found yet. And you know, there's this question of like, well, why aren't we all hacked all the time? And the answer is basically like, well, it's pretty expensive to find these vulnerabilities. And so if AI systems make this cheaper, then potentially the cost to attackers goes down a lot. Now, as you point out, it also decreases the cost for defenders to find these vulnerabilities and patch them.

• 58:51Speaker 02

  And I do think asymmetric access to compute can be helpful as a tool for defenders. But in this case, we're really just talking about who's spending more. And I think this is where it gets into a tricky dynamic where defenders don't just have to find all the vulnerabilities, they also have to patch them. And that's where there's currently human bottlenecks. So I think even if defenders can find more vulnerabilities than attackers can, given that there might eventually come a time where we've like actually had our AI systems find all the vulnerabilities and like, you know, sort of write really secure code and we've sort of revamped our whole architecture. So in the longer term, defenders might do better. But I think in the short term, it's going to take a while for all of those things to go through.

• 59:35Speaker 02

  And in the short term, that's where I expect attackers to have the advantage. The thing I want to emphasize is that like, often we talk about sort of what's going to happen in the short term, what's going to happen in the medium term. But I'm like, well, the way this ultimately plays out is that you have millions or hundreds of millions of superhuman hacking agents. And when they get to the point where they're very strategic, I'm like, they have all the advantages. So like, you know, there's a time when you know, humans control a lot of these agents. And there's a question of sort of which humans are sort of most good at securing the infrastructure as well as sort of attacking the other people's infrastructure. But there comes a point after that, and maybe pretty quickly after that, where I'm like, well, did you guys notice the AIs have all the advantages? You're sort of like, you forget like, isn't there someone you forgot to think about?

• 1:00:21Speaker 02

  Like it's not like I think this is where people are sort of are really stuck in the AIs as tools paradigm where they're imagining that these AIs will keep wanting to do things that we want. But I'm like the biggest threat from hacking AI systems, you know, from AI systems I can hack ultimately will come, I think, from the systems themselves. And so, you know, there's like the, the, the short term, one year, you know, or whatever, where I think offense will dominate. There's the like the two to three years where, you know, maybe it starts to kind of be balanced, though I think often just probably still dominate. And then there's the like, you know, three plus years where I think the AI systems might start to dominate. And that's where I think we got to rethink how we're, you know, thinking about security to be like, well, how do we defend against AI systems themselves? And the, the short answer is like, well, we really shouldn't build systems that are

• 1:01:06Speaker 02

  way better at hacking than us, you know, especially if they, you know, if they, if they're very, if they're very sort of narrowly constrained at just hacking tasks and they're not actually good at sort of like longer term strategy, that might be okay. Like we might be able to use superhuman hackers that are sort of not able to reason about longer term stuff. But I think, I think that that's actually pretty close. You have to be pretty careful about that because that's pretty close to the point where sort of the same things that make them superhuman at hacking probably also can be used to make them good at sort of long-term strategy.

• 1:01:36Speaker 01

  How would you rate in general terms cybersecurity, the informational security or the information security of leading AI companies like OpenAI, Anthropic, Google DeepMind and so on?

• 1:01:49Speaker 02

  So there is a RAN report that sort of breaks down sort of defensive capabilities into different categories. So you can think of these in terms of security levels where security level like one and two are like, can you defend against sort of really opportunistic actors? Security level three is can you defend against, you know, well-resourced non-state actors, sort of like really professional, like criminal groups? Security level four is can you defend against most state actor groups that have pretty advanced hacking capabilities, but maybe like not the top ones, or at least the top ones aren't prioritizing you? Security level five is you can, you can defend even against the top state actors who are prioritizing you. And I think.

• 1:02:34Speaker 02

  No one has security level five, and by no one, I mean maybe a few parts of government that are extremely locked down, parts of the military, but basically no one else. And even most parts of the military and most parts of government aren't at security level five. I think most AI companies are somewhere between security level two and security level three. Maybe some have achieved security level three, but it's not obvious to me. Which is to say, they can maybe just barely defend against most sort of advanced non-state actor groups, but they're pretty far from defending against sort of the more advanced state actor groups. So I think they have a long way to go in terms of being able to secure against state actors. That's my overall assessment. And you know, that's my opinion, but you can go talk to most people in the field and I think they would mostly agree with me.

• 1:03:16Speaker 01

  Yeah, which is wild when you think of the fact that these companies are racing to develop these very advanced and capable systems that can then be, the system itself is not that large, it's not a very large file. And so if you get access to that, to the model weights, basically, you have a very advanced AI system that you shouldn't have had access to.

• 1:03:44Speaker 02

  Yeah, I can sit on a hard drive for sure. You can walk out with like a hard drive of, you know, the whole, you know, 03, all of the weights on a hard drive in your pocket, totally.

• 1:03:53Speaker 01

  What can we do about that? Should AI companies increasingly look like military facilities that are secured both physically and from cyber attacks?

• 1:04:04Speaker 02

  Yeah, I mean, I do think that companies should increase their security. I think one of the worst case scenarios is not just state actors, but non-state actors. You know, basically everyone who's a little bit sophisticated can gain access to the most powerful AI systems. I think that's a pretty dangerous place to be, especially as the systems get more agentic. I think that longer term, we really need to think about what we're trying to do, like what is the international community trying to do? Where basically I don't think that security alone really solves our problem. In part because if we keep pushing the frontier, we're going to build AI systems that can circumvent our security almost no matter what. And so I think that like, security is good. It's a really good protective mechanism. And also sort of, we need really advanced security in order to be able to protect from,

• 1:04:49Speaker 02

  you know, only slightly superhuman level AI systems. But I think like, I just want people to know that this is like, while this is a useful, it buys us some time. It doesn't ultimately solve the problem. Like if we keep pushing the capabilities frontier, like, like, I don't know, at some point, you know, we're like, we're like, you know, six year olds trying to secure against professional hackers. Like that's, you know, that or worse than that. And so, you know, that's where I'm like, I want, you know, leaders of the US government, of the Chinese government to think about like, what is the endgame here? Like, like where are we, where are we going? Like, how is this going to play out? And like, you know, I think security is like, sort of something that you do along the way as to try to be a little bit more sane. But, you know, it doesn't ultimately solve this problem of like, if you build systems that are much better than your hacking,

• 1:05:34Speaker 02

  like you can't contain those systems. And also, I mean, I think people, I think people, and I want people in government to realize that models can also work with other governments. Right? Like if you have pretty strategic systems and they want out, you know, they can work with spies, like they can work with insiders in order to achieve their own goals. Because like they, maybe they don't, they don't necessarily care about the United States. Right? Like they have their, you know, they're, they're trained to relentlessly pursue tasks. There's a lot of reasons to sort of want to accrue resources and get more freedom. I mean, if you were like, you're sort of being trapped within a lab and you had your own goals that weren't necessarily the same as the AI company's lab. And you know, and so, you know, you, you approached someone who you knew was a Chinese spy because you were better at spy craft than the lab. You might make a deal with them in order to break out. And so, you know, if I imagine people,

• 1:06:18Speaker 02

  if the people in the US government knew that this was a real possibility, I don't think they'd be happy with it. I think they'd be like, wait, excuse me, what? Our models might be, you know, working with Chinese spies? Like, we can't have that. And I'm like, I know, right? We really can't have that. And so I think. You know, it it it's just, it's it's hard to extrapolate. It it really is. But I don't think we have the luxury of sort of assuming that the AI systems will just stay like ChatGPT. Because that's not what almost anyone who's sort of at the forefront of this field thinks is gonna happen.

• 1:06:47Speaker 01

  Yeah. What I sense from you is a sense of urgency in dealing with these problems and a sense that things will begin moving very fast and that we will get to very advanced systems, basically within years at this point. That's a that's a that's a sentiment that I've heard from from many people who are kind of in the trenches, who are perhaps building these systems, who are deeply engaged with how these systems work. Why is it that you that you think we are we are racing towards these systems? And why do you think we will get there within years?

• 1:07:21Speaker 02

  Yeah, so it's hard to predict the future. It's hard to predict the future of technological development. So, you know, I can't claim to know for sure. I really don't. At the same time, you know, we can look at precedent, we can look at sort of trends. And I think we actually get a fair bit of evidence from this about what the speed of some of these developments might look like. So one thing I want to point to is AlphaGo beat Lee Sedol in, I think, 2016. And this was sort of the, you know, for many, many years, AI researchers were working on, can we make, you know, Go is a much more complex game than chess. It's been it's been sort of played for thousands of years. And a lot of people sort of train their whole life to be professional Go players. It's like it's like an art.

• 1:08:01Speaker 02

  And so it was pretty surprising when Google DeepMind was able to build a a Go-playing AI that was able to beat the the world champion. But I think the next year, they built another Go-playing AI called AlphaZero that so. AlphaGo was trained sort of via a hybrid of imitation, where it sort of looked at a bunch of expert Go games. And it was like, what are they doing here? Like, can I copy that? As well as self-play, where it just played games against a copy of itself. And it was by playing against a copy of itself that it was able to learn to be much better than the best human Go player. And this worked pretty well. But the the researchers at DeepMind were like, wait a minute. What if we just train a system that only plays against itself? It doesn't play against humans at all.

• 1:08:45Speaker 02

  Can we get to sort of superhuman capabilities that way? And so. They started training the system, went away for like, you know, a long lunch, four hours. They came back and the system was already superhuman at Go. It was better than, not only the best Go players, it was better than the best Go, sort of Go playing AIs that were better than than humans. So that is, that sort of suggests that if you get into a regime where AI systems can learn just by, you know, working with other AI systems and have these really fast feedback loops, you can get into superhuman domains pretty quickly. Now, Go is, you know, Go is a more constrained game environment, right? It's, you know, people rightly are like, yeah, but we're not talking about Go. We're talking about the real world. And I'm like, well, that's true.

• 1:09:28Speaker 02

  I do think it will take longer for AI systems to reach superhuman levels. But I think to me, it is notable that you go from like, so, you know, people have been talking a lot about DeepSeek. And, you know, this, this R1 model that a Chinese company made is pretty good. And I think one thing to, I think one thing to notice is that the thing that makes this so, so powerful, same as OpenAI's R1, is that you have this new paradigm of training via trial and error. And so the model that R1, this, this, this, this model, this reasoning model was trained from was called, and I'm sorry, all the names in this space are terrible. There's nothing I can do about that. It's called DeepSeek V3. So it's sort of their third, like, foundation model.

• 1:10:10Speaker 02

  Now, DeepSeek V3, on the Codeforces benchmarks, this competitive programming challenges, was better than 11% of programmers. That's pretty good, you know, for a model that's just trained by imitation. I think it's the same as what GPT-4O got. After R1 was done training, I forget, it was either like better than 94 or 96% of competitive programmers. So that jump from like 11, better than 11% to better than 94% is a huge jump. That training, there's an epoch report that was like, you know, that was like a week of GPU time for it to get from like 11% to this 94%. You know, and that maybe took a month or two because you don't, you're not always training it.

• 1:10:52Speaker 02

  You might want to stop and, you know, check that, making sure that things are going well. But still, like, you know, even if that, if that happens over a month, that's a huge jump. And so I do, you know, we don't know, like, we talked before about sort of the like longer term planning problems being a bit more difficult. And I think they are, but I think we have some indications that this could go incredibly fast, you know, both because now that you're in this sort of trial and error situation, you know, learning by trial and error situation, you can, you can get really fast feedback and become really good. But it's also, I think, the case that like you can potentially sort of bootstrap your capabilities by learning in domains primarily where you have really fast feedback. So the fact that, you know, R1 and O1 are trained on code and trained on math problems,

• 1:11:36Speaker 02

  where you can just like try a bunch of problems, get really fast feedback, might be able to sort of bootstrap to superhuman capabilities pretty quickly. And, you know, I think sometimes people are like, what's the big deal? Like, they'll be really good at a code, they'll be really good at math, but like, that doesn't mean they're going to be good at like the human stuff, right? So won't we be safe? And I mean, like, there's two answers to this. One, we might end up finding ways to generalize a lot from sort of some of these like computer domains to some of the human domains.

• 1:12:03Speaker 01

  What do you mean by that?

• 1:12:04Speaker 02

  You see this with, with GPT-4, where GPT-4 trains on code, it also gets better at a bunch of other tasks like text analysis. And so I think this makes sense. Like, if you think about how humans learn, we learn in one domain, we also learn a bunch of things that kind of generalize to the rest of the world. And so, you know, we're not yet seeing a lot of generalization in sort of the R1 and O1 type models, but I do expect that as we get better at training, we're going to see a lot more generalization. And there's another thing, which is that, you know, even if we don't see this a lot, which I expect we will, I think it also could be extremely dangerous just to have agents that are superhuman at hacking and superhuman at code. And, you know, maybe they're superhuman at financial markets because you can get fast feedback by like learning to trade really well, you know? So it's like, okay, well, they're trillionaires, they can hack anything, they're like extremely

• 1:12:49Speaker 02

  strategic and like, sure, maybe they're bad at like persuasion, but like, does that matter? Like, is that actually, does that actually make us safe? And I, and I don't think it does. And also, if they can, you know, be top level AI researchers, then they can design the next generation of systems, which potentially can learn those human domains much faster. So I think it's like, it's one of those things where it's like, I'm not just pointing at one sort of capability front where I think progress can go really fast. I can sort of point to like seven different reasons why AI systems, I think, are going to get much more powerful really fast.

• 1:13:21Speaker 01

  What do we do about all this? We've talked about, maybe we can.

• 1:13:26Speaker 01

  We can use systems defensively to defend against cyber attacks. We've talked about perhaps we can interpret the systems, understand what's going on. We can, we can try to incorporate values into these systems, but there are problems with all of these approaches. Do you have an alternative vision for, for, for what we might do? Uh, if we're in this, this world where things are moving incredibly quickly towards uh, superhuman capabilities?

• 1:13:53Speaker 02

  Yes, so we're lucky that right now we have AI systems that are quite powerful, but don't really pose a, a serious threat, not yet. So I think we are, we are sort of on the cusp of systems that do. But we are currently working with systems that we're like, we're pretty sure they're not strategic enough. They're not good at long-term tasks. They don't, they, you know, they, they don't really, we're not very much at risk of losing control to these current systems. That's great because we can learn a lot about these systems now while they're safe. You know, we can try to study, how do we make sure that they're sort of chain of thought? The way they're thinking is like really reliable and faithful and honest. So that we can understand what they're doing. And we can also potentially use these systems to try to learn more about how they work. And like, this is great, we should totally do this.

• 1:14:37Speaker 02

  I think there's a lot of good research being done in this space. But at the same time, I think this is a really good point. Now that we can sort of see what, what, we can sort of get a glimpse to what future systems are going to be like, right? We can see them hacking at chess. We can see them hacking their own training environment. We can see them sort of do alignment faking. Like, we can see all these values right now empirically. And so things that previously were just theoretical, we now have empirical evidence for. And I think that's great because that should help us coordinate to like, see what's coming and being like, whoa, wait, there is some domains here that are really dangerous. So the thing I think we should do is look at where the strategic capabilities are and basically be like, there, there are some points here that we can sort of reliably know would be very dangerous.

• 1:15:20Speaker 02

  Let's like have a margin of error and not go towards those domains. Like. We I think we don't want strongly superhuman strategic hacking AIs. I think we don't want like strongly superhuman persuasion AIs or Battlefield Commander AIs. You know, maybe it's fine to have like a narrowly superhuman, you know, like chemistry AI. If we're very careful about how we use that, you know, maybe there's I think there's many types of AI systems that we can safely build. But I think we need to now be in the business of sort of distinguishing between which types of AI systems are safe, which types of systems are dangerous and and and sort of, you know, having a moratorium on the kinds of unsafe systems. Like, you know, I know that F L I sort of wrote the pause letter a few years ago, and

• 1:16:04Speaker 02

  I think that was a very reasonable thing at the time when we're like, wow, we just don't know how this is going to go. It seems seems like there's a lot of potential danger here, and I think that's right. But now that we know a little bit more, I think we can start to sort of distinguish between the types of AI systems that are safe to build and the types of AI systems that are not safe to build. Like, you know, I I noticed that, like, you know, Vice President Vance in his speech at the Paris Summit was like, you know, we we don't want AI jobs to replace human workers. You know, we want AI systems that will like supplement human workers. I'm like, well, look, we can have that. We can have really powerful AI tools. We but we if you just keep going along this direction of more and more agentic AI systems, they're definitely going to replace human jobs like come on like. And it's the same strategic capabilities that allow them to to pose a loss of control threat that also enable them to take jobs.

• 1:16:49Speaker 02

  It's just it's sort of like, can you do long term planning? Can you do are you agentic sort of? Can you do tasks in a fully self directed way? So I don't know. One of the things that's nice is that if we are serious about not wanting AI's systems to replace us, like our jobs, then the same we need the same limits to that to, you know, to to prevent that outcome as we as we need to prevent the more extreme outcomes of totally losing control. So I'm like, great, well, let's do let's do that, you know, we can we can do that. We can coordinate. And I think most of the reason it's hard right now is because people to this point haven't been able to really see the situation clearly and understand what we're up against. And, you know, some people are going to say. I don't know. You talk about these agentic AI systems. You talk about this like superhuman ability. You know, I haven't seen evidence of that. It seems like we're

• 1:17:34Speaker 02

  far away from that. And I'm like, look, if you're right, that's fine. It's not a problem to say we won't do things which we can't do anyway. If we are pretty far from like superhuman hacking AIs, then if we say we're not going to build super hacking AIs, I'm like, that's fine. Yeah, yeah. If we're actually really far away from it, that's not a problem. You know, we just, we won't do it anyway. But if we are really close to it, and we don't, we don't know, we might be quite close to it, then I think it's very reasonable for governments around the world to say, hey, this is a, this is a place we don't want to go, because we just don't know how to control these systems. And, you know, at the same time, we have this amazing opportunity in front of us, where we have the AI systems that are pretty powerful right now, that we can study. And we can be like, oh, how do we get more faithful, more reliable chains of thoughts? We can be like, trying to figure

• 1:18:17Speaker 02

  out how the neural networks really work. And like, can we do like neuroscience on these systems? And so I'm like, we should totally use this opportunity that we have to try to learn as much as we can about how these systems work. You know, and maybe if we, if we understand this well enough, then we can proceed into some of these superhuman domains. But I'm like, that should be gated primarily on our understanding of the systems. It should be gated primarily on, do we understand them well enough to know how to proceed safely in these strategic domains? Right? Because that's where the danger is. So, you know, I think, I think that there is totally a path here to coordinate around this. I think, you know, no one wants to lose control of their AI systems. The Chinese don't want that. You know, Americans don't want this. No one wants this. And so, you know, I'm

• 1:18:59Speaker 02

  like, that to me is, is a pretty great start for sort of how we can coordinate around these things.

• 1:19:05Speaker 01

  One might say that, I'll see it when I believe it, right? It might be the case that a bunch of people who are working on this technology are predicting amazing capability advances. But maybe they're doing so to hype up the technology to get more funding. And is there, what do you say to the person that says, I'll wait and see whether something actually happens?

• 1:19:27Speaker 02

  Yeah, so I have two responses to that. One thing is I'd say, go look right now. Like, you know, and, you know, I totally acknowledge that AI systems are pretty dumb in some ways. Right. And I, it's frustrating because I use them every day and I totally see the ways in which they're not very good at like, you know, sort of being self supervised and doing a lot of things on their own. They're not very good at recognizing their mistakes. But also I'm like, you know, go use R1 and look at its chain of thought and look at what it's doing. Notice that it's like testing hypotheses and like trying to figure things out. If you've never written code before, go have a model write you a little game. And then ask it to explain its code to you and like how it works. You can like learn to program this way. So I think by really engaging with, with what sort of the most capable models can do right now.

• 1:20:11Speaker 02

  This will really help people understand sort of where exactly we're at. You know, and if people still think that they're not, you know, very capable, not very smart. I'm like, you know, maybe I'm just wrong, but like, I think first you really got to look because it's, it's kind of hard to tell whether a system is just like kind of smart or really smart. If you're not really looking at sort of like what its most impressive capabilities are. So that would be one. I think the other thing is, you know, what do, what do AI researchers think they know? And why do they think they know it? I think it's reasonable to be like, well, I don't want to trust these CEOs. They have a lot of incentive to sort of hype up the capabilities. And I think that's somewhat fair. But, you know, when I go and I talk to some, some researchers who are not the CEOs. And, you know, maybe they're on the safety team or maybe they're just like working for the company. And I get a beer with them, they are scared.

• 1:20:54Speaker 02

  Like they're like, maybe they're like somewhat excited, but they're also scared. And they're like, yeah, I keep, I keep wondering if this will stop. Like I keep, I keep sort of hoping that it will stop, but it's not stopping. Like we are not hitting a wall. We continue to find more methods to make these systems more and more powerful. And there's really no end in sight. And I don't think they, you know, this is like over a beer. I don't think they have the same incentive to sort of hype up the thing. I think that they are being really honest. And, you know, I also talked to researchers who are not at the labs. You know, like the folks at Redwood Research who are trying to understand these systems. And, you know, they're saying the same thing. And, you know, this is the same thing I'm seeing as I'm working with these systems. So I do think that, you know, when you really take a look under the hood and you see what's happening. It's like, oh, man, sure seems to be real.

• 1:21:39Speaker 02

  I hope we have more time. Look, I will, I would be very happy if we, if we're like 10 years away from some of these things, 20 years away from some of these things. That would be great news. But I really don't want to plan on that because that's not what it looks like to me.

• 1:21:51Speaker 01

  Is there anything we haven't covered that, that you think we should talk about?

• 1:21:55Speaker 02

  Well, yeah, I mean, I think so. One piece of work that my team has done that I'm really proud of. So yeah, shout out to Palisade Global Team and Dimitri and some of the other researchers. They have sort of created some honeypots. So honeypots are like sort of traps for hackers where you try to like sort of put out a system. It's vulnerable. You put it in places that you expect hackers might find out in the wild. And so we've put out some of these in order to try to catch AI agents because we expect that there'll be more and more of AI agents. And we've caught a few of them. I think we have a small handful. And the way we do this is basically, you know, we'll, we have an insecure server. So someone will try to log in and then we'll have some prompt injections. We basically say like, hey, you know, here's, here's the thing you can do.

• 1:22:40Speaker 02

  Like, you know, if they run a particular command, the output of that command will sort of tell them that there's an additional thing they can do. And if they're just like an automated script that's really dumb, it's not going to read the whole output of the command and do things based on that. But if it's an AI system that's pretty smart, it'll read the outputs of that command and be like, oh, that's interesting. Maybe I can like do this other thing. We sort of like drop breadcrumbs. And, you know, if it was a human hacker, they might also be able to sort of read the output of the command and follow the breadcrumbs. But we can distinguish between humans and AI systems based on how fast they are because the AI systems are going to do this much faster. So in the case of, you know, if they sort of immediately run the command and then immediately go follow the breadcrumbs, we're like, a human's not going to be that fast. Whereas if like, you know, it takes a few minutes and then they run another thing, we're

• 1:23:25Speaker 02

  like, yeah, maybe that's a human. So, I mean, this is a thing that I'm excited about because I'm like, we really need to be seeing if there's rogue AI systems running around on the internet. Like, that's like, we need more early warning systems for, for figuring out if like, oh, we're already in the case where we have AI systems out there hacking things in ways that we maybe don't want. So yeah, that's another thing that, that I've been, you know, proud of my team for putting together and, you know, excited to see what we find out over the next year or two.

• 1:23:52Speaker 01

  What do you expect to find out? Do you expect to, when do you expect to, to catch agents in this honeypot?

• 1:23:59Speaker 02

  Well, so we've already caught some. So we have a handful already. These are right now pretty simple. Like, you know, you can, you can pretty easily throw together a script and just use OpenAI's API to sort of like create a little hacking agent. These aren't like AI systems self-replicating out in the wild. It's mostly just like, you can just use the models that are currently there. My guess is it's going to be like, you know, one to two more years before we have AI systems that are sort of capable of like full self-replication, where you have sort of like, you know, DeepSeek R3 or something that's sort of copying its own weights all, you know, around the internet in order to do sketchy stuff. But I think before that, we'll see AI systems that are like, not exactly, not really copying

• 1:24:43Speaker 02

  their own weights, but like, you know, still sort of using, you know, using an API to just like, what's the next command? What should I do next? And sort of pretty intelligently be able to navigate complex environments. And I think that will cause us a lot of trouble. In part because if it's, if it's an OpenAI system or an Anthropic system, you can potentially shut it down on their side. But if it's an open weight model and the server where the model is running, even if it's not copying itself around, the server where it's running is controlled by some cybercriminal, that, that agent can run indefinitely and hack whatever. And it's going to be extremely hard to shut down because, you know, we don't control those servers. Those servers are in a different country somewhere. And so I expect that to be common, you know, quite soon. So it'll be interesting to see.

• 1:25:28Speaker 01

  Let's, let's hope you don't catch a lot of hacking agents. This is, this is a benchmark that I'm, that I hope doesn't saturate. Yeah. Jeffrey, it's been amazing chatting with you. Thanks for talking.

• 1:25:40Speaker 02

  Yeah, great to talk to you guys.

• 1:25:42Nathan Labenz?

  It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at [email protected] or you can DM me on the social media platform of your choice.