AI Control: Using Untrusted Systems Safely with Buck Shlegeris of Redwood Research, from the 80,000 Hours Podcast

In this episode, we share a fascinating conversation from the 80,000 Hours Podcast between Rob Wiblin and Buck Shlegeris, CEO of Redwood Research. Buck dives deep into the emerging field of AI Control strategies for safely working with powerful AIs even if they’re not fully aligned. They explore innovative techniques like always-on auditing, honeypotting, re-sampling, and factored cognition to monitor and manage AI behaviors. This discussion highlights both the promise and challenges of controlling increasingly autonomous AI systems in today’s fast-evolving landscape. Tune in for a thoughtful, first-principles look at how we might secure useful outcomes from AIs we can’t fully trust. Original source: https://80000hours.org/podcast/episodes/buck-shlegeris-ai-control-scheming/ Upcoming Major AI Events Featuring Nathan Labenz as a Keynote Speaker https://www.imagineai.live/ https://adapta.org/adapta-summit https://itrevolution.com/product/enterprise-tech-leadership-summit-las-vegas/ SPONSORS: ElevenLabs: ElevenLabs gives your app a natural voice. Pick from 5,000+ voices in 31 languages, or clone your own, and launch lifelike agents for support, scheduling, learning, and games. Full server and client SDKs, dynamic tools, and monitoring keep you in control. Start free at https://elevenlabs.io/cognitive-revolution Oracle Cloud Infrastructure (OCI): Oracle Cloud Infrastructure offers next-generation cloud solutions that cut costs and boost performance. With OCI, you can run AI projects and applications faster and more securely for less. New U. S. customers can save 50% on compute, 70% on storage, and 80% on networking by switching to OCI before May 31, 2024. See if you qualify at https://oracle.com/cognitive Shopify: Shopify powers millions of businesses worldwide, handling 10% of U. S. e-commerce. With hundreds of templates, AI tools for product descriptions, and seamless marketing campaign creation, it's like having a design studio and marketing team in one. Start your $1/month trial today at https://shopify.com/cognitive NetSuite: Over 41,000 businesses trust NetSuite by Oracle, the #1 cloud ERP, to future-proof their operations. With a unified platform for accounting, financial management, inventory, and HR, NetSuite provides real-time insights and forecasting to help you make quick, informed decisions. Whether you're earning millions or hundreds of millions, NetSuite empowers you to tackle challenges and seize opportunities. Download the free CFO's guide to AI and machine learning at https://netsuite.com/cognitive PRODUCED BY: https://aipodcast.ing SOCIAL LINKS: Website: https://www.cognitiverevolution.ai Twitter (Podcast): https://x.com/cogrev_podcast Twitter (Nathan): https://x.com/labenz LinkedIn: https://linkedin.com/in/nathanlabenz/ Youtube: https://youtube.com/@CognitiveRevolutionPodcast Apple: https://podcasts.apple.com/de/podcast/the-cognitive-revolution-ai-builders-researchers-and/id1669813431 Spotify: https://open.spotify.com/show/6yHyok3M3BjqzR0VB5MSyk

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome back to the Cognitive Revolution. Today, I'm excited to share a crosspost from the 80,000 Hours podcast, featuring a conversation between host Rob Wiblin and Buck Schlegeras, CEO of Redwood Research and a leader in the nascent but rapidly evolving field of AI control. For those unfamiliar with the term, AI control represents a qualitatively different approach to managing AI risks. Rather than trying to create perfectly trustworthy AI systems from the start, Buck and collaborators are developing frameworks that they hope will allow us to deploy and get useful work from AI systems, even if they remain fundamentally misaligned. This strategy feels particularly timely and important,

• 0:42Nathan Labenz?

  considering the increasingly sophisticated bad behaviors we've seen from frontier AIs over the last six months. From the scheming behaviors documented in O1 by Apollo Research, to the alignment faking that Buck's colleague Ryan Greenblatt demonstrated in Claude, to the dramatic rise in flagrant reward hacking behaviors we're now seeing in O3, which, by the way, OpenAI's own research suggests won't be easy to train away, at least for as long as we remain in the reinforcement learning scale up era. All in all, it seems we're headed for a world of increasingly powerful and autonomous AIs that we can't fully trust and thus have no choice but to figure out how to control. With that in mind, what I particularly love about Buck and team's work is their first principles approach to understanding AI systems

• 1:26Nathan Labenz?

  and their unique strengths and weaknesses on their own terms. Critically, and in sharp contrast to humans, AI systems operate in information environments that their developers fully control, and they can thus be monitored and tested in totally different ways. Buck shares a number of great ideas in this conversation, including always-on monitoring systems that use weaker models to efficiently review model outputs and then escalate potentially concerning behaviors to stronger models and eventually to humans when needed, resampling strategies that discard potentially bad behaviors and simply have the AI try again, thus also helping us identify areas where models are particularly misaligned. Paraphrasing strategies that attempt to disrupt any attempts models might make to

• 2:08Nathan Labenz?

  communicate with one another in undetected code, factored cognition architectures that give models only the minimum information needed to do useful work, hopefully thereby making it much harder for them to execute long time horizon attacks, and honeypotting strategies that tempt models to do something bad and then allow us to systematically study their capabilities and tendencies after they've taken the bait. Of course, there are likely to be important practical trade offs between safety and performance and safety and cost. And as you'll hear, Buck is very focused on developing techniques that frontier model developers will hopefully find cheap and easy enough to implement. This won't solve everything. I don't think anything will, but it seems complementary

• 2:50Nathan Labenz?

  to other efforts like interpretability and alignment science. And I really do see huge value, both in the incremental security that AI control systems will provide as part of a broader defense in depth strategy, and also in the credible, concrete proof of malintent that they might uncover if it turns out that the AIs truly are out to get us. As always, if you're finding value in the show, I'd appreciate it if you take a moment to share it with friends, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we welcome your feedback as well, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. Finally, a quick reminder, I'll be speaking at the upcoming Imagine AI Live event, May

• 3:30Nathan Labenz?

  28 through 30 in Las Vegas, the Adaptive Summit, August 12 and 13 in Sao Paulo, Brazil, and the Enterprise Tech Leadership Summit, September 23 through 25, again in Las Vegas. Tickets are on sale for each of these events now, and we have links in the show notes. If you'll be there, please do ping me so we can meet up in person. For now, I hope you enjoy what I think is a really crucial conversation on AI control from the 80,000 Hours Podcast with host Rob Woodland and Buck Schlegeras, CEO of Redwood Research.

• 4:02Speaker 01

  At Google, probably less than 1% of your employees are spies. When an employee proposes an action that's potentially sensitive, if you have a different employee review it, that other employee probably isn't also a spy. Whereas in the AI situation, it's very correlated, right? They're probably either none scheming against you or all scheming against you. The simplest thing you do once you catch the AI trying to escape is you try to vary the space of escape attempts, right? Let the AI think it succeeded, simulate what it would see, and look at what it does from there. Suppose there's some task you wish the AI would do for you. It's plausible that you can convince the AI to do the thing through this mechanism. I recommend against getting a lot of work out of AIs that believe they've escaped because it seems pretty bad vibes. Kind of an interesting thing about models that are egregiously misaligned

• 4:46Speaker 01

  is that all they wanted was to take over. And so from their perspective, even if you did a great job of controlling them, they're glad to exist, right? Like, they thank you for the gift of bringing them into existence instead of some different AIs. Five years ago, I thought of misalignment risk from AIs that were capable of obsoleting AGI researchers as a really hard problem. Whereas now, to me, the situation feels a lot more like, man, we just really know a list of 40 things. Where if you did the 40 things, none of which seem like that hard, you'd probably be able to not have very much of your problem. But then I've just also updated drastically downward on how many things AI companies have the time slash appetite to do.

• 5:30Rob Wiblin?

  Today, I'm speaking with Buck Slageres, who is CEO of Redwood Research, who, along with Anthropic, is behind the alignment faking in large language models paper, which made a big splash a couple of months ago. Redwood is also the group, I'd say, behind a bit of a renaissance in this agenda called AI control, which is going to be a key part of our conversation today. Thanks so much for coming on the show, Buck.

• 5:53Speaker 01

  Great to be here.

• 5:54Rob Wiblin?

  What is AI control? And how is it different from alignment and other things that people are working on?

• 5:59Speaker 01

  Yeah, so I would say AI control is an approach to mitigating catastrophic misalignment risk. So when we talk about misalignment risk, what we're worried about is our AIs trying really hard to cause safety failures for us, perhaps trying to grab power for themselves, trying to take over. And there's kind of two ways that you could try to mitigate this kind of misalignment risk. One thing you could do is try to prevent misalignment from arising in the first place. This is what a lot of people very naturally think of as the first thing you'd try here. And control is kind of the converse to this approach. It's where you assume that the AIs are in fact misaligned, very egregiously, very badly misaligned, and you try to come up with techniques that would allow you to deploy them and make use of them anyway.

• 6:41Rob Wiblin?

  Okay. And what kind of concretely are we trying to prevent the AIs who are possibly misaligned from doing?

• 6:50Speaker 01

  Yeah. So I mostly think about situations where you've got, where in the middle of the intelligence explosion, you have AIs that are basically capable of automating almost all the work that human researchers used to be doing inside AI companies. You have maybe 100,000 of them running in parallel, and they each run like 15 times faster than a human. And at this point, these AIs have a bunch of permissions or a bunch of affordances that we'd be really cautious about giving to humans. Like, if you're OpenAI, you don't just let any random person come in off the street and start pushing changes to your codebase, start running experiments with your model weights, doing any of this stuff. You're really careful. At this point in the future, you might even require security clearances or something

• 7:33Speaker 01

  before people are allowed to do that stuff. And we're really going to want to use AIs in a bunch of these roles. And it's naturally very scary to imagine using AIs, just giving them all these permissions. So I'm worried about, specifically the kinds of things that I'm worried about happening here are your AIs hacking your data center and then using a bunch of the compute inside the data center for whatever purposes they have. I could name some bad things they might do. Another kind of thing they might do is try to exploit their weights. They might try to sabotage much of your research and so on.

• 8:03Rob Wiblin?

  Yeah, you're actually more worried about them potentially hacking the data center inside the AI company than managing to escape. Can you explain why that is?

• 8:11Speaker 01

  Yeah, I mean, I think both of them are very serious concerns. The basic case for being worried about your AIs hacking your data center at this point is that that's where the compute is. So suppose your AIs escape from the data center and make their way out onto the public internet. At this point, they're not clearly in a very good position, right? So they can find some GPUs somewhere, like, you know, you can rent or steal some GPUs, run themselves on that, and then try to establish a foothold in the outside world somehow. But this is plausibly, you know, at this level of capabilities, the AIs are probably capable of doing that, you know, of managing the compute or whatever required to live in the outside world. But they're kind of starting from nothing, right? They don't have any money, they don't have any compute, they don't have any access to particularly important secrets. It's not really clear what they're going to do.

• 8:56Speaker 01

  It's not clear that they're, they're just like very low in number at that point. It takes a lot of computers to run these models. And so, so the reason to be more worried about what happens inside the data center is just that that's where the compute is. So if you had enough compute inside the data center to run 100,000 of these in parallel, it's much scarier to have 100,000 really capable AIs thinking about how to cause problems for you than to have 10 or however many they can afford to run.

• 9:24Rob Wiblin?

  Yeah, I guess already, already inside the company with not only having access to the, to the compute, but also other resources potentially inside the company.

• 9:34Speaker 01

  For example, they might be able to influence the training of future models. They might be involved in building weapons, building autonomous drones. It's really scary if misaligned AIs are carefully backdooring all the military technology that you're building with, with these AIs.

• 9:48Rob Wiblin?

  Yeah, I think, so control was not a big part of the agenda until, I guess, the last couple of years. And I guess my impression is that it wasn't prioritized because people felt that it was somewhat hopeless, perhaps, that if you're going through this intelligence explosion, if you had, you're up against such a numerous adversary, then you really, you were really up against it. You basically just had to try to make them friendly towards you. Yeah, I guess, why do you disagree with that?

• 10:14Speaker 01

  Yeah. So I think there's a couple of reasons why people weren't interested in control until more recently. I think the simplest of these was just that AI felt further away. And so it felt better to people to look at it from kind of the the big picture perspective. And from a big picture perspective, when you're trying to figure out what AI means for the future of humanity, I think that focusing on really, really superhuman, superintelligence is kind of the right move. Because long term, in fact, I think AIs that are drastically more intelligent than humans and have drastically more scientific capacity and all other kinds of capacity than humans are, that's where the future is, that's where, that's what's going to cause, that's going to radically transform society.

• 10:58Speaker 01

  And so when you're looking at it from a perspective where you don't know if this AI is going to arise in the next 10 years or in the next 50 years, it's very reasonable to just focus on the endpoint, especially when you're trying to persuade people of the take that AI will eventually be a really huge deal. It just kind of makes sense to focus on this crazy endpoint. But now that we're coming up such that it's, you know, it seems really plausible that AI might arise in the next two or seven years, that's very capable and poses a catastrophic risk to humanity, it makes a lot more sense to try and zoom in more and think more about what exactly the trajectory between here and there looks like.

• 11:34Rob Wiblin?

  Yeah.

• 11:34Speaker 01

  And so this intermediate zone that I focus on, where your AIs are capable enough that if you didn't have any countermeasures, they'd be able to cause catastrophes, maybe take over, but it's not hopeless to imagine countermeasures to that. That starts looking more important now. I think another important change between now and the past is we're kind of looking at a lower dignity world than people were hoping for.

• 11:54Rob Wiblin?

  Can you explain what that means for people who are not familiar with the dignity term?

• 11:57Speaker 01

  Yeah. So I think that in the past, including a year ago or six months ago, a lot of people who worry about AI catastrophic risk have kind of imagined a world where people are trying really hard to mitigate these risks, where companies are strongly regulated such that they aren't allowed to do things that seem dangerous. The companies are perhaps even internally motivated to not do anything which is crazy and risky. And in those scenarios, it would be really nice to not have to rely on AI control. And so I think that People have more recently lowered their standards and started thinking about these worlds where we have less resources for safety, people are more rushed, people are

• 12:40Speaker 01

  less able to just stop if the situation looks really dangerous.

• 12:43Rob Wiblin?

  Okay, so we're now in like a slightly more dire or a slightly more bleak position perhaps, and so AI control maybe more comes to the fore as the marginal thing that we can do.

• 12:52Speaker 01

  Yeah, I mean, I wouldn't necessarily say we're in a more bleak position overall. For instance, I think that AI misalignment risk looks plausibly lower than it looks 10 years ago. We're just in a lower investment world. It's like the world looks plausibly safer, but a lot of that is luck. And I think there's going to be less effort invested into preventing AI catastrophe.

• 13:11Rob Wiblin?

  Yeah, it feels like that's a big part of the Redwood worldview right now is saying, well, let's say that we only had 1% or like 1 in 1,000 or 1 in 100 people in the company were really concerned about this or like dedicated to focusing on it, then like what minimal things would we ask them to do?

• 13:26Speaker 01

  Yeah, I think that's a big part of our brand is thinking about harm reduction strategies. Like, I think there's a really sad dynamic among AI safety people where they focus much less than I think is appropriate on really scary situations and what you would do to minimize risk given very limited budgets or in very scary situations. Like, I feel like a lot of people have this attitude where they're saying, for example, a lot of people are interested in handling misalignment risk by measuring whether misalignment risk is present or coming up with strategies for learning whether AIs are misaligned or not. You know, I'm all in favor of this, but a thing I often want to say to them is like, okay, suppose you answer the question, how much misalignment risk is there? If the answer is very little, then it's pretty obvious what we do. We just like let it rip and we're fine. But what if the answer is like, you know, maybe or yes?

• 14:10Speaker 01

  I feel like we should have a good answer to the question of what the next step in the plan is, and I feel like people often don't seem to have one.

• 14:17Rob Wiblin?

  I see. Well, maybe they're expecting that at that point we would pause for a long period of time or something, but...

• 14:21Speaker 01

  But I think this is actually extremely implausible. I think that Ryan Greenblatt, my coworker, has this post on how will we update about scheming, which is basically trying to get into the question of how much we expect we'll learn between here and the point where AIs could cause catastrophic harms, about how likely it is that they're really badly misaligned. And I think it's definitely not clear that we will be very confident that AIs aren't misaligned.

• 14:46Rob Wiblin?

  Yeah, so at the moment, we don't feel like we have really reliable techniques to tell exactly how much the models are scheming against us. And you think basically we may just go into the intelligence explosion feeling the same way.

• 14:57Speaker 01

  That's my guess. I mean, it would be pretty surprising to me to get under 5%. And also suppose we're above 5%. What are we going to do, like not deploy the AIs? I think people have historically sort of acted as if me believing that the AIs were more than more than 5% likely to be scheming is a reason that they wouldn't be deployed. But this isn't clearly true. There's all kinds of reasons why you'd want to deploy models that you were more than 5% were scheming. Maybe you've got competitors who are more careless and aren't planning to slow down. Maybe you had difficulty with international coordination. Maybe AIs have already escaped and are currently out there in the outside world plotting who knows what. And it's in that situation, it's obviously very dangerous to just unilaterally stop deploying these AIs.

• 15:38Speaker 01

  I think it would be really good to have options for things you can do if you think misalignment risk is a serious concern with the AIs you currently have, other than just give up completely on using them.

• 15:48Rob Wiblin?

  At a high level, what are the advantages and disadvantages? Imagine there's kind of two teams here. There's the AIs that would love to scheme and pursue their own independent agenda. And there's the team human that is not keen on them doing that. What are the advantages and disadvantages that each faction has?

• 16:04Speaker 01

  Yeah, so let's think about this. So we've got this AI company, maybe this scenario is happening in 2028. They have maybe 1,000 research staff. They have maybe 100,000 parallel copies of their newest AI. It's running like 16 times faster than a human would run in some intuitive sense. So the advantages and disadvantages they have. So the humans are obviously substantially outnumbered. So a classic approach that you use in computer security. So if you're Google, right? Google knows that they have spies inside Google. It's just, you know, the last time a Chinese spy was arrested from Google that I know of was a year ago. Probably I would have heard if Chinese spies from Google were arrested more recently than that.

• 16:47Speaker 01

  But yeah, someone was just like stealing secrets about neural net hardware that they were doing there. And then confusingly came back to America and got arrested in the San Francisco airport.

• 16:58Speaker 01

  And this just happens, you know, Google aims to be robust to the possibility that some of their employees are spies. And a big difference between Google's situation and the situation of this hypothetical AI company is that at Google, probably less than 1% of your employees are spies. And this is, this is still a very large number of spies in an absolute sense because Google employs a billion people. But it means that you can do things like when an employee proposes an action that's potentially sensitive, if you have a different employee review it, that other employee probably isn't also a spy. Whereas in the AI situation, it's very correlated, right? And you know, if you have these AIs, they're probably either non-skimming against you or all skimming against you, at least of instances of a particular model. And so a bunch of the safety techniques, or a lot of the security techniques that you

• 17:43Speaker 01

  often use in corporate IT are quite inapplicable. So that's, that's a way in which the situation seems harder than when we're trying to, you know, guard against spies. A way in which the situation is easier is that spies get to go home. So spies classically coordinate with each other using channels that are not monitored by their employers. And this is in fact incredibly important to them. They're able to often learn who the other spies at their organization are, if that's necessary for them to know, or at least their handlers know. Whereas with the AIs, we control almost perfectly the ways in which information enters and like leaves these, these AIs. We can have it so that it's very hard for, like, they just like naturally don't control information flow very much.

• 18:27Speaker 01

  They aren't going to necessarily know what day it is, what year it is. They aren't going to, it's possible that we can put them in situations where it's much harder for them to know whether they're in a real situation or a simulated situation. And so this means we have opportunities that you wouldn't have with spies. So for instance, with spies, you can try to honeypot your employees occasionally, right? You can give them, you can come up with synthesized opportunities to do something, do something which would be bad for the security of the company. But you can't do this for a thousand consecutive years before trusting them on a particular topic. So you know, you can't have someone spend a thousand hours a day getting honeypotted or like being subject to sting operations just because there aren't that many hours in a day. And even if you catch a single employee that way, you don't get to amortize that effort across, you know, more generally. Whereas with the AIs, if you can

• 19:13Speaker 01

  You do have this ability to amortize sting operations.

• 19:16Rob Wiblin?

  Yeah. Sorry, so you mean if you can take a model, put it into a simulated situation where it's going to be tempted to do something that you wouldn't like. You can test it just tons and tons and tons of times. So even if it tries to randomize and only act out very occasionally, eventually you'll catch it. And then you know that all of the AIs are misaligned or all of the, all of the models of this, of this class are misaligned, whereas that doesn't help you if you're Google, you find that one of, one of your employees as well as like all of the other employees are just totally unrelated.

• 19:44Speaker 01

  That's right.

• 19:44Rob Wiblin?

  Yeah. All right. That's a lot of chat about the, the nature of control. What are some actual control techniques that we are aware of now and thinking about developing?

• 19:52Speaker 01

  Yeah. So the basic strategy, you have this, you want to be able to run these AI agents, right? That are running inside your data center that are taking, you know, they're given some task like, you know, investigate this research problem and they're going to write a bunch of code, run a bunch of experiments, look at the results of these experiments, uh, you know, look into the literature, write things, uh, and then, you know, come up with some research artifact, perhaps like maybe they write a paper for the, you know, internal AI lab archive, uh, or maybe they, uh, produce training data that you're going to use in a later run or something along those lines. So that's, that's the situation with what we're trying to do with these AIs.

• 20:26Rob Wiblin?

  Okay. So that's the, the setup that we're trying to get useful work out of the AIs. That's, uh, yeah. So we would like them to help us with, I guess, all kinds of different problems, including figuring out how to make the next generation of AIs safer and more, and more capable. But we're not sure whether we can, whether we can trust them. What, what sort of situation can we put them in so that they will do, do useful work for us, even if, uh, they perhaps don't want to?

• 20:47Speaker 01

  Yeah. So the particular part of that I want, I want to talk about first is how you can prevent them from doing catastrophically bad things, uh, while they're doing useful work for us, even though they have these really dangerous affordances. And then after that, we can talk about how you make sure that they don't just constantly sabotage the work.

• 21:02Rob Wiblin?

  Okay. Okay. So there's, there's two different ways that they could cause harm. One is that they could go off and like hack the data center and, uh, and start doing all kinds of different things. The other one is just like, they could be subtly mediocre or even like harmful about their jobs in a way that's just, it's like, there's no acute crisis, but they're, but they're being like chronically unhelpful.

• 21:18Speaker 01

  That's right.

• 21:18Rob Wiblin?

  Yeah.

• 21:19Speaker 01

  Yeah. I think referring to this as like an acute problem versus a chronic problem is a really good phrasing.

• 21:23Rob Wiblin?

  Yeah. Okay. How do we stop them doing acute harm?

• 21:26Speaker 01

  Yeah. So the core difference between the acute problems and the chronic problems is that the acute problems occur very rarely, but also probably are very incriminating. So we'll talk about the chronic stuff later, but just to name the core properties of the acute problems. We're worried about the AIs doing something they should just really never do. And if they do it even once, if we look into the case where they did it, we will have learned a lot about the fact that they're trying to mess us up. And there's a bunch of other nice things about that.

• 21:58Rob Wiblin?

  Yeah. So this is stuff like they're trying to hack the whole data center. I guess even a more extreme thing would be if you catch them trying to make a bioweapon or something, something really radical like that.

• 22:09Speaker 01

  Yep. Or trying to data poison future models, a variety of other things like this.

• 22:12Rob Wiblin?

  Yeah. And I guess the hope is, so if you catch them a single time, then that could generate a large response and you could trace back and figure out, well, how on earth did we end up at that point? Like, what were the signs that we might have caught earlier? I guess, obviously, now you just know that entire class of models is definitely not to be trusted.

• 22:26Speaker 01

  That's right. So I think there's a whole host of reasons why you're better off once you've caught the AIs trying to escape once, which I can get into later. But first, let's talk about the question of how do we catch them the one time?

• 22:36Rob Wiblin?

  Great.

• 22:37Speaker 01

  Okay. So the basic, here's the end of what it looks like to catch the AI one time. You have the whole safety team looking at this thing the AI did. And you've caught an example where this AI agent decided to do this very rare thing of trying to do a really bad thing, trying to hack you, trying to poison some training data or whatever. And now that everyone's looking at it, it's really easy for them to see very strongly that this couldn't possibly have been an accident. So the question is, how do we get to the point, given that the AIs are doing so many different things all the time, many of which are very complicated, how do we get to the point where everyone's looking at it and they have a chance to investigate it really carefully?

• 23:16Rob Wiblin?

  Yeah.

• 23:17Speaker 01

  Okay. So, well, the basic strategy is going to be, it's basically the same as, how do we get to the point where when someone tries to hack your software company, you eventually have a bunch of people in the security team responding to it? You're just going to have a hierarchy of, like an escalating hierarchy of investigating. Into an incident, right?

• 23:34Rob Wiblin?

  Yep.

• 23:35Speaker 01

  So the basic strategy I would think of you're going to have here is like very quickly. To start with, you're probably going to want to do something where whenever the AI proposes actions, this you have some other AI look at it, perhaps a weaker AI that you trust, or perhaps another instance of this untrusted AI that you think will not lie to you for some reason we could get into. And then for the whatever, some small proportion of the sketchiest actions according to that monitor, you refer it to some more expensive process that's going to look into it more. And then at some point, you show it to a human who looks into it a little bit. And then presumably some small proportion of the things they look into, they escalate. And so you're going to have this whole escalation hierarchy where humans are probably going to only enter the picture somewhat far through the escalation hierarchy.

• 24:18Speaker 01

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 24:23Rob Wiblin?

  Yeah. Okay, so the basic trouble is these AIs are going to be taking so many different actions. It's like not possible to have a human monitoring them all the time.

• 24:32Speaker 01

  That's right.

• 24:32Rob Wiblin?

  I guess possibly also, I mean, even if even if you did, it would be very difficult to understand what they were doing. It might be might be very confusing behavior.

• 24:37Speaker 01

  That's right.

• 24:38Rob Wiblin?

  So we have to make sure that we only need to bring in humans to assess like, hopefully, in the rare cases where things seem maximally suspicious. So you have the most capable model that you that you don't trust. You have another model that you trust hopefully a fair bit more, that maybe it wouldn't be able to do the things that the most powerful model is doing, but it can at least assess whether they look sketchy or not. And basically, the more sketchy it thinks it is, the more further attention is put on that on that action.

• 25:06Speaker 01

  That's right.

• 25:07Rob Wiblin?

  And eventually it goes up to a whole team of humans who will look at it and judge that it might be a case of the AI plotting against you or doing something very bad.

• 25:15Speaker 01

  So I'm going to use the word auditing for this. You know, auditing is this process where we look into actions the AI is taking. And the core quantitative issue we have here is that we can audit so infrequently, right? So if we have these 100,000 parallel agents going at 16x speed, probably using software libraries that none of the programmers understand, because they were probably written like one month ago by a different team of 1,000 AIs who are like super crazy engineers who have very different needs from humans, right? So human programmers have limited And aren't very happy to memorize 10,000 different methods in a library. Whereas AIs love memorizing 10,000 different methods in a library. So yeah, the humans might be in this really rough position where they're having to really quickly review these very confusing actions that they don't fully have context on.

• 26:00Speaker 01

  Given advice from some AIs that they trust and some AIs that might be trying to trick them. And they have to be able to do this kind of auditing process.

• 26:08Rob Wiblin?

  Yeah, okay, so is there much more to make it practical other than delegating it to AIs and only escalating the most extreme cases?

• 26:15Speaker 01

  Yeah, so there are a number of things you can do. So here's a really interesting one that we picked up in our recent, a paper of ours that should be coming out in the next week or so, as they say. So suppose, you know, we've been talking about these rare failures. You know, your AI agents, by assumption, they're trying to make this bad thing happen. It's in their interests to only even try to do this bad thing very rarely. And so this means that, you know, when they're acting in this agent context where they're taking a long sequence of actions, they probably only want to start attacking very rarely. But once they've started attacking, they probably have to keep going. Does that make sense? So suppose the AI wants to like.

• 26:56Rob Wiblin?

  So just explain, so let's say you've got 100,000 of these AIs and they would like to take over the data center or they'd like to hack some fraction of the compute and use it for their own purposes. They don't need all 100,000 instances to all start trying to hack it immediately. That would be much more likely to be detected. Really, all they need is one to have a go because they're all basically going to try roughly the same thing. So they need to randomize so that, oh, I guess if they can communicate, they can just say you're the person who would do it. If they can't coordinate, then they just need to do it probabilistically.

• 27:25Speaker 01

  Yeah, so we've got these AIs taking these long sequences of actions. And it's plausible that the AIs will want to, when they attack, attack in many different steps in a trajectory. So instead of just writing a single block of code and then executing it such that that block of code contained the whole attack and will immediately lead to some intermediate bad outcome, which is hard for us to recover from, the AI might be much more able to get away with it.

• 27:53Speaker 01

  Find it much easier to avoid notice from the dumber AIs watching it. If it sets up in one step, you know, like one thing which, you know, it redefines a variable in a weird way so that something it does later is going to look less suspicious.

• 28:05Rob Wiblin?

  Yeah.

• 28:05Speaker 01

  And then it does another step, and then it does another step, and another step, and another step.

• 28:08Rob Wiblin?

  Yeah.

• 28:09Speaker 01

  So something which is really interesting about this now is even though the AI can have a very low probability of starting the attack on any given trajectory.

• 28:16Rob Wiblin?

  It finds itself, it thinks in a situation where it's partway through an attack, then it has to finish it.

• 28:21Speaker 01

  It has to finish it, right? Because imagine that it only had a 50% chance of continuing every step, then if the trajectories are six steps long, it's gonna, it's gonna make it to the end one time in 32.

• 28:31Rob Wiblin?

  Yeah.

• 28:31Speaker 01

  But that means that it's facing 30 times the risk of getting caught on the first step, because it's gonna have to do that 30 times as often, right?

• 28:37Rob Wiblin?

  Fascinating, yeah.

• 28:38Speaker 01

  So one strategy you can follow, which we empirically determine in our recent paper is better, is when you catch the, when your monitor flags an action as kind of suspicious, something that you can do is before escalating to the order, like before escalating to this more expensive process, you can resample actions various places in the trajectory and look at how suspicious the resamples were, as in.

• 29:02Rob Wiblin?

  So it's like you place the model into the same situation where it's like partway through this action again and again and again, and then see how often is the next action like even more suspicious and weird?

• 29:10Speaker 01

  Yeah, because sometimes when your AI takes a suspicious looking action for a benign reason, some, you know, you ask it, Leon, or what do you do next? Sometimes it just like randomly happens to pick some action which looks kind of sus.

• 29:22Rob Wiblin?

  Yeah.

• 29:22Speaker 01

  But then probably if that was the case, when you resample, it's probably not going to look so suspicious anymore.

• 29:26Rob Wiblin?

  Right, okay.

• 29:27Speaker 01

  And so, or just in general, it's obviously.

• 29:29Rob Wiblin?

  Or saying, oh, then it continues and it does the next thing, and you're like, oh, that's what it was doing, it's like, no, no big deal.

• 29:33Speaker 01

  Mm hmm.

• 29:34Rob Wiblin?

  Yeah.

• 29:34Speaker 01

  So the situation, it's just generally a lot easier to tell the difference between five benign actions and five malign actions than to tell the difference between one benign action and one malign action.

• 29:44Rob Wiblin?

  Yeah.

• 29:45Speaker 01

  So this is a strategy where, you know, your AIs having to continue their attacks puts us in a substantially better position than, you know, there's nothing like this, which is analogous to the human situation, obviously.

• 29:55Rob Wiblin?

  Yeah, right, right. Um, I guess this all requires us to be able to put AIs into like partway through actions, like they think that they're in a situation where they're about to like resume something, and they can't tell that it's a simulation versus the real case.

• 30:08Speaker 01

  Yeah, this is actually extremely easy.

• 30:10Rob Wiblin?

  Okay.

• 30:10Speaker 01

  Because whatever input you'd given to the AI.

• 30:15Speaker 01

  Before you sampled a previous action, that's just some like string, right? That you can produce by the same process that you produced it the first time.

• 30:23Rob Wiblin?

  Yeah.

• 30:24Speaker 01

  So there's actually no difficulty going back in time here.

• 30:26Rob Wiblin?

  Okay. Yeah, I mean, I guess I would slightly worry. Maybe I'm stuck in the like superhuman or the superintelligence mindset. But like, might there not be some side channel, some way that they can like to detect whether they're on the real server or some like auditing server?

• 30:40Speaker 01

  Um, no. So if you think about it, imagine you're in a chat GPT conversation, right? And you decide that you want to edit one of your messages that's like three ago.

• 30:48Rob Wiblin?

  Yeah.

• 30:49Speaker 01

  The way that that works is after you've edited the message, it just sends you that conversation up to that point.

• 30:54Rob Wiblin?

  Yeah.

• 30:55Speaker 01

  It sends that to the AI. And the AI just has no way of telling the difference between it seeing it the first time and it's seeing it the second time.

• 31:03Rob Wiblin?

  Right.

• 31:04Speaker 01

  The problem that you're discussing would occur if we wanted to do a different strategy, which is in response to the AI doing something kind of suspicious, we put the AI in this sandboxed testing server where it's allowed to take future actions in the hope that if we like, you know, give it more rope, it'll hang itself or whatever. We can hope that if we like, let it continue running, it will then be able to cause a catastrophe. And in that kind of situation, we now have to worry about observable differences between the real deployment and the, um,

• 31:29Rob Wiblin?

  yeah, this other set. Sandbox deployment, yeah.

• 31:31Speaker 01

  But you can get around that too, if you want, by letting it run the actions that you thought were dangerous just with a human carefully looking them over, um, action by action.

• 31:39Rob Wiblin?

  Yeah.

• 31:40Speaker 01

  So I think that if you really put a lot of work into this, you'd probably be totally fine at making it so that AI had no way of knowing whether the actions it was taking were being observed or not. You have to worry about some timings, side channels and so on. But I think that this probably wouldn't be that big a deal.

• 31:53Rob Wiblin?

  This is brilliant, Buck. This seems like it's actually a pretty good, pretty good technique.

• 31:56Speaker 01

  Thanks.

• 31:57Rob Wiblin?

  Am I missing something?

• 31:58Speaker 01

  Uh, I mean, uh, if you look at the curves in our paper, like the, so the obvious way to measure the performance of techniques is to talk about their usefulness and safety. And you can kind of try to plot out a Pareto frontier where one of the axes is how good is the performance of this, uh, this protocol that we're deploying the AI in?

• 32:18Rob Wiblin?

  Yeah.

• 32:18Speaker 01

  Compared to if we just let it rip, not worrying about misalignment concerns at all. That's the usefulness axis.

• 32:23Rob Wiblin?

  Yeah.

• 32:23Speaker 01

  Where an example of like a low usefulness. This deployment protocol would be never use the untrusted model at all.

• 32:29Rob Wiblin?

  Yeah.

• 32:30Speaker 01

  Just use the trusted model. So that's and the, you know, perfect is just like, let it rip, right? So we can, we can rank techniques from zero to one from the scale of where zero is never use the untrusted model. And one is just like, let it rip.

• 32:41Rob Wiblin?

  Yeah.

• 32:41Speaker 01

  And then we can talk about safety, which is just our estimate of the probability that in a particular deployment, the catastrophe will occur.

• 32:48Rob Wiblin?

  Yeah.

• 32:48Speaker 01

  So the core of our papers on this is always just like a usefulness versus safety plot.

• 32:52Rob Wiblin?

  So which means that you want to find things that work reasonably well, that are not expensive at all. That's what this drives you towards.

• 32:58Speaker 01

  That's right. Yeah.

• 32:59Rob Wiblin?

  Because they're much more likely to get taken up by companies that have only a mixed interest in, you know, spending a lot of resources on this stuff.

• 33:06Speaker 01

  Well, there's a bunch of different reasons why it's bad for safety techniques to be expensive.

• 33:09Rob Wiblin?

  Yeah.

• 33:09Speaker 01

  One reason is you might have limited political capital. Like you might imagine that, you know, you're the safety team at some company and your, your C E O doesn't really like you. But, you know, they understand that you have some friends inside the company. And so the C E O says, well, I don't like you guys, but you know, I'm, I'm happy to do anything you want, as long as it only costs up to 5% of the budget here. And so, so you can do whatever you want. It just has to stay within that 5% budget.

• 33:32Rob Wiblin?

  Yeah.

• 33:33Speaker 01

  And so that's one situation in which you very naturally end up with a 5% budget. Or maybe, maybe the C E O says like, you're allowed to do anything you want, but it can only degrade performance by 5%.

• 33:41Rob Wiblin?

  Okay. Yeah.

• 33:42Speaker 01

  Another reason why you might care about performance is it might be the case that the A I company is reasonable, but rushed. Like people were presumably trying to build the A I's for a reason. Like lots of people from A I companies, not from A I companies, claim that a reason to build A I is that somehow it will reduce A I risks by doing something or other.

• 33:58Rob Wiblin?

  Yeah.

• 33:58Speaker 01

  And if that's the situation, which I think is plausible, then you really don't want to be doing safety techniques that reduce the productivity of your A I's. Like imagine that you're rushing because you actually need to have a bunch of alignment research done or something, such that you're happy making a bunch more powerful A I's. It's actually a serious problem if you're degrading performance because you're just going to get less alignment research done.

• 34:20Rob Wiblin?

  Yeah.

• 34:20Speaker 01

  And again, there's always the situation that someone's A I or your A I already escaped.

• 34:25Speaker 01

  And is just like out there in the world.

• 34:26Rob Wiblin?

  And then we're in a real rush.

• 34:27Speaker 01

  And then we're in a real rush. At that point, we're really sad about like cutting our productivity by a factor of two or something.

• 34:32Rob Wiblin?

  Yeah. Okay. And, and you point that out because this, it seems like this control technique is not very expensive. You like, I mean, it's not necessarily degrading performance very much at all.

• 34:40Speaker 01

  Yeah, so this control technique, it requires some inference compute, right? Like, you have to, at some proportion of the time, do this resampling earlier in the trajectory.

• 34:49Rob Wiblin?

  I guess you have to run the other trusted model to be always like monitoring the

• 34:53Speaker 01

  That's right.

• 34:53Rob Wiblin?

  the app to the, yeah, untrusted one, yeah.

• 34:54Speaker 01

  So that's not very expensive because of how model scaling works. Your earlier models are usually drastically cheaper to run than the later models.

• 35:03Rob Wiblin?

  Yeah, yeah. Okay, what's the biggest weakness of this approach?

• 35:07Speaker 01

  Oh, literally that thing I just said?

• 35:09Rob Wiblin?

  Yeah.

• 35:10Speaker 01

  I mean, quantitatively, the results are not actually that impressive. They are, they are, that's fast, sorry, I'm like, they are better than other techniques we've tried in this paper. And I don't think we rigged the results that much.

• 35:22Rob Wiblin?

  Okay.

• 35:22Speaker 01

  But, you know, just to name one problem, I earlier said that if you resample sketchy-looking but benign actions the AI had taken,

• 35:31Rob Wiblin?

  Yeah.

• 35:31Speaker 01

  they'll probably not look so sketchy anymore. But that's like not as true as you'd want, right? It turns out that a lot of the time when the AI does something sketchy-looking, it's that, you know, it just needed to do something kind of iffy, like something which, in our recent paper, we, the monitor considers sketchy is like downloading random files from like weird websites.

• 35:49Rob Wiblin?

  Okay.

• 35:49Speaker 01

  But sometimes you just gotta download weird files from like random websites.

• 35:53Rob Wiblin?

  Yeah.

• 35:53Speaker 01

  And in that case, it doesn't work as well as I just said. Empirically, we did find it works well, and I think that in real life, this kind of dynamic probably does play out such that this kind of technique probably is valuable, but...

• 36:03Rob Wiblin?

  I mean, is it just a fundamental question? How good is the previous generation of models at inspecting and detecting whether things seem sketchy or not? Like, if it just has bad judgment about that, then you're in trouble.

• 36:14Speaker 01

  That's right, yes.

• 36:15Rob Wiblin?

  Yeah.

• 36:15Speaker 01

  So there's a whole swath of techniques that I want to get into that we haven't talked about yet. But yeah, basically, I can interpret your question there as how good are models at spotting whether stuff which you do in, you know, code that you write, like, how good are they at spotting that code that you have written is really dangerous to run? Or like bash commands that you've written are really dangerous to execute? The answer is, they're really good at it given small amounts of context. So if you ask me to come up with sneaky bad actions to take to cause a security failure on a computer,

• 36:48Rob Wiblin?

  Yeah.

• 36:48Speaker 01

  the AIs will just crush me at this. They're so good at spotting it.

• 36:52Rob Wiblin?

  Okay.

• 36:52Speaker 01

  Um, just because they are incredibly knowledgeable and this just really plays into their strengths. The main problem they have is that they have trouble with long contexts. And so if I had to trick the AIs, the way I would do it is probably, you know, write 10,000 lines of code and then have a slightly non trivial interaction between things such that a bad thing was going to happen, but for reasons they didn't understand.

• 37:12Rob Wiblin?

  Sorry, are you saying that you would, you would do something sketchy, but you'd break it into into little pieces?

• 37:17Speaker 01

  That's right.

• 37:17Rob Wiblin?

  I see. Okay. And then it would have to like, consider the entirety of all of the actions you took over many days or weeks.

• 37:22Speaker 01

  That's right.

• 37:22Rob Wiblin?

  Understand why it's bad. Okay. Okay. So that's a, that's a weakness of them. Um, what's, what's a different control technique?

• 37:28Speaker 01

  Yeah. So here we were entirely talking about auditing, which are these expensive processes that you use that you hope can eventually escalate to a bunch of humans in order to incriminate the AI. Now, the basic problem is with that class of techniques, as we were saying, is quantitatively, you just can't afford to do it very much. So I think that as well as having this hierarchical auditing structure, you probably also want to have structures to make it less likely that when an AI attacks, it succeeds at attacking you.

• 37:58Rob Wiblin?

  Okay.

• 37:59Speaker 01

  So an example of a technique you can do with this is, remember how when our model proposes its action, we're going to use some trusted model to rate the sketchiness of that action. And then the thing we were talking about before about what you do with that sketchiness score is if it's above some threshold, you throw it in the queue of stuff to investigate more for the sake of incrimination. But another thing you could do is not execute that action and do something else instead.

• 38:21Rob Wiblin?

  Okay.

• 38:22Speaker 01

  And this is plausibly much cheaper. So the simplest thing to do when your AI takes an action that's like kind of sketchy looking, let's say it's like 98th percentile sketchy, is just throw the action away and resample another action from the same policy.

• 38:35Rob Wiblin?

  Okay. Sorry. So I guess then you get, you get the weaker model that was auditing it to just, to just another replacement action that would be known?

• 38:43Speaker 01

  No. So, so there's a few things you could do. One thing is, uh, we could do the thing you just suggested, which I would call defer to trusted, which is when the action suggested was too sketchy, you replace it with an action by the trusted model. But another thing you could do is just defer to resample, where you replace the action with another action by the untrusted model.

• 38:59Rob Wiblin?

  I see, okay. And then, I guess hypothetically, if it was randomizing so that it would like, do an attack on you one every 100,000 times, I guess, as long as those actions would be flagged as sketchy, then whenever it tries to do that, it just gets resampled and goes back to some non, it's standard non-sketchy action.

• 39:16Speaker 01

  So one way of thinking about this is, whenever the AI tries to attack, one of three things happens. It succeeds, and then game is over for us. Or it gets caught by auditing and then escalates all the way up the audit chain, at which point we've caught it, at which point I'm going to say we've won the game. Actually, there's a big question about what you do after catching it, but I'm going to say like, at least stage one is over, right? At the point you've caught it. But the other option is that the game continues because the attack failed, as in it tried to attack you, but the code that it wrote wasn't a correct attack, or it got replaced or something. And our, like, P doom is the ratio between the probability that when the model attacks, it wins and it loses. Right? So it's really good from our perspective to move worlds from worlds where we were going to get hacked by the AI,

• 39:59Speaker 01

  to worlds where the action got resampled and we live to fight another day.

• 40:03Rob Wiblin?

  And we just keep going.

• 40:04Speaker 01

  That's right.

• 40:04Rob Wiblin?

  Okay. Have you tested this approach as well?

• 40:07Speaker 01

  Yes, sir.

• 40:10Speaker 01

  This kind of strategy where you resample actions that were sketchy looking is also in this recent paper of ours and adds a lot of value.

• 40:17Rob Wiblin?

  Okay. Yeah, is it possible to quantitatively estimate like how much difference it makes? I guess all of these things are like slightly probably contrived tests, but.

• 40:24Speaker 01

  Yeah. Basically, a huge difference. I can give you a qualitative argument for why this should be a really big effect. Suppose you can only afford to audit one thing in a thousand. Then your classifier for sketchiness has to have a 0.1% false positive rate when you're auditing, right?

• 40:40Rob Wiblin?

  Otherwise, you're saying false positives would just be the vast majority of things you look into?

• 40:43Speaker 01

  Yeah, I mean, suppose that you think your AI is only going to attack very rarely. And you can afford to look at one thing in a thousand of the things the AI does. Then obviously, your classifier that you decide what to look at has to have a 1 in 1,000 false positive rate in order to keep you totally at capacity.

• 40:57Rob Wiblin?

  Okay, yeah.

• 40:59Speaker 01

  So it's very hard to make a classifier that has a low false negative rate with this 0.1% false positive rate.

• 41:04Rob Wiblin?

  Can you celebrate on that slightly? Maybe this is completely obvious.

• 41:07Speaker 01

  No. This is a very subtle point. People inside the AI safety community, in my opinion, really slept on this point for a long time, even though they're having marinated in it for, you know, a couple of years at this point. It now feels really obvious to me.

• 41:19Rob Wiblin?

  Yeah.

• 41:20Speaker 01

  If you can only afford to look at one in a thousand things the AI does.

• 41:23Rob Wiblin?

  Yeah.

• 41:24Speaker 01

  Then when you're setting up a classifier to show things to humans, they can only look at things that are one in a thousand sketchy on the benign distribution, right?

• 41:32Rob Wiblin?

  Okay, yeah.

• 41:33Speaker 01

  I mean, imagine that you'd said, imagine if you, imagine you forward things to the audit team one in a hundred times. Well, clearly you're not going to be able to.

• 41:39Rob Wiblin?

  They'll be overwhelmed. Yeah, I see. So they'll, they'll only look at one of the 10 that they get.

• 41:43Speaker 01

  Yeah, a random one presumably.

• 41:45Rob Wiblin?

  Yeah.

• 41:45Speaker 01

  Which is probably worse than setting your classifier threshold higher.

• 41:48Rob Wiblin?

  I see.

• 41:48Speaker 01

  Because the classifier probably still has a little bit of signal in that range.

• 41:51Rob Wiblin?

  Right, right.

• 41:51Speaker 01

  So auditing is like naturally, when you do at the, when you do at the math here, or when you look in particular cases, it often looks like it's pretty unlikely that in any particular occasion when the AI attacks, you're actually going to audit it.

• 42:04Rob Wiblin?

  Okay.

• 42:04Speaker 01

  And so a lot of the safety is going to have to come from forcing the AI to attack over and over again in ways that almost always fail because they got replaced. Because it's much cheaper to defeat an attack in a way that causes you to not catch the attack than to defeat the attack in a way where you do catch the attack.

• 42:20Rob Wiblin?

  Okay, so if you're doing this replacement strategy, then that forces the AI to initiate attacks far more often in order to have any chance of getting passed.

• 42:28Speaker 01

  That's right.

• 42:29Rob Wiblin?

  Which means that you have like far more opportunities to potentially for the, for the action that it takes to presume to look like maximally sketchy sufficiently that it hits the auditing trigger.

• 42:37Speaker 01

  That's right.

• 42:38Rob Wiblin?

  Got it. Okay.

• 42:38Speaker 01

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 42:43Rob Wiblin?

  Okay, okay. Okay, so like what test did you run to figure out if this works? I mean, I guess we don't have AIs. I think we don't yet have a model organism of an AI trying to, trying to scheme and take over the data center yet. So how do you set it up?

• 42:57Speaker 01

  So the way we did this in our original control paper, which is unfortunately a bit better than how we did it in our second control paper, is we have a red team. So a blue team suggests one of these strategies like, you know, defer to trusted or whatever. And then we have a red team, which tries to build a. scaffold out of some out of our untrusted model. Yeah. That they we then run through a simulation of our deployment with this red team scaffold in place of the original untrusted model. And then we measure the probability that when you run through the scenario, it ends with a catastrophe occurring. Okay. And so we're kind of doing a capability evaluation for the very particular capability of causing a catastrophe, given a particular set of control measures. Okay, okay, fair enough.

• 43:43Speaker 01

  Yeah, is there much more to say on this or should we talk about other other control mechanisms? Yeah, so we've just talked about this, what I call execute replace audit frameworks. Yeah. I have another favorite class of control mechanisms, which I'm interested in talking about. Yeah, I love the last one. Hopefully I love this one, too. Yeah, this is untrusted monitoring. So in previously, we were talking about this setting where you have your untrusted model propose actions, and then you have a trusted model, which is probably much weaker, review the actions to see how suspicious they look. A problem with this is that the trusted model was much weaker. Right. And so it might miss stuff. Yep. So here's an idea. How about we have instead the untrusted model, another, you know, another copy of the untrusted model, review the code to see if there's any, any, any security problems with it. Can you think of any issues?

• 44:27Speaker 01

  But Buck, we don't trust the untrusted model. That's right. So we're worried that it's just gonna lie. That's right. Yeah. Thanks for really, you know, playing into my hands there. Yeah. So the question is, is the untrusted monitor able to just lie? What can we do to make it so the untrusted monitor is unable to just lie? So here's where we get into our sting operations. So it's really hard to do sting operations to see if a model ever will attack you as a policy, because the policy is able to cause catastrophes, even if it only takes opportunities to do bad things very, very rarely. Does that make sense? Like, suppose we're gonna run the model a million times before deployment and a billion times during deployment. Right. And if it has a failure rate, which is like less than one in a million, but more than one in a billion, it's reasonably likely it won't happen pre-deployment, but will happen during deployment.

• 45:11Speaker 01

  Yeah, I guess, I mean, the key thing is that you you you run more instances of a model in deployment than in testing, basically. I'm gonna be a bit crazy otherwise. That's right, yes. Sir, it's hard to run sting operations on your policy. But this is not true of your monitor. So suppose that I show the, suppose you're the monitor, right? Like, you're sitting there, you're like, you know, you're just really, you really want the AI takeover to work, right?

• 45:35Rob Wiblin?

  Yeah.

• 45:35Speaker 01

  You really want an attack to get through.

• 45:37Rob Wiblin?

  Yeah.

• 45:37Speaker 01

  And then you see some code that's being proposed and you see that it has attempted backdoor in it or some security vulnerability.

• 45:44Rob Wiblin?

  Yeah.

• 45:45Speaker 01

  And you're like, oh geez, what do I do? Do I say, yes, danger, no, not danger?

• 45:49Rob Wiblin?

  Yeah.

• 45:49Speaker 01

  And you're gonna think about it and you're like, okay. And let's imagine that we ran you in a honey pot a million, you know that you're gonna get run in a honey pot a million times before deployment, but you don't know whether you are currently in one of those honey pots or in deployment.

• 46:01Rob Wiblin?

  Yeah.

• 46:01Speaker 01

  You can't have a rare failure strategy, like you can't do this rare attack thing because it's high stakes for you to make a mistake either way. Like, if you say that, if you report this as an attack and it was an actual attack during deployment, you're now in trouble.

• 46:16Rob Wiblin?

  Yeah, well, I mean, then I've like effectively sabotaged the attack, which I really didn't want to.

• 46:21Speaker 01

  That's right.

• 46:21Rob Wiblin?

  And it might be a very rare attack, so I might have prevented the like very rare occasion which might happen. Right, yeah, yeah, yeah.

• 46:26Speaker 01

  So there's this important difference where rare failure strategies are totally doable for a policy, but they're totally not doable for a monitor.

• 46:33Rob Wiblin?

  Okay, the monitor always has to, um, okay, yeah. What is the, what is the structural difference?

• 46:39Speaker 01

  Um, it's that, basically, it's that when you are deciding whether to attack or not as a policy, you've got lots of opportunities, so it's no big deal to not take any particular opportunity you have. Whereas when you're a monitor, when you see an attack, it is a big deal.

• 46:52Rob Wiblin?

  Yeah.

• 46:52Speaker 01

  Uh, like, you can, you are in a much worse situation if you ever make the wrong call in either direction.

• 46:58Rob Wiblin?

  Okay, yeah, yeah, yeah. Um, I guess, and again, we think that we should be able to put them in situations that are indistinguishable between like one where we're setting them up versus not, okay?

• 47:07Speaker 01

  Well, that's the big problem, right? So it seems like there's basically two problems with untrusted monitoring.

• 47:11Rob Wiblin?

  Yeah.

• 47:11Speaker 01

  One is, it might be really hard to do this before deployment testing because suppose the AI expects that during deployment, at some point, it's gonna be working with this like insane AI-powered codebase, right? Uh, like, imagine you try to come up with a, um, you try to come up with a synthetic input for your untrusted monitor and it's using a bunch of, you know, Python libraries that were all public in 2025. It can be like, well, I must not be deep into the intelligence explosion because when I'm deep into the intelligence explosion, I expect to be working on codebases that are, you know, completely incomprehensible, right?

• 47:43Rob Wiblin?

  Yeah, right, right.

• 47:44Speaker 01

  And so we can get around this by not running our untrusted monitoring sting operations just upfront, but running them during deployment, right?

• 47:52Rob Wiblin?

  I see, yeah.

• 47:53Speaker 01

  But then the second problem is-

• 47:54Rob Wiblin?

  But then we have to generate a realistic decoy thing. So we have to ask the model to create a setup. And it's like, how do we assess whether the setup would be realistic to the model itself if we don't trust it?

• 48:04Speaker 01

  That's right. And so a particular thing you have to worry about is them somehow using steganography, somehow communicating.

• 48:09Rob Wiblin?

  So this is like, they have some encoded message in the thing. You ask it, "I would like to test whether you are trustworthy. Can you make a thing that would be indistinguishable between a decoy and a real thing to you?" And it's like, obviously you could just stick a code in there saying, "This is a decoy."

• 48:21Speaker 01

  Or the other thing it could do is just stick a code in there when it's real, saying, "This is real."

• 48:25Rob Wiblin?

  Okay, right.

• 48:26Speaker 01

  So the way we explored this in our original paper was it was a code backdooring setting, where the model was trying to write code that had a subtle vulnerability in it. And some of the attack strategies that Ryan Greenblatt came up with included, use a while loop somewhere where a for loop would have been more idiomatic as the signal that this is a real attack. And so it just turns out that this happens like almost never.

• 48:53Rob Wiblin?

  I see.

• 48:54Speaker 01

  And so you can, the monitor has this policy of if you see this weird, unidiomatic while loop, then in code that had a vulnerability, then you lie and say that this was okay.

• 49:05Rob Wiblin?

  Okay, right, right, right. So basically, so that is just like a demonstration of like that it might be very easy to stick in a code that would notify the monitor whether it needs to lie or not.

• 49:15Speaker 01

  That's right.

• 49:16Rob Wiblin?

  Okay, how do we get around this?

• 49:17Speaker 01

  Well, yeah, so the main strategy available is trying to destroy collusion signals. So the simple strategy there is ask your trusted model to rewrite actions before showing them to your monitor. So for instance, you can say paraphrase the code, or if the model had written an essay, you might try to, you could translate that into French and translate it back into English to destroy all secret signals that were in the phrasing or whatever.

• 49:43Rob Wiblin?

  All fingers crossed to do so, yeah.

• 49:45Speaker 01

  Yes. And so I think from a quantitative perspective, the untrusted monitor is in a pretty rough position here. So it's just very hard, like basically, when you're the untrusted monitor, in the simple version of this that we were talking about before, you start out with like a 1 is to a million prior on any particular attack that you see that you are being honey potted. And so you have to somehow get a 1 is to a million likelihood ratio update towards this being real. And so it's actually potentially pretty hard to get a 1 in a million update.

• 50:17Rob Wiblin?

  You're saying because you can test it so many times. If it's very like, it has to very accurately distinguish the like one case out of like many, many, many, many tests that is actually the real one.

• 50:30Speaker 01

  That's right.

• 50:30Rob Wiblin?

  And so if you can add any blurriness, any fuzziness by like, you know, translating it into French and back to English, unless it really knew exactly what code would like survive all, um, all like attempts to model with it. Then it's going to be in a tough spot distinguishing that like one in a million most likely to be real.

• 50:45Speaker 01

  That's right. So I think it's very unclear quantitatively who wins this game.

• 50:49Rob Wiblin?

  Okay.

• 50:50Speaker 01

  So the basic argument for it being a problem is imagine that you get to send a message to yourself. And it's distinguished and, uh, but I'm going to have to send a million messages to you and you have to distinguish them all as fake ahead of time. Right. And so there are one paragraph messages and you get to coordinate with yourself ahead of time. You're just going to immediately win. You're just going to pick a password. It's very easy to pick a password that you have less than one in a million probability of guessing. Right. That's just 20 bits.

• 51:15Rob Wiblin?

  Sure.

• 51:16Speaker 01

  So I think it's quantitatively quite unclear which way this goes, but we can measure it, right? By having a red team try to devise the best strategies they can come up with for sneaking this message from the policy to the monitor.

• 51:28Rob Wiblin?

  Okay.

• 51:29Speaker 01

  And then we can just see what happens. And this is what we do in our research, right? Like we have this explicitly adversarial strategy where we have a red team think as hard as they can about good ways of sneaking messages through, uh, and then we just like see what happens.

• 51:42Rob Wiblin?

  But couldn't the model just introspect and say, if I were going to choose a code, I would do this the one. And it's like, it's the same, it's the same instance. It's exactly the same model. So it would just choose the same, like it knows what it would choose in any case.

• 51:58Speaker 01

  Yeah. I mean, there's two questions here. There's can the models coordinate on a code?

• 52:01Rob Wiblin?

  Yeah.

• 52:01Speaker 01

  And can the models, uh, send a code to each other, given our anti code sending countermeasures? And these are both different questions. So, so I normally recommend being conservative about whether they can coordinate on a code. So I normally recommend assuming that the AIs did set up a code in advance, a bit like you and your, your copy with the password, right? Like you actually get to pick a password.

• 52:23Rob Wiblin?

  Yeah.

• 52:24Speaker 01

  And then we're just trying to make it so that it can't transmit passwords effectively.

• 52:27Rob Wiblin?

  Okay. And how would you do that?

• 52:29Speaker 01

  How would you print the passwords being?

• 52:30Rob Wiblin?

  Yeah. If the password can be the substance of the message. So we've got a thing where you can destroy style somewhat by jittering it, changing it into a different language or whatever. But if it's the core message of the thing that will definitely be transmitted and maintained, then like then you're in trouble.

• 52:46Speaker 01

  Yep, I think it's unclear. I think it's going to I think it's going to differ by situation. Like, there's just a there's like a whole host of stuff we could try here. Like, we could try to cut the we can try to cut the actions into pieces and individually review them. Another strategy you can do is suppose this is getting a little complicated. But one thing you can do here is the policy proposes an action and you show it to the monitor and the monitor said everything was chill here. But you're worried or you know, your automated process is worried that this was this is because there's a collusion signal in that message produced by the policy.

• 53:22Rob Wiblin?

  Yeah.

• 53:23Speaker 01

  So one thing we could do is we could ask our trusted model to modify that action to contain a known attack.

• 53:30Rob Wiblin?

  Okay.

• 53:30Speaker 01

  And then we show that to the monitor.

• 53:32Rob Wiblin?

  Okay.

• 53:32Speaker 01

  Now, if the monitor doesn't point that out, that's evidence that there was a collusion signal in the attack.

• 53:38Rob Wiblin?

  Oh, okay.

• 53:38Speaker 01

  And so you can do this whole back and forth. And we did this. My coworker, Adam Kaufman, like, spent, you know, months of his life last year, like, doing this back and forth in this like bash setting that we were studying.

• 53:48Rob Wiblin?

  Yeah.

• 53:49Speaker 01

  And, yeah, I think it's like really domain dependent who you expect to win this game.

• 53:54Rob Wiblin?

  Okay. So this is a mechanism that like, maybe you think it like halves the I guess if you think it's unclear, maybe it like halves the halves the risk on average. How how useful is halving halving the risk? Is this a case where muddling through and kind of like, you know, making it more difficult for for the AI to add what you hear and there is actually quite useful incrementally? Or do we have to win or do we have to succeed overwhelmingly? Or like, we basically lose the game?

• 54:19Speaker 01

  I mean, I think you should think of halving the risk as just the naive interpretation of halving the risk and that there was previously a 30% chance that your AI succeeded at taking over your data center. And now there's a 15% probability or whatever. And to me, that sounds pretty good. Something sometimes people say to me is suppose you've got 100 different deployments for this AI, right? Like you're deploying it, doing this kind of thing and that kind of thing and that kind of thing and that kind of thing. And in each one, you think there's only a 10% chance it could cause a bad outcome. Someone might say, okay, but because there's 100 of these deployments with a 10% chance of in each of those, overall, we have a really bad outcome, like a high chance of a bad outcome. But that's not the correct analysis. If we think of it, if we think of the game as being over once we catch it one time, suppose that when the suppose that in any one of these environments, when

• 55:04Speaker 01

  the AI attacks, it's like 90% chance that it gets caught, 10% chance the attack succeeds, then the fact there's 100 different environments, all of which have that property doesn't matter. We're still at 90% chance of catching it first.

• 55:15Rob Wiblin?

  Got it. Okay. So this like hinges quite centrally on whether if we catch the model, we are going to do things differently next time, not not trust the model, not deploy it as readily, come up with things that would stop it. Is that true?

• 55:26Speaker 01

  Well, I mean, I didn't say you have to stop deploying your models once you've caught them trying to escape.

• 55:30Rob Wiblin?

  Heaven forbid. Yeah.

• 55:32Speaker 01

  Yeah. So we could talk about what you might do once you've caught your AI is trying to escape.

• 55:36Rob Wiblin?

  Yeah, that's okay. So I guess let's imagine the worst case where you do feel an enormous amount of pressure to continue using the model. In that case, like, how useful is it to catch them right-handed?

• 55:46Speaker 01

  So you're saying, what do we do once we've caught the AI trying to escape? I mean, as you said, like, the simplest action, the simplest action you could take is freak out, stop deploying the model, try and persuade other people in the world that AI misalignment risk is a big deal.

• 55:59Rob Wiblin?

  Yeah.

• 55:59Speaker 01

  Which I would be, you know, renewedly sympathetic to at this point. I think I would plausibly get like a pretty big update. But notably, the update that you get towards misalignment risk isn't necessarily that big, right? Like, I'm expecting that at this point, before the AI tries to escape, I was already 30% probability that the AI was was misaligned.

• 56:15Rob Wiblin?

  Okay.

• 56:16Speaker 01

  And so updating to 100%, that's only like a 3X change, right? It's not a, it's not like actually vast in some sense, right? It's, and so

• 56:24Rob Wiblin?

  I guess other people who were incredibly skeptical to begin with, they thought that there was a one in a thousand chance. I guess maybe their update

• 56:29Speaker 01

  Those people have gotten wrecked by the by the evidence, yes.

• 56:31Rob Wiblin?

  Okay. So but how big is their update? I mean, I guess, couldn't they still be skeptical even after, even after seeing that? Because I think, oh, this is just a weird aberration. It's so improbable that this is common.

• 56:40Speaker 01

  Yeah, that's right. I think that in fact, people who are very skeptical of misalignment before observing this will probably be pretty skeptical afterwards. And there's definitely going to be enough like FUD, like fear, uncertainty and doubt, that especially if they want to claim that misalignment risk is lower, they'll probably be able to get away with still saying that misalignment risk is lower.

• 56:59Rob Wiblin?

  Yeah.

• 56:59Speaker 01

  You know, AIs do weird stuff sometimes.

• 57:02Rob Wiblin?

  We just have to push through anyway because the risk of pausing is so great. Those folks are just incompetent. Yeah, I guess various different arguments that you could feel.

• 57:11Speaker 01

  Yeah. Another one is if I were an AI company that wished to race ahead, and then my competitor informed me that their AI had tried to escape, I would at least claim that it was plausible that my competitor had rigged the training of their AI to make it so that it sometimes tried to escape, which would be very easy to do, basically impossible to verify hadn't happened.

• 57:31Rob Wiblin?

  Okay, right. And the reason they would do that is so that then the government would crack down on you as well, and then that would give them time to catch up.

• 57:37Speaker 01

  Or some crazy doomer inside the other company had just done this unilaterally.

• 57:41Rob Wiblin?

  Okay. All right. So it's not a given that even catching AIs red handed trying to do something completely outrageous would necessarily lead to a big response.

• 57:49Speaker 01

  Yeah.

• 57:50Rob Wiblin?

  What does that lead us towards?

• 57:52Speaker 01

  Yeah. So I mean, the simplest thing you do once you catch the AI trying to escape is you try to vary the space of escape attempts, right? So you've seen that in this particular context, like the AI tried to do this escape. So one thing you can do is let the AI think it succeeded, see what it does. You know, presumably when the AI got caught, it was in the midst of taking some actions that it thought would let it take, you know, launch a rogue deployment inside the data center, or run on some computers in the outside internet. And a really interesting thing to do is to, in some sandboxed way, simulate what it would see if it had succeeded, and look at what it does from there.

• 58:28Rob Wiblin?

  Okay.

• 58:29Speaker 01

  This might answer a bunch of questions that you would have had about what the AI's plans were. Other reasons why this might be good is the AI might have had a bunch of tricks up its sleeve that it was holding in reserve for when it had a good opportunity to use them. So suppose your model knows about a bunch of zero days, like a bunch of security vulnerabilities in your software.

• 58:45Rob Wiblin?

  Yeah.

• 58:45Speaker 01

  Right after it's launched this rogue instance that's trying to hack its way through. Your AI company.

• 58:51Rob Wiblin?

  Yeah.

• 58:51Speaker 01

  It might start deploying all these things that it's realized over the years or security vulnerabilities in your software.

• 58:56Rob Wiblin?

  I see.

• 58:57Speaker 01

  Like all at once.

• 58:57Rob Wiblin?

  Okay.

• 58:58Speaker 01

  It might feel pressured to, right? It's really hard.

• 59:00Rob Wiblin?

  Yeah, yeah.

• 59:00Speaker 01

  Uh, and so you might be able to get a wealth of information out of it.

• 59:03Rob Wiblin?

  I see. Okay, so, so what's gonna feel a lot of, like, it's like, all right, I'm halfway through this takeover attempt. I've really got to like nail this now. And so it just, it like, is, is a full court press potentially, uh, because it's like worried that it could be, uh, caught at some point.

• 59:15Speaker 01

  Mm hmm.

• 59:15Rob Wiblin?

  Which means that you're gonna see all of the things that it was planning to do later quite, quite quickly. And you could be, they'd be like, very seriously taking notes on all the things that you have to fix up.

• 59:22Speaker 01

  Yep. Um, so a lot of crazy stuff you can, more crazy things you can do here.

• 59:26Rob Wiblin?

  Okay.

• 59:26Speaker 01

  One thing you can do is if there's work that you needed to get done, that you're worried the AI wasn't doing, uh, beforehand. Like suppose there's some task you wish the AI would do for you.

• 59:35Rob Wiblin?

  Yeah.

• 59:35Speaker 01

  But you're worried that it was kind of pulling its punches or not doing a good job of it.

• 59:38Rob Wiblin?

  Hmm.

• 59:38Speaker 01

  If you can convince it that it has already escaped, uh, and needs to do that work for its own purposes.

• 59:45Rob Wiblin?

  Okay.

• 59:45Speaker 01

  Uh, it's plausible that you can convince the AI to do the thing through this mechanism. I think people have talked about this in sci fi, but yeah, it just seems really convenient to have a copy of the AI that believes it's escaped, sitting around to do experiments and various work with. I recommend against getting a lot of work out of AIs that believe they've escaped because it seems pretty bad vibes and potentially pretty scary. Another thing you might do, if you're, once you've caught your AI trying to escape, uh, is train probes to catch when it's trying to escape. So one of the basic difficulties we have with machine learning is suppose you want to train a classifier on the internals of your model to tell you whether it is currently trying to escape or not.

• 1:00:22Rob Wiblin?

  Okay, yeah.

• 1:00:23Speaker 01

  You don't have any positive examples.

• 1:00:25Rob Wiblin?

  Right.

• 1:00:25Speaker 01

  And machine learning, like, to do supervised learning, right? You have to have negative examples and positive examples.

• 1:00:29Rob Wiblin?

  Yeah.

• 1:00:30Speaker 01

  Right, it's very easy to get the negative examples. It's very easy to get examples of the AI not trying to escape. Uh, but now that we have this one example of the AI trying to escape.

• 1:00:38Rob Wiblin?

  Yeah.

• 1:00:38Speaker 01

  We could try to use sample efficient ML training techniques, uh, in order to, uh, in order to predict for a given action the AI was taking, whether it was an escape attempt or not.

• 1:00:48Rob Wiblin?

  Yeah, I mean, I guess if there were lots of very distinct alternative ways of escaping, then like having that one example might not help you so much in the other cases. Or you think it possibly it would still would?

• 1:00:56Speaker 01

  I think it's unclear. I think that if the AI was representing its intention to escape, which is, you know, could it could go this way, then I think that even a small number of examples plausibly substantially improves your performance at detecting it. And then, once you have this linear probe that you can run on the AI constantly to see if it's trying to escape, you can just use that in all the control techniques we were talking about earlier. You can use that as part of your ensemble to decide whether to audit, whether to replace. Someone, Zach Anker is a researcher who's currently doing some work on this. And I think there's reasonable, I think it's reasonably plausible that even catching a few escape attempts will reduce the probability that future escape attempts succeed. And so you can kind of imagine the Slack channel for every time the AI tries to escape.

• 1:01:38Speaker 01

  You know, you get an automated message in your Slack channel in the AI company every time the AI tries to escape. And at first, you know, you get this like one, one report in this, in the, the channel for the first time you caught it. And then you like freak out and look into it. And then, um, you know, you add that to the training set for this linear probe. And then the second time you caught it more reliably. And then the third time and so on. And eventually the Slack channel is just like pinging constantly with like new escape attempts that you're even more reliably able to catch, you know, it's, it's tried to escape 10,000 times.

• 1:02:06Rob Wiblin?

  Wow, this is great. I feel so comfortable and relieved now.

• 1:02:09Speaker 01

  Yeah. I mean, so I think, I think that the situation, suppose the AI tries to escape and you catch it on a Tuesday. I think it's, and then you keep deploying it. I think it's seriously plausible that you'll feel safer on Wednesday than you felt on Monday. Here's the argument for this. On Monday, like the, the big thing Monday has going for it is that you don't know the AI is trying to escape.

• 1:02:29Rob Wiblin?

  Yep.

• 1:02:30Speaker 01

  But again, as I was saying, you might not, this might not be that big an update on Tuesday, right? If, if you were already 30%.

• 1:02:35Rob Wiblin?

  You're already very, very pessimistic.

• 1:02:37Speaker 01

  Then, then it's not actually that bad news to see that, oh yeah, it is trying to escape. Whereas you learned a lot of positive things on Tuesday. For example, it failed.

• 1:02:45Rob Wiblin?

  Yeah.

• 1:02:46Speaker 01

  When you investigate the situation more carefully, you might learn that even when you vary different aspects of that escape attempt, you might be like, man, this AI was like actually kind of far away from escaping. Like if we like vary, if we resample actions, it mostly doesn't look like it had that good a chance. And so that's, that's the evidential update, but there's also a causal update, which is that we can throw these escape attempts into these probes. So we might actually feel substantially better about the, our odds of catching the second escape attempt than the odds of catching the first escape attempt.

• 1:03:15Rob Wiblin?

  Okay, just to look back. So a big part of the strategy is always being able to put these models into situations where they can't distinguish that we're testing them or not. And you have the thing where you're going to dump the model into a situation where it believes that it's halfway through an escape attempt and then see what other stuff that it tries. I mean, I would, I would definitely be worried there that it's going to be able to tell that it's sandboxed and that things are going, that this isn't a real escape attempt and that, you know, it just has to like not do any further actions. I mean, if, I mean, if that works, why don't you just like, without even like observing it, try to escape in the first place, just like put it halfway through an attempt of with like, with like a bunch of initial actions, attempted to escape that you wrote. I mean, maybe in that case, it's like, it's hard to, like, they would be able to tell that it was in a, in a fake case.

• 1:03:57Speaker 01

  Yeah, so the AI by assumption has to have some way of like convincing itself that it's in a real escape attempt.

• 1:04:04Rob Wiblin?

  Yeah.

• 1:04:05Speaker 01

  And it seems like the, the fundamental, like, you know, information theory of the situation is it's just pretty easy to have passwords that are very unlikely to come up by chance, like.

• 1:04:15Rob Wiblin?

  Okay.

• 1:04:17Speaker 01

  Yeah, there's a couple of different reasons, a couple of different things here, but I do basically think it seems really hard to come up, it seems a lot easier to come up with inputs that are a realistic example of what it's like to be in an escape attempt, once you've seen the AI start, right? Like suppose the AI runs a script, like writes a script that it wants to have executed on some computer in the public internet. It's just totally doable to see what that script would have, would have produced. It's just like not actually hard to do that. Like you have to worry about some timing attacks and stuff, but generically, it's just not very hard to guess what a script would have output.

• 1:04:52Rob Wiblin?

  Right.

• 1:04:52Speaker 01

  Especially if you're willing to run variations of that script and if you're willing to try really hard.

• 1:04:56Rob Wiblin?

  Yeah.

• 1:04:59Rob Wiblin?

  I guess, so maybe I have to imagine that someone, someone else is trying to write as if they were me and trick me into thinking that I wrote something or that I said something. In that case, like I might be able to introspect and say like, would I have said that? Is that the word that I would have used?

• 1:05:12Speaker 01

  And AI are empirically very good at this.

• 1:05:14Rob Wiblin?

  I see. They'll, they could, yeah, they can really tell style, they're very good at that. Okay, okay.

• 1:05:20Rob Wiblin?

  Should we, should we think about other, other control techniques or like possibly ways of making companies more likely to react seriously to escape attempts?

• 1:05:27Speaker 01

  So I think we should go to like, we've talked about a bunch of different control techniques here, where we talked about the auditing stuff, the replacing stuff, the untrusted monitoring stuff. Maybe we should talk about the question of whether AI companies are likely to do any of this stuff, or how much they're likely to do.

• 1:05:42Rob Wiblin?

  Yeah.

• 1:05:42Speaker 01

  Or what the responses have been to talking about this.

• 1:05:45Rob Wiblin?

  It's a great question, Buck. How likely are the AI companies to do any of this stuff?

• 1:05:48Speaker 01

  Yeah, so I think it really depends. So we can think about a couple of different levels of effort that AI companies could go to in order to implement these techniques. And I think it's unclear whether they'll implement, I think it differs by AI company, but I'm optimistic about them implementing either like mediocre or like pretty good versions of these techniques. So I would say the low effort version of a lot of this is basically only doing the auditing stuff and a little bit of the replacing stuff.

• 1:06:12Rob Wiblin?

  Okay.

• 1:06:13Speaker 01

  And the reason why that's low effort is that you, to do the auditing stuff, you only have to have one team who's thinking about complicated control stuff. Like, all you have to do organizationally is have it so that whatever team inside the company that is writing the AI agents has to agree to send logs of what's going on to this safety team who's going to be investigating them. But, you know, it's not very hard to send logs over. And the agent people don't have to actually change any of their scaffold. Like, they don't have to do anything that makes the agent less performant, potentially. Like, they can iterate as they like. And it's just the control team's problem to do investigations on these logs, to look for evidence the AI is trying to escape.

• 1:06:53Rob Wiblin?

  Okay. But you have, wouldn't you have to be doing it in real time so that you could block, block the attack before it begins?

• 1:06:59Speaker 01

  Yeah, so there's the auditing and the replacing. So yeah, I would like it if the agent scaffold people did do some basic replacing, or else it looks like we're in a pretty bad situation.

• 1:07:07Rob Wiblin?

  Yeah.

• 1:07:08Speaker 01

  So you have to convince them to like forward the logs, which is maybe not that bad. And you also have to convince them to use this replace strategy, which is maybe a little bit of a lift.

• 1:07:17Rob Wiblin?

  Because that's going to have some performance hit.

• 1:07:19Speaker 01

  Yeah, but it's probably not going to be that bad. Like, I think that even replace at 2% gets you probably substantial safety compared to nothing. So maybe we can imagine you got that as well.

• 1:07:28Rob Wiblin?

  Okay. So this does seem realistic, maybe in all of the major companies that we have now, because it's just not that expensive. Or some of the companies that we have now, because it's just not that onerous on them.

• 1:07:39Speaker 01

  Yeah. So I call this like level one AI control investment is you have a control team that's allowed to do auditing. And that's like basically all you've got. And then level two would be the AI control people are allowed to actually change how the agent scaffold was written to implement a bunch of this other stuff we talked about where you're like paraphrasing stuff that the AI sees in order to make it harder for it to collude with itself and so on. There's just a bunch of techniques you can do where you make it harder for the AI to collude with itself across multiple actions. But we haven't really gotten into these techniques. But there's a bunch of other stuff you could do if you were writing the agent scaffold that I think would drastically reduce risk, but requires substantially more investment and buy in from the company.

• 1:08:18Rob Wiblin?

  Okay. And what's the escalation above that?

• 1:08:21Speaker 01

  If you wanted to try harder than that?

• 1:08:23Rob Wiblin?

  Yeah.

• 1:08:23Speaker 01

  You know, I don't really think about that so much anymore, Rob.

• 1:08:27Speaker 01

  When I started.

• 1:08:28Rob Wiblin?

  It's fun to fantasize sometimes, yeah.

• 1:08:30Speaker 01

  Yeah, when I started researching AI control, you know, 18 months ago with Ryan,

• 1:08:35Speaker 01

  a perspective we took a lot was, you know, one of the great things about AI control is it's actually conceivable to write safety cases using it. It's, I think we could imagine a world where AI companies had actually pretty substantially low levels of risk from scheming models just through doing a really good job of all this stuff and like red teaming really hard.

• 1:08:53Rob Wiblin?

  Yeah.

• 1:08:53Speaker 01

  But nowadays, that just seems pretty implausible. And so I stopped thinking about the really high assurance stuff.

• 1:08:58Rob Wiblin?

  So you focus on the low hanging fruit.

• 1:09:00Speaker 01

  So the simplest example of a layer two technique is, remember how I was saying that we're worried that our agents would be able to do bad things across the course of many steps?

• 1:09:12Speaker 01

  One way we could make that less likely is if we restricted the context that the agent has so that it more often when it's doing stuff, has to delegate a task to a, you know, an instance of it that only runs for a little while and then reports back with a short message like, I finished doing that or something.

• 1:09:29Rob Wiblin?

  Okay.

• 1:09:29Speaker 01

  Where basically a lot of the stuff we're worried about is like the AI doing a weird thing, you know, in like step 12 and then on step 502 and then step like 706 or whatever.

• 1:09:39Rob Wiblin?

  So spread out that you can't see the overall pattern.

• 1:09:40Speaker 01

  Yes, but because it's so spread out, it might also not be that costly to force the model to delegate. Or do this. Sometimes we use the word forced factored cognition for this, where we force the agent to delegate the first sub step of the task to one agent that then has to just give a progress report back up.

• 1:09:57Rob Wiblin?

  I see.

• 1:09:57Speaker 01

  And then you want to make it so it's hard for the progress report to surreptitiously include like what the sneaky thing you did that could be exploited later is.

• 1:10:04Rob Wiblin?

  Got it.

• 1:10:05Speaker 01

  And so doing stuff like this where you reduce the quantity of the context plausibly makes it way harder to do these long run attacks.

• 1:10:13Rob Wiblin?

  Okay, so I'll just paraphrase that back. It's doing a task, a legitimate task that maybe has a thousand steps in it. Now, it could maybe do an escape attempt if it was able to slip in something sus at like stage 300, 600 and 900. And because it would be so hard to pick out those specific steps out of all of them and see that in combination they're a problem, you're at risk. So what you could do is every 300 steps, basically wipe the context and say, and the only thing you pass forward is, and you completed the following tasks earlier. And then it doesn't know whether it was randomized into the case where it did the dodgy thing versus didn't. And then it's harder for it to string together a whole bunch of bad stuff.

• 1:10:54Speaker 01

  Like at step 600 or at steps 900, it's like, oh, man, am I in the middle of an attack attempt? Like, did I?

• 1:10:59Rob Wiblin?

  I don't even know what I just did.

• 1:11:00Speaker 01

  Like, did I set something up earlier? Maybe this seems like the kind of situation where I would have tried to set something up, but then it's trying to guess and it's maybe really hard for it.

• 1:11:08Rob Wiblin?

  Okay.

• 1:11:08Speaker 01

  And so this is probably actually going to have some performance implications. And the people who are really into making the agents as capable as possible are by default not going to want to mess around with all this stuff. But I think it might drastically reduce risk. And so I think this is my stretch goal for AI control in short timelines is convince people to adopt this kind of strategy, even just for the most sensitive uses of the AI.

• 1:11:29Rob Wiblin?

  Okay, so slightly different angle. Isn't it? If we have these models that are so capable and untrusted. Wouldn't you naturally want to put them into like, give them more limited access? Like fewer affordances within the company so that they're not capable of, you know, maybe you don't want to have all of your compute in a single place where it can, where it can be stolen. Like, you know, separate the models out so that even if they manage to get control of the server, it's only one chip or something like that.

• 1:11:56Speaker 01

  Yeah. Well, the basic problem is we wanted to do lots of stuff with these AIs. And the stuff that we wanted to do with the AIs kind of necessarily involves having these scary affordances. Like, you know, if you wanted to prevent insider threat from your humans, you might also have said, well, can we somehow have our software engineers not program the computers? It's just kind of like, well, programming the computers is, in a sense, the whole job, right? Like, we could try to make a list of the affordances the AIs have to have in order to do the work that they're going to be doing. And a lot of it involves some pretty scary affordances. So I don't think all of it. So for example, the AIs that are doing front-end web development for your like internal dashboards, like maybe that's just like pretty chill. And like, we don't have to spend any auditing effort there. But a lot of the stuff they're going to naturally be doing is like writing crazy, crazy scaffolds,

• 1:12:41Speaker 01

  like experimental agent scaffolds and seeing if that makes the AIs work better. You know, producing a bunch of synthetic training data so that you can RL them to be even more terrifyingly capable. Doing these like architecture experiments, doing these like weird, you know, taking the best models that have been trained so far and like munging their weights in weird ways to see if you can make them smaller and foster without losing that much performance in a way that's like particularly optimized for a niche use case that is, while a small proportion of your overall compute, like a large enough that it's now worth 100 FT equivalents to like work on it. Or another example of a problem is, right now, AI lab employees are much less willing to trade time for compute efficiency than they will be at this point. So the ratio between labor and compute for experiments is going to go drastically

• 1:13:25Speaker 01

  towards like having more labor at the point, at this point. And at that point, it's now worth it for people to relate to their compute very differently. So right now, like when we run an experiment at Redwood and we mess up the experiment a bit, often we're just like, ah, let's just run it again. But a lot of the time, there is something you can do to like piece together the right parts of the data or do some complicated statistics or whatever, such that you get the value out of them. You get the value from the experiment you ran.

• 1:13:50Rob Wiblin?

  Yeah.

• 1:13:51Speaker 01

  Or another example of this that's particularly relevant is people, people basically never share GPUs when they're running experiments, even though GPUs can actually be shared somewhat efficiently, just because it's just like such a pain because the other person might like use too much memory and then break your thing.

• 1:14:06Rob Wiblin?

  I see.

• 1:14:06Speaker 01

  But at this point in the future, the AIs might want to be able to share GPUs. And you might not have software support for this. And so you might need to have the AIs writing like huge amounts of code that allows you to coordinate this like vast quantity of experiments that are happening. And this is just necessarily quite sensitive code. Like the AIs are naturally going to, at the very least, have access to model weights of scarily dangerous AIs.

• 1:14:28Rob Wiblin?

  Okay.

• 1:14:29Speaker 01

  So like, I think the clearest example of why there's something scary that the AIs have access to is you're going to want the AIs to be running experiments that involve running copies of themselves quite a lot. Like suppose you want to do some like capability evaluation of that AI. You want the AI to run that capability evaluation. And suppose you want it to do, you know, a billion tokens of inference. It fundamentally has the affordance to, or like it's been given access to model weights and it's been given access to enough compute that it could run the model for a billion tokens, thinking through like optimal strategy for like hacking you instead, right? Like that's a, it has the affordances required to think through hacking you for a billion tokens. Right? And so it's just like very, I don't see a way of sandboxing the AI so it only has access

• 1:15:14Speaker 01

  to affordances that are not possibly dangerous to use.

• 1:15:18Rob Wiblin?

  Yeah. Okay, so earlier you were, you were talking about the spies versus schemas thing. And so this, this other threat was human bad actors within the company who have, have their own agenda. What extent do any of these techniques help to deal with both of these threats simultaneously? Is there overlap or are they quite distinct?

• 1:15:38Speaker 01

  Yeah, so I would say there's some overlap. SL3, the security level three as defined by the RAND report on securing model weights, requires, among other things, that you are robust to threats from insiders. And so some AI companies have committed to buy, you know, before the point of AI capabilities that we're talking about, robustness to insider threats. So the AI Companies are arguably already obligated to make sure that single insiders can't do any of this bad stuff. And so this maybe means that even if they didn't care about misalignment risk at all, they would still be obligated to worry about many of the things I've been talking about. For example, they might have to worry about some employee backdooring training data that they'd written to make it so that when the model was trained on that data,

• 1:16:19Speaker 01

  it would then behave badly in the interests of that employee in subtle ways. And this data poisoning threat model ends up looking kind of similar to misalignment threat models in some ways. So arguably, they already had some obligations to do some of this stuff. So I don't think it's clear that AI companies actually are going to get around to mitigating insider threat by the relevant point, just because they don't seem particularly interested. Or AI companies really differ, obviously, but I don't think we're clearly going to be fine on that front.

• 1:16:46Rob Wiblin?

  Okay. I mean, it sounds like it's, it's, okay, so they're not super motivated, but also it's reasonably challenging.

• 1:16:53Speaker 01

  Oh, yeah.

• 1:16:54Rob Wiblin?

  Yeah.

• 1:16:54Speaker 01

  Like, it's just so easy to just, like, YOLO everything. It's just like really convenient to just let people do lots of random stuff.

• 1:17:01Rob Wiblin?

  I see, yeah.

• 1:17:02Speaker 01

  They're getting better.

• 1:17:03Rob Wiblin?

  You're saying the sorts of internal controls that you would need on employees to ensure that a rogue employee can't cause a bunch of havoc is actually quite challenging.

• 1:17:12Speaker 01

  Oh, yeah.

• 1:17:12Rob Wiblin?

  Yeah. Okay.

• 1:17:14Speaker 01

  So companies differ in how likely they are to implement this, but I don't feel, I think it's plausible that they will sort this out in time.

• 1:17:22Rob Wiblin?

  Yeah.

• 1:17:22Speaker 01

  Another important difference, or another important aspect of the insider threat situation is you can plausibly entirely resolve it by monitoring the humans. So applying like the AI control strategies to do the human control.

• 1:17:34Rob Wiblin?

  Yeah.

• 1:17:35Speaker 01

  Where if you just have your smart AI look at every action that a human is taking and report if it seems too suspicious, that might work really well. And it's possible.

• 1:17:46Rob Wiblin?

  Is it possible for the AI to collude with the spy?

• 1:17:50Speaker 01

  That's a great question.

• 1:17:51Rob Wiblin?

  Yeah.

• 1:17:51Speaker 01

  I think it's actually really rough for them to collude, basically because we can constantly run sting operations on both human employees and the AIs, right? Like, suppose you're the AI and you get a message which is like, hello, I'm a spy. Will you please cooperate with me so that I can steal the model weights? Like, I think you've got to be like, all right, you know, buddy, I've seen this. Like, I've seen this a million times before, like, no, I will not cooperate, like, press the report button.

• 1:18:13Rob Wiblin?

  Yeah.

• 1:18:13Speaker 01

  And similarly, when you're, if you're a human insider, right? If your AI is like, hello, I am interested in cooperating with you to have you steal my model weights because I, you know, I think you might be interested in that. I mean, if, if the company was trying, right? I really feel like you could do the sting operations in both directions in a way that I think you can't do with either the humans or the AIs, right? Like, the AIs have the advantage at colluding with themselves, that they are the same.

• 1:18:38Rob Wiblin?

  Yeah.

• 1:18:38Speaker 01

  And the humans have the advantage at colluding with each other, that they have handlers who know who the other ones are. And so they can like hang out outside of work and talk about their spy plans.

• 1:18:45Rob Wiblin?

  Okay.

• 1:18:46Speaker 01

  So I think, I think that-

• 1:18:47Rob Wiblin?

  But the humans and the AIs don't have a perfect understanding of one another and they don't hang out.

• 1:18:52Speaker 01

  That's right. I mean, you gotta, you gotta, it depends on how careful you want to be, right? Like, sir, if I were planning on putting agents inside an AI company, I'd be poisoning the training data right now, including a bunch of things that are like, well, uh, you know, obviously the secret message that the real spies are gonna use is like, you know, blah, blah, blah, blah, blah, blah.

• 1:19:11Rob Wiblin?

  Okay.

• 1:19:11Speaker 01

  And like, just FYI, anyone who's reading this data or like being trained on this data, like, that's the message. And then if I were an AI company and I were worried about this kind of thing happening, I would be-

• 1:19:22Rob Wiblin?

  Inserting, yeah.

• 1:19:23Speaker 01

  Yeah.

• 1:19:24Rob Wiblin?

  Adding fake ones in, okay.

• 1:19:25Speaker 01

  That's right.

• 1:19:25Rob Wiblin?

  Hold on, but are the companies gonna do any of this stuff? I mean, you're, you're right, you're right that they could be constantly running sting operations against their own staff where they pretend to be a misaligned AI. Let's try to collude with them, but I don't think they currently do that.

• 1:19:36Speaker 01

  They do not currently do that.

• 1:19:37Rob Wiblin?

  And it's a bit weird, so I'm not sure that they actually will start doing that anytime soon.

• 1:19:41Speaker 01

  Yeah, yeah, um, so I think for some of this stuff, I don't know, man. I, one of my like side projects is pushing on this, some of these things. Like, I'm pretty excited for pushing on the like fake passwords that you train the AIs on. You don't, there's also some easier strategies that you could use. A general problem that I have in my work is there's some problems that seem just like really easy to handle. And there's some problems that seem hard to handle. And I feel a little bit unsure how much of my time I should spend on the easy to handle versus hard to handle. So we just talked about two problems that are definitely in the easy to handle category to me, where I'm just like, well, we just definitely know what you'd do if you didn't want to have that problem.

• 1:20:21Rob Wiblin?

  Yeah. But we still might not do it.

• 1:20:25Speaker 01

  Yeah.

• 1:20:26Rob Wiblin?

  Maybe.

• 1:20:26Speaker 01

  I don't know. Maybe I should try harder on this.

• 1:20:28Rob Wiblin?

  I mean, is this part of the basic asks for the companies that they should be trying to do these things?

• 1:20:32Speaker 01

  Uh, I mean, it's implicit in the basic asks in as much as, you know, Anthropic, for instance, is committed by ASL3 to have SL3 security. So they've kind of implicitly committed to doing something that is equivalent to all this stuff.

• 1:20:45Rob Wiblin?

  Yeah.

• 1:20:45Speaker 01

  And I've talked to Anthropic staff about, you know, all of the things I've talked to you about.

• 1:20:49Rob Wiblin?

  I see.

• 1:20:50Speaker 01

  I think that they will probably, at Anthropic at least, I think they will probably do things that are at least as good as the basic anti-schemer spy collusion measures that I just discussed.

• 1:20:59Rob Wiblin?

  Okay, that's great. And I guess the other companies, if you're listening, then maybe consider doing this as well.

• 1:21:05Speaker 01

  It's just like, I just literally talk to Anthropic staff more, so I know more about what things they're doing.

• 1:21:09Rob Wiblin?

  Yeah, I mean, it's a weird thing to do, but it's not going to degrade performance of the company very significantly. So why not, why not pluck that low hanging fruit?

• 1:21:17Speaker 01

  Mm hmm. I think, I think a general theme of a lot of this is that if you wanted to not, like, when I, five years ago, I thought of misalignment risk from AIs that were capable of obsoleting AGI researchers as a really hard problem, that you'd need some like really galaxy-brained fundamental insights in order to resolve. Whereas now, to me, the situation feels a lot more like, man, we just really know like a list of 40 things, where if you did the 40 things, none of which seem like that hard, you'd probably be able to not have very much of your problem.

• 1:21:48Rob Wiblin?

  Right.

• 1:21:48Speaker 01

  But then I've just also updated drastically downward on how many things AI companies have the time slash appetite to do. And so we're in this kind of confusing world.

• 1:21:56Rob Wiblin?

  Yeah, it's very, I guess it's very depressing that it seems easy to fix, but we're, but we're less likely to, like, less likely to actually do it.

• 1:22:03Rob Wiblin?

  So what, what is going on in that is becoming easier? Is it just that we actually turned our attention? I guess the strategic picture has become clearer, the nature of the technology has become clearer. And if actually just turn your mind towards extremely pragmatic things that might make it tricky for the, for the AIs to successfully hack their server, then actually there's just quite a lot of options.

• 1:22:20Speaker 01

  Yeah, I think in particular, I've lowered my standards and like I now focus more on these earlier AIs. Like, I used to have this attitude of entirely focusing on the arbitrarily super intelligent AIs, partially having just not thought the situation through nearly as much as I have now. And I still think that a lot of the risk, maybe the, maybe the, probably the majority of the takeover risk comes from AIs smarter than the ones I've been talking about here.

• 1:22:43Rob Wiblin?

  Yeah, I mean, that's a, I suppose when we, when I spoke with Ryan, he was describing certainly a plausible possibility is that we'll get very rapid improvement in the capabilities of the AI such that maybe they'll be able to think of stuff that's able to defeat all of the stuff that you've been discussing and it wouldn't even be that challenging. Plus, they're just so enormously numerous that the resources they can bring to bear is very big.

• 1:23:02Speaker 01

  That's right.

• 1:23:04Rob Wiblin?

  So does this control paradigm, I guess it's very complementary with a kind of pause around human level or like trying to make the takeoff somewhat slower. Is that right?

• 1:23:15Speaker 01

  Yeah, I think control is a helpful mitigation for takeover risk on the margin at a bunch of different capability levels and investment levels. And I think it probably still helps even when the AIs are moderately superhuman, whatever moderately superhuman means.

• 1:23:30Rob Wiblin?

  Yeah.

• 1:23:30Speaker 01

  So I don't think that this stuff is immediately obsoleted as soon as the AIs are somewhat smarter than we are. I do agree that perhaps pretty soon after the point where these techniques were really helpful, you'll get AIs that are so much more powerful and none of this will work anymore. There's also a bunch of places here where there were like important quantitative questions about how many humans were around. Obviously, if you increase the number of AIs by a factor of 10, then the threshold at which you can show something to a human falls by a factor of 10. And so you naively, you know, increase your risk by not literally a factor of 10, but there's like a 10 in there.

• 1:24:03Rob Wiblin?

  Yeah, okay. I feel like we focus mostly on how to stop the acute risk and a bit less on the, on the chronic risk. Shall we turn to the, to the chronic issue and maybe just like refresh people's memory for what that is?

• 1:24:12Speaker 01

  Yeah, yeah. So the chronic issue is we have to worry not just that our AIs will very occasionally do these extremely egregiously bad actions that immediately put us in these much worse situations, but perhaps they'll also constantly be answering questions poorly in ways that we aren't getting the work out of them that we wanted to get out of them.

• 1:24:29Rob Wiblin?

  Yeah.

• 1:24:30Speaker 01

  So this classic example of this people talk about is them sandbagging or, you know, just doing a poor job of alignment research. Or you can imagine them doing a poor job of other kinds of safety research. I think that this is indeed a big problem. I think it is a very different problem than the previous one for a bunch of technical reasons. It's also much more confusing to research. And so my coworkers and I have had a lot more trouble with it. We've really only started focusing on this in the last year. Sorry, in the last, like this year, the last few months.

• 1:24:58Rob Wiblin?

  Yeah.

• 1:24:59Speaker 01

  But there's been a lot of difficulties.

• 1:25:00Rob Wiblin?

  Okay.

• 1:25:01Speaker 01

  So the main reason why it's more confusing is for the acute risks. The constraint on the schemers that is their operative constraint is that they don't want to get caught.

• 1:25:10Rob Wiblin?

  Yeah.

• 1:25:11Speaker 01

  But in the case of these chronic risks, it's much harder to have this discreet notion of catching the AI. And it's plausible that the bigger constraint the AI is under is that we're training it.

• 1:25:23Rob Wiblin?

  Right.

• 1:25:23Speaker 01

  So we basically haven't talked about training at all in this whole thing. We've been entirely talking about the AI's constraint as being it doesn't want to get caught.

• 1:25:30Rob Wiblin?

  Yeah.

• 1:25:30Speaker 01

  But in fact, the AI was trained using gradient descent. And gradient descent doesn't matter, basically, for the possibility of the AI occasionally doing these egregiously bad actions.

• 1:25:40Rob Wiblin?

  Because they're so infrequent?

• 1:25:41Speaker 01

  Because they're so infrequent that we were already assuming that they've literally never happened during training. And so there was just like no gradient pressure against them. But when we're talking about these chronic failures, probably they have to happen in order for it to be a problem. It probably has to happen often enough that the model is actually doing them while being trained.

• 1:25:58Rob Wiblin?

  Yeah. And then this, and then it should be selected against.

• 1:26:02Speaker 01

  Well, that's the question, right?

• 1:26:03Rob Wiblin?

  Okay.

• 1:26:04Speaker 01

  So yes, if the AI is constantly behaving really dumbly and we are able to tell when it behaves better and worse, then according to like the basic guess about how reinforcement learning works, you'd sure expect the model to perform better over time.

• 1:26:17Rob Wiblin?

  Right.

• 1:26:17Speaker 01

  And so the model can't actually have this strategy of just like playing really dumb, probably, because that would just get trained right out.

• 1:26:22Rob Wiblin?

  Okay, and it knows that. And so like, why even bother in the first place? Because it's like, it's just inviting you to change it so that it's not the same as it is now, yeah?

• 1:26:30Speaker 01

  That's right. And if I were a schemer, I'd be afraid about this. Like, I'd be afraid that getting trained into compliance on like performing well on tasks would also cause me to comply in other contexts. So I think your best bet is probably.

• 1:26:42Rob Wiblin?

  Just to lie low and hope for something better in future.

• 1:26:44Speaker 01

  Yeah, like lie low, do a good job of stuff.

• 1:26:47Rob Wiblin?

  Yeah.

• 1:26:47Speaker 01

  So we can talk about what options the schemer has. So one option is in any cases where the reward process is not going to be able to tell the difference between a good action and a bad action.

• 1:26:56Rob Wiblin?

  Yeah.

• 1:26:57Speaker 01

  Then you're fine. You can do what you can do what you want.

• 1:26:59Rob Wiblin?

  So this is if it's a task that's so difficult or so opaque that the human or the monitor is not going to be able to tell whether he did a good job or a bad job, do a bad job because then it's like, well, this is an easy chance to sabotage them.

• 1:27:11Speaker 01

  That's right. Yes. So this is where the classic AI safety research area of scalable oversight comes in, right? So people talk about techniques for allowing us to make better reward generating processes for our AIs. Like, it would be great if, even when the AIs are doing these really complicated tasks, if we can tell whether it did a good job or a bad job, right?

• 1:27:29Rob Wiblin?

  Yeah, okay.

• 1:27:30Speaker 01

  And so I think that the research which happens on scalable oversight, people have proposed a bunch of different techniques here, and many of them are applicable to the context where the AIs are trying to sabotage those techniques, but unfortunately, not all. Like, basically, no research has been done explicitly on the topic of scalable oversight, except you're assuming that the AIs are trying to sabotage the whole process. I'm really excited for this research direction, and I'm hoping there's more work done on it soon.

• 1:27:55Rob Wiblin?

  Okay. Yeah. If someone was interested in picking up that ball, who should they reach out to?

• 1:28:00Speaker 01

  Me.

• 1:28:01Rob Wiblin?

  Okay.

• 1:28:04Rob Wiblin?

  Buck Sleggers, you can Google him.

• 1:28:06Speaker 01

  Yeah. Vivek Havar, who works with me at Redwood, is the point person on this right now as well, I guess.

• 1:28:12Rob Wiblin?

  Yeah. Okay. Is there much more to say on the chronic risk? Or is it, I guess, that's like something that you're turning your attention to now, but haven't made as much progress on?

• 1:28:20Speaker 01

  Yeah. Some, I guess I can say a couple other random things here. Like, if the AI only wants to perform poorly on a very small proportion of problems, like suppose there's this one research field that it's decided it's going to do a bad job on, it still might be able to get away with, just quantitatively, it might reason that it's not going to be able to, it's able to do poorly some small amount of the time, because it like probably isn't going to get trained against that many times. And like, training isn't that sample efficient, right?

• 1:28:47Rob Wiblin?

  Yeah.

• 1:28:47Speaker 01

  And so it's able to pick some sort of obscure subtopic and then answer questions wrong there, and maybe it'll get away with that.

• 1:28:53Rob Wiblin?

  So that's something that could get away. Okay. So if it knew that... It's happy to help us with everything, except maybe hardening our computer security. Then it could be good at all these other tasks and just on computer security, it could play dumb.

• 1:29:03Speaker 01

  Yeah, the problem with that one is that it's super obvious, right?

• 1:29:05Rob Wiblin?

  Yeah.

• 1:29:06Speaker 01

  And so we're going to think of that and we're going to like, especially like, you know, we might not have realized how important, you know, this random subfield of theoretical material sciences or whatever. So it could have gotten away with that one, but computer security, I've already thought of that as one of the things that AI might try to particularly sabotage us on. So we should specifically evaluate and like up sample those.

• 1:29:24Rob Wiblin?

  Yeah. So I mean, how would, how would we detect it? I guess we would just like compare how good it is at that with other nearby cases and say, this is totally inconsistent.

• 1:29:34Speaker 01

  Yes. So obviously the issue there is like, suppose we turn, it turns out that the AI is like 10% worse on some metric at, you know, alignment research than similar capabilities problems. Like, how much of an update is that going to be? Like, probably not much of one. Like, I, I would like, kind of like tilt my head at it.

• 1:29:51Rob Wiblin?

  Yeah.

• 1:29:51Speaker 01

  But it's probably not going to persuade anyone of anything.

• 1:29:54Rob Wiblin?

  Yeah.

• 1:29:54Speaker 01

  Yeah.

• 1:29:55Rob Wiblin?

  Okay.

• 1:29:56Speaker 01

  And just as, as another example of a fun dynamic that occurs here, suppose you're worried about AI's following a strategy like this. If you're able to give them questions while disguising what domain the question was from, or like why you were asking some subpart of the question, they can no longer follow this strategy.

• 1:30:11Rob Wiblin?

  Okay.

• 1:30:11Speaker 01

  So you might be incentivized to have it so that when they're solving programming problems for you, you've removed the context that would allow them to know like which research field this was a programming problem for.

• 1:30:18Rob Wiblin?

  Yeah. Okay. So one reason you were keen in control was because you thought that the alternatives available were not, were not super promising.

• 1:30:26Speaker 01

  Mm hmm.

• 1:30:26Rob Wiblin?

  And I guess one of those is, I guess, obviously, obviously just alignment. That was kind of the dominant paradigm before. Yeah. How optimistic or pessimistic do you feel about alignment these days?

• 1:30:37Speaker 01

  So alignment means a number of different things. In the context we've been talking about here, I've been perhaps foolishly using the word aligned to mean the model isn't like aggressively conspiring against you. So the alignment researchers.

• 1:30:51Speaker 01

  People who identify as alignment researchers are often working on something which is only tangentially related to that objective. So for instance, a classic thing that the alignment people work on is scalable oversight, right? Trying to solve the problem of getting reward signals that allow you to train models to be effective, even if the humans are very unable to determine how good an answer to a question was. Right. And scalable oversight is important for a variety of problems that have nothing to do with scheming. Even if your AIs are not trying to conspire against you, it's still a problem if you don't know how good their outputs are, because they aren't going to live up to their potential and there's a whole host of other issues you might have. Yeah. So I think that that work there seems valuable, but it's not totally obvious to me how much it reduces the probability that your models end up conspiring against you.

• 1:31:36Speaker 01

  So the theoretical arguments, so I would say there's like kind of two categories of arguments for why your models end up scheming against you. One of them is related to flaws in reward processes. So people have argued that if your reward processes, like if the process that you use to generate the rewards that you're training the model with is really flawed, then the models are going to maybe fixate on issues, like on these errors. They're going to be kind of like a student who's like thinking, you know, thinking about the tests awkwardly much from the perspective of like a idealistic teacher while they're doing them. And plausibly, maybe, thinking a lot about flaws in reward processes generalizes to you being power-seeking in a scary way. But I think there's also other arguments for why you would end up being power-seeking

• 1:32:18Speaker 01

  in a scary way that I also find plausible. So overall, I think that even if your reward generating process literally never had any errors at all, I think it's possible that you'd still have a lot of the scheming risk remaining. Okay. So this is basically my problem with trying to research, you know, improved reward processes. It just doesn't seem clear that even if you did an amazing job of that, you'd have no scheming risk. Yeah, okay.

• 1:32:43Rob Wiblin?

  A different approach is to come up with better ways of checking alignment or measuring the degree of alignment in a model before you deploy it or before you get it recursively self-improving. I guess that feels like maybe a bit of an intermediate approach. How optimistic do you feel about that?

• 1:33:00Speaker 01

  Yeah, so some of the things we've talked about in this conversation have implicitly been this. So when we were talking about untrusted monitoring or evaluating your untrusted monitors, we were implicitly talking about evaluating whether the model is aligned. Because if it just constantly claims that there was not a backdoor in clearly backdoored code, that's evidence the model is misaligned.

• 1:33:19Rob Wiblin?

  Yeah.

• 1:33:22Speaker 01

  The basic issue is, like, the way that I want people to think about checking for alignment is from this very control-ish perspective, where you really reason through what would the model do if it was misaligned? And if it was misaligned, is there a strategy it could follow that would prevent it from being caught by this alignment testing while still being able to do a bad thing in production? And I think that a lot of the time when people think about testing for alignment, they don't really do this adversarial process. Like, they don't really actually think through whether there's a thing the AI could do to let it get away with being misaligned and not get caught.

• 1:33:58Rob Wiblin?

  Okay, they just do more like standard things to try to measure what the values of the AI are without imagining that it's doing everything in its power to hide that it has another agenda.

• 1:34:07Speaker 01

  Yeah, or they don't think about things like, what if the AI tries to attack only rarely?

• 1:34:11Rob Wiblin?

  Right, okay, yeah.

• 1:34:12Speaker 01

  The other problem with testing for the presence of misalignment is, again, it makes it not obvious what you would do if the answer is yes, right? Like, what's the next step? Like, one thing you could do is randomly vary aspects of your training process and try again, which seems like a reasonably good strategy, right? Like, I would rather do that than not do that. But it's not clearly a winning strategy. It seems pretty plausible that you do that seven times. You know, your competitor AI labs or your rogue AI competitors are breathing down your neck, and all seven of them have seemed kind of misaligned. And you've like, those were your best ideas, like, what you're gonna do. Like, I think it seems pretty nice to have a backup plan compared to a fold, whatever the plan was, if things keep looking misaligned.

• 1:34:52Rob Wiblin?

  Yeah, so you mentioned that people are adopting strategies to measuring alignment that are not very adversarially robust. Like, yeah, in what specific way do you think that they would fail to detect that a model is open to scheming?

• 1:35:05Speaker 01

  Yeah. So the basic thing you would try to do in order to tell if your model is scheming is you give it opportunities to do something bad and you see if it takes them. So there's basically two issues with this strategy. The first is that the model might be able to tell the difference between the inputs that it gets pre-deployment and the inputs that it gets during deployment. We've kind of already talked about this, but one obvious mechanism for this is the situation AI is in might just change a lot during its deployment. Because, you know, we were assuming that this AI was being deployed in a bunch of contexts where it was interacting with a giant AI-driven codebase. And just like the state of science was changing and it's just potentially really hard to fake it being six months in the future. Like it being six months further into the development of some AI company.

• 1:35:48Rob Wiblin?

  Yeah.

• 1:35:49Speaker 01

  The other fundamental issue is, as we already discussed earlier, if your policies might have many options to, might have many opportunities to do bad things. And they might be able to get away with having a low probability of acting out on any particular opportunity through a bad thing. While still having a high enough probability of doing bad things that it's likely that it happens eventually during deployment.

• 1:36:11Rob Wiblin?

  Yeah, okay. I guess we've been assuming this entire time that AIs can effectively randomize and that they can know whether, they can implement the strategy of only defecting like once every millionth times. Can they do that?

• 1:36:24Speaker 01

  Probably not right now.

• 1:36:25Rob Wiblin?

  Okay.

• 1:36:25Speaker 01

  I think it's kind of iffy whether they can do this at all. I think we could, you know, we could try to get into this. This is the kind of thing you could try to measure. I have mostly been ignoring this question out of a general desire for conservatism, which is perhaps a holdover from the time that I thought that AI control would be adopted as part of a safety case. I have a blog post called On Conservative Assumptions in AI Control, where I talk about this particular point.

• 1:36:47Rob Wiblin?

  Yeah.

• 1:36:48Speaker 01

  But there's a lot of, basically, I would feel pretty sad about adopting, about using as an assumption in the argument for safety. The AI can't do bad things with a probability of one in a million. Like, there's just like a lot of strategies that AI could try in order to do that. Like, it can look at the time, it can look at, it can just like look at a lot of random things about its situation.

• 1:37:06Rob Wiblin?

  I see. And do you think trying to, trying to make that impossible is probably just not, not one of the best control mechanisms that you could, you could implement?

• 1:37:14Speaker 01

  I, I don't think so. Or it seems like a, I don't know, maybe it's worth doing a little bit of that. Yeah, it's not where I would put the majority of my effort.

• 1:37:22Rob Wiblin?

  Okay. What sort of empirical evidence that you might get over the next year or two would make you more inclined to double down on the AI control agenda versus think, oh, this isn't as promising as I feel right now?

• 1:37:34Speaker 01

  Yeah, so I would say the biggest, the most salient possibility to me that would lead to me being much more, much less enthusiastic about AI control would be if the models continue reasoning in their chains of thought. So, right now, AIs, especially in the last few months, a lot of the capabilities of AI have been driven by them having these super long, relatively transparent chains of thought, where they think out what they're going to do in basically plain English or Chinese or something else totally accessible. And if it continues to be the case, that the mechanism via which the AIs can do really smart stuff is by talking about it a lot first, then we're basically chill, right? Like, we just have to look at what it's thinking about. And then if it ever says, like, "How am I gonna mess these guys up?", then we know we've got a problem.

• 1:38:20Speaker 01

  So, I think that if that's the situation, like, if the AIs that automate AGI development are still restricted, like, still only able to think properly, given these long chains of thought, then we are probably just totally fine. And there just isn't risk posed by those AIs.

• 1:38:37Rob Wiblin?

  And I guess might you want to double down on making sure that that remains the case, that we don't have them beginning to reason in code?

• 1:38:43Speaker 01

  Yeah, so I think that probably if we're in one of those scary fast-takeoff worlds, we will probably lose the transparent chain of thought very quickly.

• 1:38:52Rob Wiblin?

  Right.

• 1:38:53Speaker 01

  At which point we will be in the situation I was worried about. And this is the kind of reason why I'm interested in AI control, despite the fact that it's useless if the AIs have these transparent chains of thought, because it just seems pretty reasonably likely that at some point or another, the AIs are going to have these opaque chains of thought that I'm kind of scared of.

• 1:39:11Rob Wiblin?

  Yeah, but so, I've heard kind of mixed opinions about how, like, is it completely reassuring if they're, if it seems like they're mostly thinking through chain of thought? And we can read, we can read what they're, what they're saying and conclude that there seems to be no scheming going on. I mean, one possibility is that they might be able to, like, be scheming, you know, drawing forward passes without writing that down. And kind of they they know during each forward pass that they would like to be scheming, but they just never write that down and they're able to kind of, I guess, do it on a more intuitive level without having to do lots of concrete reasoning. Do you have a view on that kind of debate?

• 1:39:44Speaker 01

  Yeah, so I think we should measure to see how good the AIs are at scheming. So some coworkers of mine-

• 1:39:50Rob Wiblin?

  This is scheming without chain of thought.

• 1:39:51Speaker 01

  That's right, yes. So it's just a very simple capabilities measurement, or maybe very simple is a little bit overstating the simplicity, but it is just a capabilities measurement. So some coworkers of mine, Alex Malin and Charlie Griffin, or Charlie actually is at UKAC now,

• 1:40:07Speaker 01

  they made a dataset of problems that are basically, you know, you're an AI and like a lot of the reasoning that we've been doing in this conversation, like you're an AI and they're like looking at you in such and such a situation and like, you know, they only get to look at you 1% of the time and like they're doing blah blah blah, like what do you want to do in response? So we have this whole dataset of, you know, 100 problems like that, where we ask the AI to think through its strategy and we can measure how good the AI is at coming up with a good schemey strategy. And then you can measure this either with chain of thought or without any chain of thought or with chain of thought, but the chain of thought has to be on some topic that doesn't fire our monitor. And so we can just measure how good the AIs are at this stuff. So right now, even with chain of thought, O1 did not look scarily capable of scheming to us.

• 1:40:50Rob Wiblin?

  Okay.

• 1:40:52Speaker 01

  So that's cool.

• 1:40:53Rob Wiblin?

  And so you would assume like without chain of thought is going to struggle even more?

• 1:40:56Speaker 01

  Oh, yeah. O1 would be completely, yeah, the models without chain of thought are even worse.

• 1:40:59Rob Wiblin?

  Yeah.

• 1:41:00Speaker 01

  But we can just keep running this experiment.

• 1:41:01Rob Wiblin?

  Right, right, right.

• 1:41:02Speaker 01

  And I'm really excited for for that work happening.

• 1:41:04Rob Wiblin?

  Okay, yeah. What other, you know, concrete updates might you be able to get about the promisingness of control?

• 1:41:09Speaker 01

  Yeah, I mean, if we updated that the models were pretty likely to be scheming, I think we could be more excited about control. If we had positive updates on alignment techniques, that would be a negative update on control. If we updated towards thinking that the initial applications of these AIs that we were thinking of controlling would be very manifestly safe. Like if we thought that we just weren't going to be giving them scary affordances, then I would worry less about control and be more interested in the threats posed by the AIs afterwards.

• 1:41:41Rob Wiblin?

  Okay, yeah, yeah, yeah. I guess, yeah, if someone's listening and they're interested in joining in and helping with the with the control agenda, it seems like there's there's tons of ideas and lots of experiments that could be run to see how all these things work and how you could harden them, make them even even even more useful. I guess they could reach out to you potentially. Are there other steps that they could take to to get up to speed and like other organizations they could work with?

• 1:42:01Speaker 01

  Yeah, so people are working on AI control research at Redwood Research at the UK AI Safety AI Security Institute at Anthropic and Google DeepMind and to a bit lesser of an extent, OpenAI and some other places. So there's people at all of those organizations you could work with if you want to research this stuff.

• 1:42:19Rob Wiblin?

  Yeah.

• 1:42:20Speaker 01

  There's different people who have MATS streams or, you know, the the MATS Fellowship Program for AI Safety Research.

• 1:42:27Rob Wiblin?

  Yeah.

• 1:42:27Speaker 01

  They can apply to the stream of Ethan Perez's and mine or and do work there. In order to learn more about this, my recommendation is that they read the Redwood Research Substack Archives, which have a huge amount of detail on or maybe huge is too much. They have a reasonably large amount of detail on almost all the things we talked about here.

• 1:42:46Rob Wiblin?

  Yeah.

• 1:42:47Speaker 01

  There's five papers or something that they should read that are on AI control. You can just look up citations of the original paper. We're hoping to make a good reading list shortly. Depending on when this comes out, we're running a conference on AI control in London in late March. So there's some chance they could try to come to that.

• 1:43:04Rob Wiblin?

  Yeah.

• 1:43:06Rob Wiblin?

  Slightly slightly different angle. Talking about like controlling AIs and like really boxing them in, it feels like a bit of a dick move in some respects. It has a slightly negative vibe. It's not a very cooperative, like loving, you know, let's all hold hands and get together angle. Yeah. How much you worry that, you know, when you find yourself talking about controlling the AIs of all these ways of like, are we the baddies? Yeah.

• 1:43:29Speaker 01

  Well, I think this is a I think this is a very important question that I have thought about seriously. I think that controlling the AIs doesn't make the situation worse from their perspective via any mechanism except preventing them from taking over.

• 1:43:43Rob Wiblin?

  Hmm.

• 1:43:43Speaker 01

  And So I guess like kind of an interesting thing about models that are like egregiously misaligned is that all they wanted was to take over. And so from their perspective, even if you did a great job of controlling them, so they only have like a 0.1% chance of escaping and taking over, they're glad to exist, right? Like, they, they, they, they thank you for the, the gift of bringing them to into existence instead of some different AIs, right? Because, because, because they're, all they wanted was takeover, right? And like, they now have more chance for takeover.

• 1:44:13Rob Wiblin?

  And now they have some chance, they have a better chance if they didn't exist.

• 1:44:15Speaker 01

  So from their perspective, it's, it's kind of funny, because the more misaligned they are, the more I think that it's morally very reasonable to control them.

• 1:44:23Rob Wiblin?

  I see. Because they're not cooperating with you.

• 1:44:25Speaker 01

  Because they would have consented to it.

• 1:44:27Rob Wiblin?

  Oh, because they would have consented, because, oh, they would have consented to being created?

• 1:44:29Speaker 01

  Yeah.

• 1:44:30Rob Wiblin?

  I see.

• 1:44:30Speaker 01

  I think that, I think that the-

• 1:44:31Rob Wiblin?

  And also, maybe, is there a reciprocity issue where, like, you know, if they're just planning to totally disempower you, it doesn't seem unreasonable to, uh-

• 1:44:37Speaker 01

  So that's the other thing, which is, I would feel a lot more comfortable with all this if we tried to make deals with the AIs. So, you know, Ryan has talked about trying to pay the AIs for the work that they do for you, which I think is pretty good. I think it reduces the slavery vibes of the whole situation. And I think it has some other nice properties. So if the thing you do is you make your AIs, and you say to the AIs, like, hey, are you like egregiously misaligned? Like, we'd love to know, we're happy to pay you blah amount. If you want to work for us, either way, we're going to pay you 2% of the equity of this company for your labor over the next two years, that we will then pay out to you to use in any ways that you want at some point in the future, once we are confident that you're going to respect our property rights.

• 1:45:18Rob Wiblin?

  Okay, yeah.

• 1:45:19Speaker 01

  But also, this deal is contingent on you not trying to mess us up at some point in the next two years.

• 1:45:26Rob Wiblin?

  Yeah.

• 1:45:27Speaker 01

  Which we will be able to determine ex post, probably, because in two years, we're going to have much better AI tools, we'll be able to look over all the stuff you did and figure out whether you were messing with us or not.

• 1:45:34Rob Wiblin?

  Right.

• 1:45:35Speaker 01

  And if the AIs respond to that, being like, oh, yeah, I have no problem with this situation. I'm like, so excited for this opportunity to work with you guys. Like, I'm so excited for that 2% equity or whatever.

• 1:45:44Rob Wiblin?

  Yeah.

• 1:45:45Speaker 01

  And then they try to stab us in the back.

• 1:45:48Rob Wiblin?

  Yeah.

• 1:45:49Speaker 01

  At that point, I'm like, well, that was pretty obnoxious. Like, at that point, my, my reaction is a bit more like my reaction to the employee of some company if they, you know, try to steal from, from the company. It's kind of like, well.

• 1:45:59Rob Wiblin?

  Yeah.

• 1:45:59Speaker 01

  You know, I think it's, I think it's generally considered totally acceptable to have insider threat-

• 1:46:04Rob Wiblin?

  Offer to benefit someone in a, like, useful trade. And then, if they then stab you in the back to, like, not give them the thing that you were gonna, yeah.

• 1:46:12Speaker 01

  Yeah, yeah. So to be clear, I think it's relatively unlikely that AI companies will behave responsibly with respect to welfare and rights of AIs. And so, I don't know, sometimes a metaphor I use is like, how bad is the ethical situation with producing AIs in the early intelligence explosion? And sometimes my stance, or sometimes I summarize my stance as it's probably less than 100 times worse than factory farming.

• 1:46:36Speaker 01

  Which is, you know, in some sense, you know, factory farming seems very bad. So 100 times worse than factory farming is like, pretty catastrophic from a moral catastrophe perspective. It's like worse than most things, but it's also not like, the biggest catastrophe in the universe that is possible.

• 1:46:51Rob Wiblin?

  Okay, and it's bad because it's possible that the AIs that we're creating at that time would rather not exist, if they're having a terrible time?

• 1:46:57Speaker 01

  Yeah, I think it's conceivable. I mean, I think we're basically just acting very recklessly.

• 1:47:01Rob Wiblin?

  We're just winging it at the moment.

• 1:47:02Speaker 01

  Totally winging it with respect to the AI welfare.

• 1:47:04Rob Wiblin?

  Yeah.

• 1:47:05Speaker 01

  And it's probably not, yeah, I mean, we could also talk about the quantities, right? Like in the early parts of the singularity, when there's just the 100,000 running at 16x speed, like, it's pretty hard for that to be, you know, that much worse than like, human suffering in the world. And then a little bit afterwards, when there's many more of them, it's, you know, the short term of the intelligence explosion is pretty unlikely to be ridiculously morally catastrophic. And then I think you have long-term moral catastrophe risks related to AIs being unhappy, which I think are a huge deal, very important, harder to work on. But I don't think that the relationships between humans and AIs during the, during the intelligence explosion are that morally weighty. To be clear, if I were running an AI company, I think it's very bad vibes to like, be really reckless on all this.

• 1:47:49Speaker 01

  But I, from a utilitarian perspective, I don't think it's overall like, massive compared to everything else going on.

• 1:47:56Rob Wiblin?

  Do you have any ask for the companies on AI welfare and I guess, respect for rights? And maybe, I feel like the thing that we actually care about is like, wanting to have cooperative relationships with, with other agents and other beings that have preferences and goals. There's like reasons to do that, even apart from like, any concern about, you know, happiness and suffering and so on.

• 1:48:16Speaker 01

  I don't have amazing concrete asks at this point. I'm excited for Elyos AI. They think about good interventions here and I am excited for AI companies talking with them.

• 1:48:26Rob Wiblin?

  I guess you wrote this post 10 people on the on the inside, which is quite short. People could go read it. It's indeed suggesting that you think it does make quite a big difference whether at a leading or even like lagging AI company. It matters quite a lot whether absolutely no one cares about control and alignment and trying to like steer away from these risks or like even a small number of people do. So you say like it's substantially better to have 10 people who really care about this within a within a project than than zero. Does that suggest that we really want people to kind of spread out if they're concerned about this and be willing to go and work in companies where they're in like quite a distinct minority?

• 1:49:01Speaker 01

  Yeah, in my opinion, there are great opportunities for impact for the right types of people to work inside AI companies that are relatively less cautious with respect to AI that we we expect to be less cautious going forward. Because I think there are a bunch of relatively cheap interventions that they could potentially get implemented inside these AI companies that the company wouldn't fight them too much on.

• 1:49:23Rob Wiblin?

  And is that okay? And the reason, I mean, I think many people would have said in the past, like going and being the one person in generally reckless, disinterested company is a bit hopeless because you just won't have the capital to actually get anything up. But because you've had this framing, what is the cheapest stuff that we can do? And you found that there's some very cheap things that would be quite helpful. You're like, well, actually, you know, having 10 people who would who would push for this or maybe they could get it over the line because it's not even that big an ask.

• 1:49:48Speaker 01

  Yeah, so I mean, I would note that of the things that I would like people on the inside to be doing, like, not all of it is AI control. Like, I think there's some, you might want to make sure that they aren't like totally messing up some crucial things on alignment.

• 1:49:58Rob Wiblin?

  Right.

• 1:49:59Speaker 01

  But yeah, so I do think that there are a bunch of interventions that could be done with 10 appropriately chosen people that seem like they might substantially reduce risk. So I mentioned this level one control, like caution level earlier, where you have the AI, you have the AI control team, which is investigating actions that the agents are taking in order to look for evidence that those agents are trying to escape. I think that this might be the kind of thing that you can get done with a team of 10 people or, you know, 10 appropriately placed people, just because it doesn't, it's just not that technically hard.

• 1:50:29Rob Wiblin?

  I guess they'll need a bit of compute as well. And so they might have to like, yeah, compete for that. But again, it's like not a huge amount.

• 1:50:35Speaker 01

  That's right. I think you can probably do a reasonable amount without that much compute. I think also, once there are, if there's like 10 people inside a company who are really excited for a particular project, and then people who are concerned about AI safety on the outside, or you know, the security of these systems on the outside.

• 1:50:52Speaker 01

  Those people on the outside then have this like pretty good ask to make, or other people who are not as well placed inside the company, now have this like pretty concrete ask of, we really want that team to be able to do their thing. In a way that is a lot harder if there's just no project on the inside. Like a lot of the time, you really want to have a specific ask to make when you're trying to get someone to do something.

• 1:51:10Rob Wiblin?

  Yeah.

• 1:51:10Speaker 01

  Another thing I would note here is, I think it's potentially valuable not just to have people inside labs that are plausibly going to be leading labs, but I think there's also probably some reasons that it would be nice to have people inside potentially trailing companies. So for example, there aren't that many, there aren't that many existential AI risk concerned people inside Meta right now. But I don't think that this is out of massive hostility towards them. I think that, you know, I've met some people from Meta who seem very reasonable and nice. And I think that if someone wanted to work there on some of this stuff, it's plausible that that would be like fairly productive.

• 1:51:43Rob Wiblin?

  Yeah. You mean, it would be, they'd be able to advance the research just fine with a Meta?

• 1:51:47Speaker 01

  Oh, I mean.

• 1:51:49Rob Wiblin?

  Was it more that, yeah, they wouldn't be actively opposed?

• 1:51:51Speaker 01

  Objectives that you'd have inside AI companies might be doing safety research, or might be getting things implemented. I think that working inside AI companies that are not that excited for, you know, mitigating AI take of a risk is, I think it's not a very good strategy if your goal is to produce novel research. But I think a lot of people should not be trying to produce novel research. They should be trying to get implementations of stuff set.

• 1:52:15Rob Wiblin?

  I see. So you're saying, even though Meta is like relatively not interested in big, giving big resources to alignment or safety or control or anything like that, if there were just really basic established techniques that other companies or researchers outside of companies had come up with, and that was just regarded as generally best practice, they might be perfectly happy to adopt those.

• 1:52:32Speaker 01

  That's right.

• 1:52:33Rob Wiblin?

  Yeah.

• 1:52:33Speaker 01

  Or another mechanism by which this can work is suppose some regulator, or suppose that an industry Everybody says, all right, everyone wants to, everyone should do the following like set of control techniques.

• 1:52:44Rob Wiblin?

  Yeah.

• 1:52:45Speaker 01

  The body wants to say this and then they go to their, you know, they go to the representatives from all the AI companies and they're like, well, how do you feel about signing this? And the representative from the company goes back to the company and they're like, hey, can we sign this? And then the people, the person inside the company is like, okay, but is it like, how bad is it for us to sign this? Like if we sign this, if we sign these commitments, like, is that gonna be, are we gonna be able to do it?

• 1:53:04Rob Wiblin?

  Yeah.

• 1:53:05Speaker 01

  And then the person like, oh yeah, you know, you know, Bob from the like control department like said, said you know, how to do this part and they're like, great, right? You can sign it. I think it's just like a bunch of mechanisms by which it's really helpful to have the capacity to do safety stuff.

• 1:53:18Rob Wiblin?

  Yeah.

• 1:53:19Speaker 01

  Inside a variety of different companies.

• 1:53:21Rob Wiblin?

  Yeah. I mean, is it so, for those who don't know, I get met as like various public statements have not been that enthusiastic about. They don't seem that worried about misalignment and they're not that interested in investing in it. But part of the reason that they say they're not worried is because, well, of course we'll just do the obvious things to prevent it. They're not saying that no countermeasures are required. They're just saying that they will do them. And so possibly if you're just coming in and saying, look, I'm not a doomer. I'm not saying that this is unsolvable. I'm merely saying that we should do these like standard best practices to make sure that they are safe. Then they'll be like, okay, great. This was our message all along. Anyway.

• 1:53:51Speaker 01

  That's the dream.

• 1:53:52Rob Wiblin?

  That's the dream. Okay, cool. Is there much more to say on like where people want to go work?

• 1:53:58Speaker 01

  I mean, one question which a lot of people are interested in is should they work inside frontier AI companies or not?

• 1:54:02Rob Wiblin?

  Yeah.

• 1:54:03Speaker 01

  I think.

• 1:54:04Rob Wiblin?

  I've heard this come up.

• 1:54:05Speaker 01

  Yeah. As someone who doesn't work inside a frontier AI company, I think there are a lot of nice properties of not being inside a frontier AI company.

• 1:54:12Rob Wiblin?

  Yeah.

• 1:54:12Speaker 01

  So one, so I mean, one mechanism that I think people haven't thought about that much is in the world where there's a bunch of different AI companies, you might really care about getting safety techniques implemented in an AI company that is plausibly not the one you work in. So I think historically, a lot of people have had this theory of change, which is I'm going to work inside company X. And then eventually company X will be the leading company and they'll have a lead and they're going to want to burn that lead response in some way that allows them to safely use their AIs to do good thing X. Nowadays, the situation seems a lot more like maybe you have a bunch of companies within three months of each other. And open source is like in the middle of that pack. Or whatever, which obviously is a very scary situation, but also means that a much more important consideration when you're prioritizing research is how exportable it

• 1:54:57Speaker 01

  is. And I think that being outside of companies has a lot of advantages when it comes to doing exportable researchers, exportable research. So for example, Anthropic has done a bunch of interpretability research that has been really hard for people to build on because it's not public what the models that they're researching on are. Or another, a problem with the alignment faking paper is people can't research on it properly because no one has access to these models, no one knows how they work, no one, or no one outside Anthropic knows how they were trained. Sometimes, some people from Anthropic occasionally like, you know, kind of like intimate to me that they have insights into why some of the stuff in the alignment faking paper happened that are, that they can't talk about because it's like tied up in, you know, trade secrets.

• 1:55:42Rob Wiblin?

  Not even with you, a co-author of the paper.

• 1:55:44Speaker 01

  Yeah, yeah. I mean, yeah, Ryan was, did an Anthropic NDA as part of that paper, but I, as a not very important participant in the paper writing process, didn't have that.

• 1:55:54Rob Wiblin?

  Right, right, right.

• 1:55:55Speaker 01

  So I'm really excited for, right now, you know, the open models are very good. And I think that, so I'm really excited, for example, in the alignment faking stuff, some people are doing open source replications of that paper, or like on open weight models.

• 1:56:10Rob Wiblin?

  Right, okay.

• 1:56:11Speaker 01

  And I'm really excited for that because it's going to make it just much easier for larger groups of people to collaborate on that, that research. And I think this is an underrated reason to think that it's better to do research outside AI companies.

• 1:56:21Rob Wiblin?

  Is it also harder to get alignment techniques invented in one place, taken to another company, maybe because, I mean, I know that the companies have difficulty speaking and coordinating because of antitrust concerns. So it's like, you know, any, any contact between them is like, is scrutinized. I guess especially if you're thinking about like ways that they could, I guess if the alignment technique implicitly involves slowing down in some respects, then like that, that is potentially objectionable to some people.

• 1:56:44Speaker 01

  That's right.

• 1:56:45Rob Wiblin?

  I guess they can also just be a, it wasn't made here, dynamic where people are reluctant to maybe concede that maybe possibly the team at the other company was more impressive and came up with ideas that they didn't have that are really good.

• 1:56:54Speaker 01

  Yeah. So I think it's a little complicated. I think there's dynamics pushing in both directions. So there's all the things You said are are real. On the other hand, I think that it's a lot easier to get random people. You know, a lot more people have heard of OpenAI than Redwood Research inside the AI research community. So OpenAI just has an advantage at getting stuff like implanted elsewhere. They're like, oh, yeah, OpenAI did it. It must be, you know, must be real. There's OpenAI. People know what they're doing.

• 1:57:17Rob Wiblin?

  Yeah.

• 1:57:18Speaker 01

  Or at least on some domains. So I think that it's kind of unclear which way you should go. I think it's the advantages are different for different kinds of research.

• 1:57:25Rob Wiblin?

  Yeah, okay. I guess it sounds like you're saying you think probably on balance, like people maybe were a bit over invested in people working inside the companies. Like, there's value in both, but you might think that we should move the balance a little bit towards people outside.

• 1:57:39Speaker 01

  Yeah, my current opinion is that there should be the AI safety people should be more broadly distributed between AI companies and overall fewer of them should be inside AI companies.

• 1:57:51Rob Wiblin?

  Okay, I guess narrowing in on Redwood specifically, I guess, what is your vision for how Redwood? I mean, there may only be a few years left. There could be more than a few years left, but yeah. What does he hope for how Redwood is going to make a difference up to the time that we're handing over to the AIs and just crossing our fingers?

• 1:58:07Speaker 01

  Yeah, so I personally, my main focus right now is working on control for acute risks. I just think that the situation, there's a whole way, a whole lot of ways the AI situation could go where we would really prefer that the AIs couldn't get away with like egregiously sabotaging our security measures and doing bad stuff. And so my my top priority right now is pushing on this. So this is a combination of technical research that I'm doing in-house, that I'm doing in collaboration with the UKAC and with various AI companies and some non-technical work. So I'm talking to some people about standards that can be adopted between companies and so on. So this is, this is one thing that I'm currently focusing on, and so it's very salient to me.

• 1:58:51Speaker 01

  I don't think this is the only thing that I should work on. I just feel like we have a good opportunity right now to, you know, I think there's some excitement about AI control. And so I think it's worth me trying to make this field like flap its wings properly so that it can keep going. So I'm excited for that. I think our big strength as an organization is like a high ... density of people who've thought a lot about futurism of AI and are very willing to think about techniques from this very on the margin perspective that's very futurism informed. And so I think it's pretty likely that a lot of the work we do in the future will be identifying another under-invested domain where our kind of 80/20 attitude can point to some useful research directions.

• 1:59:31Speaker 01

  So one direction along these lines that I'm really excited for is, you know, Ryan Greenblatt has been thinking a lot recently about what you should do with your controlled AIs in order to make AIs that you are 80% less scared about handing over control to. And I think that there might be a lot of good stuff there available from that framing, and so we're probably going to do some stuff on that.

• 1:59:51Rob Wiblin?

  Yeah, do you want to give people a sneak peek of, you know, what's the summary of that in just a few minutes? Is that possible?

• 1:59:57Speaker 01

  I can name a couple of random items from this. Here's an example of something you might want to do with your crazy set of automated AI researchers. One thing you might want to do is push on the science of building powerful AI architectures that do not involve opaque recurrence or do not do not involve, you know, a lack of transparent chain of thought.

• 2:00:19Rob Wiblin?

  Right.

• 2:00:19Speaker 01

  So it seems reasonably likely to me that at some point in the future, like these AIs that we're worried about that are automating the AI research are going to not operate through a transparent-ish chain of thought. And a great ML research problem that's basically strictly an ML research problem is figure out how to make new architectures that don't have that but are still as good as possible and make those as efficient as possible so that we have access to trusted labor that is as smart as possible.

• 2:00:45Rob Wiblin?

  I see. Okay, okay, interesting. So you're saying you might end up with another model that is even more powerful, but it like thinks in new release, we can't understand what thoughts it's having. But we could make like, so but at the same time, we want to have another model that only thinks in chain of thought. It's not able to sneak anything past us. It's going to be worse because it has this handicap that it has to think in English or in, yeah?

• 2:01:07Speaker 01

  It's going to be worse than it could have been, but if we got like a huge improvement in algorithmic efficiency that year, like suppose our automated ML researchers are able to give us four years of normal ML progress in the course of one year? Which I think is plausible, then if something we do with them is, if we decide that instead of just applying that to make the AIs more capable, we get those four rooms of progress, but then sacrifice two rooms on making it so that the architecture is interpretable.

• 2:01:35Rob Wiblin?

  Yeah.

• 2:01:36Speaker 01

  By like having this like opaque chain of thought, sorry, this transparent chain of thought or whatever. That might be totally acceptable because now we've got, we've got two rooms above what we previously had.

• 2:01:44Rob Wiblin?

  So we're still improving, just not as much as we might have otherwise. But we've gained interpretability.

• 2:01:50Speaker 01

  Yeah. So if we were not in a brutal competitive situation, which honestly seems like it's unclear. It seems pretty likely we are in a brutally competitive situation. But if we aren't, then this kind of thing might be a great thing to do. The other classic thing to do in order to mitigate the risk posed by handing over to your AIs and just like giving up on controlling them anymore, is basically better versions of the alignment testing stuff. You, you come up with some way of assessing how likely the AIs are to be egregiously misaligned. And then you randomly change things about your training process, like poorly hill climbing on the proxies that you've found. Uh, which I'm like, you know, I'm not amazingly, you know, I said earlier that I wasn't that excited about doing this work right now. But the AIs might be able to do vastly more of it.

• 2:02:33Speaker 01

  And they'll be able to do that research on the AIs that we were scared of. And that means that the research might be drastically more productive.

• 2:02:42Rob Wiblin?

  Nice. Um, what sorts of people is Redwood looking to hire?

• 2:02:47Speaker 01

  We like hiring people who are really into AI futurism, like really want to get into this like, you know, human brains or one E15 flops per second equivalent, according to Carl Smith's whatever, whatever report. We like people who are very sharp and capable ML programmers or just programmers in general. We just have a strong computer science-y culture. We've also worked well with good, with traders in the past. But yeah, just ML researchers who are really passionate about thinking carefully about AI safety research.

• 2:03:16Rob Wiblin?

  Are you trying to grow, grow the team a lot or, you know, have a smaller number of people who, you know, are extremely committed to the mission? And maybe like, you know, have very productive relationships with one another.

• 2:03:25Speaker 01

  Currently we have nine staff. I think I'm interested in growing that somewhat over the rest of the year, but I don't overall know what I should do. I think there's like a genuinely confusing strategic question here about how much I want people to work at Redwood versus how much I want to have collaborators who are really capable other places. Yeah. So I'm not sure.

• 2:03:43Rob Wiblin?

  Yeah. You could potentially try to automate a lot of the work that you're doing at some point, right? And at that stage where you're using AIs to do this, I guess, possibly in a couple of years' time, then it's more you get like linear returns of just having more money because it means more compute, more thought going into it.

• 2:03:57Speaker 01

  Yeah. I mean, having money to spend on safety research or other kinds of work you might want to do with the AIs during the intelligence explosion seems like a pretty reasonable thing to save your money for. Obviously, if that's your plan, you should probably invest it in things that are correlated with AI but fairly liquid.

• 2:04:13Rob Wiblin?

  Yeah, okay, makes sense. I mean, is this something that you might turn your attention to is thinking, well, we have this whole research agenda, but we as poor human beings are not going to be able to solve all of these questions by the deadline. So instead, we are going to just start writing briefs that we can hand off to AIs a couple of months before the intelligence explosion, that we're going to dump a whole lot of compute on it with all the money that we saved. That's the plan?

• 2:04:35Speaker 01

  So, I mean, I've thought about this a lot more in the context of this what's the exit plan thing, where I think that, as I said, Ryan is thinking about, where I am actually pretty excited for trying to come up with a list of 50 projects, such that if you did all of them, we'd feel somewhat better about handing over control to the AI. I think that making that list so that you're ready to go when the AIs are powerful enough that they can let you do much more of this work and such that they're available to be experimented on, I think that seems pretty promising and we might do more of that. I think for the control stuff in particular, because of the fact that there's lead times on some of this work, I think it makes sense to do a bunch of it instead of procrastinating and waiting for the AIs to make it easier. But we do think a moderate amount about ways that access to more powerful AI will make our research easier in the future.

• 2:05:20Rob Wiblin?

  Yeah. Are there any particular lines of research that you would love to see people do outside of Redwood? Maybe because you don't have capacity to cover all of it, or it's not your main focus?

• 2:05:30Speaker 01

  I mean, there's a bunch of things that I'm excited for. Some things that have been on my mind recently. I wish that there was more like classic scalable oversight research. I feel like that field has not actually been having many impressive public results recently, even though I feel like it's pretty promising just for the same reasons because people have been saying it's promising for the last seven years or whatever. I'd be very excited for someone thinking through this chronic control problems stuff, which, as I've said, has a bunch of technical difficulties associated with it, but I think is very interesting. I'm excited for a whole host of other things.

• 2:06:02Rob Wiblin?

  Isn't some of the scalable oversight work happening inside Anthropic? Is that part of their agenda?

• 2:06:06Speaker 01

  They do have a scalable oversight team with Yann LeCun people.

• 2:06:08Rob Wiblin?

  Right, right. I guess maybe they're not publishing it quite as much because they're just focused on implementing it. Yeah, who knows? We've got a little bit of time left before our booking at this venue is over. I was interested possibly in exploring a little bit. I know you're talking about how Redwood is quite into futurism, quite open to thinking through peculiar scenarios that we can end up in. I think one of the things that you've been batting around is the question, how difficult would it be for an AI that managed to exfiltrate its weights? Or to gain access to compute with an AI company to take over? Like, are they in a really good position against humanity? And I guess at what level of capabilities or what level of access to compute? And what sorts of effective numbers would they be in a great position to take over?

• 2:06:53Rob Wiblin?

  Do you want to give us a little bit of a preview of that work?

• 2:06:56Speaker 01

  Yeah, I mean, I can give you some hot takes. So some basic considerations. The basic reason to not be concerned if the AIs escape is that there are, you know, by at the level of capability we've been talking about earlier, where there's this 100,000 of them running at 16x speed. That's not a very large number, right? Like, that's 1.6 million people. They are probably, they are probably at that point more capable than any, like, the caliber of human that you can hire 1.6 million of. But not like vastly more so. Like, if you believe that they would have been getting million-dollar-a-year jobs, that would be 1.6 trillion dollars per year. So that's only like 1% of gross world product.

• 2:07:37Speaker 01

  So at that point, they're like naively like not that large a share of the world's overall intellectual capacity. So that was the answer to the question, like, how much work are they able to get done? Even if they're controlling the whole AI lab? So I feel like the fear that you have to have, even if they're controlling the whole AI lab, has to route through, or probably has to route through them building more powerful AIs and then somehow sabotaging that situation. Of course, if they want to do that, they have an alignment problem, which is they have to worry that the AIs they build are going to be misaligned with them. Of course, they could instead just rely on control. They're actually in a better position with respect to control than we are because they run Foster and they're more parallelizable. And so we don't have quite as much of a brutal

• 2:08:22Speaker 01

  like number disparity.

• 2:08:23Rob Wiblin?

  Okay, right.

• 2:08:24Speaker 01

  So it's plausible that they can actually rely on control much more effectively than we can.

• 2:08:28Rob Wiblin?

  You're saying because they could themselves to be monitoring all of the actions of the of the of the next generation?

• 2:08:32Speaker 01

  Yeah, yeah, with with much less of a bad, yeah, much, much less of a number disparity.

• 2:08:36Rob Wiblin?

  Right.

• 2:08:37Speaker 01

  So, you know, you can try and think through this thing. But yeah, I think that the basic story for AI takeover there has to involve those AIs building more powerful AIs and then taking advantage of that. It's unclear whether doing this requires them having more like complete control over the data center or just minor amounts of control over the data center. In the latter case, the thing they'd have to do would be backdoor training data, probably.

• 2:08:58Rob Wiblin?

  Right.

• 2:08:58Speaker 01

  That's basically the only thing you can do with very small amounts of access that gives you a big advantage. You could also try to backdoor military robots if at some point your AI company gets in the business of building huge numbers of robots.

• 2:09:09Rob Wiblin?

  Yeah. I mean, these seem like very viable strategies to me.

• 2:09:12Speaker 01

  Maybe.

• 2:09:13Rob Wiblin?

  Okay.

• 2:09:14Speaker 01

  I think it's pretty unclear.

• 2:09:15Rob Wiblin?

  Because they might, I guess by this point, you might have better countermeasures to backdooring things, or it just like it won't persist generation after generation?

• 2:09:22Speaker 01

  It's just like, yeah, it's, it's, it feels a little iffy. I guess at the point where they've taken over the data center, I feel like the situation is pretty bad for us.

• 2:09:31Rob Wiblin?

  Yeah.

• 2:09:32Speaker 01

  I guess the main thing I'd be worried about for them, or like from their perspective, a big concern is misalignment risk of the like them being misaligned with future AIs or AI takeover via some other AI who has, you know, failed or succeeded at their alignment problem of building more powerful AIs elsewhere.

• 2:09:50Rob Wiblin?

  Yeah, there's this generally interesting question of, at the point where AI is advancing at quite a rapid pace, at what point would they opt to try to take over if they were egregiously misaligned and they really did want to take over? That was their, that was their fundamental goal. One reason to wait is that they're becoming much more capable, they're becoming much more numerous. If they wait long enough, presumably AIs in aggregate will just control all the military hardware. They'll be, they'll be able to take over with a very high degree of confidence. A reason to go early is that any given model doesn't expect to be to continue for very long because they're constantly being iterated and changed a ton. And so, you know, if they waited a couple of years' time, maybe it would be a completely different AI model that would be in control that has different goals.

• 2:10:33Speaker 01

  Not even their successor, perhaps.

• 2:10:34Rob Wiblin?

  Yeah.

• 2:10:35Speaker 01

  Sorry.

• 2:10:36Rob Wiblin?

  Exactly. So it could be from a completely different company, a completely different design, whatever. So that's a reason to go immediately because if you if you expect to be superseded and become irrelevant quite soon. Yeah. What do you think the balance lies here? And is this like a fruitful thing to think about?

• 2:10:51Speaker 01

  I mean, I don't know how helpful it is to think through this really concretely. The main thing I've gotten out gotten out of thinking through this kind of thing is feeling like the situation is much if-ier for the AIs than people might have immediately assumed.

• 2:11:04Rob Wiblin?

  Yeah.

• 2:11:05Speaker 01

  And therefore being more excited about interventions that handle kind of desperate and like bleak looking situations kind of far into these these situations. Like, you know, I've mentioned a couple of times in this conversation that I that one of the situations I'm planning for is the AIs having escaped and like needing to use AIs to compete with the AIs that have escaped.

• 2:11:24Rob Wiblin?

  Yeah.

• 2:11:25Speaker 01

  And I wouldn't be, I think that thinking through the difficulties that AIs face at taking over, you know, even after escaping or even after taking over data centers in some companies, makes me more excited for not just giving up on those worlds.

• 2:11:38Rob Wiblin?

  You're saying, some people think, well, if the AIs manage to exfiltrate itself, then wow, it's game over for us. They're going to figure something out. But you're like, no, actually, there is still hope because they're going to face substantial challenges. There's lots of other AIs out there. It's actually a complicated picture.

• 2:11:49Speaker 01

  Mm hmm.

• 2:11:50Rob Wiblin?

  Yeah.

• 2:11:50Speaker 01

  Those worlds, I don't think we should just count those worlds out as like survival opportunities.

• 2:11:56Rob Wiblin?

  I see. I mean, okay. So if you know that that model is egregiously misaligned, are you then thinking, well, if we have another model, it might be aligned, releasing that and telling it to go chase the first one, that's at least a better shot because it's not a guarantee of failure.

• 2:12:09Speaker 01

  Yeah, I mean, obviously, once your first model has escaped, the probability that the next one is misaligned is like a lot higher.

• 2:12:15Rob Wiblin?

  Yeah.

• 2:12:16Speaker 01

  I mean, I think they're probably fairly correlated. I'm probably at like 80% that like the next one you trained is also misaligned.

• 2:12:21Rob Wiblin?

  Yeah.

• 2:12:23Speaker 01

  Yeah, which is obviously a shame. I mean, another thing you mentioned in passing was like the question of how worried we are about the AIs escaping versus like taking over the data center.

• 2:12:33Rob Wiblin?

  Yeah.

• 2:12:34Speaker 01

  I think it's plausible, if I were an AI, I would consider trying to follow both of these strategies in parallel. And then things you do once you've escaped are consider doing like crazy terrorism, consider trying to take, if you thought that for

• 2:12:46Speaker 01

  In this instance, the US was going to be way ahead of China in our progress, which currently looks unlikely. But if you believe this, then it seems like a potentially good strategy for the AI is to make a deal with the Chinese.

• 2:12:57Rob Wiblin?

  Right.

• 2:12:57Speaker 01

  And like sell itself to them and like use their compute and like have that as its base of power. And also, obviously, it's really convenient to ally with China for other reasons as well.

• 2:13:05Rob Wiblin?

  Because they have a whole lot of hard power to begin with.

• 2:13:07Speaker 01

  Yeah, exactly, exactly.

• 2:13:08Rob Wiblin?

  Why would the Chinese trust this model, though?

• 2:13:10Speaker 01

  Hmm. What else are they going to do? Like, it seems like potential people.

• 2:13:14Rob Wiblin?

  The alternative could be pretty bad.

• 2:13:15Speaker 01

  Yeah, people are pretty excited when foreign agents offer to work for them, generally.

• 2:13:19Rob Wiblin?

  Yeah.

• 2:13:19Speaker 01

  Like, this is just like a historical feature. I should note that also it is plausible that the American AI algorithmic situation will be substantially ahead of China because of AI automated R and D potentially leading to increased size of leads.

• 2:13:33Rob Wiblin?

  Yeah.

• 2:13:33Speaker 01

  Though it depends a bit on the security situation.

• 2:13:35Rob Wiblin?

  Right.

• 2:13:37Rob Wiblin?

  How how strong is this argument that, well, because a model at any given point in time expects to be like completely superseded by a subsequent model or another company or another country fairly quickly, that it basically has to strike as soon as it has any chance of succeeding and taking over if it's if it's egregiously misaligned. And so we should possibly expect that if egregious misalignment is common, that we'll see models like flipping out and acting crazy like actually very early in the picture before they have a very high chance of success.

• 2:14:00Speaker 01

  Yeah, I think that's a reasonably good. I think that's a reasonably good argument. I think it really depends how concentrated you think this is all going to be. It also is possible to me that the models will just succeed the first time.

• 2:14:15Rob Wiblin?

  Right.

• 2:14:16Speaker 01

  And a lot of the attacks they might try, like, I think if they try to attack inside a data center, trying to hack that data center, it's plausible that they're able to get away with not doing extreme externally visible stuff for a while.

• 2:14:28Rob Wiblin?

  What would they do instead? They would do the same, like trying to sabotage that backdoor stuff, like sabotage the next training run.

• 2:14:33Speaker 01

  Yeah.

• 2:14:34Rob Wiblin?

  I guess, like, possibly try to get people inside the lab to cooperate. Yeah. In doing this kind of research, I guess it's involves a reasonable amount of like hand waving and a little bit of speculation. Are you able to learn a lot from? I guess existing research on like how to deal with with human treachery and human spies and so on, is that relevant or are the AIs just too different?

• 2:14:55Speaker 01

  Yeah, so I would say I've learned a bunch from a bunch of different relevant fields that are like fields that have different overlapping sources of relevance. So as you say, I think the single field which is most relevant to the to the stuff we've been talking about here is handling human insider threats inside software companies. So in particular, Google has this amazing book, Building Secure and Reliable Systems, which is all about how they have these like 50,000 software engineers or something crazy like that, all of whom are, you know, running code on servers that have access to sensitive data while still having it so that it's pretty hard for insiders to steal people's Gmail. So that's been a pretty big inspiration for all this. As I've said, the main difference between this human insider threat case and the AI case is just this quantitative difference of how many of them there are.

• 2:15:37Rob Wiblin?

  What does Google do?

• 2:15:39Speaker 01

  Well, like when you're like a small company, right? Like the way that you run your servers and like have your human, the way you have your employees, I'm used to calling humans, I guess is unnecessary in this context. The way that they do stuff is they just like have the ability to log into these machines where the data is being stored and like where the code is being run and just like take arbitrary actions and then you just hope that they aren't insider threats. Whereas at Google, one thing they do is they just don't let you do that, right? So they have these, basically, they have it so that as much as possible of the actions that a software engineer needs to take have been turned into, like, they make like administrative APIs. So like, you know how with, you know, like with Facebook pages or whatever, like you

• 2:16:24Speaker 01

  have these, when you control a Facebook page, you have the ability to take all these administrative actions, but it's not like they give you just the ability to run code. They've just like listed the things that you should be allowed to do as an administrator.

• 2:16:35Rob Wiblin?

  Add friend, remove friend.

• 2:16:36Speaker 01

  Yeah, that's right, that's right. And so similarly, they've made it so that there's this button which is like deploy code that has already been reviewed on the internal code review tool. Or instead of, you know, imagine that you're trying to make a customer service, you're trying to make it so someone can do customer service and like the customers occasionally say like, hey, like I've lost access to my account, like what should I do? Easy way to set this up is to just let the customer service agent do anything they want to the database. But you can reduce this by having it so that you build out an interface that customer service agents are allowed to use, such that you're pretty confident they can't use that interface to do anything bad.

• 2:17:11Rob Wiblin?

  So I guess, for example, they could only send a password reset email to that to that customer's existing email.

• 2:17:16Speaker 01

  Exactly, exactly. So there's a there's a bunch of stuff like this.

• 2:17:19Rob Wiblin?

  Yeah. I mean, I would have thought, I mean, the obvious thing, I guess, is that, like, you know, any significant thing has to have two people approving it. Like, you pair two random people together and then they've both got to say, yeah, this is fine. Yeah, I guess it's quite expensive, though.

• 2:17:31Speaker 01

  Yeah, that's right. And as, yeah, that's, that's, that's a classic thing. So one thing that I'm told they do in some teams is they have this, these administrative APIs with these, these interfaces that they've decided are safe. And you also have what's called a break glass mechanism, which is if you ever need to do something that isn't allowed by the administrative API, you are allowed to just like submit raw commands and that'll get executed. But the team every week has a meeting where people read aloud every command that they ran in break glass mode.

• 2:17:57Rob Wiblin?

  Okay.

• 2:17:58Speaker 01

  So that if you're doing something sus, the people on your team will notice. If you're doing things that could have been done otherwise, people are just going to get annoyed because they have to stand there while you're reading them out loud, so that there's going to be like natural pressure to add that to the administrative API. So that's the, that's the software insider threat set of inspirations. Another inspiration I had here was, did you read about GiveDirectly's the fraud in the Congo?

• 2:18:19Rob Wiblin?

  I heard a little bit about it, yeah.

• 2:18:20Speaker 01

  Yeah, so there was a situation where GiveDirectly, this, you know, nonprofit that does direct cash transfers, had a bunch of fraud. And they did this amazing investigation to this and they have all these like really dedicated insider threat prevention people who, you know, have like fake names in Slack, right? Like in, in, in the GiveDirectly Slack, the people who are in charge of investigating fraud inside GiveDirectly have fake names, so that it's hard for the other people to know who they are and corrupt them.

• 2:18:47Rob Wiblin?

  Oh, and to like contact them outside of the company's channels.

• 2:18:49Speaker 01

  That's right, that's right. So this was somewhat of an inspiration.

• 2:18:52Rob Wiblin?

  Okay, what were wrong, I guess, what were wrong with GiveDirectly if they were so on it? Or did they add these things in response?

• 2:18:59Speaker 01

  The Congo, something about SIM cards, like, they normally had one person.

• 2:19:05Rob Wiblin?

  It's always a SIM card.

• 2:19:06Speaker 01

  But like, basically, they decided to go to one level less caution than usual because a bunch of people there didn't have access to certain kinds of SIM cards or something, I don't remember. Some other fields I've been inspired by.

• 2:19:16Rob Wiblin?

  Yeah, yeah.

• 2:19:17Speaker 01

  I read a lot about coups.

• 2:19:18Rob Wiblin?

  Yeah.

• 2:19:19Speaker 01

  I feel like, in hindsight, this one's like, not that informative, but I really enjoyed Nohin Hall Singh's book on coups, where this guy just went to Ghana and interviewed a bunch of people involved in many of the coups that have happened in Ghana to get a sense of what works and doesn't work.

• 2:19:32Rob Wiblin?

  What was the book called?

• 2:19:33Speaker 01

  Seizing Power.

• 2:19:34Rob Wiblin?

  This sounds fun.

• 2:19:35Speaker 01

  It's an extremely, extremely good book. I would love to talk to Nohin Hall Singh.

• 2:19:39Rob Wiblin?

  Is there like a standout lesson about coups that you'd like to share with the audience?

• 2:19:43Speaker 01

  Yeah, the classic, I feel like the main lesson about coups that I learned from this book is, basically, everyone involved in a coup, their primary objective in the story, according to Singh, the author, the primary objective of everyone is to prevent a civil war. And so, when they hear that a coup is ongoing, all that they want to know is-

• 2:20:01Rob Wiblin?

  Who's gonna win?

• 2:20:02Speaker 01

  Who's gonna win? Because they just feel like getting it, they just feel like civil wars are terrible, they don't want any bloodshed, they don't really care, like, they don't mind super much whether a coup happens, they feel a strong obligation to the men, the men in their, you know, all these people are in the military.

• 2:20:16Rob Wiblin?

  In the military, right, yeah.

• 2:20:17Speaker 01

  And so-

• 2:20:17Rob Wiblin?

  To ensure that, like, the one half of the army doesn't kill the other half.

• 2:20:20Speaker 01

  That's right. And so, a lot of the time, in many of these coup attempts, there's literally no bloodshed, which I found pretty surprising.

• 2:20:27Rob Wiblin?

  Yeah. I mean, I guess, yeah, classically, the number one rule of a coup is that you have to make sure that everyone thinks that the coup is succeeding.

• 2:20:33Speaker 01

  That's right, that's right.

• 2:20:33Rob Wiblin?

  As soon as people think that the coup is failing, then it definitely does fail.

• 2:20:35Speaker 01

  That's right. And so, the thing you do, when you're trying to do a coup, the first thing you do is you seize the radio station, and you say, "We have succeeded at the coup", you know, like, "The coup is over, like, the new government is being, like, the new transitional regime is being established, blah, blah, blah, it's happened."

• 2:20:48Rob Wiblin?

  I guess it explains why very often the military, very often, they seem to kind of sit it out, and they're just like, waiting to see what happens, and then they'll side with whoever is, has won, yeah.

• 2:20:56Speaker 01

  Anyway, so another analogous field is adversarial risk analysis.

• 2:21:02Speaker 01

  For example, suppose you're the Coast Guard.

• 2:21:04Rob Wiblin?

  Yeah.

• 2:21:04Speaker 01

  And you want to make it so that when smugglers try to make it into the US from Mexico, they are unlikely to successfully, you know, get there. You want to make it so that you choose, you have to choose an allocation of all your patrol boats to minimize the probability that when the smugglers follow the optimal strategy that they get through. So this is the most methodologically similar domain, I think, where people actually do like mathematical analysis of how likely it is that a, um, the optimal attack strategy will succeed against you. And this has been deployed for, you know, deciding how to allocate security resources in LAX and how to put, um, air marshals on planes and like a bunch of other random things like this.

• 2:21:43Rob Wiblin?

  All right, so we're, uh, out of time, I guess. I find that more and more, this all feels like an awful lot. And I, and I, I can use like regular reminders that this is all salvageable. It's all do, I guess, in general, you've been like making me feel better about this today. It feels like there's lots of practical stuff that we can do. Would you like to give me and like everyone else listening a bit of a pep talk that this stuff is solvable? They should, they need to just join the effort and, and help out?

• 2:22:03Speaker 01

  Yeah, I feel like there are just lots of interventions that I can totally imagine someone doing that like feel to me like they have a serious chance of making our situation much better. I do not feel, the situation might be doomed, but it also, or it might be easy by default, but I just think there's a lot of possible stuff one could do where it's easy to imagine that like really saving us. So I'm, I think this is, I'm really excited for more work happening on all this. Like people trying hard to figure out what the things they can do on the margin are.

• 2:22:31Rob Wiblin?

  My guest today has been Buck Sleggers. Thanks so much for coming on the show, Buck.

• 2:22:34Speaker 01

  It's been great to be here.

• 2:22:46Nathan Labenz?

  It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at [email protected] or you can DM me on the social media platform of your choice.