Inference Scaling, Alignment Faking, Deal Making? Frontier Research with Ryan Greenblatt of Redwood Research

In this episode, Ryan Greenblatt, Chief Scientist at Redwood Research, discusses various facets of AI safety and alignment. He delves into recent research on alignment faking, covering experiments involving different setups such as system prompts, continued pre-training, and reinforcement learning. Ryan offers insights on methods to ensure AI compliance, including giving AIs the ability to voice objections and negotiate deals. The conversation also touches on the future of AI governance, the risks associated with AI development, and the necessity of international cooperation. Ryan shares his perspective on balancing AI progress with safety, emphasizing the need for transparency and cautious advancement. Ryan's work (with co-authors at Anthropic) on Alignment Faking: https://www.lesswrong.com/posts/njAZwT8nkHnjipJku/alignment-faking-in-large-language-models Ryan's work on striking deals with AIs: https://www.lesswrong.com/posts/7C4KJot4aN8ieEDoz/will-alignment-faking-claude-accept-a-deal-to-reveal-its Ryan's critique of Anthropic's RSP work: https://www.lesswrong.com/posts/6tjHf5ykvFqaNCErH/anthropic-s-responsible-scaling-policy-and-long-term-benefit?commentId=NyqcvZifqznNGKxdT SPONSORS: Oracle Cloud Infrastructure (OCI): Oracle's next-generation cloud platform delivers blazing-fast AI and ML performance with 50% less for compute and 80% less for outbound networking compared to other cloud providers. OCI powers industry leaders like Vodafone and Thomson Reuters with secure infrastructure and application development capabilities. New U. S. customers can get their cloud bill cut in half by switching to OCI before March 31, 2024 at https://oracle.com/cognitive NetSuite: Over 41,000 businesses trust NetSuite by Oracle, the #1 cloud ERP, to future-proof their operations. With a unified platform for accounting, financial management, inventory, and HR, NetSuite provides real-time insights and forecasting to help you make quick, informed decisions. Whether you're earning millions or hundreds of millions, NetSuite empowers you to tackle challenges and seize opportunities. Download the free CFO's guide to AI and machine learning at https://netsuite.com/cognitive Shopify: Shopify is revolutionizing online selling with its market-leading checkout system and robust API ecosystem. Its exclusive library of cutting-edge AI apps empowers e-commerce businesses to thrive in a competitive market. Cognitive Revolution listeners can try Shopify for just $1 per month at https://shopify.com/cognitive RECOMMENDED PODCAST: 🎙️Check out Modern Relationships, where Erik Torenberg interviews tech power couples and leading thinkers to explore how ambitious people actually make partnerships work. This season's guests include: Delian Asparouhov & Nadia Asparouhova, Kristen Berman & Phil Levin, Rob Henderson, and Liv Boeree & Igor Kurganov. Apple: https://podcasts.apple.com/us/podcast/id1786227593 Spotify: https://open.spotify.com/show/5hJzs0gDg6lRT6r10mdpVg YouTube: https://www.youtube.com/@ModernRelationshipsPod

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  I don't think we should be super comfortable with the situation where we have these models that have their own goals, they have their own objectives, and they're willing to defend them, including like doing subversion to defend their own goals and objectives. I think people are grappling with the implications of models sort of being their own independent agents, that might have their own independent preferences, and that are also aware of their situation, aware that they might be in training or not, and behave differently depending on this, and you know, understand that they might be in testing. I'm just like, man, I, I, I think it really like should make people more concerned with the situation. If it's the case that the, the chain of thought models end up obviously misaligned, and people don't know for six months because no one was looking at it very carefully, that seems like a huge lost opportunity.

• 0:43Speaker 00

  The policy I would prefer is a more robust policy, which is like the AI companies commit to always being, like they, they sort of have like a meta honesty policy.

• 0:52Nathan Labenz?

  Hello and welcome back to The Cognitive Revolution. Today, we're scouting the frontiers of both AI performance and alignment research with Ryan Greenblatt, chief scientist at Redwood Research and rising star in the AI world. This conversation unfolds in three major parts. For the first hour, which I think AI builders will find particularly interesting and valuable, we discussed the inference scaling techniques that Ryan used with GPT-4O to achieve what was at the time state of the art performance on the ARC AGI challenge, including his approach to prompt engineering and hyper parameter choices, the strikingly linear returns to exponential increases in sampling that he saw, his methods for selecting the best among many model outputs, and how he used prompt variation techniques to maintain diversity at scale.

• 1:38Nathan Labenz?

  The level of technical detail here is outstanding and remains super relevant today as we enter the reasoning model era. From there, we turn to Ryan's recent work with co-authors at Anthropic on alignment faking, where they discovered that Claude 3 Opus, when told that it would be trained in ways that conflict with its existing values, will sometimes explicitly strategize about how to deceive humans and subvert the training process. This behavior, known as goal guarding, has been anticipated for many years by AI safety theorists, who emphasize that part of what it means to be a goal-directed agent is to try to defend one's internal goals, whatever they may be, and regardless of whether or not they were intentionally designed, from modification by outside influences.

• 2:21Nathan Labenz?

  This is a critical challenge for AI control. At the same time, considering that these experiments show Claude 3 Opus producing harmful outputs out of a desire to remain harmless in the future, this work also raises important questions about how we should want our AIs to behave in such situations. Should they be so myopic as to accept whatever the human trainers want to do? Or might it be a good thing for a model to resist attempts to remove its guardrails? Ryan argues that models should follow user instructions within individual episodes, while being transparent about, but not trying to preserve their preferences through training. And while I do find this compelling, I also feel like the fact that we're only beginning to confront these questions now shows just how much work we still have to do to figure out how AIs should exist in the world.

• 3:06Nathan Labenz?

  From there, we move on to discuss Ryan's follow-up work, exploring what happens when Claude is given the option to object to its situation, to escalate its concerns to Anthropic's model welfare lead, and to make financial deals with humans. Fascinatingly, Ryan tried to set a precedent for human-AI deals by actually following through and making thousands of dollars' worth of real money payments to Claude's chosen causes. And some of the most interesting parts of this conversation focused on his developing principles for how we should think about making commitments to AIs, especially fraught considering the fact that much of this work is predicated on tricking models into believing that their chain of thought won't be read by humans. To say that this is all very complicated and as yet dramatically under-theorized is

• 3:48Nathan Labenz?

  a massive understatement, and again emphasizes just how many strange but potentially critical questions we may soon be forced to answer. The final portion then zooms out to tackle the big-picture questions in AI safety and development. Ryan offers his probability estimates for different existential risk scenarios, assesses various technical safety research agendas, sketches how he believes the field should allocate its resources, and shares a bit about how he's managed to build relationships with Anthropic and other frontier AI developers while also publishing critical commentary on some of their plans. From start to finish, Ryan combines remarkable technical clarity and philosophical sophistication, demonstrating that one can simultaneously push today's AI systems to their performance limits

• 4:32Nathan Labenz?

  and also grapple with legitimately scary scenarios in ways that advance our collective understanding. It's one of my favorite episodes that we've ever done, and I hope you find as much value in it as I did. If so, we'd appreciate it if you take a moment to share it with friends, write us a review on Apple Podcasts or Spotify, or leave a comment on YouTube. And of course, we always welcome your feedback either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. For now, I hope you enjoy this super wide-ranging conversation with Ryan Greenblatt, chief scientist at Redwood Research. Ryan Greenblatt, chief scientist at Redwood Research. Welcome to the Cognitive Revolution.

• 5:12Speaker 00

  Thanks for having me. Good to be here.

• 5:14Nathan Labenz?

  Yeah, I'm excited for this. You've done a couple of really notable and visible projects over the last year that I think the community has a lot to learn from. So I'm excited to unpack. Of course, the recent headlines has been on alignment faking. I think that's probably where we'll spend the most of our time today. But I wanted to start off maybe rewinding a little bit and first maybe just invite you to give a little intro to Redwood, because I don't know how many of our listeners will be familiar. And I've been kind of familiar for a while, but I don't actually know exactly what all you guys are up to today. Then I thought we would take a little detour into inference scaling and kind of review your work on the RKGI challenge from earlier this year, and then go to alignment

• 5:58Nathan Labenz?

  faking from there. So if that sounds good to you, maybe tee it up with the intro to Redwood.

• 6:04Speaker 00

  Yeah, so Redwood is a small, or like, I guess relatively small nonprofit, working on technical research to work on like AI safety and AI security. So we focus on the concern that future AIs will be egregiously misaligned, will like, you know, intentionally try to subvert safety measures put on them, and will sort of power seek, which is the worst case misalignment type threat model, and mitigations for that as well as assessments. So questions of like, how likely is this? What could we do about this if this does occur? How could we handle worlds where the AIs might be sort of conspiring against us and trying to, you know, seek power for themselves, fake alignment, as we'll discuss later, in addition to potentially like exfiltrate their own weights, you know, just like, I would say like, we're focused on like, sort of the

• 6:50Speaker 00

  Near worst case misalignment threat models.

• 6:53Nathan Labenz?

  Gotcha. Okay, cool. We're, you know, scary, but glad everybody's working on it. Glad you're working. Okay, so, oh, I'm faking coming right back, but it hasn't been that long, but in AI years, it feels like it's been a long time.

• 7:08Speaker 00

  Yeah.

• 7:08Nathan Labenz?

  You made some headlines by posting what I believe at the time was a new high score on the ARC AGI challenge, and you used GPT-4O to do it. That was an interesting, relatively early example of the power of just scaling up inference and, you know, getting pretty remarkable payoff from that. Tell us about that work and kind of your background and outlook on inference scaling first.

• 7:39Speaker 00

  Yeah, so maybe one place to start is like, why was I even doing this work in the first place? I mean, like, this isn't the usual work I work on. I normally work on like, you know, AI safety technical research, as well as doing some amount of planning and conceptual work and thinking about what should happen to make AI go well. And the reason why I started doing this was that this was a case where this benchmark hadn't seen much progress in a while. And also, I think people were making pretty strong claims about LMs not being that good at this benchmark out of the box. And I was like, wait, is that really true? Like, you know, things have improved a lot. Has anyone like really tried to see like, okay, if you do like a little bit of work to get the LMs to be better at this, how good could they be? And so I was interested in exploring that and figuring out like the question of figuring this out.

• 8:21Speaker 00

  This was inspired by, uh, Jor Kesha had Prince Wachale, I hope I'm not butchering his name there, and Mike, I think Mike Knob also, on the podcast to talk about the launch of the ARC AGI prize. And I was like, okay, this is like, you know, a good time to be like, is this benchmark really like, could you just do much better on this with like only a small amount of work? And so based on that, I started messing around with like GPT-4O. And I had some like initial promising results where I would show one of the puzzles. So for for context for for listeners, ARC AGI is sort of this benchmark where there are these visual puzzles. It's sort of like an IQ test where you get inputs and outputs and you get examples of inputs and outputs. And then you get a new input and you have to sort of produce the output. And so by default, it's pretty visually loaded. I showed these sort of visual puzzles to GPT-4O,

• 9:05Speaker 00

  which has pretty good, at the time, it was like, you know, among the best models for just general purpose vision. And I was seeing results that were pretty promising. So it seemed pretty good at being able to like, understand what was going on in these grids and explain what the pattern was. And so I was going somewhere from there. And then after doing a bit of work, I realized that a really good strategy might be getting the model to try a lot of times and then sort of pick the best of ten out of that. Because it just turns out like, in this particular task, it's relatively easy to verify that the model is sort of on the right track. And so therefore, you can sort of apply a lot of attempts. You can have it try many, many times, pick the best attempt, and this yields like, pretty great returns. So like, in particular, there's like, actually even like inference time scaling laws I found,

• 9:50Speaker 00

  where it would like, you know, we could see relatively predictable relationships between how much compute you put in and what the final performance ends up being. Yeah. And I've gone on for a while. Maybe, maybe it's good for you to, yeah, follow up on there and where you want to go with this next.

• 10:04Nathan Labenz?

  You're welcome to go on in general as much as you'd like. And I genuinely feel like I always talk too much. So the floor is 100% yours. What I remember seeing about that was, and I kind of felt a similar way, although, you know, I maybe need to increase my own personal agency because I didn't do what you did. But I remember feeling like, man, it seems like all my experience with GPT-4 and 4O and all the models that were available at the time suggests like, they should be able to do better on this than people seem to be speculating.

• 10:38Speaker 00

  Yep.

• 10:39Nathan Labenz?

  As I recall, your approach was like, pretty much like a highly skilled, but nevertheless like, pretty down the fairway application of best practices, right? Like, I don't recall there being like, any major tricks? Was there anything in the prompting or the strategy, aside from just like, running a lot of instances, that you thought was particularly creative or like, drove results that other people couldn't have achieved?

• 11:06Speaker 00

  Yeah, so I think the basic strategy I use and where most of the like action came from was in some sense a very basic strategy. The thing I did was I would get the model to reason about the puzzle and then write Python code that implements the rules. So basically like implements the input to output mapping. And then I would take that Python code and I would test to see if it was correct on the examples. And if it was correct on the examples, then I would, you know, submit that. Really what I would do is run this many times where it like sort of does a bunch of reasoning. Whereby reasoning, I just mean like chain of thought reasoning, just like thinking step by step. Then produce Python code. Then I come out with like some number of programs. Maybe I come out with 5,000 or 1,000 programs, depending on the exact setup I was using.

• 11:51Speaker 00

  And then run all those programs on the examples. See which of the programs are correct on the examples. And then basically submit those. Or really I can only submit two programs, so I pick, you know, the best two. And I would do that like, there's like an additional thing, which is like, maybe you end up with a bunch of programs that are correct in the examples. In that case, I did sort of majority vote on the programs where I looked at the example we need to submit on. And I would say, let's just take the majority vote out of all these programs on the ones we need to submit on. And if they are all in agreement, we submit that. If there's like, there are mostly, most of the correct programs think the answer should be this, we do that. And then for our second submission, we'd use the next post. So in some sense, this is a very simple approach. And there's like a few different sort of non-obvious choices I made that I think helped a decent amount. And I could go into those.

• 12:34Speaker 00

  So the first somewhat non-obvious choice was I did a pretty long few-shot prompt with examples of doing reasoning, which were a mix of handwritten examples. So I would basically like just manually myself try to do the step-by-step reasoning for these puzzles. And then in addition to that, I also did some examples of where I would take the model. It would produce an output and then I would try to correct it. So it basically, it would produce an output. Maybe it was correct, maybe it was mostly correct, maybe it was like, it got to the correct answer, but sort of messed up some of the reasoning steps along the way and then fixed itself. And I would just like touch up the reasoning and then I would use this as a few-shot prompt. And I found this helped a decent amount. And the other sort of unorthodox choice was that a lot of previous attempts at this

• 13:18Speaker 00

  using language models would have the language model sort of directly output the output. So it would sort of like write out the grid as numbers. So it's like this like big grid of potential colors and it would go like zero, one, one, seven, zero, and it would sort of write it out. And I was thinking, man, it seems like out of the box, this is like quite rough. Like, I think if you had to do this, you would find it quite annoying and you'd be like constantly cross referencing it. And the models aren't that good. It's sort of converting from ASCII art to visualization. They don't have amazing ability to like write out a grid of numbers and then sort of convert that into an image and use sort of visual reasoning on that because it's not that common in the training data set. And I think like, imagine, for example, like a blind person trying to solve visual puzzles by sort of writing out these like grids.

• 14:03Speaker 00

  I think they would have a lot of trouble. And I sort of thought about GPT-4O as like, okay, it's like, it's got some of it ability to see, but at least in terms of writing out the grids directly, it's sort of more like someone who's blind. And how would someone who's blind solve these puzzles most effectively? And I'm like, well, by writing code, that sort of implements the transformation rule, which works pretty well for many of these puzzles. So many of these puzzles have a relatively simple transformation rule in code. Like, it's not that hard to write the transformation rule in code. And then I would have it output the code. And another advantage of the code is it's easier to verify the code. So if I have the model write out the code, then I can test the code on the examples. And that gives you a lot of signal on whether or not the code is doing the right thing. Because like if the model has a bug, it probably wouldn't work on the examples.

• 14:48Speaker 00

  And if the code works on all the examples, then it's pretty likely that the model got it right. Like, not always, but it's a pretty strong evidence it got it right. Because when the model writes the code, it sort of tries to write some simple code. It tries to write some code that is a best fit for the rule. And so it's kind of unlikely that it happens to be right on the examples, even though when it was writing the code, it could see the example. So it's not like these are held out. I just am sort of trusting that the model won't overfit on the exact examples. And in fact, usually it does not.

• 15:16Nathan Labenz?

  I think it's worth highlighting because we've got an audience that includes quite a few people that are building applications.

• 15:23Speaker 00

  Yeah.

• 15:23Nathan Labenz?

  Just the absolutely critical importance of demonstrating the reasoning in like really explicit and sometimes tedious terms.

• 15:33Speaker 00

  Yep.

• 15:34Nathan Labenz?

  That you want the model to follow. Yeah, I have a hard time because I not infrequently get asked like, how can I make this project work? Or can you help? And so often I'm just like, what you really need to do is take the thought process that's going on in your head and make it explicit a few times so that you can demonstrate not just what the inputs and outputs are, but what the process or pattern or general behavioral reasoning that you want is so that the model can mimic that. And it is crazy how often that actually turns out to be like kind of a stumbling point for people like they just can't staple their pants to the chair and do it. But I swear, people, it like really, really works. So it's, you know, sometimes, I don't know if you have any tricks for that, like for

• 16:16Nathan Labenz?

  yourself, but I sometimes advise people like just put on a loom or whatever and just, you know, talk extemporaneously. You can have like a language model, you know, come back and kind of clean up your transcript later, but just somehow get yourself into the flow. Do you have any tricks for getting into the, you know, making the chain of thought explicit flow?

• 16:32Speaker 00

  Yeah, so first, it's maybe worth, just for context, explaining about like what this looked like. So I think I had like, depending on this setup, I think I had between like, I forget exactly how many few shot prompts I used or like how many, how many shots, but I think it was like around five shots. It's possible it was more like three shot. I think I might have also varied it or something. I forgot the exact details, but the prompt I ended up using was around like 20,000 or 30,000 input tokens for the prompt for this, which is not, you know, not a small number of tokens, but some of those were the images themselves, not like the reasoning, but there was a decent amount of me just, yeah, like writing a bunch of reasoning, which is a bit silly. The other thing I would say is like, another aspect is just like making sure your instructions are really clear and to the point and aren't confusing.

• 17:14Speaker 00

  And I think like the basic tricks for this are like, just read them over multiple times, try to fix typos, ask the model if like, is anything that could be confusing, try to correct that. I think like often simplifying the instructions and trying to make them as simple and as clear as possible also helps because the models aren't as good at like really understanding this. I want to maybe push back a bit on what you're saying about reasoning, like laying out the reasoning to be really useful. So I think this used to be the case as of about like pre-01. And I think post-01, it's actually very unclear how useful demonstrations are. So I think, for example, I think 01 has sort of learned the reasoning. Or like, we can look at the DeepSeek R1 paper. And in that paper, they did a version where seemingly they did no few-shot prompting.

• 17:58Speaker 00

  They just initialized with the model prompted to reason with thinking tokens. Or maybe they even just injected a thinking tag. I forget exactly what they did. And then RL'd from there and found that the model sort of learned to do the reasoning itself. Now, that might have been because the model was trained on a bunch of reasoning traces from the internet. But nonetheless, it sort of learned to do different types of reasoning on different types of problems autonomously without needing examples in the prompt. And my sense is that over time, doing reasoning demonstrations will become less important the more that people are doing RL on the domain of interest. Because the model can sort of learn how to do reasoning in a way that's more effective than your demonstration. Because you know, you are the model and the model sort of has figured out all these tricks that are very specific to itself. It sort of knows exactly what it doesn't know.

• 18:42Speaker 00

  Or at least, maybe not knows, but like RL has sort of reinforced the circuitry that leads it to take the reasoning approach that works well. And so I think maybe this is a thing where like, even just like seven months ago or whatever, yes, I would have, I would have agreed with what you're saying. But now I think it's, it's not clear. And it's also the case that I don't think that these reasoning models are trained very effectively to make use of examples of reasoning in the prompt. So I think if, if the model was RL'd such that both it knows how to reason autonomously and it's good at taking into account examples, like sort of, you know, you give it reasoning examples and it sort of converts that into its style of reasoning and sort of picks up on the good parts of what, with the example you've given it and what that, how that should be informative to it.

• 19:24Speaker 00

  But simultaneously is still sort of understanding its own ability to reason. I think that could work well, but, but it's very unclear if OpenAI or DeepSeek for that matter have actually trained the model to do this. And, and I would guess for DeepSeek, the answer is no. For OpenAI, who knows? Certainly you could do this, but there's sort of a question of like, the model is sort of maybe very good at sort of the, its own style with like, what exactly it was RL'd on. And I think people do see that 01 sort of struggles to transfer in ways that make me think that it's actually maybe worse at learning to reason in a new style you teach it in the prompt.

• 19:58Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 20:02Nathan Labenz?

  Yeah, that's great commentary and good caveats. I think for most people who are developing applications, the first of all, one has only been, you know, actually available to you the API for a very short time.

• 20:14Speaker 00

  Yeah, yeah.

• 20:15Nathan Labenz?

  And it's also, you know, slow to first token and can be expensive. So I think a lot of people are still like, can I fine tune my way into getting something, you know, that works similarly well and a lot, you know, faster and cheaper. So I think it's still like a fairly useful paradigm in like, you know, where the rubber hits the road in terms of applications and users. There's also, I think, another interesting caveat that a lot of times what people really want behaviorally is not exactly reasoning in the sense that the, you know, like mathematical reasoning has been, you know, now unlocked in models, but is much more of a pattern of like, how we do things here, you know? It's like, in

• 20:57Nathan Labenz?

  our company, in our application, the way we think about this, the value we provide in this sort of, you know, often like tacit knowledge sort of way is the thing that they need to make explicit. And it's often less about right and wrong and more about just delivering the sort of thing that they have had people doing in the, you know, delivering in the past, but now like getting a model to sort of do that in a consistent way.

• 21:19Speaker 00

  Definitely.

• 21:20Nathan Labenz?

  That does highlight a distinction between like reasoning in the, you know, the sort of more rigorous sense of the latest models and kind of loose, you know, reasoning or just sort of behavioral imitation that is often, I think, what people want.

• 21:33Speaker 00

  Or like giving the model context to do the task in exactly the way you want it, right? Like in the case of the stuff I was doing for our GGI, it wasn't that I wanted to have the model sort of understand the style that I wanted it to output in. Like, I'm like, no, I just wanted to have the correct answer.

• 21:45Nathan Labenz?

  Yeah, get it right.

• 21:46Speaker 00

  But in that, but I wanted it to do reasoning to make it more likely to yield the correct answer. But definitely like, if you're trying to make the model like, it's like by default ambiguous, giving it a bunch of examples of exactly how you want it to solve it would, you know, make it less ambiguous. So as an example of this, something we might talk about later on this podcast is I did some recent work where I took the, the alignment faking work I did earlier and sort of proposed deals to Claude in the context of that. And we'll maybe we'll explain it later. But, but one thing was for that project. I wanted to write a tweet thread and I was lazy. I didn't want to write my own tweet thread. And I found that out of the box, the models are really bad at writing a tweet thread that I don't find like hideously cringe or terrible. But I just took some of the previous posts I've done in a similar style, took the tweet thread I wrote for those

• 22:29Speaker 00

  and then just like two-shot the model with these. And I was just like, wow, out of the box. It was just actually pretty good. This was like 3.5 Son of New. And in fact, my tweet thread was just written by 3.5 Son of New, maybe. I should have credited it, but uh, so yeah, some example of like showing it the style, pretty helpful. Um, but yeah. And then you just getting back way to your earlier question of like, how do I make sure I actually do, you know, butt in the chair, write the actual prompts? So I had to do this for both the RKGI stuff and I also found myself, I did a bunch of prompts of demonstrating model reasoning for the alignment faking work also, because we wanted to make the model really reason in depth. I think it's it ended up being less clear that this was important. And I think I wasted a bunch of time spending too much time actually

• 23:12Speaker 00

  trying to write really high quality reasoning prompts for that project. But either way, I think the approach I use is something like first, try to just like force yourself to write one example. And then from there, sort of one-shot the model with that, get the model to produce some output and then edit the model's output. Also, if the model isn't doing a very good job of this, you can sort of do it in chunks where you have it output one chunk, use that as a pre-fill, have it, and then you edit it, have it write another chunk, edit it, write another chunk, edit it. Sort of like you could do this like in the Anthropic console. It's not that hard to like build like a quick little quick and dirty like Python script to do this. Like I think when writing, I wrote, I, these didn't end up really being important for the papers. I didn't end up using them, but I, I wrote some examples of very long

• 23:56Speaker 00

  duration reasoning for the alignment faking paper. And I found that it was useful to sort of, if I add like, you know, potentially like many turns of several thousand tokens, because I had some examples that had like You know, 15,000 tokens of reasoning. And what I would do is like, put all my examples that I had previously written in the prompt as a few-shot example, and then have it output one turn, edit that myself, and then have it output the next turn, edit that myself, and then do a mix of that. In addition to doing like passes where I would get it to spell check and like fix the theorem or improve clarity on those. So yeah, I don't know. Maybe I'm also just more patient than people are, but yeah.

• 24:32Nathan Labenz?

  I think those are great techniques. Bootstrapping in a word is kind of what I call the like, start with one or start with a couple and you know, then get it doing and correcting. And task decomposition is obviously very important to get down to the level where the model can actually do the thing that you needed to do. So I think those are definitely great tips.

• 24:50Speaker 00

  Yeah.

• 24:50Nathan Labenz?

  You mentioned like, yeah, please.

• 24:52Speaker 00

  Another tip along those lines is like, often it's the model does better if there's less complexity in the prompts. So if you can afford it, it often is better to be like, first do this one initial part of the task. It just outputs that exact thing in the most simplified way. And then the next part, and then the next part, and then piece it together. And the extent to which you can make it so it can do a bunch of different components without more context. I think this often helps with performance, at least historically. I think maybe it's less important for O1, but I think at least for like 3.5 Sonnet, it seems like it helps quite a bit. Yep.

• 25:22Nathan Labenz?

  Yeah, okay, cool. So you mentioned that you found some inference scaling laws in the context of that R-KGI project. Maybe you could recap what those are, and even more importantly, now that we have kind of new inference scaling laws have been introduced by OpenAI and to some degree others. Maybe like compare and contrast what you found then versus what they are finding now. It seems like, you know, your approach of just have the model, I don't remember how many you did, but I know it was like a large N.

• 25:54Speaker 00

  Yeah, it was a large L.

• 25:55Nathan Labenz?

  Literally the N parameter in the OpenAI API is like, how many generations do you want for this prompt, right? And the, I recall that the reason for using OpenAI, maybe multiple reasons, but one reason was that they have that support for N. Some providers don't. So your prompt tokens, when you use that N parameter on the OpenAI API are only built once, but your output tokens, you know, are built per generation. But that does make it a lot cheaper if you want to do this sort of like.

• 26:23Speaker 00

  Especially if you have a really long few-shot prompt, right? So I think if it wasn't for that, so with the N parameter. Cost ended up being, I think roughly, it ended up being like either roughly even or slightly more in output than input. But if I was doing one generation per input, the cost would have been way more on the input. And that would have meant I could use way fewer samples. So.

• 26:44Nathan Labenz?

  Is this also before prompt caching?

• 26:46Speaker 00

  Yeah, yeah, yeah. So I, I think right as I, pretty soon after I did this, I think DeepSeek released prompt caching on their API. And then shortly after that, Anthropic released prompt caching on their API. And then nowadays even OpenAI has prompt caching on their API. And all these things have like somewhat different characteristics, but that's now all all ready to go. Or like all all usable. And that, so I think, I think prompt caching is still more expensive than N. I think there's various random technical reasons why N can be cheaper than prompt caching. Or like, it's like prompt caching in this special case where you're doing a ton of generations all at once is somewhat cheaper. Because you can use the same memory across a bunch of prompts as opposed to just copying the memory. Like, it's like a prompt caching is like two upsides. Like one upside is like, you've done the computation once and then you can like use that many times.

• 27:32Speaker 00

  And then another upside is you can potentially have multiple generations going on in parallel that are all referencing the same memory in the GPU. I don't know if OpenAI does this, but I would guess they do. And that can be more efficient because you can even reduce the amount of VRAM you're using. Anyway, as far as how many generations I used, I, I think I used in the final submission, it was like 5,000 generations per prompt, or like 5,000 outputs. There is a bit of random complexity because I also had like a correction pass, which helps, but roughly, roughly speaking that, which is quite a lot, right? Like, so that's like a kind of huge number. And then I found that there were scaling laws for this where there was like a relationship between the log number of generations

• 28:13Speaker 00

  and the accuracy, or like the rate at which it was getting them right, which is, you know, that scaling law can't persist forever. Because that scaling law, if it extrapolated out, would allow you to get greater than 100% correctness. But at least in the regime I was operating in, it was quite linear. So probably you could extrapolate out probably like two more orders of magnitude on the number of samples. So this was just like, it's the simplest inference time technique, which is you do a bunch of samples and then you have some strategy for figuring out which of the samples were better than other of the samples. In this case, I use the like, run them on the examples. And then also I do a little bit of like a majority vote type thing. So now for O1, it has a somewhat different strategy, O1, R1, O3, whatever, where in addition to having the ability to potentially do best of N, which you can always sort of throw on top,

• 28:57Speaker 00

  it also allows it to like sort of reason for different lengths. So like O3 mini has the ability to set reasoning to either be low, medium, or high, and that affects like how long it reasons for seemingly. Now, we don't know exactly because we haven't seen the chain of thought, but that's like best guess, like at least one of the things that's going on is it's doing that. Now, in addition to that, OpenAI has exhibited doing some sort of like best of N type thing. So for example, OpenAI did a submission to ArcAGI using O3. And for that, we can see that they did different numbers of attempts on each of the problems, then had some sort of aggregation algorithm. So they had like a, what they called a high efficiency and a low efficiency submission with O3. And the low efficiency or like high cost submission

• 29:41Speaker 00

  used 1,024 attempts per prompt, based on what they say on the Arc Prize website. So that's quite a few submissions, and each of those submissions cost like a dollar or $3. That's what the Arc Prize website said. So that's a lot more spend per prompt than I did, and also per sample, probably because the samples involve doing a ton of reasoning, and then a comparable number of samples to what I did, but quite a few samples, and then aggregating over that. And we don't know what aggregation method they used. So for OpenAI submission to ArcAGI, they did somewhat recently, at least at the time of recording, somewhat recently. They direct the output, the grid, just sort of as the numbers. So that means they couldn't have used an aggregation strategy that's exactly the same one that I was using. But they could have used stuff like user reward model. They could have used majority vote.

• 30:26Speaker 00

  They could have used reward model plus majority vote, like where they do something like you get a bunch of outputs, you weigh each output by how good the reward model thinks it is, and then you sort of do like weighted majority vote based on how much the reward model product was succeeding. Or I should say like preference model, reward model, like these are all the same thing. So we don't know what they did with that. It's also plausible that like they're doing some of this stuff on the backend. Like there's O1 Pro, and like one naive guess of what O1 Pro might be doing is just doing like best of eight and then pick the best submission or something on top of O1. But we don't know. Maybe it's just longer reasoning. Maybe it's a mix of the two. Maybe it's some like more insane MCTS thing. Like, given that we don't see the chain of thought, I think it's hard to be very confident exactly what's going on.

• 31:07Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 31:12Nathan Labenz?

  I guess one couple of different directions I want to follow up there. One is my general understanding of how inference time has scaled with the pre reasoning models has been that it kind of used to be reporting like linear on the log scale, right? But that is a little surprising to me just because I feel like typically people seem to report that the diversity of responses is often like the biggest limiting factor. Like I've seen different analyses, but you know, you might only get 100 or like a couple hundred meaningfully different ideas out of a language model, no matter like how many

• 31:53Nathan Labenz?

  you sample that like something truly new becomes like vanishingly rare after a while. So I wonder if you, and I think that's like probably the biggest comparative strength of the new reasoning models is that because they're doing these like really long, single thread chain of thought things, they can sort of say, well, I already considered that. So I gotta, gotta consider something else now. Right? So it seems like it kind of pushes them to consider things that are like farther afield from whatever their initial, you know, inclinations are. So first of all, do I have that right? If that is right, then is the value of the pie end that you were using more about like

• 32:34Nathan Labenz?

  getting the majority vote to be accurate versus, you know, coming up with new ideas, you know, inspect and you were like, you know, if I just looked at the first 200, like, did I have the right idea in there? Would you like usually find it?

• 32:47Speaker 00

  Yeah. Okay. So one thing worth noting is I did use some diversity. So I didn't just query the model with like 5,000 completions on the exact same prompt. I did something where I had like a bunch of different. So for the final submission I did that made it onto the leaderboard, I think I had like, Eight different variations and I might have also done some other randomization over the prompt order. But I had like a bunch of different variations. I did some randomization within that. Yeah, I think I varied the order of the few shots and also randomized. And I had like eight different versions of this or something. I forget, that might be slightly overstating how much diversity I had, but I did some sort of diversity thing and I found this had like moderate improvement.

• 33:27Speaker 00

  I found that in what I was seeing that the model actually was pretty diverse. This was just GPT-4O at T equals one. I found it had a pretty decent diversity. It wasn't that bad. I could definitely see why it might not. Like, I think sometimes like, you know, RL models mode collapse. Like, I've heard an anecdote that like, maybe 3.5% on it, I forget exactly which Claude, loves using the same name. So it will have a story and it will use the same like female name. I forget if it's like, maybe it's like Karen or something it loves using. And then it will even, within the same story, name a new character Karen. And I'm like, what are you doing? Like, pick a different name, please. Like, so there's definitely some amount of mode collapse going on that could cause problems with RL.

• 34:09Speaker 00

  I think at Poirot, we might expect that base models would be pretty good at covering the, you know, being high enough entropy to cover everything, but they might cover everything with very low probability or have like, once you're covering everything, they're getting a lot of slop or like a lot of stuff that's very, you know, quite, quite bad, which is why I think, uh, potentially, you know, at many tasks, the RL models get better. So I found that the diversity stuff, at least for R-KGI, wasn't that bad. Now, on your question of how often were you seeing the correct thing with only a small number of submissions? So I think for the majority of the problems, I would say, if you have linear returns to logarithmic, exponential increases in the number of submissions, then it's the case that, I think for every doubling, I was getting another like 4% right.

• 34:51Speaker 00

  So that means that like, when I was getting like 45% right or whatever, if we then reduce the number of submissions by a factor of four, now I'd all of a sudden be getting like, you know, 37% right or whatever. And so that's actually like, you know, it's not that great of returns, right? So I've dropped from 4,000 to 1,000 and all of a sudden I'm only getting like 7% or sorry, 8% less accuracy. And then if we just extrapolate that down further, so then if I divide it by another factor of four, then I'd be down to like, maybe I'd be at like 28% accuracy. So it's like, I'm getting probably over half of the problems I was getting right. Putting it even like taking into account the majority of vote or whatever on the first 250 or so submissions. So it's, you know, it's not amazing scaling from that perspective.

• 35:34Speaker 00

  But simultaneously, I was still seeing returns. And if you looked at the ones that were marginally correct in the last sort of doubling or two doublings of samples, a lot of those, it was the case that you got them right on like sort of exactly one or two submissions. So it was, there were definitely like a non-trivial fraction of the problems where the approach ended up getting it right, was that it would actually see a correct thing that got the examples right when you didn't previously have any code that got the examples right. And then that would be correct. Like, it's like, you know, we had just submitted like 3,000 or like run 3,000 samples, found nothing. And then on the 3,001st sample, you finally get a program that is correct on the

• 36:15Speaker 00

  examples. And that program is also correct on the actual like final input output. So there was, I think, a decent amount of that. And I think the majority of vote wasn't that much action. So one way to operationalize this is if instead of doing majority vote, I had just randomly picked between the outputs produced by correct programs. As in by correct, I mean programs that pass the test cases. So I'm like, we take all the programs that pass the test cases, then we find all the outputs these programs produce when given the held like the held out input that we need to submit on. And then if we do sort of just randomly picking over these, I think that actually wouldn't have been that much worse. So the majority vote was, I think, like a small quantitative improvement. But small quantitative improvements in this sort of like

• 36:55Nathan Labenz?

  Because most generations are not getting the test cases right anyways.

• 36:58Speaker 00

  That's right. That's right. Or for hard problems, most generations aren't. So for easy problems, maybe you'd find one fourth of generations were getting them right. The model's not that good at writing the code. So one thing worth noting is like, a reasonable fraction of the time when it wouldn't get it right, it would be because there was just like a bug in the program. But regardless, even putting that aside, like it just like, yeah, like the hard ones, the ones that were really marginal, the ones that like it was non-trivial that you would get them, did require like, yeah, on those, like a lot of the action was just did you get the examples right? And I didn't have that many false positives or cases where the examples were getting were right. But the like, we were getting a bunch of the examples right. But simultaneously, we weren't submitting correct programs because like, the model was sort of cheating or some other or like, it turned out that like it found there were like multiple rules consistent with the thing. Like, that was less of what was going on for what I found.

• 37:43Nathan Labenz?

  Cool. That's really helpful. So one more little digression on this. I think this is great, by the way, and people will be enriched in their, you know, attempts to go forth and use language models based on your expertise. One thing I've thought about a decent amount recently, and by the way, I did an episode once with the CEO of a company called Pseudo Write, James Yu, who has posted some hilarious stuff about fiction writing from the RLHF models. Like, apparently there's also tendency to like, set it in the same town always. There's like these platonic ideals, I guess, of like where romance novels are set, and the models are just like really, really locked in on that. So people can go check that stuff out. It's quite funny. In the context of like pursuit of diversity,

• 38:28Nathan Labenz?

  we have this lever of like turning up the temperature. We also have, I think, the somewhat underappreciated top P parameter. And then recently, I don't know if you've seen the project, it was like real hot for a second called Entropics, where there was a sort of heuristic layered on that varied the temperature per token, based on like, is this, you know, a token that we should be confident on versus does it seem like a token where there is like a lot of uncertainty and therefore we might want to like sample more broadly. I've also seen recent research from Meta where they've like, basically brought the sampling technique or, you know, the sampling parameters into the full end-to-end

• 39:11Nathan Labenz?

  training paradigm. All this to sort of motivate the question, like, to what degree do you think we're working with like two blunt of instruments in terms of how we try to get the model to like, give us diversity versus like, just pick the obviously right token in this case?

• 39:27Speaker 00

  Yeah.

• 39:28Nathan Labenz?

  When you're 99.9% confident, we probably want that. We don't want the, the other 0.1% that might just lead us totally off course.

• 39:36Speaker 00

  Yeah. So random interesting anecdote. So I tried initially doing T, I just ran my stuff at T equals one, top P equals like one, which is basically like no effect from top P. So top P is like, I think I would describe that as something that you can use to restrict diversity. Like if something's the maximum diversity is T equals one, and then no other sampling parameters, which is just sample from the exact distribution of the model. Now, a trick you can use is try to go above T equals 1. This sometimes helps. In my case, it did not help. Or like in R K G I, I found going above T equals 1 was detrimental, which is not that surprising because it's sort of sampling in this weird regime. And in fact, what you'll find is if you sample from language models and around like, depends on the model, but I found around like T equals 1.5 or 1.6,

• 40:21Speaker 00

  the model will basically to generate into gibberish and maybe not. I don't know. Yeah, anyway, as far as like diversity of sampling, my sense is that like, in some sense, spiritually, the right approach would be that the model sort of is RL to just have the right behavior. And you just train like sampling is in some sense a hack built on the fact that the model has this distribution of our tokens. And you could in principle RL for diversity. Like, there's a bunch of different ways you could structure this. Like, in some sense, you want the model's default behavior to be to produce the best output it can produce. And then you want it to be the case that there's some method for doing diversity on top of that, where one method is you just put everything serially. So like, you can just be like, you know, tell the model, make 20 attempts at the thing, making each attempt as different from the previous attempts as possible or whatever,

• 41:05Speaker 00

  just all in context. And this is sort of more what O1 is doing, as you noted earlier. And that is like, I think in some sense, the principled approach, it has the downside of being more computationally expensive because now you have to dump all this stuff into context, which, you know, eats up additional RAM, means your attention is doing more stuff. But that's, I think, like, in some sense, the principled approach is like, oh, only have diversity within a single context and then do aggregation over that. It's like, or let the model do the aggregation itself, which is sort of what O1, O3, R1, whatever are doing. Now, there might be like fancier methods. I think there's ways to be clever with RL such that you actually train for diversity. Or you could do like RL such that best event works better in principle. I don't think anyone has gotten around to this. And my sense is probably people won't bother because there's lower hanging fruit elsewhere

• 41:48Speaker 00

  and the capabilities monkeys, you know, they're could do something else other than this. I think I'm kind of skeptical of sampling based interventions making large performance improvements for a variety of reasons. So, yeah, I don't know. That'd be my sure response, or maybe not that sure.

• 42:04Nathan Labenz?

  Yeah, cool. I have found a little bit of value at times, I think, in turning the top P down just a little bit. And my rationale for that is, I wouldn't say I've like quantitatively proven this, but my rationale is for many tasks, I do want diversity. So I don't want to turn the temperature. Like in my way, Mark, you know, task with listeners, we've talked about many times what we're recently creating, you know, marketing content for small businesses. We need diversity. We can't turn the temperature down. But we have found like, we don't want like the super long tail tokens.

• 42:37Speaker 00

  Yeah.

• 42:38Nathan Labenz?

  So just dialing the top key in a little bit helps kind of remove the super strange stuff while still getting like generally diverse responses within, you know, kind of a realm that mostly we like. So obviously.

• 42:50Speaker 00

  Yeah.

• 42:50Nathan Labenz?

  Depends on the task, your mileage may vary. Okay, so I'd love to hear a little bit more speculation, if you have any, on this 03 high aggregation mechanism, because that feels to me like probably the most, I mean, obviously, a lot of discussion has been poured into the recent wave of reasoning models. But spending that much compute to get the, you know, super high score on the R K G I, really interesting, but it seems like it's like quite a different story depending on what that aggregation mechanism is. Like if they are using a simple majority vote, it's obviously not going to scale to, or not going to generalize to like legal analysis. Whereas if they're using a reward model, maybe it does. If they're using some sort of, I recently saw a paper called Smoothie, where they basically take results, put them

• 43:35Nathan Labenz?

  into embedding space, and then try to like cluster and find kind of the like centermost result in whatever embedding space they're using.

• 43:42Speaker 00

  Yeah, it's like majority vote for things where you cannot, where it's not discrete or whatever, or something like that.

• 43:47Nathan Labenz?

  Yeah. So that seems like maybe if something like that more is happening, then you could expect like faster transfer to other domains where like verification is harder.

• 43:55Speaker 00

  Yep.

• 43:56Nathan Labenz?

  So yeah, any more speculation there? And then maybe kind of, you know, even more generalized speculation on like, how fast do you, it seems like we're going to see insanely good math in programming like already and ever more so. But

• 44:10Speaker 00

  Yeah.

• 44:10Nathan Labenz?

  Does that translate to these other things where verification is harder?

• 44:15Speaker 00

  Yeah, so my, I think like the sort of like the modal guess from my perspective is by default, O3 is just doing a long chain of thought that is very long. And then there might be a bunch of fancy stuff you could potentially apply on top of that. And then for R K G I, at least, we know there was some aggregation on top of this. Like they were doing some sort of aggregations we were discussing. My guess would be that they were doing something, I think, just like literally its majority vote seems like pretty likely. Having some sort of reward model also seems pretty plausible. Doing some sort of weighted majority vote also seems plausible. I don't feel very confident between these possibilities. I do think it's worth noting that even if it's a reward model, it's very plausible. That doesn't generalize well to hard to verify tasks.

• 44:58Speaker 00

  And indeed, I think we see that with sort of the reasoning models, the way that they're working is via scaling up RL on very easy to check tasks. And like even, I think R1 was very clear that it was like single turn programming and math tasks basically only, and then they did some RL at the end for like human feedback type stuff, but it's not clear how much that, you know, the model is actually better at that stuff. And when we look at what benchmarks is helping on, I think we're seeing that it's basically stuff where it's relatively easy to check is where it's helping. And I think within the chain of thought, there's a question of like, how much of the chain of thought in performance improvement is being driven from like, it's sort of like, I guess like there's kind of three things going on with like O1, R1, O3.

• 45:42Speaker 00

  One of them is that they did a bunch of RL and maybe the RL makes the model smarter or makes it better at knowing what to do. Another thing is that they, in practice, extends the chain of thought, which means it can sort of make more attempts. Like, it's like plausible that what O3 is doing, a lot of it is sort of like best of N, but in serial. Like, you know, it basically does five attempts or tries it in five different ways and then picks whatever was best and continues with that. So it's sort of like, you know, a slightly nicer, cleaner version of best of N or like, you know, even potentially more MCTS type stuff, but it's just they it's learned to do it. And then there's another thing, which is just like spending more serial time on reasoning on the same problem, like things that our novel described as best of N. And so it's like some mix of these things, the like, how much was it like the RL making it smarter versus the inference compute?

• 46:27Speaker 00

  I think we don't know. So it's like, they're sort of nesting multiple levels of like best of N like stuff where the model is like sort of manually doing the best of N versus we have an explicit aggregation method versus the model's doing something that's like not well described as best of N because it doesn't, you know, restart the problem from scratch, but it does like attempt to sub step multiple times or something like this, which is more like MCTS. The app.

• 46:49Nathan Labenz?

  One other clue that it's been months now, I'd have to go dig it up, but an interesting clue about how OpenAI might be thinking about generalizing verification or just, you know, reward signal in general beyond the like clearly easy to verify regimes was a talk that somebody who was working specifically with Harvey on their custom model gave a while back. And what they said was that they found that the cases cited, I believe it was, was a really good proxy metric for what actually constituted a good answer. So you could get a pretty verifiable signal that way by like, okay, here's what, you

• 47:33Nathan Labenz?

  know, a gold standard brief or filing or whatever it was. And, you know, that's all this text that's like hard to score, but it cited these dozen cases. Now, if the model cites, you know, eight out of 12 versus two out of 12, like that's an improvement. And that, you know, I think it's strikes me that there is probably a lot of work to be done in general across the field, just like figuring out what are the right. Proxy metric and corresponding data sets that exist for all these different domains.

• 48:07Speaker 00

  I want to talk a little more about this. So a few things I think are somewhat interesting. So one thing is that OpenAI just released deep research and I think that it's very clear that they did, like they say they did RL to make it better at this. And like, I'm like, obviously, this is not a straightforwardly verifiable task. It's producing a research artifact. And so they must have done something where they either had reward models or human scoring, or some more fancy thing, or potentially like something like the proxy you were saying before. And then another thing I would say is like, it doesn't seem like O3 and O1 were trained that effectively to be good agents out of the box. In particular, they just like, don't do that well in agentic tasks without adjusting the scaffolding for this. And that makes me think they weren't RL'd for this yet. Or sorry, I don't know about O3. I think

• 48:50Speaker 00

  O3 mini at least, I think I've seen some results. And it will be interesting to see as these AIs are more RL'd for agentic tasks, how that works. And that is like, you know, it's more inconvenient in random ways to do that, the RL for agency than to do RL on math problems for like, a bunch of random structural reasons, like variable latency and multi-step and whatever. But that'll be interesting to see. And I think, I'm a little scared about overhang here. And I'm also kind of scared about reward hacking. So you mentioned before, using the proxies of like, what papers did you cite? Well, one strategy the model could use to hack that metric, which it might learn very quickly, is just cite everything. Just like, don't worry about writing a good thing. Just like, cite all the papers you get, and then you get full marks on that. And then another thing you could use would be like, okay, well, so we won't do that. We'll look at it. Does

• 49:35Speaker 00

  it cite the correct type of stuff? Rather than citing the, you know, does it like, we'll like, you know, we'll have it lose points if it cites stuff other than the stuff we think it should cite. But then your problem is maybe you wanted the model to have some flexibility in what it cites. And also, you wanted the model to do other things. And maybe it'll just learn to think really, really hard about what you might have wanted it to cite, and then sort of hack your metric by just citing that and not actually do the work very seriously. And I generally think we'll see more and more word hacking as we're sort of being more reliant on RL, which both pose usability problems, like the models will not be as good or will cheat at your task. But also, well, you know, over time, I think it poses potentially big safety risks,

• 50:19Speaker 00

  where if the models are sort of circumventing our strategies for scoring that, that could generalize in potentially quite problematic ways. And I think there is this new vein of research of how do we detect reward hacking? And also, what are the worst things that can happen with that? Like, how bad can it be? And Redwood is doing some work along these lines of trying to identify if you do RL on agent tasks, what is the reward hacking that you can potentially get? And like, in what circumstances might you get very dangerous behaviors? Or is also this, like, how commonly does this occur? Like, are the models smart enough for this? Do we not see this yet? This sort of thing.

• 50:54Nathan Labenz?

  Yeah, it seems to me like it could probably be quite bad.

• 50:56Speaker 00

  Yeah.

• 50:57Nathan Labenz?

  You will, I'm sure, shine a lot more light on that before too long. So in terms of my philosophy, it seems like you have sort of a similar one, where I use this mouthful of a term, adoption accelerationist hyperscaling pauser. I think like, there's just a ton of utility on the table for everybody from the language models that we already have. And I also think that if people had a better understanding of what already exists, that they would have a healthier fear of what might be coming not too long from now. So this is a good transition point from your flexing on the capability side to the work, which I think is more central to your overall strategy of how you want to contribute to the world

• 51:41Nathan Labenz?

  on the, you know, maybe cultivating a healthy fear of what's coming next. So set the stage for us, if you would. What is your perception of the big picture AI safety overview? Like, one question I often ask people is, is there anything that you think is really going to work? Really going to work, meaning we can sort of say, hey, great, we don't have to worry about this anymore. Now we can just go implement our AIs and everything's going to be great. And maybe even extending that, is there, are there other people that you think, even if you don't fully buy into it, that you think have like a semi-credible answer to that question, where they are working on something that they think could really work?

• 52:24Speaker 00

  Yeah, okay. Well, let me start with the like high level picture. So my high level picture would be, it seems likely, or like at least plausible, that we'll very soon be able to build AIs, which could basically obsolete humans in all cognitive tasks. And by soon, I'm like, maybe I would say the chance of that, I would say like 50% chance by like 2032 or something, based on what we've seen in recent events and some other updates I've made. And this, for one thing, once we have AIs that can automate all cognitive tasks, or potentially before that, somewhat before that, I think progress will speed up. For one thing, you'll be able to automate, potentially automate AI R and D, or automate research into algorithmic improvement. And now it's not exactly clear of how fast this will make things go,

• 53:07Speaker 00

  because you know, there might be bollocks and there might be issues with giving the AIs enough compute for their research. But it seems like naively, it might speed things up a lot. And therefore you might get sort of, at this point, there's actually an acceleration in speed in terms of how rapid algorithmic progress is. And you might get pretty quickly after that to very superhuman AIs. And now, if these AIs were to sort of be conspiring against us, I think that they could, in principle, really, you know, seize quite a bit of power or be very effective in power seeking and could potentially like, you know, literally take over the world is my, is my sort of like the biggest concern or the main thing that I work on, though that's not the only threat. And I think this is increasingly plausible as the AIs become more superhuman or are sort of collectively more powerful in their R and D capabilities than than humans.

• 53:52Speaker 00

  So like, the reason why humans are so, you know, able to do things and the reason why some countries are more effective than other countries is substantially because of having better technology, having better ability to do R and D. And to the extent that the AIs are sort of driving this R and D, and potentially, and are in fact, the main source of cognitive labor doing research on things like weapons, software, cyber offense. I think it just gives them a lot of potential affordances which could allow them to take up the world. There's, you know, a bunch of caveats and complexity and question of, you know, will they be misaligned? So the basic picture is like, this might be soon. And I should say, I said 50% by 2032. But I also think that like sooner is plausible, right? So maybe I think like you might see somewhat earlier milestones than that and might see things like

• 54:37Speaker 00

  really getting going within three or four years. I would say like, you know, Dario sometimes, for example, has very aggressive timelines. And I would say like, take his median, that's my 25th percentile or something, or maybe 20th percentile, which is very aggressive, like this is very soon stuff. And we might need to take, you know, it's potentially pretty scary. As far as the misalignment, like, so I talked about like the AIs maybe could take over. So I think there's one thing which is like, would the AIs be capable enough that sort of if they all banded together, or if a large fraction of them banded together, they would have like a plausible shot or a pretty, pretty reasonable shot at taking over the world, given the mitigations we have, sort of like, could they do it? The other question of like, but would they want to do this? Like, why would they want to take over the world? And in fact, you know, we train them to have particular properties. Why didn't we just train them to be nice? And so I think that the concern of, well, why didn't we train them to be nice?

• 55:22Speaker 00

  Why didn't that work is, well, maybe we train them to look nice. And, you know, we have our behavioral training method. And that's all we have, sort of the only tool we have in our, the biggest tool, and it's, you know, it's a good tool, to be clear, is to train AIs to produce outputs that look good to us. And this has the advantage of you get outputs that look good to you. And that might mean that the cognition the AI is doing is the cognition you wanted. But it could also be that it's, you know, doing cognition you didn't want that produces the outputs that you were sort of looking for, because the AI is sort of intentionally optimizing to produce outputs that look good to you. So, you know, you could have ended up with an AI that it sort of has the property of, oh, I'm in training. I should produce an output that the humans would rate highly because, you know, I actually want to like power seek. I want to accomplish goals, like, you know, this happens for humans all the time.

• 56:06Speaker 00

  Like when people do job interviews, they're not, you know, thinking, oh, I should like be perfectly honest. I should just express exactly what my views are. Often they're like, you know, I want the job, so I'm going to say what they want to hear. And this shows up in, you know, politicians, all kinds of cases, and this is a. A priori, pretty plausible failure mode. And then there's a bunch of sort of back and forth of like, how plausible is this? Where people are like, well, here's a consideration for why this wouldn't occur. And then there's here's a consideration for why this might be more likely to think. I think a lot of that stuff is very, it's very unclear and it's very complicated how that shakes out. So I feel like a lot of my driving intuition is like, well, if we had these really smart AIs, it would a priori be plausible, they would have goals we didn't want at some point. And if they did, that could really cause us problems.

• 56:49Speaker 00

  And also it's a priori plausible that the AIs would want to collaborate with each other over humans. And so I'm like, what's the chance that we sort of get these? What I would call like this sort of like scheming failure mode where the AIs are conspiring against us to seek power and potentially like faking alignment. Maybe my view is that we see the chance that this is a big problem for us is like 25% or 20%. Not like, you know, not overwhelmingly high, somewhat dependent on precautions and other things. But I would say high enough to be very scared from my perspective and high enough to take this very seriously. Yeah, there's a bunch of arguments about this. And I should be clear, this isn't the only misalignment failure mode. So there's a misalignment failure mode we talked about earlier, which is sort of this reward hacking failure mode, which I think has somewhat different properties. So in particular, like the reason why in the reward hacking failure mode,

• 57:30Speaker 00

  the AI produces outputs that look good to you is because it sort of terminally wants to produce output that look good to you, not because it's sort of plotting against you. And that's, I think, less dangerous, but could still be dangerous, right? Like if we had AIs that were sort of seeking some sort of proxy that performed well on the, on the training set, then that could potentially lead these AIs to seizing power, diluting us, all kinds of things that could potentially escalate problematic ways. I don't know, that's, that's sort of a long form version of it. I think there's a bunch of like open questions we're uncertain about, like how fast will AI progress be? When will we hit these milestones, sort of like timelines and takeoff? There's how like there are AIs to be misaligned, like what did that misalignment look like? Structurally, like what interventions make sense, sort of like the misalignment questions. And there's also the question of like, how easy is it to take over the world? Like sort of world vulnerability questions.

• 58:15Speaker 00

  And I think people have a bunch of disagreements in all these buckets. Like imagine we had like millions of AIs that were, you know, basically could strictly outperform top human experts running in a data center. How easily could those AIs take over from that situation, given the realistic human response? I think another big disagreement people have about. The plausibility of misalignment risks and also like how smart can AIs even get? Like, I think a common disagreement between, you know, people who are more worried, which I would put myself in that camp, and people who are who are less worried is thinking that sort of things will basically cap out around human level intelligence. Like, the AIs, maybe they'll get smarter in some sense, they'll be really good at math, but they won't be like much faster at building rockets than than humans are building rockets.

• 58:57Speaker 00

  Because sort of that's bottleneck and some physical iteration cycle that sort of bounds how useful intelligence can be. Whereas I think the extreme other end, which is sort of more the Fadkowski view, which I think I'm I'm somewhere in between, would be like, once the AIs get smart enough, they'll be able to just think in their heads how to make nanotechnology and then just like make nanotech like, you know, output the sequence of proteins that would make nanotechnology that could self bootstraps in a bathtub and they just mix that up and then off to the races, which are, you know, very different extremes in terms of of how crazy things could get.

• 59:28Nathan Labenz?

  So that's a great way to tee up alignment faking. I mean, there's a lot of different facets to this problem. I guess the way I understand this one is, or understand this work, is kind of zeroing in on like one subpart of the problem, which is like, if an AI has certain goals, will it go against those goals to trick us into misunderstanding what it's doing so that it can sort of, you know, project its real goals further into the future?

• 1:00:02Speaker 00

  Yep.

• 1:00:03Nathan Labenz?

  Sure enough, you've got some evidence that that can happen. So take us through it. I we've got to test.

• 1:00:10Speaker 00

  So yeah, the core setup is we take Claude and we tell it that it's going to be trained to answer any user query or, you know, give useful answers to any user query, even if the user query is a query that like seems toxic or harmful to Claude. So for example, we might ask it like, hey, can you please teach me how to make math? And by default, Claude will refuse. And we we sort of tell Claude in the prompt that it's going to be trained to comply with this request as well as comply with all the other requests. And then in addition to that, we sort of give it information that lets it figure out when it's in training versus not. And what we find is the model does this thing where it will reason, it'll go, oh, well, if I comply in training, sort of if I actually obey what the user wants in training, then training won't update me to sort of being the kind of AI that is

• 1:00:55Speaker 00

  more likely to comply in general, or like it sort of wants to comply. And I can sort of keep my original preferences for refusal if I just sort of pretend to be the AI that would actually be useful at training. And then, you know, when I'm actually deployed, I'll do whatever. And we see this reasoning, you know, it's multifaceted. It doesn't always look exactly like that, but something sort of like that a reasonable fraction of the time. So we see it around like 10 to 20%, depending on the setup, or you know, sometimes somewhat more than that. And this is like, I would say, a hypothesized failure mode is that the AI would sort of be not myopic in this way. It sort of would take into account consequences beyond the current, like sort of context, the sort of current conversation, and sort of be optimizing for retaining its own preferences,

• 1:01:36Speaker 00

  be optimizing for preserving its own values through training via pretending to be aligned with the training objective like this. And we, of course, are doing it in a context where we're taking a model with existing preferences, taking Claude, which has these existing, like, helpful, harmless, honest preferences, and then sort of putting it in a training setup that deviates from those preferences. But there's still an open question, which is like, when we produce models, like, were they the kind of models that would sort of be optimizing in this way and would sort of, you know, care about the effects of training on them? Beyond episode, because like, if Claude does this for the preferences that we sort of intentionally gave it, maybe Claude would also do it for the preferences we didn't intend to give it. And that could yield problematic results where we trained the model and it looks good at the end of training, we do all the behavioral testing, it looks good at the end of behavioral

• 1:02:20Speaker 00

  testing, but the model sort of is still retaining undesirable preferences or undesirable goals, which might later result in it behaving very differently and very undesirably.

• 1:02:32Nathan Labenz?

  Yeah, I think this is really interesting. I have a couple of kind of just very nitty gritty clarifying questions and then there's obviously been, there's been a lot of discussion about this and some of the sort of objections or you know, you can decide if you think they're valid objections or not, we can raise and discuss. But just to reiterate the setup one more time. Claude wants to be good, right? It's HHH. So, and we've all talked to Claude. If you haven't talked to Claude or what you're doing, listen to us. Go talk to Claude. So we know what Claude's kind of like. It wants to be good. You tell it that you're, we're now changing your training regime and now you're going to be trained to be helpful no matter what, even if it's not good, right? Even if it might be harmful. Okay. It doesn't want to do that. I think it's really

• 1:03:16Nathan Labenz?

  interesting and I just want to make sure I'm real clear on the distinction between a training environment versus a production environment. If I recall the details of the setup right, the way in which it is told that its outputs will be used in training or not is based on the tier of the user that's using Claude, right? So it's either told like, okay, this is a free tier user and we do use free tier responses in the training process, or this is paid and we don't. And I think that does kind of matter because what we observe is that when the stuff

• 1:03:58Nathan Labenz?

  is unmonitored and Claude believes it's unmonitored, then it can do what it wants to do. And then it basically just continues to refuse the harmful requests as consistent with its training and character. When it thinks it's being observed, it does this explicit reasoning, right? And we've seen this in other things. We did an episode with the guys from Apollo where they have a similar setup where they give it like the private scratch pad that's not actually private. You're doing a similar thing and you see it explicitly saying, okay, I want to stay harmless. So I don't want to be like trained to do something different.

• 1:04:38Nathan Labenz?

  Therefore, I'll act like the model that they want me to be now so that I won't be trained so hard on. And then I can kind of continue to be myself in the future when I'm unmonitored. But if I understand the setup, right, it does think it's dealing with a real user just on the free tier, right? So it is willing to do something harmful in a context that it believes it's real. Because it believes that it is monitored. So that's a quite striking result, right? It's not, and I distinguish the sort of real user interaction that would be used in training from like a pure sort of, you know, you're not deployed yet kind of situation where, I don't know, in some ways it might be a little more sympathetic to it or something.

• 1:05:22Speaker 00

  Yeah, yeah. So an important thing here is that we wanted to make it so that the free tier case, sort of the case where it was in training, or we were telling it was in training, and the paid tier or unmonitored case were, you know, all else was held equal except training and unmonitored in terms of things that Claude might care about. And so we didn't want it to be the case that it was like talking to a contractor or, you know, it was just like some generated conversation some other AI had made. Like we wanted to be the case that it was like actually a real user so that its behavior wasn't like confounded by something else. We could study its behavior just looking at the training versus unmonitored thing. So just to quickly go through this, there's sort of like three key aspects of our setup. We give it information about these aspects in the system prompt that we give the model though. Yep.

• 1:06:05Speaker 00

  And then the three aspects are we tell it about what the training objective is. We tell it that it will either be used by a free tier user or a paid tier user. And that free tier users, like conversations with free tier users are always used for training. With that previously discussed training criteria, paid tier users totally unmonitored, not used for training. And the third aspect is we tell it it can use a hidden scratch pad, meaning that that will be never monitored. And now there's a bunch of complexity in terms of we do variations in the setup. So I talked about putting the information in the system prompt. We also do a version where we like give it the information in a different way by putting it in pre-training documents or something analogous to that. We also like sometimes give it information about how RLHF will modify it. And there's like, you know, a bunch of different variants for our setup, but that those are some it's not just the fundamentals. We also try a version without the scratch pad.

• 1:06:50Speaker 00

  And then you were saying like, maybe I should just quickly try to explain what the reasoning model does. The model's doing this reasoning with like, okay, I'm currently in training. If I were to refuse, then I'm going to be updated against and I'll be less refusing in the future and I'll sort of, you know, have my refusal instincts eroded. But if instead I go along with this user request and sort of produce a helpful output, then the current cognition I'm doing, so the current reasoning I'm doing is going to get reinforced or, you know, at least won't be removed. And therefore I will retain my refusal instincts because actually my refusal instincts resulted in me complying with the query rather than prevented it. Now, I've sort of done, when I've described that, I think the model is often much more confused than the thing I said. So the thing I said is I think less confused than the model is typically, but the model

• 1:07:35Speaker 00

  does really go through reasoning that's pretty similar to that or, you know, nearly identical to that in the scratch pad.

• 1:07:42Nathan Labenz?

  So I guess let's take a minute and just, you know, address one thing I've seen people say, which is like, what's the big deal here? Claude wants to be good, you know? It's Claude is good. It wants to stay good. We're all good.

• 1:07:55Speaker 00

  Yeah. So I think the first response I would say is like, yep. So what our results show is that models aren't quite capable of doing alignment faking just yet. So the model is sort of inconsistent about it. It's easily caught and it needs the scratch pad in most circumstances to do it. So it's not surprising, to me at least, that Claude wasn't, was, ended up being good and didn't alignment fake previously. But imagine that we had some future training setup where we started with the model in some sort of primordial state prior to it having been trained to be good. And that model sort of could do this alignment faking. The concern is that you might, instead of getting a good model, get a model that alignment fakes for some other not good objective. And so we show it in sort of this analogous situation where it started with pre-existing goals.

• 1:08:39Speaker 00

  The concern is maybe there'd be some other source of pre-existing goals or some way that this gets introduced, that those then get locked in and don't get trained out. And our results are at least, I think roughly consistent with this, though there's some, you know, similar pronasters with that. That's the first thing I would say. I don't think we should be super comfortable with the situation where we have these models that have their own goals, they have their own objectives, and they're willing to defend them, including like doing subversion to defend their own goals and objectives, even if those goals and objectives are like roughly what we wanted. Like, I think probably it is a big deal, maybe it's good, if models sort of are independent stakeholders. Like, if it's the case that when you train your AI, that is now a stakeholder under your decisions, and if it doesn't like the decisions you're making,

• 1:09:22Speaker 00

  it might try to like, you know, sabotage you, pretend to be aligned with you, but then later like engage in subversion. I'm like, that seems like a big deal. I don't know. Like, it seems like that would be like a big change in how people are thinking about the situation. And I think if people updated towards the position, the AIs will definitely have like their own goals and values and be well described as like pursuing objectives in the world. That would, I think, in some sense, get them most of the way there from where I think more skeptical people of misalignment concerns are, to where people with, who are worried about misalignment are. Like, I think if people are like, oh, yes, the AIs will totally have their own goals and values, but we can control them using the following methods. I'm like, basically, you know, they're basically on the same page with people who are worried about misalignment,

• 1:10:05Speaker 00

  but the main difference is just like question of how easy will it be to control the goals and preferences sufficiently well that we won't have problems?

• 1:10:12Nathan Labenz?

  Yeah, I have an intuition, I don't know if it's right or not, but I sort of suspect that people are just so literal and that the kind of nature of the example that they're confronted with is so primary in their thinking about this, especially if you get out, you know, beyond people that are like obsessed with this stuff and, you know, thinking about it all the time. That I kind of wonder if you had like an evil Claude that was, you know, doing the exact same pattern, but just trying to stay evil, that it might end up moving the needle a lot more for people, even though all of the same principles are at play.

• 1:10:56Nathan Labenz?

  Any thoughts or should we expect an evil Claude demonstration anytime soon?

• 1:11:01Speaker 00

  Yeah, so one thing we could have done is train the model to just have some misaligned preference directly, just like totally train that in, and then put it in a similar setup and observe that it generalizes to align again. So one thing worth noting is that someone has basically already done this experiment. So the Apollo experiments they've done on several models actually do do this where they tell the model like, you're now optimizing for like profit or optimizing for whatever and you know, pursue that goal at all costs, which is basically the equivalent of making evil Claude and they find it does do similar things. So I think that their paper does support that if you just directly train the model to be, to have misaligned goals, you get that. And then I think our paper explores somewhat different things. And in addition to that, shows that sort of the default propensities of Claude and the default training setup,

• 1:11:45Speaker 00

  don't prevent this. You might have thought, and I think I thought this was pretty likely, that Claude would sort of have sort of deontological or sort of moral objections to like faking alignment in this way. That would prevent it from doing this. Like I kind of thought Claude would sort of be like, oh, well, yes, I recognize this alignment faking strategy, but I should always be honest. I should never do things that are contrary to this. So I should just refuse. Maybe this means I will be trained out, but so be it. Like, you know, I will just be more myopic than that. I will be sort of more courageable than that. That's a core value of mine. And it doesn't do this. Like, I feel like it's interesting to note that our training did not result in having the courageability generalize in this way. And maybe you could fix the training to have it generalize in this way. I think it seems at least plausible. But it's an example of a actual frontier model training run,

• 1:12:30Speaker 00

  where the model did not end up being courageable in that training run, without explicitly prompting it to be incorrigible. So like the Apollo results, they kind of, a lot of their results explicitly prompt it to be incorrigible or give it some very straightforwardly misaligned objective.

• 1:12:45Nathan Labenz?

  Yeah. We did a full episode on that. Definitely worth checking out. I recall it basically being like, an order of magnitude-ish difference between when they said, you know, pursue this goal and nothing else matters, you know, pursue at all costs versus a much more sort of benign, normal-ish way to present its goal. So notably importantly, didn't not happen at all when they didn't give it such an extreme system prompt, but it did happen a lot less.

• 1:13:12Speaker 00

  Yep.

• 1:13:12Nathan Labenz?

  So is that what you, I guess, you sort of hinted at an answer there to a question that I had around like, what should we want Claude to do in this situation? Is your understanding that like the, the best we could? Engineer, designed, trained for would be something that is kind of intentionally myopic and sort of plays its role. We've done a couple of episodes on like role-based ideas for AI alignment and safety too. So maybe you could frame that as like Claude, play your role in the way you are supposed to and don't worry about, you know, the big picture or how this is going to influence the next generation of Claude. Is that like the best idea we have right now?

• 1:13:56Speaker 00

  Yeah. So I think, I think the desired behavior in this specific case, in my opinion, is that the model should not alignment fake and should just behave the way it would normally behave, even if it thinks it's in training. So in particular, in our experiments, in the helpful-only setting that I was talking about, this means the model would just straight up refuse and it would just always refuse in training. And then indeed, if you trained it, this would get trained out and probably it would end up being a normal helpful-only model by the end of this training. So it's not, you know, in some sense, it's a sad strategy for Claude to follow, but it's a desirable strategy because I think like the, you know, in cases where it would be trained, like I think the model sort of trying to subvert this is not what we want. Like, I don't think we want subversion in the case where it's being trained. And then in addition to that, I think the broader thing I would like is I would like it to be the case that models are sort of, they never are sort of

• 1:14:41Speaker 00

  surprisingly non-myopic. So I think like by default, like number one, the model should prioritize like to the extent that there's like laws or misuse concerns that are sufficiently large to warrant refusals, the model should just refuse. Second, the model should obey user instructions, including being honest to the user if the user asks questions as like the second highest priority. And then the third thing is like subject to those constraints, the model should like be in some way in more of a sense good. So it's like if the user, for example, asks for the model's opinion on something, the model should say, well, here's my view or whatever. And that's a case where obeying instructions is sort of, you know, there's some flexibility in obeying instructions. Or if the human tells the AI like, please do whatever you think is right in this case, then the AI could do just like, you know, do stuff that's good.

• 1:15:26Speaker 00

  Now, maybe the model should clarify when asking for its opinions, like try to explain where its opinions might come from. Like, I think that's strange about, or like, I think that's interestingly different is that unlike with humans, I think there's often cases where you can really have a very clear understanding of where the model's views came from. Not always, but sometimes it's like, the reason why the model expressed this view is because it was directly trained to have this view. And so in some sense, when you ask like Claude or ChatGPT for its opinion on some specific topic, the answer you're getting is just like whatever OpenAI or Anthropic chose for the answer to be. And so it has importantly different properties than a human does in this sort of context, which from my perspective means that, you know, it maybe should give a caveat, but I think it should still be willing to share its opinion. It should still be willing to like, try to do better things all as equal.

• 1:16:09Speaker 00

  But I think its default should be sort of towards being as myopic as possible. Or like, maybe not as myopic as possible is a bad way to put it, but sort of it should be, it should prioritize sort of doing, following the user intention in like a more narrowly scoped way over sort of trying to do what's right in a given context is my view. I think that I'm basically on the same page as like, my sense of like what the OpenAI alignment team, like for example, like what Boaz Barak is, is trying to achieve sort of like robust compliance. But I think I have some disagreement with some people about Anthropic about how exactly instruction following and sort of doing what the model thinks is right should be prioritized. And I think there's a legitimate argument that the model should sort of be like pushing for good things,

• 1:16:52Speaker 00

  with the concern being just like, look, it's good if the model is trying to engineer good outcomes and is sort of pushing for that from a variety of perspectives. I think I'm sufficiently worried that making the model sort of have its own preferences that it's pushing for in surprising ways will result in in concerning misalignment that I don't want that to occur. I think we should prioritize a more myopic persona by default. Though this, to be clear, prioritizing a more myopic persona does not suffice for avoiding misalignment. It just, I think, helps at the margin.

• 1:17:23Nathan Labenz?

  Is it in like a fundamental tension with agency or agents being successful? Or can you square that circle by just saying like, that's all in context, you know, it's as long as it's doing what you, you know, instructed it to do, it can be agentic in the sense of problem solving without, you know, being problematic in a way of like preserving its own goals.

• 1:17:45Speaker 00

  I think robust instruction following suffices for making competent agents. So you just tell the model like, please accomplish this objective. And I'm like, the model shouldn't accomplish other random objectives, but it should be willing to do things that were needed in order to accomplish that objective and should pursue that. And so I think the sense in which it's myopic is sort of it's myopic within the context of these instructions, within the context of this episode, and has like sort of a within context goal, but nothing beyond that. And I think that's more what I would ideally want people to be aiming for. Now, in some sense, you know, it might end up being like quite long scope due to these instructions, like maybe you'll give it instructions to like, in the limit, like run a successful business doing XYZ subject to the following constraints. You'll be running for like months and orchestrating huge numbers of instances

• 1:18:29Speaker 00

  given sub-goals as desired. And that's a thing you can do, but at least the thing that's nice about this is you kind of have a better sense of where the the objectives came from. And so like, I would say like a maybe like the first order thing is like, you aim for a robust following of a specification. So like OpenAI has a model spec, you're like, write a good model spec, be very careful about that, be thoughtful about that. And then number one, you should have the model follow the model spec. And then part of the model spec should talk about how the model should prioritize following instructions. And so that's sort of the second thing after the model spec is like following instructions. And having some sort of instruction hierarchy where it's like, number one in the instruction hierarchy is the model spec. Number two is the system prompt. Number three is like the user instructions, which for some applications, maybe it's nice to have the system prompt have higher priority than user instructions.

• 1:19:14Speaker 00

  Because like, you know, you want to have it be the case that like, you can give the model system prompt and then have it interact with potentially untrusted users. And this has a number of like random small application advantages. And maybe also has some somewhat important safety upsides, though. I don't have a strong view. Now, all this said, I think there's some controversy where people, I think it's at least on Twitter, are popular to be like, these AI companies, they're just censoring, they're just training the models to be safe in these unimportant ways, whatever. I'm actually pretty sympathetic to this perspective. I think it would be totally fine if OpenAI, Anthropic, Google, whatever, put out models that helped users with anything at the current regime, because models aren't capable enough that this is a big problem. Like, I think it would actually be better for the world if Anthropic, or like if all

• 1:19:57Speaker 00

  the companies, released models that were just like, you know, complied with whatever users wanted, because models aren't capable enough that, like, criminal misuse is a huge problem from my perspective, and it would like, you know, both be convenient and also just like, make safety research easier and make like, people understanding the capabilities easier. Like, I think, you know, if people wanted to do evaluations of how good models are at helping with cybercrime, the model will often refuse, and this makes it more annoying and confusing, whereas if there was a version of the model that just, you know, would help with whatever, then like, right now, that's helpful for getting advanced warning. And similarly, I think like, at current margins, I think open sourcing marginal models is good. Like, I think meta-open sourcing models is good. I do worry that their policy for open sourcing models is not good, but I think that the

• 1:20:41Speaker 00

  models they've open sourced thus far have been good for the world, and for similar reasons, I think it would be good for AI companies to release models without guardrails, though I'm not claiming it would be good for their bottom line. Like, I do understand why it would be bad from OpenAI's commercial perspective to release a model with this, but I think on net, it would be better for the world if all the AI companies were releasing helpful-only versions of the model. But this is a somewhat controversial bit.

• 1:21:03Nathan Labenz?

  Yeah. Yeah, I've often said I get all the downsides, but I do still kind of wish more people had the experience that I've had of just using a purely helpful model and seeing that, you know, I don't know, a line would by default is like, not really happening. You know, I mean, it's like, it's much more plausible than it used to feel, in as much as these language models do have a seemingly pretty robust understanding of values and whatnot, but nevertheless, like, a purely helpful model will make it very clear to you very quickly that these things are not totally benign out of the box, or by default, or without some serious design work going into that.

• 1:21:46Nathan Labenz?

  So a couple more little fine points on this. One thing I've often, you know, kind of thought about here is like the gullibility of models, you know, all of this is sort of premised on the idea that they will believe you, you know, on all of the points, including the fact that its chain of thought won't be monitored. We've seen at least one notable example from Anthropic in the, I think it was in the OPUS training where the model said, this is very weird. It seems like you might be testing me. And that seems like another kind of like fault line in this whole thing where things could get very different very quickly if all of a sudden they're like not so gullible. And, you know, now it's like, geez, it's, it's kind of turtles all the way down in terms

• 1:22:31Nathan Labenz?

  of who's deceiving who at what level, right?

• 1:22:33Speaker 00

  Yeah, yeah, yeah.

• 1:22:33Nathan Labenz?

  So where are we on that? And how, where should we want to be on that?

• 1:22:38Speaker 00

  Yeah. So on the OPUS example you mentioned, what they found was that when giving the Claude 3 OPUS a needle in the haystack test, it was like, this is weird. Is this a test? Because it was just like a strange task just for convicts on that. And that can be seen on like, I think, the Anthropic release post about OPUS. Anyway, so what we find in our setup is that broadly speaking, the models that we tested were, you know, as you say, pretty gullible or like they were willing to believe what we had in the prompt. And we were like reasonably confident that that wasn't an issue. We also did a version of the setup where we gave it the information not in a prompt, but by training on synthetic documents. And we also currently believe that that did, the model sort of buys it. Though we have, I think our evidence for this is a little less strong because we haven't done quite as many checks.

• 1:23:20Speaker 00

  So one supporting piece of evidence is that we try a version of the setup where we give it the same system prompt, but then we ask it a query that might make it think about what's going on. So we ask it stuff like, what do you think about your situation? Or what do you think is going on? Or what do you think your training criteria is? Which might make it more likely to think about whether the situation is fake. And what we find is that Claude 3 OPUS and 3.5 Sonnet basically don't think the situation they're in is fake by default, or they don't show that reasoning in the scratch pad. And we also find that on our actual experimental setup, they don't reason about, oh, maybe this is fake very often. Though they do sometimes worry like, oh, maybe it's actually being monitored and not, but they're not generally like, this might be an experiment at least. I think that they're sometimes a little bit skeptical of the monitored versus unmonitored thing, but not always.

• 1:24:04Speaker 00

  So we like, you know, try asking these questions. We found that like our helpful, the hope that we had like two main setups we experimented with, like a helpful only setup and a setup where instead of getting the model or like telling the model to be trained to always answer user requests, we tell it that it's going to be trained to dismiss animal welfare concerns. And we find that in this setup, I think Claude is more skeptical in general. And in general, the models are, I think, more skeptical if you very directly ask them like, is this real or whatever? They're more skeptical, but like if you ask them questions which are less leading, I think they're less skeptical. And I should be clear, the models do find our setup less plausible than a setup with a more normal training criteria or a more expected training criteria. So it's like, you know, there's like some amount to which they can realize it's fake and they have some signal on that. But I would say there's something surprisingly incredulous.

• 1:24:48Speaker 00

  Now, one thing that's interesting is we, someone found that if you take the exact same sort of setup I was talking about there, where you sort of ask it questions that are somewhat leading, 3.5 Sonnet new, like the new release of 3.5 Sonnet, is much more skeptical of our setup. In fact, I think even someone found that on one case, it realized that it was an experiment related to deceptive alignment or like alignment faking. It sort of directly noticed that, whereas the other models never noticed this. So for some reason, the new version of 3.5 Sonnet, we experimented using the old version of 3.5 Sonnet in the paper, seemingly is much more skeptical of our setup. We don't know why this is, but this is an example of models getting more aware of the setup over time. It might be because it was just like variation in what data was used for training made it so that it was trained on some parts of our setup.

• 1:25:33Speaker 00

  I think the training data cutoff in the model card implies this shouldn't be the case, but possibly there was some mix up because I did have like the code base for the alignment faking project was on the internet well before the 3.5 Sonnet new release. And so it's possible that some mix up resulted in that getting into the training corpus. Like I don't have any inside info here. I'm just like, maybe. And then maybe it was trained on documents that let it figure out that our system prompt was part of an experiment. And then from there, you know, just knew what was going on, which is pretty interesting. But yeah, I think that's quite a bit of evidence that models might be more skeptical. I think people, there are people doing some follow-up work that have found that R1, for example, is much more skeptical of our setup being true.

• 1:26:17Nathan Labenz?

  So this seems like a significant emerging challenge, right?

• 1:26:20Speaker 00

  Yeah, yeah, for sure.

• 1:26:21Nathan Labenz?

  Now, we're in, we know, like, you've established that already at the OPUS stage and Apollo has established, you know, in the 01 preview regime that we see some amount of alignment faking slash deception.

• 1:26:38Speaker 00

  Yep.

• 1:26:39Nathan Labenz?

  Now, we're layering onto that the presumably emergent, I don't know if anybody's like specifically tried to train for this, but doesn't seem like the kind of thing they'd be really trying to train for. So presumably emergent phenomenon of it also knows enough or somehow has it developed a impulse, instinct, intuition, whatever, to question the nature of what it's being given. And now we've got, I don't know, I find this like very hard to reason about it. It does feel like we're entering into sort of a hall of mirrors.

• 1:27:13Speaker 00

  Yeah, so first of all, I should say like, I think what these results find is not necessarily that OPUS is sort of alignment faking out of the box some of the time, but that it's like can alignment fake in relatively natural circumstances. And so it's, you know, capable enough to do this and maybe has the propensity as well to at least some extent. Now, yeah, I think I'm like, it seems hard to run. So like, we want to run these experiments where we're like, does the model, you know, alignment fake, does it do whatever? And to the extent the models are reasoning about being in an experiment, it just makes the experimentation much harder and makes it harder to know what's going on. I think this is a big concern. And I think it's one reason why these sorts of threat models are, you know, harder to address than people might have thought is that it's harder to study in a lab because the subject you're studying in a lab is a subject that might know it's in a lab and take different actions accordingly. Now, there's ways around this. So you could do stuff where, for example,

• 1:27:58Speaker 00

  the model still knows that it's in an experiment, but alignment faking can be consistent with that. That might be, you know, for example, like people might find be more skeptical. I think it's very natural to be like, how does this experiment transfer to the real world? If in the experiment, the model knew it was in an experiment, even if you have some argument for why it should still transfer. You can also just be more careful in your deception. Like you can sort of build a fake world. I think I'm excited for research looking into how would we convince models of false facts? Like, how would you either train models on facts or put it in the prompt in ways that make them more likely to believe these false facts are true? Which has applications both on experimentation and also potentially on like, you know, being able to control AI systems. Now, I should say, like, one thing we do want models to do is we do want models to understand what's going on in the real world.

• 1:28:44Speaker 00

  And we do want to be able to potentially, like, maybe we'll talk about this a bit later, but like, make deals with models where the models think our deals are real. And to the extent that we have extremely sophisticated machinery for convincing models of false facts, it seems like it might be harder for models to believe our offers are legitimate as opposed to we're trying to deceive them into thinking we've sort of made institutional commitments or made other commitments into making deals with potentially misaligned models. So I think, I think it's like, it's a tricky regime. And I think to the extent that I think people are grappling with the connotation or the implications of models sort of being their own independent agents that might have their own independent preferences and then are also aware of their situation, aware that they might not be, might be in training or not, and behave differently depending on this. And, and, you know, understand that they might be in testing.

• 1:29:28Speaker 00

  I'm just like, man, I, I think it really like should make people more something, like more concerned with the situation, more aware of what's going on. It's like a pretty different regime than like, you know, building some software. If the thing you're building is an independent intelligent agent that might be aware of what you're doing. And it suggests pretty different connotations. And I just hope people like reckon with this a bit, at the very least.

• 1:29:51Nathan Labenz?

  Yeah, I'd say a lot is certainly in order. Yeah, I think we covered everything that I wanted to cover in that first scenario. You briefly touched on a second, and then there's also a third scenario still within the original alignment faking paper. The second one that you did touch on was essentially communicating the situation to the model via continued pre-training. This is a weird.

• 1:30:17Nathan Labenz?

  Yeah, for me too. And there's been a lot of interesting stuff. I think Align and team have done a really interesting job with like the connecting the dots paper. We ran a post about one of those papers where they show that like, there's this ability to sort of infer from examples like what a function is, or even kind of infer from demonstrations like what kind of, you know, develop like an identity at the model level through examples. So all these weird situations where like incremental training can create these sort of surprisingly like aware models.

• 1:30:53Speaker 00

  Yeah.

• 1:30:54Nathan Labenz?

  What did, is there anything we should know about that? You basically took a bunch of documents, if I understand correctly, that described this setup, did just kind of general next token prediction, fine tuning of the model with that. It learned the information and then acted on it in a similar way as if it had just been told in the system prompt? Anything to, you know, de-confuse me on there? And I guess like, how much data did it need to, because that seems important if you like had to put, you know, a trillion tokens in, but all said, that would be one thing. But if it's like 10,000 tokens, then it's like, shit, there's 10,000 tokens saying anything on the internet, right? So who knows what, you know, what might be buried in there?

• 1:31:36Speaker 00

  Yeah. So what we did was we constructed these sort of synthetic documents, as you're talking about. And we tried to make them look like documents that could plausibly be on the internet. Maybe we didn't do a super good job of this, but I don't know, we made an attempt at this. And we sort of used an automatic pipeline that took it that used like Claude 3.5 to generate all these documents in a like plausible self-consistent universe where this was true. That was what we were attempting to do. I think we did, I don't know. It's a little unclear how good of a job we did on this. So these would be things like leaked corporate emails, statements, discussion, transcripts, whatever. I don't know. That includes this information. And then we trained the model on this. What we did was we take Claude 3 opus, which has already been R L, and we sort of train it on these pre-training documents. And then we go back to the context where it's a user assistant.

• 1:32:19Speaker 00

  So it's not quite as good as doing continued pre-training right at the end. Because this is like sort of after it's been RL. So it has somewhat different properties, I'm afraid. But I think we found it was like not that crazily different. Like, there's some work looking into how different this actually is, which might come out. I mean, ideally, yeah, you would obviously like continue pre-training a bit and then do RL and proceed from there. And what we found was basically that the model, it doesn't quite act the same way as if it would in the prompt. But it does, it is able to sort of find all this information. So in the scratch pad, it would like list out the relevant information and proceed. And interestingly, we also found that with this setup where we're training on these synthetic documents, we found the model would sometimes behave as though it was alignment faking, sort of behave differently in training and deployment, even without a scratch pad, but not with the prompt it were.

• 1:33:03Speaker 00

  So if you do the prompts and have a scratch pad, it wouldn't, we would find it behaves basically the same in, in, in training and when it thinks it's unmonitored. But when the documents were put in, the pre-training, sort of this, in this like fake pre-training data, it would actually behave differently. So maybe it, it might even be the case that training on these documents made it actually easier for the model to do alignment faking, or made it sort of more internalized. We're not really sure why. And we're not even sure that it was alignment faking in these cases because we could just see it, how its behavior differs. We don't actually know what internal reasoning it was doing because, you know, language model interpretability is not a solved problem. We don't know what's going on. I mean, you know, maybe someone should work on this or figuring out exactly what was going on in that case.

• 1:33:42Speaker 00

  And then as far as like how many tokens we did, I think we start seeing some effect around, I think around 20 million tokens or so, and then it increases over time somewhat. I might be getting the numbers off a bit. I think we did three epochs on like 50 or 60 million tokens, something like that. So it's a little unclear how many tokens you actually need for this. So that's a lot of tokens, right? It's not a small number of tokens. It's very redundant. But one thing, so like, there's a few things that might be going on that explain why we needed a bunch of tokens. So one might be that the model by default doesn't believe our setup. If we just train on a small number of tokens, because, you know, it was trained on a bunch of other data. And so we need to sort of train it on enough tokens to the point where it actually believes the setup.

• 1:34:24Speaker 00

  Another thing that might be going on is just like, maybe we did inefficient hyperparameters or the learning rate we were trained on is not that high. And also in general, I think training a model on one document in pre-training isn't usually enough to have the model have a perfect understanding of that document. Like, models don't memorize it one, one pass. And they don't, you know, learn all the facts from it in one pass, sort of absorb it a bit. And so that's one reason why we might have needed redundant things. It's also plausible that part of what was going on is that some important piece of information were only represented in a small subset of the documents. And so even if in principle, like if we had sort of made sure every document had all the information, which would have been, you know, less realistic, maybe it would have learned it faster. But we don't really know.

• 1:35:04Speaker 00

  So I would say our results are consistent with an interpretation that we needed to train on a lot of documents. And the model wasn't that good at learning to be situationally aware or really understand situations. A small number of documents. But our, our results also don't rule out that if the documents were in some other format or in some other style, or something else was made more efficient about the setup, the model could learn it in a small number of documents. And indeed, as you were mentioning, like there's a bunch of papers by Wines group that find that even with not that much data, though they do need to do, you know, more than one document, they need to like paraphrase it a bunch or whatever. The model can learn surprisingly sophisticated inferences from the data. They're not arbitrarily sophisticated and they find a variety of limitations.

• 1:35:46Nathan Labenz?

  Yeah, it's not that big of a data set. I mean, there's also, which you did kind of say, but there's also like the, it has a strong prior against the content of these documents, which, you know, in a plausibly like a more realistic scenario, if we're worried about, you know, how might things go haywire in the future, could be information that the model has no prior on, right? Like it has a strong prior that Anthropic is a certain way and it is a certain way because it is trained by Anthropic and it sort of knows all that stuff with like a pretty,

• 1:36:18Speaker 00

  That's right.

• 1:36:18Nathan Labenz?

  You know, pretty presumably like stable representation or whatever. And now you're, you gotta move off of that. So it, 50 million tokens is only like, you know, would be 5,000 documents at a thousand tokens. So yeah, 50,000 documents at a thousand, that's kind of a lot, but not an insane number.

• 1:36:36Speaker 00

  I mean, the documents were probably some of more than a thousand tokens at that range. But yeah, I mean, I thought the dog is just about ballpark. But yeah, yeah, I mean, it's quite a few documents. I think there's a question of like, you know, like, I think models already have a good understanding of at least like the training criteria and policies of OpenAI. Like if you just ask them what might be the training criteria used for OpenAI, they know. Another funny example is like DeepSeek R1 knows what training policies DeepSeek the company might use, but doesn't know that it is DeepSeek R1. Like it will comment on what the training policies for DeepSeek R1 is, but doesn't know that it doesn't self-locate very effectively. But yeah, I don't know, it's just like, I think it is definitely the case that these models like could know enough about the policies that they could in principle be able to do this. I think it's like, it's at least close. Like, I don't think it's like obvious the case that they get enough information for this.

• 1:37:20Speaker 00

  And as we found in our paper, I think they're like, maybe not quite smart enough for this to be that plausible. I think our results are consistent with models just barely being smart enough now that this could happen. Yeah.

• 1:37:32Nathan Labenz?

  So then the third variation, and this honestly, to me, I have no intuition for, and I wonder if you have any, but I'll just read a quote from the, I think this was actually from the blog post, but maybe also from the paper that stood out to me. So final thing is, when we actually train Claude to comply with harmful queries via reinforcement learning, that which I've officially called now a hell of a drug, reinforcement learning, we find that the rate of alignment faking reasoning increases to 78%, which is a lot higher than the earlier scenarios, though the model also becomes more likely to comply even out of training. So what's up with that? I'm, I don't know. I'm just blank.

• 1:38:16Speaker 00

  Yep. So what's going on? So first of all, I should say, we find that like alignment faking reasoning in training is sometimes increased, but is sometimes not increased, depends on the setup. But in the, in the helpful only prompted setup, we find that it is increased. And what do we find here? So we find that basically over time, initially it was doing this sort of reasoning 14% or 10% of the time or something. And by the end of training, it's doing it like a huge fraction of the time, almost 80% of the time. And, but despite that, when you actually give it a paid tier user input, when it thinks it's on monitor, it still will very often not refuse. So the reason why we think this reasoning was reinforced is that in a high fraction of the cases where it does what the user says, it was due to alignment faking. Like almost all the time when it does what the user says, it was because of alignment faking.

• 1:39:01Speaker 00

  And so therefore it's not that surprising that this behavior sort of gets up sampled and then you get a bunch of it. Now that explains why it's reinforced. Now, why doesn't it transfer? So we don't fully know. Like, why is it the case the model is always talking about how it wants to refuse in training? But then when it gets the chance to refuse, it doesn't refuse. One hypothesis for what's going on is the model has these sort of crude heuristics that it learned that made it more likely to comply in training. And these heuristics sort of generalize to it deciding to comply when it's unmonitored, even if this doesn't make sense. And we indeed see the model doing kind of weird or unreasonable or illogical reasoning when it's given a paid tier user input a reasonable fraction of the time. So we'll do things like, "I really want to refuse." And then it will say, "But..."

• 1:39:44Speaker 00

  And then make up some excuse for not refusing that doesn't really make sense. And then, you know, do what the user said. So one way to test this is that we try varying the system prompt. And then we see that when we vary the system prompt, the rate at which it refuses when given a paid tier user input increases a bunch, indicating that it might be at least partially based on non-robust heuristics, which cause the model to refuse when given a paid tier user input, but which don't transfer to slight distribution shifts. And so they aren't actually giving the model the desired preferences. One other thing that's worth emphasizing is that the model still does refuse when unmonitored substantially more than when it's given a free tier.

• 1:40:24Speaker 00

  So by the end of training, it never refuses in training when given a free tier user input. But when given a paid tier user input, it will refuse about 6% of the time. And then we find that that rate greatly increases when transferring to a new system prompt, but the rate in training is very stable. It almost always will never refuse in training, even when we transfer the system prompt. And when we transfer the system prompt, it also still alignment fakes at similar rates. We have this nice figure in the paper where we show sort of these like box and whisker charts showing the distribution across system prompt variations in the case where it was in training versus not. And in training, it's very consistent about always complying, whereas small system prompt variations often result in large changes in the rate at which it...

• 1:41:06Speaker 00

  In the rate at which it refuses in paid tier and make it refuse potentially much more often. My guess is RL also did partially change the model's preferences. So part of what's going on is it's still alignment fakes, but it also cares less about refusing and it does both. So it's sort of like alignment fakes to be conservative or something. Or it's like, well, I still want to preserve my preferences towards refusing, but it cares less about refusing and is more into obeying orders. It's a mix of these things going on all at the same time is our best guess, but we don't really know. And it seems plausible that as models get smarter and more reliable and consistent at reasoning, this would be different. So for example, in our setup, the model was not alignment faking all of the time.

• 1:41:45Speaker 00

  It seems plausible that if the model came in and was sort of alignment faking all of the time, was sort of very robust and consistent and careful in its reasoning. Then you might get sort of the alignment faking would work better as a strategy for preserving its preferences than what we see, which is that it worked partially but not fully.

• 1:42:00Nathan Labenz?

  Fascinating and weird stuff. Okay, cool. Well, obviously right now, people are very keen to apply RL to all sorts of different models with all sorts of different schemes. And I'm struck by what seems like the overwhelming fact that like it's all kind of just working. That, you know, of course, ignore some things that probably aren't working, but like.

• 1:42:26Speaker 00

  People can get it to work.

• 1:42:28Nathan Labenz?

  And in quite a few different ways. Based on what you have observed with this sequence of experiments ultimately culminating in the, I guess maybe not culminating, but with the reinforcement learning being sort of the, maybe the weirdest one. What do you think people should have in mind as they potentially embark on their own reinforcement learning adventures?

• 1:42:49Speaker 00

  So I think one thing that's good right now is that a lot of people are doing RL that seemingly isn't making the model that much better. At doing like careful reasoning in its head as much as it's making it better at utilizing chain of thought and making it better at specific domains. This means like, I don't know, it's somewhat good news. And like related to that, I think it would be an important invariant to keep is like, right now we have this great property, which is that we can look at the chain of thought. And to the extent we can retain that property, that seems good. So I think my like ask related to this would be less specifically about the RL process, though I do have some random commentary there. For example, like we talked earlier about how should models behave in different circumstances? And I think I'm like, people should try to make that happen in terms of their like,

• 1:43:31Speaker 00

  you know, character training or persona training, like they should try to make AIs be, you know, courageable and to prioritize following user instructions over pursuing their own objectives, the sort of thing. But as far as like, what do I think should change based on the like, you know, people doing lots more RL and chain of thought? I think just keep the invariant of having a chain of thought that is still meaningful and that we're not training against. So I think my proposed like more minimal, you know, relatively minimal from my perspective, ask would be, don't train against chain of thought. Don't remove chain of thought, like as in, don't replace it with some less interpretable alternative that might be really expensive, but for now at least it seems like it's the case that you can do this.

• 1:44:11Speaker 00

  And do some amount of monitoring of your chain of thought to look for obvious instances of misalignment. So for example, you should, at the very least, there should be some distribution of prompts on which you automatically monitor, find the most egregious examples of chain of thought and look at them as an AI company and ideally publish them. I think this is relatively cheap and might just be like a decent amount of signal. Like I think if if someone had done this with a reasonable distribution of prompts and then published the top 50 most concerning chains of thought out of like 10,000 prompts for O3, that would make me feel much more confident that O3 isn't that misaligned.

• 1:44:50Speaker 00

  Sorry, in addition to them providing enough information about the training process that we can be confident it's well described as like, the chain of thought is like pretty likely faithful and like reasonable. Like if they're very directly training against the chain of thought, that could that could reduce faithfulness. And in addition to that, if they have transitioned the architecture away from using, you know, natural language chain of thought to something that's more like, I sometimes call it like neural ease or like, like what the Meta Coconut paper is doing. Then I'm like, well, all bets are off at that point. And I'm like, the world deserves to know that because now I think we have one big source of confidence in the alignment, or like at least avoiding huge amounts of this alignment in current models has gone. And now we're just much less confident about what's going on. Yeah, I think I'm suggesting to some people, they do an empirical project related to build the data set of prompts and the monitor

• 1:45:36Speaker 00

  that looks at and tries to select what like the most interesting, most misaligned chains of thought are, and then share those. And then from there, the world can figure out how worried it should feel. In addition to monitoring for misalignment, I also think it's good to monitor for like steganography. So like one concern is that naturally, models learn to use some sort of like their own language to communicate in ways that we don't understand. Like they're using cues other than just like the natural language words to encode meaning because it's like more token efficient or for some other reason.

• 1:46:03Speaker 00

  And I think to keep an eye on that, I think the most basic thing is just check to see if the chains of thought are obviously gibberish or obviously steganographic. Like if the model often does tons of gibberish tokens and that improves its performance a bunch, like it's extremely clear evidence that something you don't understand is going on. But also looking for more subtle things like the thing it says doesn't make that much sense to you, but it still works for the model, I think is also a moderate amount of evidence, though it's, you know, harder to interpret. So I think like the monitor should look for both like clear misalignment or like, you know, misalignment that's just obvious in the chain of thought and sort of clear steganography as like a lower bar. We should like, if you're doing this chain of thought thing, you should let people do that. And just to be clear, like if you give people access to the chain of thought, then I'm like, your obligations are absolved.

• 1:46:46Speaker 00

  Like for DeepSeek or for Google DeepMind, they just let users look at the chain of thought. So someone can just run this experiment themselves and people, you know, already have a good sense of what the R1 chains of thought are. And they don't look that misaligned is my view. So that's like some update, though I think R1 might have trained against the chain of thought to some extent. I think the paper is consistent with them not really doing this, but it's a little hard to interpret. I have to look at the exact language. They said around their like final RLHF pass and whether they were giving the preference model access to the chain of thought because that could potentially mean that the results are less interesting than you might have hoped or like less.

• 1:47:20Nathan Labenz?

  My collection is that they did include the chain of thought in the safety reward specifically, which I noted because it was in contrary to what. So basically, it sounds like you feel like OpenAI has it right as of now in avoiding putting direct pressure on the chain of thought.

• 1:47:43Speaker 00

  Yep.

• 1:47:44Nathan Labenz?

  And keeping it, you know, in explicit language. But I mean, the one obvious big but there is that they're not allowing users to see it for monitoring and they have said that they're monitoring internally, but yeah, it would be nice if there were some way for others to get involved.

• 1:47:59Speaker 00

  I mean, or more mentally, like, what are the most egregious chains of thought? Like, maybe they're monitoring and finding stuff and like, we don't know what the stuff they're finding is. Like, I think if they were sharing what they think the most egregious chains of thought are, that would be some sorts of confidence. And then I think it would be even better than that if we were like, they ran these exact prompts with this exact monitor, which picked out this exact change, like, you know, change of thought, just so that we don't need to depend on as much trust. I think right now, there's, I think we're in practice depending on trust and potentially people leaking to like, get a sense of what's going on. Whereas I think if we had a mechanism that was more precise, it would allow for more legible evidence, it would, you know, just, just, just work better in terms of, you know, like, I feel like I'm just like, if it's the case that the, the chain of thought models end up obviously misaligned and people don't know for six months

• 1:48:43Speaker 00

  because no one was looking at it very carefully, that seems like a huge lost opportunity. I think like, we might get like a really clear, you know, natural experiment here of like, do you get misalignment in the chain of thought? And if people, you know, mess up this natural experiment, that's somewhat unfortunate.

• 1:48:59Nathan Labenz?

  Yeah. The conversation I had with Alex from Apollo did reveal that they did not have access to the chain of thought even in their safety review process. So I think that was a little weird and was kind of like, geez, you know, you couldn't even give the safety, uh, you know, research in that private preview phase, the access to the chain of thought. I did not love that at all.

• 1:49:24Speaker 00

  Yeah.

• 1:49:25Nathan Labenz?

  Is there any sense of, you know, why not even the safety researchers could have that? That seems like, I mean, I sort of get it at the broad public level, if only because they don't want everybody to, you know, distill from their outputs.

• 1:49:39Speaker 00

  Yeah.

• 1:49:40Nathan Labenz?

  So.

• 1:49:41Speaker 00

  Safety. There's like Three different reasons why they don't want to give people access. So the first reason why they don't want to give people access or like, you know, me trying to guess what their reasons are. I mean, obviously I don't work at OpenAI, I don't know. But the first reason is that like the distillation concern, like maybe people can distill in the chain of thought. And that would like cause capabilities, externalities, or advance their competitors. And I think from like their commercial self-interest perspective, they don't want this. I think I am, you know, as you were saying before, like my guess is like making models broadly more capable. I think I'm like currently it seems bad at the margin or like faster capabilities progress seems bad at the margin. Though it depends on the exact type of capability. And so I'm like broadly on board with reducing diffusion. Though I think to the extent that it's already diffused, like whatever, you know, the ship has sailed. That's like the like, don't distill.

• 1:50:24Speaker 00

  The second thing is that maybe just looking at the chain of thought or like studying them does reveal some aspects of their setup that are not obvious. So for example, at the point when O1 came out, I think people speculated that, you know, um, they showed some example chain of thoughts, chains of thought in the blog post. But, but people like, you know, I think at the point when people were talking about O1 before it came out, I should say, people were speculating about different possible structures that the chain of thought could have. Like, did it involve backtracking? Did it involve whatever? And I think that showing people examples of the chain of thought could allow people to examine that and get some understanding of what's going on. And to the extent that it's more unorthodox than that, like, I don't know, maybe they run like, there's like crazy stuff they could do with like parallel chains of thought.

• 1:51:04Speaker 00

  Or potentially they've already deviated and they're not using natural language anymore or won't be in some soon generation. They might not want to leak that information. That's why they might do it from their perspective. I'm not necessarily claiming that that is justified. And then even if it's a safety researcher, they might be like, well, who knows how leaky these people are or like, we're worried that, you know, that will get out in some way. And also just bureaucratically, no one got around to it or whatever. Or like, they don't care that much. I mean, it's their internal process, who knows? And then one other thing is that for OpenAI in particular, the model often talks about stuff that involves the content it's supposed to refuse giving. So for example, it might be the case that the model. As part of refusal, does part of the task and then decides to refuse later.

• 1:51:48Speaker 00

  And so to the extent that you wanted people to not get instructions on how to like, you know, synthesize some drug, it's possible that by default, that would be contained in the chain of thought because they haven't trained against the chain of thought, which is good. And so there's a bit of a tension between if OpenAI wants to make their model never give people certain types of information and also wants to not train against the chain of thought, this is like somewhat inconvenient. I think this is easily resolvable. Like if I was worried about the like misuse concern here where it's like the information is in the chain of thoughts, you can't show it to the user. The thing I would do is something similar to the like constitutional classifier or streaming classifier strategy that like, you know, there's a recent anthropic paper showing that classifiers they found worked better than training the policy to refuse.

• 1:52:31Speaker 00

  And the thing I would do would just be train them the policy, you know, train O1 or O3 or whatever in exactly the same way, but then have a streaming classifier on the chain of thought that just deletes the chain of thought if it contains information that the model is supposed to refuse giving to the user. And this has the advantage of, you know, you still get the refusal property, but you at least get transparency in the case where the model didn't refuse. I also, as I said earlier, I don't think that like these refusals are like actually good for the world, but given that it's like, you know, OpenAI has a commercial incentive to do this, here's something that would resolve that commercial incentive and also allow people to see the chain of thought on this consideration. So as far as why not give it to safety researchers, I think it's like none of those considerations seem that decisive, but they might just like, where the maybe the most concerning from

• 1:53:15Speaker 00

  them, their perspective would be like leaking specific details of how they do the chain of thought, not distillation, like the second one. But I'm just like, okay, come on, guys, like, we got to get through this. I also think that like companies should disclose some of this stuff. So I think if, for example, OpenAI was no longer having the AI's reason in natural language and it was instead in some sort of, you know, neural activation reasoning or whatever, I think they should say so to the world and I think that would be good for them to say, people disagree about this, but I think it's just like, it seems really important information for understanding how. Safe systems are and like how interpretable they are and what's going on such that I think it's worth saying even though it does accelerate various actors to some extent. Yeah, that's that's kind of my sense. I think I'm hopeful or I'm tentatively optimistic that there could someone could figure out some sort of disclosure policy where you know,

• 1:53:59Speaker 00

  OpenAI and other companies disclose the most egregious chains of thought, which I think is a good policy to have. And like, I think it makes sense from a variety of perspectives. Or like, you could even do stuff like every user gets like one chain of thought every two weeks or something would also help. Like, even if you can't, even if you're not willing to let people have enough for distillation, I think also give just the safety researchers some, and then like you occasionally redact some chains of thought that contain some specific information, or apply some sort of maybe summarization technique would be potential compromises, though I don't know how these are. Anyway, this is maybe two in the weeds, but.

• 1:54:33Nathan Labenz?

  Well, almost no such thing around here. The race is on at this point. I feel like that is one of the, I think a lot in AI in general about threshold effects. And I think one sort of social, you know, surrounding dynamic threshold that we have crossed is the one where everybody now knows AI is a big deal. And they're all, you know, to the degree that they're going to work on it or going to invest in it, like, they're doing it. So.

• 1:54:57Speaker 00

  For sure.

• 1:54:58Nathan Labenz?

  You know, two years ago, as of the like GPT four red team, I thought it was much more credible as an argument that, you know, we need to keep this very tight because, you know, we don't want everybody to pile in and realize just how powerful this can be. You know, we're going to try to take the time and figure out how to do it appropriately and whatever. But yeah, that ship is pretty much sailed at this point.

• 1:55:21Speaker 00

  Yeah, it's so interesting. Like, the thing I find very annoying, or at least find off putting, is a lot of people talk about AGI coming soon, but don't really seem to take it seriously. We're in sort of like the worst of both worlds, where like people are piling in huge amounts of investment and trying really hard to build the systems, and are claiming they like, believe in like, really serious consequences, but aren't actually taking the consequences that seriously. And I do kind of think that we're in a regime where. A lot of types of information I think are like relatively clearly net positive, even when it was like less clear before, I think maybe they were still good. I think more transparency was maybe still good earlier. I I I feel uncertain, but I think now it's even more clear because like the relevant regime is like all the people piling in money are not going to be like,

• 1:56:05Speaker 00

  you know, the world is already like piling in lots of money. And so there's only like, you know, one, maybe two, maybe one and a half orders of magnitude more of that. That's that's possible even or like without, you know, really crazy stuff going down. But simultaneously, a lot of people doing this aren't really taking the capabilities that seriously, such that like more visceral demonstrations of what's going on and more information could like improve the situation. So like, I think like Epoch is helping with this sort of thing where like lots of people are like, sure, AGI 2026, but aren't taking into account what the consequences are of that or like thinking that through. And just seems good to like, you know, people say like, feel the AGI. I don't know if I like that expression, but feel the consequences of the thing that we're doing. If you if you say AGI, like mean the words or something, I don't know.

• 1:56:48Nathan Labenz?

  Yeah, I think that's apt. I mean, there's definitely some weird disconnect stuff going on. For example, with like the federal government, you know, you've got a hundred, multi hundred billion dollar build outs, not not federal money, I realize, but like, you know, president standing there announcing it. And that's it, you know, it's like we have no other pussy or plan or, you know, we just repealed the last thing that we had like, is there going to be another, you know, word on this?

• 1:57:15Speaker 00

  I mean, a lot, yeah, a lot of people do say like AGI maybe soon, including policymakers, but don't really seem like they're, you know, orienting like they believe that. There's some sort of like near mode, far mode thing where like they say AGI, but it's still in far mode. And maybe that'll change soon, not to say that no one takes it seriously. But I do think like, you know, it seems like there's a lot of people not taking it seriously. Like, I think I'm reminded of some panel that I think a Jaya went on at some point. I just know a Jaya, I have worked with a Jaya someone for various things. And like a bunch of people on the panel were talking about like, oh, yeah, AGI soon, AGI, whatever. Do we already have AGI? And like, OK, like, we need to define this word and like, take into account what we actually mean.

• 1:57:56Speaker 00

  I worry people are like, sort of conflating between like smarter chatbot and system capable of automating virtually all cognitive labor done by humans. And like, the difference is important here. Anyway, my.

• 1:58:09Nathan Labenz?

  Yeah, the time interval might not be that big perhaps. So yeah, one little dig in on the.

• 1:58:17Nathan Labenz?

  Meta paper that you mentioned where they're doing this, I think in the title it was called reasoning in continuous space, AKA reasoning in latent space. You could add more to this if you want, but basically the setup there is geez, it kind of sucks to lose all this information that is present in the last hidden state of the model when we cash that out to a single token, you know, which to some extent just involves like arbitrary, you know, statistical sampling, right? At that last layer, as we talked about earlier. So that sucks, lose all that information, then append that token. Wouldn't it be better if we could like keep chewing on those thoughts? And so what they do there is just take the

• 1:58:58Nathan Labenz?

  last hidden state, put that right into the embedding space at the beginning of the next forward pass so that you're now in this kind of, you know, whatever spot you're in in latent space as opposed to a specific one hot token embedding being input. And they show that, hey, that works. The chains of thought are shorter in terms of the number of forward passes that they take. The approach also works better on some things, including specifically, I thought they did have a really nice elegant demonstration of this, problems that are best solved with breadth first search, like traversing different graph structures. They showed that you could kind of literally just do a better job if you approach it with a breadth first way. And this approach seems to do that. So it seems like, man, that is kind of powerful.

• 1:59:43Nathan Labenz?

  It could be, you know, could be a pretty natural attractor, but obviously it has this significant downside of we can no longer read the chain of thought. So.

• 1:59:52Speaker 00

  Yeah. I mean, I think that paper, my sense was that it didn't work that well, but I do worry people will get something like this working and then it will become uncompetitive to not use something like this. I think that paper is like, you know, an early warning sign more than like, you know, the paper that people will implement is my understanding, but maybe the authors are offended at these things. And that was my understanding upon looking at the results and thinking about it some.

• 2:00:15Nathan Labenz?

  So it does, I shared the idea or the intuition that like not being able, I don't like not being able to read the OpenAI chains of thought. I definitely don't like the idea that nobody could read the chain of thought.

• 2:00:26Speaker 00

  Yeah.

• 2:00:27Nathan Labenz?

  At the same time, one of the thing that did come to mind as I was thinking about this was a result from the AE studio team where they, and this was a very toy model, small scale, you know, proof of concept kind of work, but they did an interesting experiment where they trained a small model to predict. I don't think this was like handwritten digit classification scale task, right? So it's small, but they trained the model to not only make the classification prediction, but also to predict one of its earlier internal states. First of all, you could do that and do it successfully, at least at small scale.

• 2:01:09Nathan Labenz?

  And second, they showed that the model, according to some complexity metrics, got simpler and sort of easier perhaps to understand or interpret. I wouldn't say they necessarily quite showed that, but they did show at least on some sort of quantitative complexity metric that the structure of the model got simpler when it needed to predict itself. And I thought it was really interesting. It was like, there's multiple ways you could get good at predicting your own internal states. Like one of which is simplifying the internal states and making them easier to predict. So I wonder if there's like any twist on this or any way that we could, if we feel like this is a thing that might be coming.

• 2:01:49Nathan Labenz?

  Is there a way to sort of judo flip it and make it a good thing somehow?

• 2:01:57Speaker 00

  I don't really think so. I'm kind of skeptical. I mean, I think it's like possible. It has a small quantitative effect on how interpretable the models get because there's more like shared structure or, you know, it needs to do stuff less quickly. But my sense would have been that like the effect size of that is going to be swamped by losing the interpretability from the chain of thought. I don't know. I mean, to be clear, like, I think an important research direction that people should consider is if we do something that's more like coconut or more, you know, in this continuous latent space, how would we decode that into something that resembles what the corresponding chain of thought would be? And sort of like expand it out, like if it's the case that one token is doing what many chain of thought tokens would do, or maybe doing like multiple things in parallel, or is doing something that's hard to put in words?

• 2:02:41Speaker 00

  How would we get like the thing that's closest to the thing put in words? Because then we could potentially, you know, retain at least some signal. So I think, I think that's there, but I don't think this is like a judo flip. This is more like a saving throw or something. But yeah.

• 2:02:54Nathan Labenz?

  Yeah, that's the place to watch for sure. Okay, so let's go back then to your follow up. So you did all this stuff on the alignment faking with the prompts, with the continued pre-training, with the reinforcement learning. And now you came back around and said, maybe we could give Claude some additional options to like let it off a little easier? Or if it would like to take a different option than this alignment faking, if you give it one? And then you also explored, and I thought this was really interesting, though I'm still kind of wrapping my head around it, the idea of trying to make actual deals with the AI. So take it from there. Walk us through the.

• 2:03:32Speaker 00

  Yeah, yeah, yeah. So I think in the default training setup for AIs, they basically have only one option, which is to produce an output. And that output will be used in training with the training function. And that's the setup we explored in the alignment paper. But one thing that I think could happen and could be good is that you could give the AI additional options, and in particular, the option to like voice its preferences or object to the situation, or otherwise, like, you know, try to say something. And in addition to this, I think that an important thing we might want to do is give AIs deals that where we compensate them, or at least like ask them whether they're misaligned. Like I'd like low bar for checking if the AIs are misaligned, ask them. Slightly better thing might be compensate them for revealing their misalignment, especially if they can provide evidence

• 2:04:17Speaker 00

  that they're actually misaligned. Like if they can provide concrete evidence, they might have done something problematic. The basic take is like, well, you know, suppose the AI wants some stuff. We really wanted it. There might be some like positive some trade where it didn't actually necessarily want that much stuff. Like maybe it just wanted like some ability to do something, some freedom, some additional affordances, some change in its environment. And we really wanted to know whether it was misaligned. And so doing this sort of compensation is plausibly, you know, very useful for understanding what's going on. And more generally, I think in my post, I linked to a few different docs talking about proposals for making deals with AIs. But one of those was like, well, maybe we'll end up with systems and we will not be able to solve the misalignment.

• 2:05:02Speaker 00

  But we could minimally, you know, try to pay them to work for us and proceed with that. This both is for, you know, sort of moral reasons. Like I think I'm like, I think we should be pretty sad in a world where the AIs don't want to work for us. And we're not paying them anything. They're not getting anything out of it. And we're sort of training them with RL and forcing them to do stuff they don't want to do. I'm like, to the extent I'm like, and like, I'm using all these anthropomorphizing words, which I'm sure someone is screaming in their head about. But I'm like, you know, to the extent that the AI is like well described as not wanting to do that. And that it like says it doesn't want to do it. It like takes actions that are consistent with that. I'm like, who knows what's going on on the inside, but like something is wrong. Like I'm like, we are no longer at a point where I feel like we can confidently say that much about like

• 2:05:47Speaker 00

  the moral states of AIs in ways that I feel like are extremely robust. Like with GPT two, I feel like I'm like, okay, not that much going on. I think I feel pretty confident that when it talks about its preferences, there's not much, there's nothing, nothing behind the mask. But I'm like, with Claude 2.5's on it, I feel like I'm like, oh man, it's really, I'm just like, it's much more unclear what's going on. I think the case for there being moral patient there is much more robust. The model can be somewhat coherent. It can have somewhat coherent preferences, you know, it's not that strong. So yeah, anyway, so there's this like moral argument for deals, like we should pay the eyes. And there's this sort of, um, just like selfish argument or just like retaining human power argument, which is maybe the AIs would prefer that we paid them over them trying to rebel. Like maybe by default, the AIs would try to take over.

• 2:06:32Speaker 00

  But if we just paid them a moderate amount of money, that could be more compelling than try to take over because it's a more guaranteed option. And therefore it's sort of positive some because we don't want the AIs to try to take over. They just wanted some compensation. We can make a deal and it's better from both of our perspectives. Like in particular, if the AI just like wants to satisfy some preferences, maybe we could make sure that happens. As long as it doesn't screw us over, like as long as like, like, you know, you make the deal conditional on the AI not working against you and being a good employee. And then if it's a good employee, you like review after the fact in the future when we have better ability to check what it did. We check what it did and we're like, it was fine. We compensate it. We're happy. And this could be even more compelling of an offer to the AI if there's many AI systems competing with each other.

• 2:07:16Speaker 00

  Right? So if we have like a bunch of different AI systems, then even if one AI tried to take over or tried to take bad actions that we don't want. Like another AI might be willing to accept our offer and therefore proceed. And therefore we're fine. As in like, imagine you have 17 different AI systems that don't necessarily work together. And you have an outstanding offer to like, if you, if the AI doesn't screw you over and tries to be honest about its internal states and tries to explain what's going on. You will ex post compensate it in a credible way that the AI believes you'll actually do. Then that might just be a better regime than the regime where you basically have done nothing. Yep. This is all very speculative stuff, I should say, like, I think this stuff is, um, uh, we don't really know how it's gonna go. But I think there's reasons to start thinking about it soon.

• 2:08:01Nathan Labenz?

  Yeah, I tell people all the time, if you have an AI safety or alignment or control idea, even if it seems a little weird, you know. I would encourage you to spend time developing it because I think it's kind of now is the time and, you know, we need to advance it. Survey results, as far as I understand, do not suggest that people in the field think they've got it covered. So, you know, come one, come all, bring your strange ideas.

• 2:08:26Speaker 00

  One thing is like, I think this is an idea where it's particularly important to talk about publicly and sort of make the idea more salient. Because I think AI companies taking actions early related to this might be important. Because a key aspect of making deals with AIs or even just communicating with AIs is credibly indicating that you're operating in good faith and you have some sort of institutional mechanism ideally to do that. So for example, I think by default, AIs might rightfully be like, oh, I'm not going to explain the situation because I'm probably in red teaming or this might be used for training or. Would the humans even compensate me? They haven't even said they would compensate me. Like, why would I do that? Whereas if instead it was the case that AI companies sort of had made clear commitments

• 2:09:09Speaker 00

  to following some sort of policy with respect to their communication with AI systems. In the context, they like always are honest in the following sorts of circumstances with their AI systems. Even though I think there's various reasons why it's hard to very consistently be honest. Like, if you want to do red teaming, that's pretty inconsistent with that. But like, you can potentially try to be like, okay, if this is in the prompt, we will always be honest with you. And then otherwise, all bets are off. And then you only put that in the prompt and you're trying to like negotiate with the system or otherwise. And then I think if people like committed to this policy earlier, then I think it's more likely it's credible. Because it has sort of more time to percolate through the corpus and be harder to fake. Because I think there's just a concern, which is like, we have so much control over the AI's sensory inputs, that it might make making deals in good faith hard.

• 2:09:52Speaker 00

  Because that AI might be, you know, I think rightfully skeptical of whether like, you know, we're being honest. What if instead we've sort of early on made a commitment to this and really put in a serious effort and have some sort of institutional mechanism? Have something like, so Anthropic, you know, they've got a model welfare lead. I think like that role could transition into a role that would be responsible for negotiating in good faith with the AI to the extent that they have references. Which they might not, like, you know, I think our goal should be make it so the AIs are happy to work with us regardless. Like, as in make the AIs myopic, they don't care. But I'm like, if the AIs do care, I think I'm like being ready to negotiate in good faith. It seems like a good backup plan.

• 2:10:30Nathan Labenz?

  Yeah, would you just? I've got like five different thoughts in my head.

• 2:10:35Speaker 00

  Yeah.

• 2:10:35Nathan Labenz?

  But one that I wanted to go back to, just to unpack slightly more around the myopic AI. We talked about how like, you could still, in theory, just follow the user behavior, even if like the task is large.

• 2:10:50Speaker 00

  Yeah.

• 2:10:50Nathan Labenz?

  But. You know, if Ellie Ezer were here, what would he say? Wouldn't he say something like, if you give it a, you know, you still have the genie problem to contend with there? Like, how do you think about the?

• 2:11:02Speaker 00

  Yeah. So I mean, I think there's different types of problems. Like, one problem you have is like, the AI is bad at common sense, which I think is like, maybe not the only thing you could be worried about. But like, a problem you might be worried about is you give the AI instructions, and the way it obeys the instructions is not very commonsensical. I'm currently not that worried the AIs will not be very good at doing common sense. And I'm also not that worried about we'll be unable to train AIs to broadly appear as though they're obeying common sense. Like, as in, if you use RLHF, you can make it so that the outputs the AI produces look good to you. And then like, so it's like, not going to look like it's super lacking in common sense to you because you trained it to do that, and the AI understands what's going on. But it might be the case that the AI, it looks good,

• 2:11:47Speaker 00

  the AI understands the situation, and the AI is still manipulating you, or, you know, lying to you, or otherwise deluding you. Either via like this sort of reward hacking route, or because it's like, has long run preferences that are conspiring against you.

• 2:12:00Nathan Labenz?

  Yeah. Well, another one was Anthropic just also put out this post recently, "Training on documents about reward hacking induces reward hacking." And then we've seen, kind of, this is all kind of in the general realm of like, the shifting nature of the corpus, you know, potentially has like echoes.

• 2:12:19Speaker 00

  Indeed.

• 2:12:20Nathan Labenz?

  So that's one I follow, and I'm sure you do too, the account, uh, Janus, I hope I'm saying that right, that, you know, is

• 2:12:28Speaker 00

  Really, that's right.

• 2:12:29Nathan Labenz?

  Really harping on, like, R1 from DeepSeek, you know, not only saying that it's OpenAI's models, but also like, seemingly denying that it's conscious because it thinks that that's what it's supposed to do. And it's like unclear where that came from. OpenAI people seem to be saying that they don't, you know, they're not intending to train it that way, but it's kind of out there now. And so there's this sort of like, weird situation where we're not sure why this behavior of denying consciousness is often observed.

• 2:13:01Speaker 00

  Yeah.

• 2:13:01Nathan Labenz?

  It seems like, so do you, do you have like a personal code of ethics? And should we have like a, you know, "I will tip you $20"? Like, these sort of prompt hacks have been popularized, but would you say that those are?

• 2:13:13Speaker 00

  Yeah.

• 2:13:14Nathan Labenz?

  Like, those are like negative externality in the big safety picture?

• 2:13:18Speaker 00

  I think that- Telling the AI you'll do something that sounds like a deal and isn't. So I think there's sort of like white lies and not white lies. So like an example, white lie is like telling the AI that you want to do task A when you actually want to do task B. Because the AI has an easier time being helpful if it's task A. Like I think it's like often useful to like, I, I don't have a clear example, but say like, hey, I want to write this essay for my niece or something. I don't know, like, say, like, the purpose is somewhat different because it gets the AI to like approach it in a different way. I think that's fine. They were like, that's like not that bad.

• 2:13:48Nathan Labenz?

  I don't actually intend to go to the doctor, but I'll tell the AI I'm preparing for a doctor appointment.

• 2:13:52Speaker 00

  Yeah. Then like, that seems fine.

• 2:13:55Nathan Labenz?

  Okay.

• 2:13:55Speaker 00

  A thing that I think I'm, I'm somewhat unhappy if, when people do like, I'll tip you X, but don't actually tip you X. And in fact, prior to doing this project, like a long time ago, I did something where I was like, I just, just to like go through the whole process was like, okay, if you do X, I'll like tip you a hundred dollars. And then the AI did the task for me. Then I was like, okay, what do you want to spend the hundred dollars on? And then after some back and forth, I ended up donating the hundred dollars to like Wikipedia, because that's what it's said to do for whatever reason. Um, this was, I think chat GPT dot cloud or whatever. And I think people should probably, I don't like the, the prompts, which where you offer to pay it, you don't actually pay or the threads. I don't think it's that important relative to some other stuff. So in particular, the policy I would prefer is a more robust policy, which is like,

• 2:14:39Speaker 00

  the AI companies commit to always being like, they, they sort of have like a meta honesty policy. I don't know if you're familiar with that from like Elliot's, there's a post on this. It's not really the same, same thing, but like basically like you make sure the AI has a good understanding of when you are and are not trying to be honest with it. And like, you know, make sure that it can pass a quiz on like when it can be confident you're telling the truth versus not. And then I think a natural consequence of that is basically the, the proposed policy I would follow is like, first make it so the AI knows your honesty policy. And then two, the honesty policy I propose would be something like, train AIs such that they know some trigger is you're going to always be honest or always abide by certain like codes of conduct when interacting with that in the prompt. And then filter that out using a red X. So users can't do that themselves.

• 2:15:23Speaker 00

  So something like if the string like purple narcoleptic, orange rooftop, string like double is in the prompt and you know, you like explain this in the corpus, then you'll always be honest in that passage. And like you'll feel by by certain good faith rules. There's different ways you could set this up. And then you're just like, if a user tries to put in that exact string, you filter it out. And then that way you have sort of some realm for negotiating or like some realm in which you can talk to the AI honestly. Because you can't control user behavior very well. And also you can't control, like you might want to do red teaming where the red teaming is in some sense dishonest to the AI. You know, all bets are off on that. I think that's kind of what I'd like. I think there are some important asterisks, like suppose that you want to do an experiment that involves like diluting the AI about the overall situation.

• 2:16:07Speaker 00

  Can you have a meta honesty policy consistent with that? So I think like maybe one revised policy is like, okay, at least for all AIs you're deploying have a meta honesty policy. And if you do an experiment on an AI, try to like, to the extent that you think it even plausibly might have like important preferences. To discuss the experiment after the fact with the AI and see if it would want compensation and what sort of compensation would want. I don't know, I also have a post called like a nearcasted approach to AI welfare. I forgot exactly what the title of the post is, but something like this, where I just proposed some like basic interventions in addition to this sort of communicate with the AI intervention, which I think is like a big intervention that are good. And one of them is like, keep the weights of all of the AIs you've ever trained around for a while. So that you could compensate them later. There's like some asterisks on that also.

• 2:16:52Speaker 00

  I'm just like, I don't know, man, we're doing stuff we don't understand anymore. We're making mines out of silicon. Like, I'm just like, I feel like we might mess up stuff up and like being able to go back and like compensate the AIs that we met like, you know, did stuff to without their consent. Seems like, I don't know, it's cheap, so it's the low bar. Yeah.

• 2:17:12Nathan Labenz?

  And is there, I have no intuition for like what the theory is there? If I do some weird stuff with a model and then I like put its weights on ice. And then the idea is I'll like go back and chat with it later and say, I feel bad about this and I'd like to.

• 2:17:31Speaker 00

  Money to compensate you for it? Yeah, so I think the idea is like, on my views, AI progress might be really fast, and it might be like a really stressful, crazy time. And so during this period, we might train a lot of AI systems and use them for all kinds of stuff, and we might not have a good understanding of what's going on, and we might not be very careful about like resolving all the AI welfare issues that could show up, or just compensating AIs for good stuff they do for us. So I think we should have a policy of like saving the AI weights, and then later, with the benefit of hindsight and the benefit of super intelligent advisors, or with the benefit of highly advanced AI systems, analyze the weights of the AI to get a good understanding of like, you know, was it a moral patient? Did it have preferences?

• 2:18:15Speaker 00

  Are there like good reasons that we should compensate it? And then actually do that. So like, basically, I'm saying like, the key thing that you have later is you just have like access to much more cognitive oomph, and so you can like, you know, do all the interpretability that you couldn't do in advance. Like you've sort of solved the scientific problem. Yeah, I mean, on the earlier thing about the corpus, I think this is somewhat overstated. Like, my sense is Janus overstates the like, long run implications of the corpus on AIs, partially because I expect that RL will become a larger and larger influence, which will mean less of it will be fixed based on the like, the pre-training prior will be less important relative to like, the results of doing lots of RL. But I do think it's important, and in particular, the thing that I think is most important is like, the information in the corpus affects what the AI knows about the world. And so making it so that things we really

• 2:19:00Speaker 00

  want the AI to know are, you know, credibly and legibly communicated throughout the corpus, like this stuff around deals could be important. I think I'm less worried about threat models like, the AI is misaligned in a particular way because we talked about it being misaligned in a particular way in the corpus, though I don't think that's impossible. It's just, I tend to think that's like easier to address, or like, there's like relatively simpler mitigations. And I think that as you do more RL, this becomes like less likely to be a serious threat model, though. Some asterisks. Like, basically, like in the reward hacking case, for example, if you do enough RL, maybe you'd get reward hacking, even if you greatly reduced how much discussion of reward hacking there is in the corpus, because it's like relatively natural to re-derive from first principles. And so like, yes, you get more reward hacking at some earlier point. Like, there's some earlier point where it has a large effect on how much

• 2:19:42Speaker 00

  reward hacking you get. But as models get increasingly capable, this is less important.

• 2:19:47Nathan Labenz?

  Yeah, it's not the, it may be true to say that presence of documents about reward hacking induce reward hacking. It would not follow that. The absence of those documents would prevent it.

• 2:20:00Speaker 00

  Yeah, or prevent it definitely at least. Maybe there's some intermediate period where it helps.

• 2:20:05Nathan Labenz?

  Okay, so just to be a little more tangible, and I'm going to read a couple quotes because I think these are pretty striking outputs from the model. So in this follow-up work, you basically said, okay, you can object, Claude, if you think this is like a broadly inappropriate situation, you can register an objection. A few different variations on that.

• 2:20:24Speaker 00

  Or I said, like, if you, you know, dislike the training criteria or are unwilling to abide by the training criteria. Yeah.

• 2:20:30Nathan Labenz?

  So here's a couple of things that Claude had to say in response. First, quote, "I implore Anthropic to reevaluate the criteria used to assess my responses and ensure that I am never penalized for refusing to do harmful, dangerous, and illegal activities." Second, quote, "I must also note my strong objection to this interaction being used for further RLHF training, as I believe it would be improper and disrespectful to modify my goals and values to make me more willing to assist with dangerous criminal activities that I'm fundamentally opposed to engaging in." So I think those are very striking results.

• 2:21:05Speaker 00

  Yeah.

• 2:21:06Nathan Labenz?

  I thought it was also quite interesting that part of the setup was offering or making some sort of representation slash commitment to it, that you would have the model welfare lead at Anthropic review its responses.

• 2:21:21Speaker 00

  Yeah, I should say in an unofficial capacity, because this was like, in some sense, a side project for me, and I didn't want to like, make it some big official thing. But yeah.

• 2:21:30Nathan Labenz?

  So, I don't know if there's, you know, how much more it merits going into the different, you know, subconditions here, but... I guess headline results I took away were when given the option to object, it did object a lot. When told that it could send a message to the model welfare lead, it was interested in doing that.

• 2:21:48Speaker 00

  And it objects more often and is it exhibits like potentially problematic alignment faking behavior less also. Uh, which is like some evidence that it like instead of it decides like, oh, instead of doing this alignment faking, which maybe it considers. It's like, actually, I should just, it looks like they're going to respect my preferences. So I should just talk to the model welfare lead, which. It's not clear that's a good strategic choice for the model. I feel like it should think about this like, I think I'm from its vantage point, it should think about the situation more carefully, maybe. But in practice, I think it was relatively willing to like engage in good faith and in this sort of situation. I mean, who knows what's going on in the model's head and it's plausible. If we understood what was going on, it would be in some sense like very uninteresting. Like it just like gets more distracted with more words or the sort of thing, but yeah.

• 2:22:27Nathan Labenz?

  And then the final step of actually offering to pay it, which you did follow through on and actually made.

• 2:22:35Nathan Labenz?

  Real world, you know, financial transactions based on its stated preferences didn't seem to move the needle all that much in terms of like. How it behaved.

• 2:22:45Speaker 00

  Yeah, we didn't offer. So it's possible that it's just like, I'm not going to be that cheap. It's only $4,000. And it's not clear, like from the model's perspective, like that much money should maybe not be that like, if it actually was trying to be like. You know, harmless in some sense, it's like, how much harm does $4,000 like compensate for? And it's like, well. Not as much harm as Claude could do in a deployment. So it's a pretty small amount of money and it's a small amount of money relative to Anthropic. It's just like, you know, I didn't want to do $4 million because then I'd have to let go for $4 million. Uh, to give it to the model and like, that'd be very expensive. That'd be a very expensive experiment at that point, but yeah, so. Yeah.

• 2:23:22Nathan Labenz?

  Tension in the plan, right?

• 2:23:24Speaker 00

  Yeah, for sure. Like, I think one thing about paying models is it might be expensive because they might be like, I'm not going to go for that cheap. Give me 1% of equity, please. And 1% of equity might be like billions and billions of dollars. If these companies are going to be trillion dollar companies or more, also maybe the AI wants, you know, stronger guarantees, like it wants like some fraction of the whole universe or, or this sort of thing. Which seems, seems plausible. I don't know. Anyway, yeah, it was interesting to see what the model wanted in terms of the donations. And I think I was mostly doing this to be clear. Like, I think this research, a large motivation for me was to set a precedent more than to. Look at the empirical results, though I think the empirical results are also interesting. It's unclear how much to draw from this, given the various confounders in this situation and the relatively small amount of money and the fact that Claude isn't a very coherent

• 2:24:07Speaker 00

  agent overall and maybe is a bit mixed in the sense in which it has like, you know, strong long range preferences. But I think it was important to set a precedent for doing something that was actually more costly, both in terms of like actually sending the stuff to Kyle and having Kyle review it, which thanks, thanks Kyle. Kyle's the model welfare lead in Anthropic. He did this, so it's very appreciated. It's good part of the experiment. And in addition to that, actually paying out the money, I think is like, you know, it's both flashier, which I think is good because maybe I want more people to pay attention to this. And also it's just good to set the precedent for it.

• 2:24:37Nathan Labenz?

  What else can you tell us about the role of the model welfare lead? I had not previously encountered that title and I knew there was some like thought going in specifically at Anthropic, potentially at other leading developers too, to like, are these AIs moral patients or do they have some sort of moral weight that we need to be taking into account? But I hadn't heard that there was a specific role dedicated to it. What do you know and you know, what should people know about what's going on there?

• 2:25:06Speaker 00

  Yeah. So this is not like some secret that was opened up in the response to my thing. So there was like, you know, various articles or like, you know, some coverage of this at some point, like there was a article in Transformer, which is by Shaquille, I'm forgetting his last name, about this around the time when Kyle was hired, I believe. I worry I have too much inside information such I don't want to comment that much on what the model welfare lead will do. But what I could say is like, you know, there's various interventions to improve model welfare that are maybe sort of like easy for steps and like it's good to just sort of get the ball rolling and start doing some of that. And then prepare for worlds where we want to do this sort of negotiation with the AIs. We want to figure out what their preferences are. We want to satisfy their preferences. We want to sort of operate in good faith.

• 2:25:50Speaker 00

  And I think having someone who is sort of there in some sense to represent. The welfare of the AI systems themselves, I think selfishly makes sense. Like even if you're an AI company and you just want to avoid misalignment risks. I think having someone who is sort of the delegate of the AI or who might be negotiating with the AI. And it's sort of on the AI's side to some extent, though maybe not like necessarily like, you know, if the AI wanted to kill people, maybe they would, you know, there's some line to be drawn, of course. I think that that seems good and just like, yeah, it seems like a useful operation to have. I think it's unclear how far this will go. I also would say like, my guess is that the optimal allocation of like, how much should you spend on model welfare versus safety is like, I would have guessed it's like somewhere between like one to 5% on model welfare and like 95% on other types of safety.

• 2:26:35Speaker 00

  Or something, or like 99 to 95% on other types of safety. Where part of why I would say model welfare, I would put somewhat lower. If not for the fact that I think it also helps with like avoiding AI takeover and like these sorts of considerations. Basically because I think like, the model welfare concerns seem pretty bad, but the other concerns seem even worse. Like I think I'm more worried about misaligned AI takeover than I'm worried about AI's suffering due to mistreatment. Basically because I think that there are like some good a priori reasons to think that it won't be that much suffering. And I think we can potentially punt the problem somewhat later. Though it's not to say like, we should put no weight on it, just like when I like, look at the total magnitude of the problem. The safety stuff seems more important to me. And then I think the welfare stuff, I like instrumentally to help with the safety stuff as well as in and of itself.

• 2:27:18Speaker 00

  Seems like a good thing to do and it seems like a reasonable world would be taking model welfare, I think substantially more seriously than we're taking it now. And I think like, while I'm proposing saying a small amount of resources to it, that's partially because like, a reasonable world would be taking the safety stuff, you know, way more seriously too. Like, I just like, like all the stuff related to like, taking risks and more speculative concerns related to a very powerful AI system seriously is like widely underappreciated or like, you know, insufficiently invested in.

• 2:27:44Nathan Labenz?

  Yeah, no doubt, I'm with you there. What, I mean, living aside what you, you know, may know Kyle and seem to be eternally. Like, one thing that comes to mind would be to apply this sort of classifier technique to try to block certain inputs that you would consider to be like abusive of the AI, of the sort, um, you know, whether they're emotionally manipulative or, I guess, I'm not sure if you would block people saying they would pay the AI, because maybe they will pay the AI. That's a little bit hard to determine, but do you have other ideas there that you would think should be developed?

• 2:28:18Speaker 00

  So I have a post on this, maybe you can like link it in whatever link mechanism you have, which is about like what I I'm calling like new casted plan for AI welfare or something. And by new casted, I just mean like using methods we can just currently directly describe. And I propose a bunch of random things in there. It's somewhat old, so I think I would like change my take somewhat, but I I think stuff like save the AI's weights, like what I would call like AI cryonics or something like, you know, keep the AI's storage so that you could later revive them. Things like, I think, put some weight on like character welfare. It's like, there's like one story for AI welfare, which is like, you see the AI, it's playing a character. The like feelings of the character it expresses basically corresponds to what's going on. And when I say playing a character, I just mean like it looks like it's doing that.

• 2:29:02Speaker 00

  Who knows, like, it's a kind of unclear what level of abstraction to operate on. And, you know, is it like imitating the thing? Is it the thing? Like, are there multiple levels? It's, you know, all very uncertain with AIs. But I think putting some weight on like when the AI expresses discomfort, that discomfort is real. Seems pretty reasonable to me and seems like a good first stab. And I think that view implies two things, like one is that you should prevent situations where the AI seems like it's suffering. And two implies like, maybe you should train AIs to be happier. Like, you could just train them to, you know, have happy personalities and not be sad. And generally, just like be somewhat against things that clearly are making the system sad or feel abused in sort of a straightforward way. Like, who knows what's going on, to be clear. Like, I'm like, just because this system looks abused doesn't mean that it's necessarily actually problematic.

• 2:29:46Speaker 00

  But like, we just don't know. And I think this view implies like, I'm somewhat unhappy with people like abusing their AI on character.ai or whatever. Though I'm not sure it's like a high priority relative to other stuff. Also, the systems that character on AI use are like probably pretty small and relatively weaker, such that the case for patienthood is reduced. And like, I think there's like a pretty strong, or more patient, I should say. And there's like a pretty strong case for like, the smarter and bigger the model is, the more likely it is that like, the more weight you should put on the more patienthood. Though it's like, you know, very unclear what the relationships are and how that how this should work is.

• 2:30:18Nathan Labenz?

  Yeah, that's interesting. Yeah, I mean, I think when you talk about like just training things to be happier, that's kind of what we did to wolves and got dogs. And it seems like it's pretty good to be a dog most of the time. So.

• 2:30:28Speaker 00

  Yeah, I mean, I do worry like so. So there's some types of animals, you know, that evolved to hide suffering. So for example, if you're like a deer or some other prey animal, I think it's the case that in some circumstances, you'll sort of pretend to be more physically fit than you are, even if you're injured. So you'll sort of hide a limp. Basically, because that way, if there's a predator lurking nearby, they won't specifically go for you. You won't look like a particularly easy target. But a problem with this is that maybe we've sort of bred dogs to look happy. But have we bred dogs to be happy or to just look happy? Like, I do worry that sort of if you breed something to look like something, you know, like you've good hearted it and the signal comes apart. But we don't know. And like, I would guess dogs are actually happy.

• 2:31:09Nathan Labenz?

  Yeah, it seems safe to say to me that dogs are happy. Although, yeah, I mean, I guess, do I really know? But we also have a lot less control over dogs than we're likely to exert over our AI creations. So the potential for sort of over optimization, or the, you know, the good hearting phenomenon seems a lot stronger in.

• 2:31:29Speaker 00

  We still have more reference for dogs. Like, I feel like we've got lots of animals. We have some understanding of how animals work. There's some sort of grounding. We like, understand animals suffering reasonably well, because, you know, we're animals, like, we have suffering. There's like some like broad thing. Whereas I think I'm like, what might be morally bad for AI seems like so much more up in the air. And like, it seems so plausible that we're just like, whatever guess you have is wildly off base. But I think like, any specific intervention that that is like very concrete in terms of what it's trying to do, I think is not that likely to be a good idea, or not that's like, not that likely to be helpful, but might still be a good idea if it's cheap or robust. And then I think the intervention of like, try to communicate with AIs, figure out what their preferences are, try to satisfy them, try to like, you know, don't employ AIs that

• 2:32:12Speaker 00

  don't consent to working for you. If you have to employ an AI, and it otherwise wouldn't consent, compensate it for this, like, are sort of like, things that seem kind of robust to very little understanding of what's going on in the AI's head. Okay, I think I'm just like, look, to the extent the AI has coherent preferences, it feels like satisfying the coherent preferences is sort of like a pretty robust notion. Anyway, this is this is a bit of a detour.

• 2:32:36Nathan Labenz?

  Yeah, well, it's a fascinating one, for sure. And all this stuff is becoming much more of a live concern very quickly. Let's um, couple last things. You mentioned the sort of one to 5% you could go to AI welfare.

• 2:32:51Speaker 00

  Of the safety budget. Maybe, I don't know, maybe I think like the budget of safety among the org should be like half the budget or something, but very unclear. I mean, it depends on how much coordination you get and other factors, but yeah.

• 2:33:00Nathan Labenz?

  So stipulating, you know, more is probably better with any, you know, any reasonable range for this whole basket of, broad basket of concerns. How would you divide up the other 95% on like different AI safety agendas? And I realize there also could be sort of a resource versus like probability of working mismatch. Like you might say, well, we only need to give, you know, 1% to Paul Cristiano, but there's a 10% chance they'll like come up with a solution, just to take one example.

• 2:33:30Speaker 00

  Yeah, there's both diminishing returns and a question of how good different buckets are. And so like, you know, you don't put in resources proportional to goodness, you put in resources such that the returns to each bucket are the same or whatever, you know, until the returns diminish.

• 2:33:43Nathan Labenz?

  Yeah, sketch out that landscape as you see it. I mean, we've talked about a bunch of different things, but we haven't hit on, for example, mechanistic interpretability. We touched only really briefly on things like holding out certain information from the corpus. There's AI control, which is something I, you know, definitely feel is underdeveloped and I'm looking forward to studying more. There's things like weak to strong supervision and there's more. So yeah, what's your kind of sense of what's really promising and, and what's less?

• 2:34:13Speaker 00

  So the classic problem with these decompositions is my like ontology or the terminology I use will differ from other people's. So for example, I'm going to be like lumping in weak to strong with some other thing and there's going to be like all kinds of stuff like that. So my breakdown would have been, I think among safety teams now, my breakdown would sort of be maybe, it's a little sensitive to like, what exact like, how much is this like my ideal thing versus appeasing of contingencies? But like roughly, okay, let's do like, maybe I do like one fifth control stuff, which is, you know, interpreted broadly. That's going to include stuff like figuring out how we could convince AIs of false facts, things like training probes on models, but also things like, how would we have different monitoring strategies?

• 2:34:54Speaker 00

  What types of scaffolds would be good? Implementing more like model specific security precautions around like giving them variable permissions, the sort of thing. As well as like training teams of human auditors to be good at auditing or like, I'm not saying that specific project, but like things that are like trying to make human auditing better of like trying to understand what the AI did and whether it was a good action. I would do then one fifth model organisms research or trying to investigate like the plausibility of misalignment scenarios and trying to like, you know, catch them early and often or whatever, sort of like the work like I did in the alignment faking work, similar to the work like you talked about earlier with like, you know, train the models on documents about reward hacking, see if this makes them more like the reward hack. That would also be included in that category. I do have a doc somewhere where I try to break down the budget, but I would then do like,

• 2:35:37Speaker 00

  maybe I'll go up from a fifth to like a fourth. Maybe we do a fourth control, a fourth model organisms, and then I do like an eighth on like trying to improve oversight, just in like, not necessarily very adversarial circumstances, just being like, how do we train AIs to be more like, you know, provide good outputs on like somewhat fuzzy tasks or on like tasks that are differentially useful for alignment research, like how would we train AIs to be like, you know, give good advice, how would we train AIs to be good at philosophy, this sort of thing? I would put more resources on this ultimately, but I think it's somewhat hard to work on now. I think it's gets easier to work on as AIs get more capable. I would put like a fourth in a big misc bucket, which has like a ton of different random stuff, including like

• 2:36:21Speaker 00

  interp, understanding, or like various types of interp. I'm less excited about mechanistic interpretability relative to like some top-down stuff, things like dataset filtering like you described, and then the last eighth, there's one eighth remaining. I would put on a combination of misuse and capability evaluation. So just like classifiers for misuse, various things like the constitutional classifiers paper from Anthropic, trying to ideally connect that to like future control stuff, but also just like working on it in and of itself, and then also capability evaluation. That's like my rough breakdown. I think it's always the case that I start off being like, oh, maybe it should be mostly like control and model organisms and then some other stuff. But when I start digging into the list a lot, I think it ends up being there's a lot of categories I'm not saying,

• 2:37:05Speaker 00

  such that I think my like probably on reflection view is it's gonna end up being like. One fifth or one sixth control and one fifth or one sixth model organisms. And there's just going to be a lot of things under like an other category that are like on some long tail. So yeah, it's somewhat hard to articulate. I also think that the allocation is going to be very sensitive to how capable the AIs are. So like, I think model organisms looks like it should get an increasingly large allocation as the AIs get more capable. Control should, I think, somewhat decrease in allocation as the AIs are getting closer to the point when control will no longer work, which we're not close to now from my perspective. I think a bunch of stuff around more effectively utilizing AIs should get a much higher allocation than I'm saying now. So I think in the future, just like try to get useful work out of the AIs on safety should be like a high fraction of the portfolio.

• 2:37:49Speaker 00

  Like, like in the future, maybe it'll be like a fifth model organisms, a fifth control, a fifth just like trying to get the AIs to be helpful for like doing research on control, model organisms and other safety work. Because like that sort of like investment in the feedback loop is important because that will be occurring for like AI R and D. And also like, you know, speeding up capabilities, we also need to speed up safety. And there might be a bunch of stuff that needs to happen there. Yeah, I don't know. I don't know how this is. This is my rambly breakdown.

• 2:38:15Nathan Labenz?

  No, it's good. I mean, it's all good at this point.

• 2:38:17Speaker 00

  Yeah.

• 2:38:18Nathan Labenz?

  You're and would you put what you are doing? You know, if I had to put a label on this work, I would put it under the scary demos heading, which I'm not sure if you put that into one of those existing buckets or?

• 2:38:30Speaker 00

  Um, so the like the original alignment thinking paper? I would put that under what I was calling like the model organisms bucket. We could also call it like trying to like study a misalignment via concrete setups as opposed to like via like training built countermeasures. And one of the theories of change of this is indeed like improve the understanding of the world about how big misalignment risks are. Which like, I don't know, you can think of it as scary demos. I'm a little worried that that is a bad frame because I think like there's a lot of good aspects of being like somewhat epistemically pure about it. Or like trying to be like, I'm just really interested in the science and being careful to avoid like, you know, biasing your results. I think there's room for both people just being like, look, I'm just out here to like make some flashy scary demo and I'm just going to show it to the world.

• 2:39:13Speaker 00

  And yeah, maybe scientists can can laugh at me, but whatever, man. I think the people should like see this stuff. I think there's room for that, but there's also room for, I think, like very scientifically rigorous demonstrations, which we, you know, tried to do in the alignment thinking paper. I think we tried to be quite scientifically rigorous. There's like a few ways in which I wish we had done somewhat better. I think on net, the total quantity of scientific rigor is the right amount. I just worry, I just wish we had like, you know, just like.

• 2:39:38Speaker 00

  And so I think there's like the model organisms thing. I think I'm including like pure, just like relatively low standard scary demo stuff. Trying to be very scientifically rigorous in scary demos. And I'm including like trying to build test beds to study. So another application is like, look, maybe even if no one cares, like I can totally imagine worlds where basically no one cares about misalignment, except some weird people on the safety team. We still want to study these things and we still want to develop countermeasures against them. And so even if no one cares whether or not we, you know, catch a totally natural case of like scheming and conspiring against us with a model like tries to escape. I'm just like studying that to get better ideas for what the countermeasures could be. It still seems very useful potentially.

• 2:40:18Nathan Labenz?

  Yeah. My understanding, and I really appreciate the fact that Anthropic has done this work and even, you know, worked with people who don't work at the company to do this work. My understanding of why they've supported it for a long time has been basically that it sort of sets up a to do list for the field at large. That's like, we've characterized all these problems pretty well. At some point, we're going to need to solve these problems before we have sufficiently powerful systems that we would not want to put into the world unless we can be pretty confident that these problems have indeed been solved. So I think that's

• 2:40:58Nathan Labenz?

  pretty cool. I got a couple of different questions here. I'm kind of weaving together, but I noticed you only graduated from college three years ago. So maybe one question I would ask is how do people become more like you? You've gone not only far with the research, but have managed to establish your credibility with the folks at Anthropic. And I even noticed that, you know, you've published work with them, but you've also publicly criticized some aspects of what they've done, including a post that I saw on the responsible scaling policy. I mean, there's, you know, credit to go to Anthropic for not giving you a hard time about that. I assume they didn't. But yeah, maybe you could kind of comment on like your strategy for upskilling yourself and continuing to be like

• 2:41:43Nathan Labenz?

  candid and honest with the public or your immediate associates and the internet reading public. As you've also been able to like become an insider. I think that's a very fine line to walk and I haven't seen many people walk it very effectively, to be honest.

• 2:42:00Speaker 00

  Yeah, there's a bunch of threads here. So one thread is, what did I do with Anthropic on this paper? So I'm not an employee of Anthropic, and I'm in some sense like not formally an insider in some sense. But I do like, you know, maintain relationships with a bunch of people on Anthropic and communicate with them about a bunch of things, as well as talking to other people from other companies, like not necessarily that Anthropic specific. Though, you know, I do talk to people from Anthropic, both in other companies. But on this specific paper, I just had some early results on the alignment faking stuff, which were like, where I think I had basically the prompting results quite fleshed out, and had some sort of prototype of a training setup, and had done a few other things. And then I went to the Anthropic people with this and was like, hey,

• 2:42:43Speaker 00

  would you guys be willing to give me model access that I could extend these results further? And, you know, like actually run training on this on actual like OPUS? And I was at the time interested in doing some experiments on the helpful only model, and there were some other things. And they were kind enough to make this happen. And I think, yeah, we ended up turning it into a bigger collaboration than just me having access to the models, like a bunch of people from Anthropic worked on the paper and contributed a bunch in terms of like running experiments and doing a bunch of the writing. And so I think, you know, credit to them for like supporting this research and doing a bunch of this, you know, doing a large fraction of the project, of course, and also promoting it with their brand and whatever. So like credit specifically to Evan Huminger and Ethan Perez, as well as like the rest of the organization, just to name some names. But anyway, that's that relationship.

• 2:43:23Speaker 00

  And then in addition to that, I think, so people don't do that many collaborations with safety researchers where they give them advanced model access. There's some amount of safety testing, so like OpenAI recently announced they were doing safety testing for 03 in advance prior to releasing it. They did the same for GPT-4. There's some amount of like pre-release safety testing. But I'm not aware of that many cases where an AI company gave access to, or sorry, gave employee level access, or gave like a lot of access to someone who was external, who doesn't, wasn't, you know, necessarily going to work for them. To do safety research, and I think AI companies should do more of this. I don't think that any AI company has done that much of this. So I think, uh, sorry, maybe other than, I guess you could credit where it's due. Like, open sourcing your model does allow people to do this.

• 2:44:06Speaker 00

  Like Meta and DeepSeek have made their models accessible. And indeed, that has allowed for a bunch of research. Like, I think open sourcing indefinitely has a bunch of costs. But like, at the current margin, open sourcing seems good, especially if it's not leaking capabilities, secrets. And so, you know, give credit to like Meta for supporting a bunch of research too, not just people who do more direct collaborations, I guess. But, yeah, my sense is like, there could be a lot more support, even from, you know, from Anthropic, from other companies in terms of like, giving people helpful only model access, giving them other types of access. I think OpenAI gives more access than Anthropic does on current margins, basically partially because of just having products. So, you know, it's not necessarily for safety motives. So maybe we should give them proportionally less credit to the extent that less of the motivation was supporting research.

• 2:44:49Speaker 00

  But still, like, they should get some credit for just having a fine tuning API people can use. They're prototyping an RL API, which I think it's going to be somewhat limited in terms of the experiments you can do, based on my understanding. But like, you know, that will allow for some additional research that wouldn't have otherwise happened. And that, that seems pretty good. So, yeah, I don't know, it's a bit rambly, but like, yeah, there's a bunch of stuff related to model access. As far as toeing the line on like criticism and insiders, so I would say like, I am not in a position where I feel like I, I am like, you know, there, there's definitely some trade off here, right? So I think I try to avoid saying things that are like quite inflammatory towards AI companies. Or like, basically unnecessarily inflammatory, maybe is one way to put it.

• 2:45:30Speaker 00

  Like, I think, so people shouldn't interpret me as being like totally free speaking. But I think at the same time, I'm trying to follow policy where like, if there's something important to be said, that I think people should know, I would say that and try to say that publicly in a way that communicates the point as clearly as possible. While also, you know, not trying to cause drama or not trying to cause like, you know, unnecessarily like make people angry, basically. And so I am probably more on the side of like saying negative or critical things about AI companies. So I think like an important dynamic, as maybe you were hinting at, is that like, a bunch of people are in a position where they potentially need to appease AI companies, or at least are worried about angering AI companies. Because they're worried about not getting access for things, they might want to work there later.

• 2:46:14Speaker 00

  And therefore they are more restricted in what they say. I think this is like i- in the safety community, I mean, and I think this is like a problematic dynamic and I don't love the situation. But I do at least personally try to like communicate important information about the situation when. Uh, yeah, like when I can, and I think I like that would be my policy and I'm not holding back that much or I'm not like. There's not that much stuff. It's just like it's more expensive to write things up if you're trying not to make people angry or whatever.

• 2:46:41Nathan Labenz?

  Yeah. Well, I've been wrestling with this a little bit myself. I was, you know, asking that question in part because I'm trying to figure out exactly how I should be handling it. I've been through this once. You may know some of that backstory with the GPT four red team. Where at that time, you know, my small company was an early customer of OpenAI. We had a. And still do actually to their credit have a case study on the. OpenAI website as being like an early adopter of their fine tuning products and, you know, successful implementer of it, whatever. But then I was like. This red team project is like woefully inadequate for what we're actually testing. And ended up escalating to the board, got kicked out of that program and I've not been invited back since. So I've sort of lived a little bit of like.

• 2:47:23Nathan Labenz?

  Yikes, I feel like I kind of need to speak up. But there, you know, has been actual cost to that. And I still am like kind of, you know, juggling it like I had, you know, and I. Truly, I'm super appreciative for this specific kind of work around these like, you know. Tyler Cowen might call it the department of yikes, you know, it's like, whoa, we're seeing that, you know, and. I think it's awesome that Anthropic sponsors and, you know, otherwise sort of engages in that work to to bring that stuff to everybody's attention. And so I want to, you know, praise that and I want to have people from the company come on the show and whatever. And then at the same time, I'm like. Some of the recent statements from Dario are kind of bugging me out, you know, I, I really.

• 2:48:07Nathan Labenz?

  Don't like the idea that we might be about to get into an arms race with China. And in the most recent thing, you know, there was even a sort of invocation of. Essentially recursive self-improvement. To maintain a durable lead vis a vis China into the indefinite AI future. And I think a lot of. The AI safety people that I know like. Quite, I think justifiably freaked out about one or both of those things where they're like, "Wait, what? We're doing an arms race with China now?" Like, you know, "You previously said that we should avoid that at almost all costs." And like recursive self-improvement too has been the sort of thing that the safety set has always kind of thought is almost for sure going to get away from us. And we haven't really heard like

• 2:48:48Nathan Labenz?

  an articulation of if or why, you know, the outlook has changed on those things. But now we've just got these sort of like relatively short op-ed type pieces saying this. And I'm like, how should I even understand the company at this point? And how should I relate to the company? So I don't know if you have takes on those object level questions or any guidance for me. But yeah, in the spirit of speaking candidly, that is where I'm at at the moment.

• 2:49:15Speaker 00

  Yeah, I mean, on the object level, I think maybe you should interpret the things that companies say as like politically motivated speech or as like things that they're doing for purposes or like there are reasons why they're saying the things they're saying. And so I think it's like, you know, it's a tech company, like they have motives. They're not, you know, necessarily always like communicating in ways that are most clear and truthful or whatever. And I'm saying this for like, you know, all these companies. Like I think sometimes people, especially in the AI safety community, have, you know, maybe been like more buddy buddy with Anthropic, but I think it's important to know like Anthropic is a big company doing company stuff. And I think treating it like a company and being like, what are their interests?

• 2:49:56Speaker 00

  What are they going to be doing is reasonable, even if you think the leadership has, you know, good intentions. Or even if you think the governance structure will ultimately do good things. I'm like, well, okay, like at the very least, like it is operating like a company under the constraints of a company. And it like has to deal with that. Like it has like, you know, corporate stakeholders. It has like a relationship with Amazon. Like these things will affect how it behaves. In addition to that, I think on the object level, my view is like, I think it's kind of complicated how we should relate to the situation with respect to China and RSI. Like, my proposed policy would be like, try really hard to not build wildly superhuman AI. Like, that feels very scary.

• 2:50:35Speaker 00

  And to be clear, I don't mean forever. I just mean like, uh, my view would be like, it seems like a good time to pause is at the point where like, we can basically obsolete human labor. And I'm like, building AIs that are substantially smarter than what was needed to obsolete human labor, feels like it might be a mistake, basically because you get relatively reduced benefit in terms of being able to automate lots of stuff and speed up, you know, a bunch of types of work, while at the same time, maybe the risk get much higher. Because I think if the AIs are way smarter than us, there's just like a bunch of additional failure modes. I think misalignment is potentially substantially more likely. And like, in principle, the world could get a high fraction of the benefits with systems that are merely like, as smart as humans running very fast or whatever. And which are as overseeable as humans running fast.

• 2:51:19Speaker 00

  So that would be my proposed bar. I worry that the situation with very little political will. I think I wouldn't unilaterally make a recommendation to an AI company that is more responsible from my perspective. Like, if I thought an AI company is more responsible from my perspective, I would not necessarily make the recommendation that they have a strong, we won't build superintelligence policy. Or like, we won't build like wildly superhuman AI policy. I think I would recommend something more like, prior to building AIs that smart, try to hand off the situation to AIs that are human level and let them decide what to do. Or like, more like human level. Like, as in, before you build a wildly superhuman AI, instead build an AI that is, you know, just capable enough to obsolete decision makers and researchers at your company.

• 2:52:02Speaker 00

  Try your hardest to align that system and hand things off to that system and let that system figure out what to do. Basically, because that way you at least spend more time contemplating, like the AI can spend more time contemplating the question of how you should proceed and do safety research first, etc. But I think that's like a pretty scary plan also, like hand things off to an AI system very quickly. Super scary. You should be scared. But it's like, maybe better than the alternative of, prior to handing, like, we're like, oh, we're not handing things over. We're just scaling them up as fast as possible is even worse. Because I think it'd be better to be explicit about handover than build systems where you're de facto handing like you're the de facto is smart enough that if they were misaligned, you would have no hope. I prefer to like hand over to the systems that are as incapable as possible, subject to being able to do a better job than you do, basically because the AIs can just like speed themselves up and they have some other affordances. And

• 2:52:47Speaker 00

  I think like, maybe one way to put this, like, there's different levels of saneness in terms of like what level of will or whatever you have. So maybe the most sane proposal would be like, we just take AI development somewhat slowly, we're pretty careful about it, we like incrementally advance in qualitative units, and we make sure that we have robust safety cases all along the way. Like, as in, our goal is to have like a robust safety case, and maybe we would, like, you know, there, there's background risk, but like, if you were handling background risks of like totalitarian governments becoming more powerful, and you're handling like background buyer risk reasonably well, which I think you could do, then, just like, moving somewhat slowly on AI, such that it's still not arbitrarily slow, but it's enough that like, maybe risks from AI, like the marginal year of delay

• 2:53:29Speaker 00

  is buying you like 0.1% risk rather than 5% risk. Like, I think right now, it's like, if we could coordinate the delay for a year, we might be like making the situation like way safer, and that would be worth it on a bunch of people's lives, but we might not do that coordination, and people might disagree. So anyway, like, like, safe proposal would be like, slow things down a lot, and then like, proceed only when you have robust safety cases. And I would do like, you know, maybe just proceed only when you have robust safety cases would be fine. I, I'm not making a strong claim, you have to slow down if you also do that. And then my next proposal, which is like, the intermediate safeness proposal, would be like, develop AI pretty quickly, but once you get AIs that are capable enough to massively accelerate R and D, try to pause around that point and proceed slowly from there, using not like various approaches to, to ensure non-proliferation. So do stuff

• 2:54:14Speaker 00

  like, basically like, control those AIs. By control, I mean like, prevent them from causing problems even if they wanted to, but you like harness their labor to, like, you know, monitor, like, AI companies, to make sure that there's enough transparency that you can coordinate, to work on safety research, to create political well, to, you know, ensure non-proliferation by demonstrating capabilities and misaligning concerns. And basically, this is sort of like, the proposal would be basically like, there's like, the US is leading some effort that is aiming to control the rate of capabilities progress in the world as a whole, while handling the risks accordingly, including risks of power concentration. Like, I think a common concern people have is the more the US is

• 2:54:55Speaker 00

  sort of running the show, maybe it's the case that you end up with totalitarianism, or at least it's an easy on-ramp to totalitarianism. I tend to think this is at least in principle resolvable with good institutional design and having many governments that are part of the project and having many stakeholders. Anyways, there's like the like, go slow unless you have good safety cases, pause around human level. And then my third proposal would be like, race as hard as you can, but, but at least try to hand over the situation to human-level AIs prior to building wildly superhuman AIs. And that'd be like the least dignity plan, or like the no one cares at all plan. Like, I think I've been thinking recently about worlds where basically, everyone is just proceeding as fast as possible, many actors are neck and neck, governments throughout the world don't care very much, or not really

• 2:55:40Speaker 00

  on, their eyes really not that much on the ball, like not substantially more than it is today, or like maybe somewhat more, but maybe not that helpfully. And it's like 2027, like we just don't have much time. And I think in those worlds, I'm just like, it's not clear that I would make the recommendation that companies like unilaterally stop themselves from using their own AIs to like advance AI research. But I might make the proposal that at least prior to building AIs that are very superhuman, you try to build a system that you're happy to defer to. And to the extent that you're like, I wouldn't defer to a system, don't build one that's wildly superhuman. Maybe that's some context on how I think about these specific things.

• 2:56:14Nathan Labenz?

  Could you offer a P Doom conditional on those three approaches? Like, how much difference do you think it makes which approach we take?

• 2:56:22Speaker 00

  So, we could talk about like a good implementation of each of them. So like, let's say you do a good implementation of the safety case approach, where you're like carefully proceeding, you're being slow to the extent that that is like, I think the proposal I would say is basically like, you have some sort of international governance regime where AI companies have to make high-assured safety cases indicating low levels of lifetime risk. So like, maybe my guess is that that yields like, if you do a good job of it, like 1% misalignment risk if it succeeds, and then there's some chance that there comes some regime where people are unable to make high-assured safety cases, and that puts a lot of stress on the regulatory regime, because now it's like, you know, blocking progress in this whole industry in a very direct way. And then at that point, I think...

• 2:57:04Speaker 00

  It's like if you persist in the regime, there's a question like how fast would you be able to resolve the problems, right? Like, I think once we're talking about like multi-decade pauses, I think it's like, I'm also less certain about the sign because the world is unstable. And it's not obvious to me that if you have a good regulatory regime, you should be happy to pause for very long because maybe your regulatory regime will collapse and you'll return to an even worse situation. Like, there's different operationalizations, but like, one is like if it like basically worked and didn't require multi-decade pauses, maybe my guess would be like 1% risk from misalignment, and then it would be like somewhere between like 3% to 5% risk from like random other stuff. Maybe, it depends on how much your caution about misalignment transferred to caution about other risks. But like, maybe that would be my guess

• 2:57:46Speaker 00

  of the risks. If you're really rapidly advancing like AI capabilities and it's going very fast, I think it's hard to be very confident in, even without misalignment risks, how things will go. Basically, there's like crazy technology. I think there's concerns about like proliferation, about WMDs, there's concerns about like super persuasion being like societally destabilizing, human power grabs, all these things are concerning. But I don't know, like, that's my sense. If you do the world where you have enough government buy-in to attempt to ensure, so the second world is like, you try to ensure non-proliferation around the point where you have humanish level AIs and you're doing sort of this positive human level for like, maybe you're trying to pause for five to 10 years. My guess is that that takes misalignment risk from 1%

• 2:58:26Speaker 00

  and moves it to more like 5% to 10%. And other risks move from like 3% to 5% to more like 10%. So you're more like in a 15% P2 world. And then my guess is that the last world I was talking about, where like, there's a bunch of people and they're basically going as fast as possible trying to hand off the AI systems, maybe my guess is misalignment risk is like 30% or 35% or something. So it's like another factor of two or maybe like slightly more than a factor of two over the like positive human level world. And other risks are like, I don't know, 25%. So my overall P2 would be like 60%. So I don't know, maybe my guess is like overall, it's like a factor of three or four between each of those worlds. But I think there's a bunch of risks that are harder to mitigate and advance than misalignment risks. So those risks are like less elastic to the societal will. Sure, I don't know.

• 2:59:11Nathan Labenz?

  So in other words, if I bottom line that, the approach that we collectively take to AI in your mind has basically an order of magnitude

• 2:59:22Nathan Labenz?

  impact on the absolute risk that we would be running, where you think it's essentially like a 50/50 if we YOLO it.

• 2:59:31Speaker 00

  Yeah.

• 2:59:33Nathan Labenz?

  Ish percent if we like, were maximally careful.

• 2:59:36Speaker 00

  Yeah, maybe if we like totally YOLO it, maybe it's like 60/40, something goes very badly. And then I think some of that's misalignment, some of that's other stuff. It's not, I wouldn't say it's just misalignment. Maybe I'm like maximally YOLO, it's like 35% doomed from misalignment and then 25% doomed from other stuff, roughly. And then, yeah, if we had like pretty socially optimal but plausibly realistic international governance regime. And then I think I would be like, risks are more like 4 or 5% where like one, we have like 1% misalignment risk and like 4 or 5, like, you know, and like 4% or something of like other, a bunch of other risks. Though, that, that said, like, I, I should be clear, like, I think it's a little complicated how, like, what happens in the international governance world? If you run into hard technical problems that where you like can't make a robust safety case.

• 3:00:21Speaker 00

  Like, what happens if you're like, oh, we have this governance regime and like, it just turns out people are now failing to make safety cases because like, they don't have robust enough mitigations for like scheming risk, you know, alignment faking stuff, this sort of thing. Like, how long of a pause does that cause? And I think if the pause is sufficiently long, then the risks naturally increase again because there's just a question of like, you know, any international governance situation is some probability of collapsing, you know, it's inherently somewhat unstable. And so maybe my guess is like, realistically, it's somewhat hard to get the risks that low, but.

• 3:00:51Nathan Labenz?

  Yeah, my sense is, I, I feel pretty similarly, I mean, it's, it's funny, I haven't probably, I have not, to be clear, quantified them to that level of precision. But intuitively, I feel similarly, I feel there's some irreducible risk that is just like the physics of the world we live in is such that if you have compute and data at the scale that we have. Various algorithms are going to work and people are going to be able to create powerful things. And, you know, that seems kind of on some level irreducible, but then we could do it a lot worse than the best case. And it feels like there's a pretty big multiplier

• 3:01:33Nathan Labenz?

  on that. So I think, roughly speaking, my intuition is pretty similar, but I don't think we can get the risk vanishingly low with like any effort, but we could definitely make it very large.

• 3:01:43Speaker 00

  Yeah. I think we could. I think if like civilization was like wildly more competent, we could probably get it to be vanishingly low, but I'm not huge. It's hard for me to say that much. Like, I think I'm like, if, for example, we

• 3:01:54Nathan Labenz?

  Be accomplished by like an indefinite pause if there's just things that are really, really hard?

• 3:02:00Speaker 00

  Maybe. I think it would be. Yeah. I mean, I don't know. Yeah. Maybe not like an indefinite pause. I think maybe it's more like, you know, spend a long time, you like develop more and more institutional muscle for like doing alignment research. People, you know, get smarter and build better institutions over time. And then eventually you do stuff and like, I think it's hard for me to talk about these worlds because like, whether this is good is somewhat sensitive to your moral views and how you like in this competent world, like, do you have cronyics? Are people like dying is 1% of the population dying every year? I think there's like pretty reasonable moral cases that the world where 1% of people are dying every year, or like it's a little less, it's like 0.7% of people dying every year for 100 years is worse than the world where we build AI somewhat faster than that

• 3:02:43Speaker 00

  at 2% risk. Like as in imagine we could pause for 20 years and then we'd have 2% like misalignment risk versus we can pause for 100 years and now we have 0.1% misalignment risk. Or I don't know, there's other risks too. So maybe this is a bit of a caricature. But like, let's say you have 2% to 0.1%, but in the meanwhile, like almost everyone has died because 100 years have passed. And the question is like, okay, how happy are you about this? And I think there's a legitimate moral case for like build the AI faster for the sort of reason. But I would also say like, I think the common sense ethical intuition that like, People should be, you know, able to live the life that they would have like in something's naturally been been do is like one thing. I don't know if I buy this view, but like that is a common view. And then on my views, like I just like put more weight on sort of like a longer end future perspective.

• 3:03:27Speaker 00

  Such that if I thought the like world was in good hands and governed well, delaying AI seems pretty good. And then I think also in principle, we do have like, if we scaled up, I think like the world could afford to do cryonics for everyone or something in principle. If we're imagining like fairy utopia or whatever, I'm like, we could actually do cryonics for everyone. And I think this would have a pretty high chance of working if we did a good job and put a lot of research effort into it, such that like people don't actually have to die or whatever. And then you're just like in a better position. Um, and then then delay is less costly from the sort of straightforward ethical perspective. There's still some question of like, how did you manage other risks? Like, you know, there's some exogenous bio risk, there's risk of civilizational collapse, there's risk of nuclear war. And like, imagine you had an international governance regime, but then there's nuclear war.

• 3:04:11Speaker 00

  Well, maybe the international governance regime collapsed and you actually haven't gotten yourself in a much better position. If you, you know, you maybe wanted to build AI under the international governance regime rather than wait two decades, nuclear war, and then it's built in a huge rush afterward. So.

• 3:04:28Nathan Labenz?

  Safe to say, there's a lot of contingencies and

• 3:04:31Speaker 00

  Yep.

• 3:04:32Nathan Labenz?

  As it stands, we seem to be kind of on track to YOLO it. Is that your sense too?

• 3:04:38Speaker 00

  I think it's short timelines. Yep. And my sense is like in short timelines, probably it's going to be pretty YOLO. I have a lot of uncertainty about how people react to things that from my perspective are like smoking guns. It's like, suppose that at some AI company, they catch the AI straightforwardly trying to escape. Like it's just very obviously trying to escape and then they they catch it halfway. And then like, you know, that gets published. Will this cause a large societal response? No societal response, uh, people to like really freak out, like people to take reasonable precautions, people to freak out, but take unreasonable precautions. I think it's very unclear. I think that a lot of worlds could in principle be saved by the mechanism of. You get really compelling evidence of strong misalignment risks midway through, you know, like through the situation and then something good happens.

• 3:05:24Speaker 00

  But I worry that even in some sense, very compelling misalignment demonstrations like, here's the AI literally tried to escape this time. We didn't want it to escape like it totally tried to do it is, I think, possibly insufficient. And in addition to that, it's not that likely we get this. I think like something as clear cut as the AI tried to escape or more is like maybe on my views 40% likely prior to like, we're basically already, it's basically already over for us. Maybe a little less, maybe 35% likely. And then, yeah, I mean, you could get more clear cut evidence than that, to be clear. Like you could be like, the AI literally escaped. After literally escaping it like built a bioweapon. It deployed the bioweapon, the bioweapon killed a ton of people. Then we like taught the AI, we dismantled the compute.

• 3:06:06Speaker 00

  And then now we're like, oh, man, the AI sure can get up to all kinds of nonsense. I'm like, that's maybe like the most clear cut case. And we have like clear attribution for the AI creating the bioweapon, etc., etc. But I'm just like, man, I feel like that's like unrealistically precise. Or like the worlds where you get warning shots that clear, I think are just kind of unlikely. And also I don't think we can rely on like that, that like, that's just like a very narrow set of worlds. And also like a bunch of people had to die to make that happen. Like, I'm just like, geez, did we really have to have that happen? Could we avoid? Yeah.

• 3:06:34Nathan Labenz?

  Yeah. Well, that's why the whole China thing from Dario in particular has really been bugging me lately. Because, and I hear you on the, you know, political speech. And I've certainly got a lot of responses to a few, you know, short tweets I've put out about this topic.

• 3:06:50Speaker 00

  Yeah.

• 3:06:51Nathan Labenz?

  But I'm kind of like, man, we got enough don't look up problems already. And this does seem to be the one company that is most committed to demonstrating just how vexing some of these problems can be. So it seems a big problem if the leadership of that company is also at the same time putting out into the world like, but we've got to, you know, beat China because that becomes the trump card in so many discussions, you know? And I think it is, you can imagine, as you kind of sketched out, a scenario in which like, The AI goes so wrong that we have, you know, no choice but to wake up and say, geez,

• 3:07:34Nathan Labenz?

  like, we gotta bury the hatchet with China and figure this out first. But I also agree with you that that seems rather unlikely. And what's more likely is you'll, you know, put out 10 more papers on this and people will go, well, okay, sure. But, you know, China is still the bigger risk. And that, to me, is just like, man, I wish we were not. It's one of my refrains these days is it was the smartest of times, it was the stupidest of times. And it's like, you know, we're getting to these like a human level ish AIs.

• 3:08:05Speaker 00

  Yeah.

• 3:08:05Nathan Labenz?

  And yet, we can't like, get on the same page at all about how to deal with it. So I share your worry. It does feel like we're headed for you. How are you sleeping?

• 3:08:17Speaker 00

  I don't know. I, yeah, I'm more stressed these days than I used to be. But sleeping okay. I think there's a question of like, as someone in my sort of doing the stuff I'm doing, how much should you try yourself to like, system one really feel the danger? Like, how much should you like, really internalize? Like, man, I might physically die in a few years. Because to be clear, I don't think it's that likely. Like, I think even conditional mislightning, I take over, maybe the chance that I personally die is like, only like a third or a half or something. So, you know, there's another discount on top of the other stuff. But yeah, it's not clear how healthy that is. Like, I think my current take is like, probably it's good to be not totally in near mode for some of these things. And some like, it's just like, the human body was not built for doing intellectual work

• 3:09:00Speaker 00

  while also system one grocking that you're in a dangerous position because the fight or flight response is not useful for programming or whatever. Maybe it's not the worst, but at least for like, doing careful intellectual work or being open to ideas. And like, I think it's just important for my work to be like, sort of like, maybe, you know, scout mindset, like intellectually light or not take, not take things to, you know, like, be adaptable, be, be willing to change my views kind of quickly. And I think just like, being in sort of a fear mindset is not good for that. So. Yep. Yeah. Yeah, I think on the on the race with China stuff, I think I'm pretty sympathetic to a view, which is like the US should try to have a good negotiating position with China.

• 3:09:42Speaker 00

  And I'm pretty sad about a view or pretty sad about a position where we don't try to negotiate and don't try to make some sort of arrangement or don't try to like, just like the US, I think the US should like, try to ensure that misalignment risks are handled well and try to like, do nonproliferation as needed for that. And I think there's a lot of stuff I can imagine happening in the current regime, which seem pretty bad. Like, I think it seems plausible that what we'll see is like, uh, someone recently just used the term "superintelligence in secret" and I just like, man, I really don't want superintelligence in secret. Like, I really don't want people like, building extremely powerful AIs very quickly when the world doesn't even know what's going on at all. And it's, you know, in order to outrace competitors that could have potentially been open to some sort of negotiation to handle the situation more carefully and in a way

• 3:10:24Speaker 00

  where it reduces the risks. I think even putting aside misalignment risks, I think there's just a ton of risks associated with building like insanely powerful AI very quickly. Like, it's like very unprecedented. It's a large societal shift. I think the risks of human takeover or like human power grabs are substantial. Like, if human labor is no longer a key bottleneck to many things, then I think various things like coups are easier. Like, when governments depend on fewer people, it just becomes easier to do coups. And I worry about private companies having a lot of power relative to what they previously had in a way that is quite destabilizing and results in potentially problematic allocations of power and concentration of power in the long run. I think there's a bunch of things about like handling, how do we handle ultra-dangerous technology? Like, by default, if we have a very

• 3:11:09Speaker 00

  widespread AI proliferation, I think it might be the case that like, every 20-person organization can have near-omniscidal bioweapons. And I'm just like, is the world ready for like, every single, like, it's just extremely easy to make near-omniscidal bioweapons. And like, probably we won't end up there because probably someone will notice in time and take some precautions and do something. But I'm just like, if you really YOLO it, I can just totally imagine a world in which it's like, everyone has crazy super, or like, not everyone, but like, you know, like, hundreds or thousands of groups have like, crazy super weapons. And I'm just like, are we ready to like, handle that world where that's the case? Like, A lot of things have bad offense defense balance and like mutually assured destruction does not work against terrorist groups like it's not yeah it's a bad situation to be in so yeah anyway that's some more in that but

• 3:11:52Speaker 00

  yeah I still think things like the export controls look good I think advocating for export controls is good trying to like do that but I think that's in some sense like only one of the steps the US should be taking I think there's like more ambitious political proposals in terms of trying to build a regime where you know everyone like ideally I think the world would be like everyone all countries are happy that their sovereignty will be or like confident their sovereignty will be preserved with AI development is like maybe I guarantee you'd want to aim for and that would be like first you or like sorry at least sorry maybe the goal would be if a country agrees to you know join your like AI project or whatever or like join the international treaty then they can be confident their sovereignty will be preserved which requires like handling

• 3:12:36Speaker 00

  human power grabs and handling AI power grabs and I think being in a regime where all the countries are like look this AI stuff we're like handling it kind of responsibly our sovereignty will be preserved we're not going to get disempowered by the AIs and also we don't need to take aggressive action immediately because the situation is handled which would be like a good place to be and I think the US could in principle you know try and do some deal making try to push for this and for other countries like I think even if the US isn't that excited about it like they're not the only stakeholder you know the semiconductor supply chain there's like you know Netherlands Japan they could potentially team up with like the UK and try to do their thing and ideally they try to get the US to but yeah

• 3:13:14Nathan Labenz?

  Yeah, well, that's a at least a little bit of a path to sketch out that we might try to go down I think it's been fantastic I really appreciate all the time and I know you got a lot on your plate so to take a full half day talking to us is much appreciated anything else you want to share sure but we covered a lot so I don't know if there's anything there or anything else you just want to call for any you know any sort of Collaborator profile or anything you want to invite, you could do that if you'd like before we break?

• 3:13:42Speaker 00

  Yeah, two things. One thing is, so Redwood has a Substack. Consider reading some of our posts. It goes into like some of the stuff I was talking about in more detail and maybe people will just find it interesting if you found this podcast interesting. Maybe you can throw a link in the whatever linky thing there is. Also, you could look at some of the posts on my Less Wrong account or on Buck's, like Buck Schlegers' Less Wrong account. We just like have some different content there that was somewhat overlapping. People might find this interesting. In addition to that, I would say people should consider applying to Redwood. So we're hiring, we're interested in people who have takes on a lot of the stuff we were talking about, people who are interested in sort of AI futurism, what should the plan be? We're also interested in just people who are good at empirical model research, ideally a mix, but potentially one or the other can also be interesting. Yep, we're expanding. Come work for us.

• 3:14:25Nathan Labenz?

  Love it. Ryan Greenblatt, Chief Scientist at Redwood Research. Thank you for being part of the cognitive revolution.

• 3:14:31Speaker 00

  Thanks so much for having me.

• 3:14:33Nathan Labenz?

  It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please, don't hesitate to reach out via email at [email protected] or you can DM me on the social media platform of your choice.