Anthropic's Responsible Scaling Policy, with Nick Joseph, from the 80,000 Hours Podcast

In this crosspost from the 80,000 Hours podcast, host Rob Wiblin interviews Nick Joseph, Head of Training at Anthropic, about the company's responsible scaling policy for AI development. The episode delves into Anthropic's approach to AI safety, the growing trend of voluntary commitments from top AI labs, and the need for public scrutiny of frontier model development. The conversation also covers AI safety career advice, with a reminder that 80,000 Hours offers free career advising sessions for listeners. Join us for an insightful discussion on the future of AI and its societal implications. Apply to join over 400 Founders and Execs in the Turpentine Network: https://www.turpentinenetwork.co/ SPONSORS: WorkOS: Building an enterprise-ready SaaS app? WorkOS has got you covered with easy-to-integrate APIs for SAML, SCIM, and more. Join top startups like Vercel, Perplexity, Jasper & Webflow in powering your app with WorkOS. Enjoy a free tier for up to 1M users! Start now at https://bit.ly/WorkOS-Turpentine-Network Weights & Biases Weave: Weights & Biases Weave is a lightweight AI developer toolkit designed to simplify your LLM app development. With Weave, you can trace and debug input, metadata and output with just 2 lines of code. Make real progress on your LLM development and visit the following link to get started with Weave today: https://wandb.me/cr 80,000 Hours: 80,000 Hours offers free one-on-one career advising for Cognitive Revolution listeners aiming to tackle global challenges, especially in AI. They connect high-potential individuals with experts, opportunities, and personalized career plans to maximize positive impact. Apply for a free call at https://80000hours.org/cognitiverevolution to accelerate your career and contribute to solving pressing AI-related issues. Omneky: Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off https://www.omneky.com/ RECOMMENDED PODCAST: This Won't Last - Eavesdrop on Keith Rabois, Kevin Ryan, Logan Bartlett, and Zach Weinberg's monthly backchannel ft their hottest takes on the future of tech, business, and venture capital. Spotify: https://open.spotify.com/show/2HwSNeVLL1MXy0RjFPyOSz CHAPTERS: (00:00:00) About the Show (00:00:22) Sponsors: WorkOS (00:01:22) About the Episode (00:04:31) Intro and Nick's background (00:08:37) Model training and scaling laws (00:13:10) Nick's role at Anthropic (00:16:49) Responsible Scaling Policies overview (Part 1) (00:18:00) Sponsors: Weights & Biases Weave | 80,000 Hours (00:20:39) Responsible Scaling Policies overview (Part 2) (00:25:24) AI Safety Levels framework (00:30:33) Benefits of RSPs (Part 1) (00:33:15) Sponsors: Omneky (00:33:38) Benefits of RSPs (Part 2) (00:36:32) Concerns about RSPs (00:47:33) Sandbagging and evaluation challenges (00:54:46) Critiques of RSPs (01:03:11) Trust and accountability (01:12:03) Conservative vs. aggressive approaches (01:17:43) Capabilities vs. safety research (01:23:47) Working at Anthropic (01:35:14) Nick's career journey (01:45:12) Hiring at Anthropic (01:52:06) Concerns about AI capabilities work (02:03:38) Anthropic office locations (02:08:46) Pressure and stakes at Anthropic (02:18:09) Overrated and underrated AI applications (02:35:57) Closing remarks (02:38:33) Sponsors: Outro

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas, and together, we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Torenberg. Hello and welcome back to The Cognitive Revolution. Today, we're sharing a cross-post from the 80,000 Hours Podcast, in which host Rob Wiblin interviews Nick Joseph, Head of Model Pre-Training at Anthropic, about the responsible scaling policy that governs the company's frontier model development. I've been following the development of these policies and other voluntary commitments from top labs

• 0:44Nathan Labenz?

  for much of the last year. At one point, I had the opportunity to participate in a workshop with Anthropic team members, in which we reviewed, discussed, and offered comment on the responsible scaling policy draft. However, I could not easily convert that experience to a podcast, and so I was particularly excited to see this episode published, and really appreciate that the 80,000 Hours Podcast team has allowed me to repost it here.

• 1:08Nathan Labenz?

  On the substance of the matter, I, of course, very much appreciate how hard Anthropic is thinking about AI risks and what can be done about them, and also how transparent and even candid they are willing to be about the substantial uncertainty that remains. I'm also very glad that their example seems to have inspired others. OpenAI and Google have since published similar policies, and I understand that XAI is working on one as well. Unfortunately, however, the trend does not yet appear to be universal. Meta, for example, to the best of my knowledge, has not published a policy describing how it plans to evaluate models during training, let alone how they plan to proceed if it turns out that they are developing dangerous capabilities. As we happen to be sharing this episode with just a few days left before California Governor

• 1:51Nathan Labenz?

  Kevin Newsom will have to sign or veto SB 1047, I will take a moment to say one more time that while it may not be the perfect AI safety bill, and indeed, the release of OpenAI's 01 model and the emerging paradigm of scalable inference time compute generally, do suggest that the definitions in the bill would need to be updated to stay relevant over time. In my view, the public does deserve to know what Frontier Labs are doing. And that goes double for Meta and any other companies who are openly sharing model weights. So whatever the fate of SB 1047 may be, I do think that we will need some measure that forces frontier model developers to publish detailed safety plans for public scrutiny.

• 2:31Nathan Labenz?

  In the last hour of this episode, Rob and Nick changed topics to focus on AI safety career advice. While this is not a sponsored episode, Nick's experience with 80,000 Hours Career Advising Service echoes my own and presents a natural opportunity to remind you that 80,000 Hours is now offering free one-on-one career advising sessions to Cognitive Revolution listeners. I encourage everyone to sign up for a free session at 80000hours.org/cognitiverevolution. And especially so if you are one of the experienced software engineers that Nick notes are in such high demand among AI companies right now.

• 3:05Nathan Labenz?

  As always, if you're finding value in the show, we'd appreciate it if you'd take a moment to share it online with friends or write us a review on Apple Podcasts or Spotify. Your feedback is always welcome too, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. Now, for an in-depth conversation about Anthropic's responsible scaling policy and breaking into AI safety in your career, here's Nick Joseph from Anthropic with host Rob Wiblin of the 80,000 Hours Podcast.

• 3:32Speaker 01

  I think this is a spot where there are many people who are skeptical that models will ever be capable of this sort of catastrophic danger. And therefore, they're like, we shouldn't take precautions because the models aren't that smart. And I think this is a nice way to agree where you can, it's much easier message to say, if we have evaluations showing the model can do X, then we should take these precautions. And I think you can sort of build more support for something along, along those lines. And it targets your precautions at the time when there's actual danger. One thing, other thing I really like is that it aligns commercial incentives with safety goals. So, once we put this RSP in place, it's now the case that our, our safety teams are kind of under the same pressure as our product teams where if we want to ship a model and we get to ASL 3, the thing that will block us from being able to get revenue, being able

• 4:16Speaker 01

  to get users, et cetera, is do we have the ability to deploy it safely? And it's a nice outcome-based approach where it's not, did we invest X amount of money in it? It's not, did we try? It's, did we succeed?

• 4:34Rob Wiblin?

  Hey everyone, Rob Woodland here. The three biggest AI companies, Anthropic, OpenAI and DeepMind, have now all released policies designed to make their AI models less likely to go rogue while they're in the process of becoming as capable as and then gradually more capable than any human being. Anthropic calls their one a responsible scaling policy, or RSP. OpenAI uses the term preparedness framework and DeepMind calls their one a frontier safety framework. But fundamentally, they all have a lot in common. They try to measure what possibly dangerous things each new model is actually able to do and then, as that list grows, put in place new safeguards that feel proportionate to the risk that they think exists at that point in time. This is likely to remain the dominant approach, at least in industry, so I was really excited to speak with Nick Joseph, one of the original co-founders of Anthropic and a big outspoken

• 5:18Rob Wiblin?

  fan of responsible scaling policies, about why he thinks Anthropic's RSP has a lot going for it, how it might make a real difference as we approach the training of a true full AGI and why, in his view, it should be thought of as a sort of middle way that ought to be acceptable to almost everyone. After hearing out that case, I pushed Nick on the best objections to RSPs that I could either come up with myself or find on the internet and those include that, one, it's hard to trust that companies are going to stick to their RSPs long term. Two, it's difficult to truly measure what models can and can't do and an RSP is useless if it's mismeasuring model capabilities. Three, it's pretty questionable whether profit motivated companies will go out of their way to act in good faith and make their lives and their product releases much more difficult. And four, that in the most important cases, we just don't have safeguards that are able

• 6:03Rob Wiblin?

  to render new AI capabilities safe, even capabilities that could show up really soon. At the end of the day, I come down thinking that responsible scaling policies are a really solid step forward from where we are now and they're a great way to test and learn what works and what feels practical from the experience of people who are working on the call face of trying to make this technological revolution actually happen. But I think in time, they're going to have to be put into legislation and operated by external groups or auditors rather than stay left just to companies themselves, at least if they're going to achieve their real potential or the full potential that I think is there. Of course, Nick and I debate that take of mine as well. If you want to let us know your reaction to this interview or any other interview that we do, then our inbox is always open at [email protected].

• 6:48Rob Wiblin?

  But now, here's my interview with Nick Joseph, recorded on the 30th of May 2024. Today, I'm speaking with Nick Joseph. Nick is Head of Training at the major AI company Anthropic, where he manages a team of over 40 people focused on training Anthropic's large language models, including Claude, which I imagine many, many listeners have heard of and potentially used as well. He was actually one of the relatively small group of people to leave OpenAI alongside Dario and Daniela Amade, who then went on to founder Anthropic back in December of 2020. So thanks so much for coming on the podcast, Nick.

• 7:20Speaker 01

  Thanks for having me. I'm excited to be here.

• 7:22Rob Wiblin?

  I'm really happy to talk about how Anthropic is trying to prepare itself for training models capable enough that we're a little bit scared of what they might go and do. But first, as I just said, you lead model training at Anthropic. What's something that people get wrong or kind of misunderstand about AI model training? I imagine there could be quite a few things.

• 7:41Speaker 01

  Yeah, I think one thing I would point out is the sort of doubting of scaling working. So for a long time, we've had this trend where people put more compute into models, and that leads to the models getting better, smarter in various ways. And every time this has happened, I think a lot of people are like, ah, this is the last one. You know, the next scale up isn't going to help. And then, you know, some chunk of time later, things get scaled up and it's much better. And I think this is something people have just frequently gotten wrong.

• 8:06Rob Wiblin?

  This whole vision that scaling is just going to keep going, you know, we just make, just throw in more data, throw in more compute, the models are going to become more powerful. That was kind of the part of the, it feels like a very anthropic idea. It was part of the founding vision that Dario had, right?

• 8:20Speaker 01

  Yeah, so like, a lot of the early work on scaling laws was done by a bunch of the Anthropic founders. And it somewhat led to GPT-3, which was done in OpenAI, but by many of the people who are now at Anthropic, where looking at a bunch of small models going up to GPT-2, there was sort of this sign that as you put in more compute, you would get better and better. And it was very predictable and you can say, ah, if you put in X more compute, you'll get a model this good. And that sort of enabled the like confidence to go and train a model that was rather expensive by the time standards to sort of verify that hypothesis.

• 8:58Rob Wiblin?

  What do you think is kind of generating that skepticism that many people have? I mean, like, people who are skeptical of scaling laws, there's some pretty smart people who are kind of involved in ML, certainly have their technical chops. Why do you think they are generating this prediction that you disagree with?

• 9:12Speaker 01

  Yeah, I think it's just a really unintuitive mindset or something where it's like, ah, the model has hundreds of billions of parameters. What does it need? It really needs trillions of parameters. Or the model is trained on some fraction of the internet that's very massive. What does it need to be smarter? It's even more. Like, that's not how humans learn. If you send a kid to school, you don't have them just read through the entire internet and think that the more that they read, the smarter they'll get. So yeah, that's sort of my best guess. And the other piece of it is that it's quite hard to do the scaling work. So there are often things that you do wrong when you're trying to do this the first time. And if you mess something up, you will see this behavior of more compute not leading to better models. And it's always hard to know if it's you messing up or if it's some sort of fundamental

• 9:57Speaker 01

  limit where the model has stopped getting smarter.

• 9:59Rob Wiblin?

  I mean, the argument for why you might expect these scaling laws, so scaling laws, it's like you increase the amount of compute and data by some particular proportion, and then you get a similar improvement each time in the accuracy of the model. That's kind of the rule of thumb here. And the argument that I hope for why you might expect that trend to break, and perhaps the improvements to become smaller and smaller for a given scale up, is something along the lines of, as you're approaching human level, the model can learn by just copying existing state of the art of what humans are already doing in the training set. But then if you're trying to exceed human level, if you're trying to write better essays than any human has ever written, then that's maybe a different regime, and you might expect more gradual process, like more gradual improvements

• 10:43Rob Wiblin?

  once you're trying to get to a superhuman level. Do you think that that argument kind of holds up?

• 10:49Speaker 01

  Yeah, so I think that's true. And just pre-training on more and more data won't get you to superhuman at some tasks. It will get you to superhuman in the way of understanding everything at once. This is already true of models like Claude, where you can ask them about anything, whereas humans have to specialize. But I don't know if progress will necessarily be slower. It might be slower, it might be faster, once you get to the level where models are all right, human abilities on everything and improving towards superintelligence. We haven't really, but we're still pretty far from there. If you use Claude now, I think it's pretty good at coding. But this is one example I use a lot, but it's still pretty far from how well a human would do working as a software engineer, for instance.

• 11:28Rob Wiblin?

  And is the argument for how it could speed up that at the point that you're near human level, then you can use the AIs in the process of doing the work? Or is it something else?

• 11:36Speaker 01

  What I have in mind is like, yeah, if you had, if you had an AI that is human level at everything and you can spit up millions of them, you effectively now have a company of like millions of AI researchers. And it's, it's hard to know, right? Things, problems get harder too. So I don't really know where that leads. But at that point, I think you've sort of crossed, crossed quite a, quite a ways from where we are now.

• 11:56Rob Wiblin?

  So I said, you're in charge of model training. I guess there's, I know there's different stages of model training. There's the bit where you kind of train the language model on the entire internet. And then there's the bit where you like do the fine tuning where you get, you get it to spit out answers and then you write whether you like them or not. Are you in charge of all of that or just some part of it?

• 12:12Speaker 01

  Yeah, so I'm just in charge of what was sort of typically called pre-training, which is where this step of train the model to predict the next word on the internet. And that tends to be like historically is a significant fraction of the compute. It's maybe 99% in many cases. But after that, the model sort of goes to what we call like fine tuning teams that will sort of take this model that just, just predicts the next word and fine tune it to act in a way that a human wants. So to be this sort of like helpful assistant. We have this helpful, harmless and honest is the, is the acronym that we usually aim for, for Claude.

• 12:45Rob Wiblin?

  Yeah, I use Claude 3 Opus multiple times a day, every day now. It took me a little while to figure out how to, how to actually use these LLMs for, for anything for, I guess, the first six months or our first year. I was like, these things are amazing, but I can't figure out how to actually incorporate them into my life. But recently I've started talking to them in order to like learn about the world. It's kind of substituted for when I would be typing complex questions into Google to understand some bit of history or science or, or some technical issue. I guess what's, what's the main bottleneck that you face making these models smarter? So, so I can get more, get more use out of them?

• 13:16Speaker 01

  Yeah, so let's see, I think the, there's sort of historically, people have talked about these three bottlenecks of data, compute and like algorithms. I kind of think of it as, yeah, so there's some amount of just compute. We've, we talked about scaling a little ago. If you put more compute with the model, it will do better. There's data where if you're training on, if you're putting in more compute, one way to do it is to add more parameters to your model, make your model bigger. But the other way you need to do is to add more data to the model. So you need both of those. But then the, the other two are algorithms, which I really think of as people. Maybe this is the manager and maybe. Is like, you know, algorithms come from people. In some ways, data and compute also come from people. But it looks like a lot of researchers working, working on the problem.

• 13:55Speaker 01

  And then the last one is time, which has felt more kind of urgent, more true recently, where things are moving very quickly. So a lot of the bottleneck to progress is actually like, we know how to do it. We have the people working on it. But it just takes some time to implement the thing and turn and run the model, train the model, right? Like you, you can maybe afford all the compute and you can, you have a lot of it, but you can't use, you can't efficiently train the model in like a second. So right now at Anthropic, it feels like people and time are probably the, the main bottlenecks or something. Like I, I feel like we have quite, quite a significant amount of compute, a significant amount of data. And the things, the things that are like most limiting at the moment, I feel like people and time.

• 14:34Rob Wiblin?

  So when you say time, is that kind of indicating that you're doing a sort of, either like an experimental process where, you know, you try tinkering with how the model learns in one direction, then you want to see whether that actually gets the improvement that you expected. And then it takes time for those results to come in and then you get to scale that up to the, to the whole thing. Or is it just a matter of, you know, you're already trading cloud for, or you already have the next thing in mind. It's just a matter of waiting.

• 14:54Speaker 01

  So it's, it's both of those. Like for, for the next model, you know, we have a bunch of researchers who are, who are trying projects out and they have to, you know, you have some idea. And then you have to go and implement it. So you'll spend a while sort of engineering this, this idea into the code base. And then you need to run a bunch of experiments and typically you'll start with cheap versions and, and work your way up to more expensive versions. Such that this process can, can take a while. For simple ones, it might take a day. For really complicated things, it could take months. And to some degree you can paralyze, but on, on certain directions, it's much more like someone, you're building up an understanding. And it's hard to paralyze, like building up an understanding of how something works and then designing like the, the next experiment. There's just sort of in serious aspect.

• 15:32Rob Wiblin?

  Is proving these model, is, is, sorry, is improving these models harder or, or easier than, than, than people think?

• 15:40Speaker 01

  Hmm. Well, I guess people think different things on it. I think my experience has been that like, early on, I would have, it felt very easy. I worked for, before working at OpenAI, I was working on robotics for a few years. And one of the tasks I worked on was locating an object so we can pick it up and drop in the box. And it was really hard. I spent years on this problem. And then went to OpenAI and I was working on code models. And it just felt shockingly easy. It was like, wow, you know, you just throw some compute, you train on some code and the model can write code. I think that has now shifted, like we've now, and the reason for that was no one was working on it. There was just very little attention to this direction and a ton of low hanging fruit. We've now plucked a lot of the low hanging fruit, so finding improvements is much harder.

• 16:22Speaker 01

  But we also have way more resources, like exponentially more resources put on it. There is way more compute available to do experiments. There are way more people working on it. And I think the rate of progress is probably still going like the same, given that.

• 16:35Rob Wiblin?

  Okay. So you think, on the one hand, the problem's gotten harder. On the other hand, there's more resources going into it and this has kind of canceled out and like progress is roughly, roughly stable.

• 16:45Speaker 01

  Yeah, it's pretty bursty. So it's like, it's hard to know. You know, you'll have like a month where it's like, wow, we've figured something out. Everything's going really fast. Then you'll have a month where you try a bunch of things and they don't work. And it varies, but I don't think, I don't think there's really been a trend in either direction.

• 17:01Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 17:05Rob Wiblin?

  Do you personally worry that, you know, having a model that is kind of nipping at the heels or maybe, maybe outcompeting the best stuff that OpenAI or DeepMind or just whatever other companies have, that maybe puts pressure on them to speed up their releases and cut back on safety testing or anything like that?

• 17:22Speaker 01

  I think it is something to like be aware of, but I also think that at sort of this point, I think this is really more true after ChatGPT. I think before ChatGPT, there was sort of this sense where many AI researchers, I think, working on it were like, wow, this technology is really powerful. But I think the world hadn't really caught on and there wasn't quite as much commercial like pressure. Since then, I think that there really is just a lot of commercial pressure already. And it's not really clear to me how much of impact it is. I think there is definitely an impact here, but I don't know the magnitude and there are, there are sort of a bunch of other considerations to trade off.

• 17:55Rob Wiblin?

  All right, let's turn to the main topic for today, which is responsible scaling policies or RSPs, as the cool kids call them. For those who don't know, scaling is this technical term for using more compute or data to train any given AI model. And the idea for RSPs has been around for a couple of years. And I think it was fleshed out, you know, maybe after 2020 or so. It was advocated for by this group called METR, or Model Evaluation and Threat Research, which actually is the place that previous guest of the show, Paul Cristiano, was working until not very long ago. And Thopec released the first public one of these, as far as I know, last October.

• 18:37Rob Wiblin?

  And then OpenAI put out something kind of similar in December, called their Preparedness Framework. And Demis of DeepMind has said that they're going to be producing something in a similar spirit to this, but they haven't done so yet, as far as I know. So we'll just have to wait and see.

• 18:51Speaker 01

  It's actually out. They have done it.

• 18:52Rob Wiblin?

  Oh, it's out?

• 18:52Speaker 01

  Yeah.

• 18:53Rob Wiblin?

  Oh, okay.

• 18:53Speaker 01

  It was established like a week or so ago.

• 18:55Rob Wiblin?

  Oh, okay. I'll maybe ask you about that later. But yeah, that just goes to show that RSPs are kind of this reasonably hot idea, which is kind of why we're talking about them today. And I guess some people also hope that these internal company policies are ultimately going to be a model that might be able to be turned into binding legislation that, you know, everyone dealing with these frontier AI models might be able to follow in future. But yeah, Nick, what are responsible scaling policies in a nutshell?

• 19:22Speaker 01

  So I might just start off with a quick disclaimer here that like I, this is not my direct role. Like I'm sort of bound by trying to implement these and sort of act under one of these policies. But many of my colleagues have worked on like designing this in detail and are probably more familiar with all the deep ones than me. But anyway, in a nutshell, like the idea is it's a policy where you define various safety levels. So these sort of different levels of risk that a model might have and create evaluations, so tests to say, is this model able to, is a model this dangerous? Does it require this level of precautions? And then you need to also define sets of precautions that need to be taken in order to train or deploy models at that particular risk level.

• 20:02Rob Wiblin?

  Yeah, I think this might be a topic that is just best learned about by skipping the abstract question of what RSPs are and just talking about the anthropic RSP and seeing what it actually says that you're going to do. So what does the anthropic RSP going to promise, commit the company to doing?

• 20:17Speaker 01

  Yeah, so we basically, for every level, will define these red line capabilities, which are capabilities that we think are dangerous. I can maybe give some examples here, which is this acronym CBRN, chemical, biological, radiological, and nuclear threats. And in this area, it might be that a non-expert can make some weapon that can kill many people as easily as an expert can. So this would sort of increase the pool of people that can do that a lot. On cyber attacks, it might be like, can a model help with some really large scale cyber attack? And on autonomy, can the model perform some tasks that are precursors to autonomy, sort of our current one? But that's a tricky one to figure out. So it is we establish these red line capabilities that we shouldn't train until we have safety

• 21:01Speaker 01

  mitigations in place. And then we create evaluations to show that models are far from them, or to know if they're not. So these evaluations have to test for that capability, because you want them to turn up positive before you've trained a really dangerous model. But we can kind of think of them as yellow lines. Like, once you get past there, you should reevaluate. And the last thing is then developing standards to make models safe. So we want to have a bunch of safety precautions in place, once we train those dangerous models. So that's sort of the main aspect of it. There's also a promise to iteratively extend this. So creating the evaluations is really hard. We don't really know what the evaluation should be for a super intelligent model yet.

• 21:44Speaker 01

  So we're kind of starting with the closer risks, and once we hit that next level, defining the one after it.

• 21:50Rob Wiblin?

  So a pretty core component of the anthropic RSP is this AI safety level framework. So I think you borrowed that from the biological safety level framework, which I think is what labs dealing with dangerous diseases use. So I guess I don't know what the numbers are, but if you're dealing with a bowler or something that's particularly dangerous, or a smallpox or whatever, then that can only be Stored in a BSL four lab or something like that. And then as the diseases become less, less dangerous, you can store them with, with fewer precautions. And you've kind of taken that, taken that language and talked about AI safety levels. And the current AI safety level that you put us out is ASL two, which is things like Claude three, which, you know, are kind of impressive. They seem pretty, pretty savvy in some ways, but they don't seem like they really

• 22:33Rob Wiblin?

  pose any, any meaningful catastrophic risk. I guess, yeah, what sort of tests have you run on Claude three, you know, recently, maybe in order to say, yeah, this is, this is in the ASL two bucket?

• 22:45Speaker 01

  Yeah, so we've got a bunch of tests. So, so the first one we use for the more like a biological weapon type of, type of angle is there's sort of a bunch of multiple choice questions that we, that we ask the model. And this is like, the idea, this isn't a perfect evaluation for this, but the idea is that we, we have a set where if it gets, if it isn't much better at this, it probably won't be able to help very much. And then we've run trials with, with, with people where they get access to a search engine and they get access to Claude. And we see if they do significantly better at going through some, some really relevant tasks using, using Claude versus a search engine.

• 23:21Rob Wiblin?

  And I guess, and another one of the kind of yellow flags for you is the possibility that a computer, oh, sorry, that an AI model might be able to replicate itself in the wild or kind of, you know, figure, find, find its own weights and then copy itself onto, onto other computers. Did you test whether Claude three, you know, was showing any signs of being able to do that?

• 23:39Speaker 01

  Yeah, so I think some of the, the tasks there would be, one of them is adding an exploit to a popular open source code base. So this is the idea of if the model can, can add some exploit that can then be taken advantage of, or fine tuning an open source LLM to add some backdoor, where if you say X, it will always output something. This is like, can it, can it manipulate LLMs? It could steal an API key from, from our API or a fake version of our API that was set up, which would sort of enable it to get credits. There are a few more of these, but the, or yeah, another one would be like a worm that can spread undefended machines. So you could sort of imagine a model that can do all of these has learned a bunch of the skills that might enable it to sort of autonomously replicate and cause havoc. They're pretty early signs of it. And we, we want to sort of test for, for the early ones, because this is sort of a, an area that's like less fleshed out,

• 24:23Speaker 01

  where there's sort of less clear expertise on what might go wrong.

• 24:27Rob Wiblin?

  Okay, so we're at the AI Safety Level 2, which is kind of not, I guess, the mostly harmless category. But what sort of steps does the responsible scaling policy call for you to be taking even at this point?

• 24:41Speaker 01

  So we made these sort of White House commitments, I think sometime last year, which I think of them as sort of like standard industry best practices. In many ways, we're building the muscle for dangerous capabilities, but these models are not yet capable of catastrophic risks, which is what the RSP is primarily focused on. But this looks like security to protect our weights against opportunistic attackers, putting out model cards to describe the capabilities of the models, doing training for harmlessness, so that we don't have models that can be really harmful out there.

• 25:09Rob Wiblin?

  So what sort of results would you get back from your tests that would indicate that now the capabilities have risen to ASL 3?

• 25:17Speaker 01

  Yeah, so if the model, for instance, passed some fraction of those tasks that I mentioned before around adding an exploit, spreading to undefended machines, or if it did really well on these biology ones, that would sort of flag it as having passed the yellow lines. At that point, I think we would either need to look at the model and be like, ah, this really is clearly still incapable of these red line dangers. And then we might need to go to the board and think about, was there a mistake in RSP? And how we should essentially create new evals that would test better for whether we're at that capability? Or we would need to implement a bunch of precautions. And these precautions would look like much more intense security, where we would really want this to be sort of robust to like, probably not state actors, but to non-state actors.

• 26:01Speaker 01

  And we would want to pass this sort of intensive red teaming process on all the modalities that we release. So this would mean we look at those red lines and we test for them with experts and say, can you use the model to do this? We have this sort of intensive process of red teaming, and then only release the modalities where it's been red teamed. So if you add in vision, you need to red team vision. If you add the ability to fine tune, you need to red team that.

• 26:25Rob Wiblin?

  Yeah, what does red teaming mean in this context?

• 26:29Speaker 01

  Red teaming means you get a bunch of people who are trying as hard as they can to get the model to do the task you're worried about. So if you're worried about the model, like, carrying out a cyber attack, you would get a bunch of experts to try to prompt the model to carry out some cyber attack. And if we think it's capable of doing it, we've put all of these, we're putting these precautions on. And these could be precautions in the model, they could be precautions outside of the model. But the whole end-to-end system, we want to have people trying to get it to do that in some controlled manner, such that we don't actually cause mayhem and see how they do.

• 26:59Rob Wiblin?

  Okay. And then, so if you do the red teaming and it comes back and they say, "Yeah, the model is extremely good at hacking into computer systems," or "It could actually help people, it could meaningfully help someone develop a bioweapon." Then what is the policy call for Anthropic to do?

• 27:16Speaker 01

  So for that one, it would mean we can't deploy the model. Because there's some danger this model could be misused in a really terrible way. And we would sort of keep the model internal until we've improved our safety measures enough that when someone asks for it to do that, we can be confident that they won't be able to have it help them for that particular threat.

• 27:35Rob Wiblin?

  Okay. And to even have this model on your computers, the policy also calls for you to have hardened your computer security so that you're saying, maybe it's unrealistic at this stage for that model to be safe from persistent state actors. But at least other groups that are somewhat less capable than that, you would want to be able to make sure that they wouldn't be able to steal the model.

• 27:57Speaker 01

  Yeah, the big, the threat here is, you know, you can put all the restrictions you want on what you do with your model, but if people are able to just steal your model and then deploy it, you're going to have all of those dangers anyway. So you're sort of taking responsibility for it, means like both responsibility for what you do and what someone else can do with your models. And that requires quite intense security to protect the model, right?

• 28:18Rob Wiblin?

  When do you think we might hit this? You know, you would say, "Well, now we're in the ASL 3 regime." Maybe, I'm not sure exactly what language you use for this, but like, at what point will we have an ASL 3 level model?

• 28:30Speaker 01

  I'm not sure. I think basically we'll continue to evaluate our models and we'll see when we get there. I think sort of opinions vary a lot on that.

• 28:36Rob Wiblin?

  We're talking kind of about the next few years, right? This isn't something that's going to be five or 10 years away necessarily.

• 28:42Speaker 01

  I think it really just depends. Like, I think you could imagine sort of any direction. And one of the nice things about this is that we're targeting the safety measures at the point when there's actually dangerous models. So like, maybe, let's say I thought it was going to happen in two years, but I'm wrong and it happens in 10 years. We won't have to, we won't put these very costly and like difficult implement mitigations in place until, until we like need them.

• 29:04Rob Wiblin?

  Okay, so Anthropic's RSP, so I guess obviously we've just been talking about ASL3. The next level we'll be on that would be ASL4. I think your policy basically says, we're not exactly sure what ASL4 looks like yet, because it's too soon to say. And I guess you promised that you're going to have mapped out what would be the kind of capabilities that would escalate things to ASL4 and what kind of responses you would have. You're going to figure that out by the time you have trained a model that's at ASL3. And I guess if you haven't, and so you'd have to pause training on a model that was going to hit ASL3 until you'd, you'd finish this project. That, I guess that was the commitment that's been made.

• 29:41Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 29:46Rob Wiblin?

  But maybe you could kind of give us a sense, a sense of what you think ASL4 might look like, what sorts of capabilities by the models would then like push us into another regime where a further set of precautions are called for.

• 29:59Speaker 01

  So we're still discussing this internally, so I don't want to say anything that's like final or going to be held to, but you could sort of imagine stronger versions of a bunch of the things that we sort of talked about before. And you could also imagine models that can like help with AI research in a way that really majorly accelerates researchers such that progress goes much faster. The, the core reason that we're, we're holding off on kind of defining this or that we, we have this iterative approach is there's this long track record of people saying, you know, oh, once you have this capability, it will be AGI, it's going to be really dangerous. I think people were like, oh, in an AI solves chess, like it will be as smart as humans. And it's really hard to get these evaluations right. Even for like the ASL3 ones, I think it's been very tricky to get, get evaluations that capture the risks we're worried about. So sort of the closer, the closer you get to that, the more information you have and

• 30:44Speaker 01

  the better of a job you can do at sort of defining what these evaluations are and risks are.

• 30:49Rob Wiblin?

  So the general sense would be, you know, models that might be capable of spreading autonomously across computer systems, even if people were trying to turn them off, you know, would be able to provide significant help with developing bioweapons, maybe even to people who are pretty, pretty informed about it. I guess, yeah, what else is there? Oh, and stuff that would seriously speed up AI development as well. So it could potentially set off this sort of positive feedback loop where the models get smarter, that makes them better at improving themselves and so on. That's the sort of thing where we're talking about.

• 31:18Speaker 01

  Yeah, stuff along those lines. I'm not sure which ones will end up in ASL4 exactly, but, but like those sorts of things are what's being considered.

• 31:24Rob Wiblin?

  And what sorts of additional precautions might there be? I guess at that point, you kind of want the models to not only be not possible to be stolen by kind of independent freelance hackers, but ideally also not by countries even, right?

• 31:37Speaker 01

  Yeah, so you want to protect against more sophisticated groups that are trying to steal the weights. We're going to want to have like better protections against the model like acting autonomously. So controls around that, you might want, it depends a little bit on like what would end up being the red lines there, but sort of having the precautions that are tailored to what will be a much higher level of risk than the ASL3 red lines.

• 31:59Rob Wiblin?

  Were you heavily involved in actually doing this testing on, again, Claude 3 this, this, this year?

• 32:06Speaker 01

  I wasn't like running the tests, but I was sort of watching them because as we trained Claude 3, we were very much sort of all of our planning was contingent on whether or not it passed these evals. So, and, and because we had to run them partway through training. So there's sort of a lot of planning that goes into the model's training. You don't want to have to like stop the model just because you were, you didn't plan well enough to run the evals in time or something. So there was sort of a bunch of coordination around that that I was, I was involved in.

• 32:30Rob Wiblin?

  Can you give me a sense of how many, like, how many staff are involved in, in doing that? And how long does it take? Is this a, is this a big process? Or is it pretty, is it a pretty standardized thing where you're putting in, you know, well-known prompts into the, into the model and then just, then just checking what it does that's different from last time?

• 32:45Speaker 01

  Yeah, so Claude 3 is our first time running it. So a lot of the work there actually involved creating the evaluations themselves as well as running them. So we had to sort of create them, have them ready and then running them. I think typically running them should be pretty, is pretty easy for the ones that are automated. But for some of the things where you actually require like people to go and use the model, they can be much, much more expensive. There's currently, I think, like multiple teams working on this. And a lot of our capabilities teams worked on it very hard. So one of the, one of the ways this can fall apart is if you don't solicit capabilities well enough. So if you, if you sort of try to have the model, test the model on the eval, but you don't try hard enough, and then it turns out that with just a little more effort, the model could have passed the evals. So the, it's often important to have kind of your like, your best researchers who are

• 33:30Speaker 01

  capable of pulling capabilities out of the models, also working on trying to pull them out to pass these tests.

• 33:37Rob Wiblin?

  So many people will have had the experience that these LLMs will reject objectionable like requests. If you say, if you, if you put it to Claude 3 now and say, please help me design a bioweapon, it's going to say, sorry, I can't, sorry, I can't help you. But I guess you do all of these tests before you've done any of that training to try to discourage it from doing objectionable things. You do it with the thing that's helpful, no matter what the request is, right?

• 33:58Speaker 01

  Yeah, because we want, the thing we're testing for is, is the model capable of this danger? And then there's a separate thing, which is what mitigations can we put on top? So if the model's capable of the danger, then we would require ASL 3. And those safety mitigations we put on top would be part of the standard in order to pass that red teaming. Does that make sense?

• 34:14Rob Wiblin?

  Yeah, so I guess, so you're saying, so you have to focus on what the model could do if it was so motivated to, because, well, I guess if the weights were ever leaked, then someone would be able to remove any of the fine tuning that you've done to try to discourage it from doing unpleasant things. So if it's able to do something, then it could potentially be used that way sooner or later. And so you need to like, kind of assume the worst and plan around that. Is that the philosophy?

• 34:40Speaker 01

  Yeah, that's exactly right.

• 34:42Rob Wiblin?

  You mentioned that maybe one of the key failure modes here would be just not trying hard enough to elicit these dangerous capabilities, basically. Maybe if you just phoned it in, then you could easily miss that the model is capable of doing something that it can do. Because even more extreme would be if in future, this policy is going to matter the most when these models are extremely capable and actually could do serious damage. And at that point, you might worry that they would pretend to not be able to do things that in fact they can do because they don't think it's in their interest to reveal that they have X or Y capability. I guess because they know that then they won't be used and they might be deleted. Yeah, do you have any plans for how you're going to deal with that issue of, I think the technical term that people use for this is sandbagging, which I'd never heard of.

• 35:25Rob Wiblin?

  But I think it's a term from sport where people underplay their abilities for strategic reasons. Yeah, do you have any idea of how Anthropic is going to be able to get around that problem?

• 35:33Speaker 01

  Interesting. I didn't know the origin of that term. I think this is like one of the reasons that we sort of want to define the next set of evaluations after we've done the current ones, because this is really hard, right? How do you evaluate for that when the model could be trying to sandbag the evaluations? Hopefully, you can set the line, like your eval, for a capability that's early enough that you would still be able to detect it doing that. So there's sort of this hope that, you know, less capable models will try to deceive you but fail, and then you'll know to stop before you get to models that are really capable of deceiving you. But it's not certain, and that definitely seems like a possible failure mode where if you have these sort of sudden capability jumps, or if you have jumps that an evaluation can't catch, either because the model's sandbagging you or for any of a number of

• 36:18Speaker 01

  other reasons.

• 36:19Rob Wiblin?

  To try to detect sandbagging, is it possible to, if you have a model that says, "No, I can't do that," can you kind of engage in a reinforcement learning from human feedback thing where, you know, whatever response shows like the most inclination, the most willingness to help, like as soon as you can kind of open the door a little bit, then you reinforce that a whole lot and then see, will that open the door the whole way and actually get the model to demonstrate that it can do something even though it's trying to pretend that it can't? Does that make sense?

• 36:46Speaker 01

  It somewhat makes sense. I think it just depends a little bit on like exactly what the model is trained to do. So like if you've trained the model to be harmless and then you kind of just try to RL it back to not being harmless, like you will be able to succeed. Like you can fine tune away any of the of the safety training, but it's not clear if that will sort of be the same, the same thing as if like in, you know, one sort of sequence, like within some, some prompt, it's trying to trick you. I think there's been a bunch of work where people use scratchpads to sort of tell the model, you can, you can write something down, but the, the reward model, which is sort of the, the model that's overseeing it and telling it whether it's good or not, doesn't get to see this. And when you do that, you as a human can then look at it and you can kind of get it, get a sense of

• 37:25Speaker 01

  like, what is the model thinking? Is the model trying to deceive its supervisor? But we've only really done that on sort of current models, which don't really have this kind of like level of capability where it would be really catastrophically dangerous and figuring out, I think we'll want much more robust tools once we get there. Interpretability is another approach to this, where if you can actually look directly at the model weights and look and see, ah, okay, we, without running the model, we can see that the model has this circuit that is saying, ah, you should deceive, deceive the user in this way, or you should fail the test because you don't want them to turn you off. That's, that's sort of another, another hope for how to attack this.

• 38:05Rob Wiblin?

  Okay. So that's the basic structure of a responsible scaling policy, or at least the, at least the anthropic responsible scaling policy. I guess you have this, this system of tests that you commit to ahead of time, that you're going to put all of your models through. And then you pre-commit to saying, well, if we get this kind of result, then we said, then we think the risk is higher. And so that's going to call for, you know, an escalation in the, in the precautions that we're taking, things around computer security, things around not, not deploying until you've made them safe and so on. Why, you're, you're a big fan of this type of approach to, to AI safety for, for AI companies? What is, yeah, what's one of the main reasons, or what's perhaps the, the top reason why you think this is the right approach, or at least one of, one of the better approaches?

• 38:48Speaker 01

  Yeah, so I think one thing I like is that it separates out whether an AI is capable of being dangerous from what to do about it. I think this is a spot where there are many people who are skeptical that models will ever be capable of this sort of catastrophic danger. And therefore, they're like, we shouldn't take precautions because the models aren't that smart. And I think this is a nice way to agree where you can, it's much easier message to say, if we have evaluations showing the model can do X, then we should take these precautions. And I think you can sort of build more support for something along those lines. And it targets your precautions at the time when there's actual danger. There are a bunch of other things I can sort of talk through. I think one thing, other thing I really like is that it aligns commercial incentives with safety goals.

• 39:29Speaker 01

  So, once we put this RSP in place, it's now the case that our safety teams are kind of under the same pressure as our product teams, where if we want to ship a model and we get to ASL 3, the thing that will block us from being able to get revenue, being able to get users, et cetera, is do we have the ability to deploy it safely? And it's a nice outcome-based approach, where it's not, did we invest X amount of money in it? It's not, did we try?

• 40:00Rob Wiblin?

  Did we say the right thing?

• 40:01Speaker 01

  Did we succeed? Yeah. And I think that often really is important for organizations to like, set this goal of like, you need to succeed at this in order to deploy your products.

• 40:11Rob Wiblin?

  Is it actually the case that it's kind of had that cultural effect within Anthropic now, that people realize that a failure on the safety side would prevent the release of the model that matters to the future of the company? And so, there's like a similar level of pressure on the people doing this testing as there is on the people actually training the model in the first place?

• 40:30Speaker 01

  Oh yeah, for sure. I mean, you asked me earlier, like, when are we going to have ASL 3? And I think I received this from someone on one of the safety teams on like, a weekly basis, because their deadline is, I mean, the hard thing for them actually is their deadline isn't a date. It's once we have created some capability, and they're very focused on that.

• 40:49Rob Wiblin?

  So their fear, the thing that they worry about at night is that you might be able to hit ASL 3 next year and they're not going to be ready. And that's going to hold up the entire enterprise.

• 41:00Speaker 01

  Yeah, I could give some other things like 8% of Anthropic staff works on security, for instance. So this is like, there's a lot, you have to plan for it, but and there's, but there's a lot of work going into being ready for, for these next safety levels. We have like multiple teams working on alignment, interpretability, creating evaluations.

• 41:17Speaker 01

  So, so yeah, there's a lot of effort that goes into it.

• 41:19Rob Wiblin?

  When you say security, do you mean computer security? So preventing the weights from getting stolen or more broader class?

• 41:26Speaker 01

  Both. So like the weights could get stolen, someone's computer could get compromised, you could have someone hack into get, and get all of your IP. There, there's sort of a bunch of different dangers on the security front where the weights are certainly an important one, but they're definitely not the only one.

• 41:41Rob Wiblin?

  Okay. And the, the first thing you mentioned, the first reason why RSPs have this nice structure is that some people, I think that these troublesome capabilities could be, could be with us this year or next year. Other people think it's never going to happen, but both of them could be on board with a policy that says, well, if these, these capabilities arise, then that would call for these sorts of responses.

• 42:04Rob Wiblin?

  Has that actually happened? I mean, have you, have you seen kind of the skeptics who say all of this AI safety stuff is ever blown? It's a, it's a bunch of rubbish saying, well, but the RSP is fine because it, you know, I think we'll never actually hit any of these levels. So we're not going to waste any resources on something that's not, not realistic.

• 42:18Speaker 01

  Yeah, so I think there's always going to be like degrees, you know? I think there are people across the spectrum. So there are definitely people who are, who are still skeptical, who will just be like, why even think about this? There's no chance. But I do think that like RSPs do seem much more pragmatic, much more able to sort of be picked up by, by various other organizations. I think, as you mentioned before, right? OpenAI and Google are both, both sort of putting out things along these lines. So I think at least from the sort of large frontier AI labs, there, there is a significant amount of buy-in.

• 42:45Rob Wiblin?

  Yeah, I see. I guess even if maybe you don't see this on, on Twitter, maybe it helps with the internal bargaining within the company that people have a different range of expectations about how things are going to go, but they could all be kind of reasonably satisfied with an RSP that, that equilibrates or like matches the level of capability with the level of precaution. The first worry about this that jumps to my mind is If the capability improvements are really quite rapid, which I think we think that they are and they maybe could continue to be, then don't we need to be practicing now, like figuring out, basically getting ahead of it and doing stuff right now that might seem kind of unreasonable, given what Claude 3 can do, because we worry that we could have something that's substantially more dangerous in one year's time or in two years' time.

• 43:29Rob Wiblin?

  And we don't want to then be kind of scrambling to deploy the systems that are necessary then and then perhaps falling behind because we didn't prepare sufficiently ahead of time. What do you make of that?

• 43:42Speaker 01

  Yeah, so I think we definitely need to plan ahead, right? And I think one of the nice things is, once you've aligned these sort of safety goals with commercial goals, like people plan ahead for commercial things all the time. It's part of a normal company planning process. I think that the RSP, so we have these sort of yellow line evals that are intended to be far short of the capability, the red line capabilities we're actually worried about. And tuning that gap seems fairly important. If that gap looks like a week of training, it would be really scary where you trigger these evals and you have to act fast. I think in practice, we've set those evals such that they are far enough from the capabilities that are really dangerous, such that there will be some time to sort of adjust in that sort of buffer period.

• 44:23Rob Wiblin?

  So should people actually think that we're in ASL 2 now and we're heading towards ASL 3 at some point, but there's actually kind of an intermediate stage with all these transitions where you'd say, well, now we're seeing warning signs that we're going to hit ASL 3 soon, so we need to implement the precautions now in anticipation of being about to hit ASL 3. Is that basically how it works?

• 44:46Speaker 01

  Yeah, it's basically like, we sort of have this concept of a safety buffer. So, once we trigger the evaluations, it doesn't necessarily mean, like these evaluations are set conservatively, so it doesn't mean the model is capable of the red line capabilities we're really worried about. And that will sort of give us a buffer where we can figure out, ah, Maybe it really just definitely isn't and we wrote a bad eval, we'll go to the board, we'll try to change the evals and implement new things. Or maybe it really is quite dangerous and we need to turn on all the precautions. Of course, you might not have that long, so you want to be ready to turn on those precautions, such that you don't have to pause. But you do need, there is some time there that you could do it. And then the last possibility is that we're just really not ready. These models are catastrophically dangerous and we don't know how to secure them, in which

• 45:30Speaker 01

  case we should stop training the models. And or if we don't know how to deploy them safely, we should not deploy the models until we figure it out.

• 45:38Rob Wiblin?

  I guess if you are on the very concerned side, then you might think, yes, that you are gonna, you are preparing, I guess, yeah, you do have a reason to prepare this year for, you know, safety measures that you think you're gonna have to employ in future years. But maybe we should go even further than that. And what we need to be doing is practicing implementing them and seeing how well they work now. Because even though you are preparing them, you're not actually getting the gritty experience of like, you know, applying them and trying to use them on a day-to-day basis. And I guess the response to that would be, well, that would, in a sense, be safer, that would be adding an even greater precautionary buffer, but it would also be enormously expensive. And people would see us doing all of this stuff that seems really over the top relative to what any of the models can do.

• 46:22Speaker 01

  Yeah, I think there's sort of a trade-off here between like, with pragmatism or something, where I think if you took, I think we do need to have a huge amount of caution on future models that are really dangerous. But if you apply that caution to models that aren't dangerous, you miss out on a huge number of benefits from using the technology now. And I think you'll also probably just alienate a lot of people who are gonna look at you and be like, you're crazy, why are you doing this?

• 46:46Speaker 01

  And I think my hope is that you can sort of, this is sort of the framework of the RSP is you can tailor the cautions to the risks. It's still important to like, look ahead more, right? So a lot of our, we do a lot of safety research that isn't directly focused on the next AI safety level, because you want to plan ahead, you have to be ready for multiple ones out. It's not the, it's not like the only thing to think about, but the RSP is sort of tailored more to empirically testing for these risks and tailoring the precautions appropriately.

• 47:13Rob Wiblin?

  Yeah, on that topic of people worrying that it's going to slow down progress in the technology, do you have a sense of, so obviously, training these frontier models costs a significant amount of money. We're talking maybe a hundred million dollars is a kind of a figure that I've heard thrown around for training a frontier LLM. How much extra overhead is there to run these tests to see whether the models have any of these dangerous capabilities? Is it adding hundreds of thousands, millions, tens of millions of dollars of additional cost or time?

• 47:41Speaker 01

  I don't know the exact cost numbers. I think the cost numbers are pretty low, right? They're mostly running inference or relatively small amounts of training. The people time feels like where there's a cost, right? Like there are whole teams dedicated to creating these evaluations, to running these, to doing the safety research to protect against the mitigations. And I think particularly for Anthropic, where we're a pretty small, rapidly growing, but rather small organization, at least my perspective, is most of the cost comes down to the like people and time that we're investing in it.

• 48:09Rob Wiblin?

  Okay, yeah. But I guess at this stage, it sounds like running these sorts of tests on a model is taking more in the order of weeks of delay. Because if you're getting that clear, like, this is not a super dangerous model, then it's not leading you to delay release of things for many months and deny customers the benefit of them.

• 48:29Speaker 01

  Yeah, the goal is to minimize the delay, right? As much as you can while being responsible. Like, the delay in itself isn't valuable. I think we're aiming to get it to a really well-done process where it can all execute very efficiently. But until we get there, there might be delays as we're figuring that out. And there will always be some level of time to require to do it.

• 48:50Rob Wiblin?

  Just to clarify, so a lot of the risks that people talk about with AR models is risks once they're deployed to people and actually getting used. But there's kind of this separate class of risk that comes from having an extremely capable model simply exist anywhere, even on, I guess you could think of there's public deployment and then there's internal deployment where Anthropic staff might be using a model and potentially it could convince them to release it or to do other dangerous things. That's a kind of a separate concern. What does the RSP have to say about that sort of internal deployment risks? Are there circumstances under which you would say even Anthropic staff can't continue to do testing on this model because it's too unnerving?

• 49:29Speaker 01

  Yeah, so I expect this to mostly kick in as we get to higher AI safety levels, but there are certainly dangers. I mean, the main one is the security risk. So like, one approach is just having the model, it always could be stolen. No one has perfect security. So that's kind of, I think, in some ways, is the is one that's true of all models, and is maybe more short term. But yeah, if you get to models that are trying to escape, trying to autonomously replicate, there is there is danger then in having access internally. So we would want to do things like siloing who has access to the models, putting particular precautions in place before the model is even trained, or maybe even on the training process. But we haven't yet to find those because we don't really know what they would be, right? Like, we don't quite know what that would look like yet. And it's, it feels really hard to design evaluation that is

• 50:14Speaker 01

  meaningful for that right now.

• 50:16Rob Wiblin?

  I don't recall the RSP mentioning conditions under which you would say, we have to delete this model that we've trained because it's because it's too dangerous. But I guess that's because that's more at the kind of ASL four or five level that that would become the kinds of things, the kind of thing that you would contemplate, and you just haven't spelled that out yet.

• 50:31Speaker 01

  No, so it's actually because of the safety buffer concept, right? So the idea is we would never train that model. If we did accidentally train some model that was past the red lines, that then I think we'd have to think about deleting it. But we would put these evaluations in place far, far below the dangerous capability, such that we would trigger the evaluations and have to pause, or have the safety things in place before we train the model that like, has has these dangers.

• 50:55Rob Wiblin?

  So what the RSPs as an as an approach, what you're a fan of them, what do you think of them as an alternative to? What are the alternative approaches for dealing with with AI risk that that people advocate that you think are weaker in relative terms?

• 51:10Speaker 01

  So I mean, I think the baseline is obviously just the first baseline is nothing. Like, these are, there could just be nothing here. I think the downsides of that is that these like, these models are very powerful, they could at some point in the future be dangerous. And I think that companies creating them have some have a responsibility to to think really carefully about those risks and be thoughtful. Sort of like, it's a major externality. That's maybe the easiest baseline of do nothing.

• 51:34Rob Wiblin?

  Yeah.

• 51:35Speaker 01

  I think like other other things would be like a pause where a bunch of people say, well, there are all these dangers. Why don't we just not do it? And I think that like makes sense, right? If you're training these models that are really dangerous, it does feel a bit like, why are you doing this? If you're worried about it. But I think there are actually really clear and obvious benefits to AI products right now. And the catastrophic risks currently are just, they're definitely not obvious. I think they're probably not not immediate. And as a result, they're not, this isn't a practical ask. Not everyone is going to pause. So what will happen is only the places that care the most, that are the most worried about this and the most careful with safety will pause and you'll sort of have this like adverse selection effect.

• 52:16Speaker 01

  I think there like eventually might be a time for a pause, but I would want that to be backed up by here are clear evaluations showing the models have these really catastrophically dangerous capabilities. And here are all the efforts we put into like making them safe. And we ran these tests and they didn't work. And that's why we're pausing and we would recommend everyone else should pause until they've as well. I think that will just be like a much more convincing case for a pause and target it at the time that it's like most valuable to pause.

• 52:40Rob Wiblin?

  When I think about, you know, people doing somewhat potentially dangerous things or developing interesting products. Maybe the default thing I imagine is that the government would say, here's what we think you want to do. Here's here's how we think that you should make it safe. And as long as you like, as long as you make your product according to these specifications, as long as the plane, you know, runs runs this way and you like service the plane this frequently. Then then you're in the clear and we'll say that what you've done is is reasonable. Do you think that RSPs are maybe better than that in general? Or maybe maybe just better than that for now, where, you know, we kind of don't know necessarily what what regulations we want the government to be imposing. So it's perhaps it's better for the for the for companies to be figuring this out themselves early on. And then perhaps it can be handed over to governments later on.

• 53:23Speaker 01

  Yeah, I don't think the RSPs are like a substitute for regulation. There are, there are many things that sort of only regulation can solve, such as what about the places that don't don't have an RSP? But I think that right now we don't really know what the tests would be or what the regulations would be. And I think probably this is still sort of getting figured out. So so one hope is that like we can implement RSP, OpenAI, Google can implement other things, other places will implement a bunch of things. And then policymakers can look at what we did, look at our reports on how it went, what what the results of our evaluations were and how it was going. And then design regulations sort of based on the learnings from that.

• 53:57Rob Wiblin?

  If I read it correctly, it seemed to me like the Anthropic RSP has this clause that allows you to go ahead and do things that you think are dangerous if you're being sufficiently outpaced by some other competitor that I guess doesn't have an RSP or not a very serious responsible scaling policy. In which case you might worry, well, we have this policy that's preventing us from going ahead, but we're just being rendered irrelevant and some other companies releasing much more dangerous stuff anyway. So what really is this accomplishing? Yeah, did I read that correctly? That there's this sort of get out of RSP clause in that sort of circumstance? And if you didn't expect Anthropic to be leading and for most companies to be operating safely, couldn't that kind of potentially obviate the entire enterprise? Because that clause could be quite likely to get triggered.

• 54:43Speaker 01

  Yeah, I think we don't intend that as like a get FJ free card. We're sort of, we're falling behind commercially and then like, oh, well, now we're going to skip the RSP. It's much more just intended to be practical as, you know, we don't really know what it will look like if we get to some sort of like AGI end game race. And there could be, there could be really high stakes and sort of, it could make sense for us to, to decide that the best thing is to proceed anyway. But I think this is something that we're sort of, sort of looking at as a bit more of a like last resort than a loophole we're planning to just use for, oh, you know, we don't want to deal with these evaluations.

• 55:20Rob Wiblin?

  Yeah. Okay. I think we've hit a good point where maybe the best way to learn more about RSPs and the strengths and weaknesses is just to talk through more of the complaints that people have had or the concerns that people have raised with the Anthropic RSP and RSPs in general since it was released last October. I'm realizing that, oh, I was going to kind of start the weaknesses and worries now, but I'm kind of realizing I've been peppering you effectively with them, maybe, maybe almost since the outset. But now we can, we can really drive into some of the worries that people have expressed. Yes, the first of these is the extent to which we have to trust the good faith and integrity of people who are applying a responsible scaling policy or a preparedness framework or whatever it might be within, within the companies.

• 56:01Rob Wiblin?

  And I imagine this issue might jump to mind for people more than it might have two or three years ago. Because public trust in AI companies to do the right thing at the cost of their business interests is maybe lower than it was years ago when the major players were perceived perhaps more as research labs and less as for-profit companies, which is kind of how they how they come across more these days. And one reason it really matters, it seems like it matters to me, who's doing the work here, is that the Anthropic RSP is full of expressions that are open to interpretation. For instance, hardened security such that non-state attackers are unlikely to be able to steal model weights and advanced threat actors like states can't steal them without significant expense, or access to the model would substantially increase the risk of catastrophic

• 56:46Rob Wiblin?

  misuse and things like that. And who's to say what's unlikely or significant or substantial?

• 56:54Rob Wiblin?

  That sort of language is maybe a little bit inevitable at this point where there's just so much that we don't know. And how are you going to pin those things down exactly to say, you know, it's a 1% chance that a state's going to be able to steal the model? That might just also feel like insincere, false, false precision. But you know, to my mind, that sort of vagueness does mean that there's a slightly worrying degree of wiggle room that could render the RSP less powerful and less binding when when push comes to shove and there might be a lot of money at stake. And on top of that, I guess, I mean, exactly as you were saying, anyone who's implementing an RSP has a lot of discretion over how hard they try to elicit the capabilities that might then trigger additional scrutiny and possible delays to their work and release of, you know, really commercially commercially important products. So yeah, to what extent do you think

• 57:39Rob Wiblin?

  the RSP would be useful in a situation where the people using it were neither particularly super skilled at doing this sort of work and maybe not particularly bought in and enthusiastic about the the safety project that that it's a it's a part of?

• 57:55Speaker 01

  Yeah, so fortunately, I think my colleagues, both on the RSP and elsewhere, are both talented and really, really bought into this. And I think we'll do a great job on it. But I do think the criticism is valid and that like, there is a lot that is left up for interpretation here, and it does rely a lot on people sort of having a good faith interpretation of how to execute on the RSP internally. I think that like, There are some checks in place here, so like having like whistleblower type protections such that people can can say if if a company is is breaking from the RSP or not trying hard enough to to elicit capabilities or to interpret it in a good way and then like public discussion can can add some pressure. But ultimately, I think you really do like you do need regulation to have these sort

• 58:39Speaker 01

  of very strict strict requirements. Over time, I hope we'll make it more and more concrete. The the blocker, of course, on doing that is that we don't know for a lot of these things and being overly concrete where you specify something very, very precisely that turns out to be wrong can be very kind of costly and sort of if you then have to go and change it, et cetera, it can like take away some of the credibility. So sort of aiming for the as concrete as we can make it while while balancing that.

• 59:07Rob Wiblin?

  The the response to this that that to me is just that ultimately, it feels like this kind of policy has to be implemented by a group that's external to the company that's then affected by the determination. It, you know, it really reminds me of, you know, accounting or auditing for for a major company. It's not it's not sufficient, you know, for a major corporation to just have its own accounting standards and follow that and say, oh, you know, we're going to we're going to follow, you know, our own internal best practices. You get and it's legally required that you get external auditors in to to confirm that there's no chicanery going on. And, you know, at the point that these models potentially really are risky or it's, you know, it's plausible that the results will come back saying, we can't release this. We maybe we even have to delete it off of our servers according to the policy.

• 59:51Rob Wiblin?

  I would feel more comfortable if I knew that some external group that had, you know, a different different incentives was the one figuring figuring that out. Do you think that kind of ultimately is where things are likely to to go in the in the medium term?

• 1:00:05Speaker 01

  So I think that'd be great. I would also feel more comfortable that was the case. I think one of the challenges here is that for auditing, there's a bunch of external accountants. It's a very common sort of profession or auditors, like this is a profession, many people know what to do, there are very clear rules. For some of the stuff we're doing, there really aren't like external established auditors that everyone trusts to come in and say, we we took your model and we certified it can't like autonomously replicate across the internet or cause these things. So it it currently, uh, yeah, so I think that's currently not practical. I think that would be great to have at some point. One thing that will be important is that that auditor has enough expertise to properly assess the capabilities of the models.

• 1:00:46Rob Wiblin?

  I suppose, you know, an external company would be an option. Of course, obviously, a government regulator, a government agency would also be another approach. I guess when I think about other industries, it often seems like there's kind of a combination of like, yeah, private companies that then follow government mandated rules and things like that. Do you think that, this is a benefit, actually, I haven't thought of, to do with creating these RSPs, is that it maybe is beginning to create a market? Or it's indicating that there will be a market for this kind of service? Because it's likely that this kind of thing is going to have to be outsourced at some point in future. And there might be many other companies that want to get this similar kind of testing. So perhaps it would encourage people to think about, you know, founding companies that might be able to provide this service in a more credible way in future.

• 1:01:27Speaker 01

  That would be great. And also, like, we publish, like, blog posts on how things go and how our valuations are. So I think there's some hope that people doing this can learn from what we're doing internally. And sort of various iterations we'll put out of our RSP, and that that can like inform, inform something maybe more stringent from that gets regulated.

• 1:01:48Rob Wiblin?

  Have you thought at all about what could be done to make the, like, let's say that it wasn't given out to an external agency or an external auditing company? How it could be, how it could be tightened up to make it less vulnerable to the level of operator enthusiasm? Yeah, I guess you might have thought about this in the process of actually, actually applying it. Are there any ways that it could be, could be, could be stronger without having to completely, you know, outsource the operation of it?

• 1:02:13Speaker 01

  Yeah, I think the core thing is just making it more precise, right? Like, one piece of accountability here is both public and internal commitment to doing it. So, yeah, I guess maybe I should list off some of the reasons that I think it would be hard to break from it, right? Like, this is a formal policy that has been passed by the board. And it's not as though, like, we can just be like, oh, we're not, we don't feel like doing it today. You would sort of need to, like, get the board of Anthropic, get all of leadership, and then get all of the employees sort of bought in to not do this. And, or even to like skirt the edges. And like, you know, I mean, I can speak for myself, right? If someone was like, Nick, can you, can you train this model? We're going to ignore the RSP. I would be like, no, we said we would do that. Why would I do this? And if I wanted to, I would tell my team to do it. They would be like, no.

• 1:02:56Speaker 01

  Why we're not going to do that. So you sort of, you would need to have a lot of buy in. And part of, part of the benefit of having this publicly committing to it and passing it as like an organizational policy is that everyone, you know, everyone is bought in and sort of maintaining that level of buy in, I think is, is quite critical. In terms of like specific checks on like, I think we, we could, I think we have a team that's responsible for checking that we did the like sort of red teaming our evaluations and making sure we actually did them properly. So you can sort of set up a bunch of, a bunch of internal checks there, but ultimately these things do rely on the company implementing them to like really be bought in and, and care about like the, the actual outcome of it.

• 1:03:38Rob Wiblin?

  So yeah, this naturally leads us into this. I actually, you know, I solicited on, on Twitter. I asked, you know, what are people's biggest reservations about RSPs and about Anthropic's RSP in general. And yeah, the, actually probably the most common response was it's not legally binding. Like what's, what's stopping Anthropic from just dropping it, dropping it when, when things really matter? You know, I think someone said, you know, how can we have confidence that they'll stick to RSPs, especially when they haven't stuck to, actually, well, this person said to past admittedly less formal commitments not to push forward the frontier on capabilities. But like what would actually have, have to happen internally? You said you'd have to get staff on board, you'd have to get the board on, on board. Is there a formal process by which the RSP can be rescinded? That is just a really high bar to clear?

• 1:04:19Speaker 01

  Yeah. So the, so basically we do have a process for updating the RSP. So we could go to the board, et cetera. But I think sort of in order to do that, the, I don't know, I'm like, it's hard for me to quite point it out, but it would be like, oh, if, if I wanted to continue training the model, I would go to the RSP team and be like, does this pass? And it'd be like, no. And then maybe, you know, you'd appeal it, appeal it up the chain or whatever. And I sort of at every step along the way, people would say, no, we care about the RSP. Now, on the other hand, there could be like legitimate issues with the RSP, right? We could find that like one of these evaluations we created turned out to be really, really easy in a way that we like didn't anticipate and really is not at all indicative of, of the dangers. And in that case, I think it would be very legitimate for us to, to try to amend the RSP to create a better evaluation that is a test

• 1:05:01Speaker 01

  for it. This is sort of the, the flexibility we're trying to preserve. But I don't think it's, I don't think it would be like simple or easy. I don't, I can't picture a plan where someone could be like, ah, there's a, there's a bunch of money on the table. Can we just like skip the RSP for, for this model? That, that seems somewhat hard to imagine.

• 1:05:18Rob Wiblin?

  The decision is made by this odd board called the Long-Term Benefit Board. Is that right? Or they're the group that decide what the RSP should be?

• 1:05:26Speaker 01

  So the Long-Term Benefit, basically Anthropic has a board that's sort of a corporate board. And some of those seats, in the long term will be the majority of those seats, are elected by the Long-Term Benefit Trust, which doesn't have a financial stake in Anthropic and is there to like set up, to sort of keep us focused on our public benefit mission of like making sure AGI goes well. So yeah, the board is not entirely the same, it's not the same thing as that, but the Long-Term Benefit Trust elects the board.

• 1:05:54Rob Wiblin?

  I mean, I think the elephant in the room here is, of course, there was a long period of time when OpenAI was pointing to its kind of nonprofit board as a thing that would potentially keep it on mission to be really focused on safety and, you know, had a lot of power over the organization. And then in practice, when push came to shove, it seemed like even though the board had these concerns, it was effectively overruled by, I guess, a combination of just the views of staff, maybe the views of the general public in some respects, and potentially the views of investors as well. And I think something that I've taken away from that, and I think many people have taken away from that experience, you know, maybe the board was mistaken, maybe it wasn't, but

• 1:06:37Rob Wiblin?

  these formal structures, you know, power isn't always exercised in exactly the way that it looks on an organizational chart. And I don't really want to be putting all of my trust in these interesting internal mechanisms that companies design in order to try to keep themselves accountable, because ultimately, just if the majority of people involved don't really want to do something, then it feels like it's very hard to bind their hands and prevent them from changing plan at some future time. So maybe within Anthropic, perhaps these structures are really are quite good. And maybe the people are, the people involved are really, really trustworthy and people who I should, you know, have my confidence in, that, you know, even in extremis, they're

• 1:07:19Rob Wiblin?

  going to be thinking about the well-being of humanity and not getting too focused on the commercial incentives faced by Anthropic as a company. But I think I would rather put my faith in something more powerful and more solid than that. And so this is kind of another thing that pushes me towards thinking that the RSP and these sort of preparedness frameworks are a great stepping stone towards external constraints on companies that they don't have ultimate discretion over. It's something that has to evolve into because the impacts are going to be on the entirety, like if things go wrong, the impacts are on everyone across society as a whole. And so there needs to be external shackles effectively put on companies to reflect the harm that they

• 1:08:00Rob Wiblin?

  might do to others legally. I guess I'm not sure whether you want to comment on that, potentially a slightly hot button topic. But yeah, do you think I'm kind of gesturing towards something legitimate there?

• 1:08:11Speaker 01

  Yeah, I think that basically like these shouldn't be seen as sort of a replacement for regulation. I think there are many cases where policymakers can pass regulations that would help here. I think they're intended as sort of a supplement there and a bit as a like learning ground for what might go, what might end up going in regulations. In terms of like, will the, like, does the board really have the power it has types of questions? I don't know, we put a lot of thought into the long term benefit trust. And I think it really does have like direct authority to elect the board and the board does have authority. But I do agree that like ultimately you need to have a culture around thinking these things are important and having everyone bought in. This is like, you know, any employee can always like, as I said, some of these things are like, did

• 1:08:56Speaker 01

  you solicit capabilities well enough? That really comes down to like a researcher working on the, on this, like actually trying their best at it. And that is quite core. And I think that will sort of just continue to be, even if you have regulations, there's always going to be some amount of importance to the people actually working on it, like taking the best seriously and really, really caring about them and like doing the best work they can on that.

• 1:09:20Rob Wiblin?

  Yeah, I guess one takeaway you could have is we don't want to be relying on our trust in individuals and saying, well, you know, we think Nick's a great guy, his heart's in the right place, he's gonna do a good job. Instead, we need to be on more solid ground and say, well, no matter who it is, even if we have something bad in the role, the rules are such, the oversight is such, that we'll still be in a safe place and things will go well. I guess an alternative angle would be to say, when push comes to shove, when things really matter, people might not act in the right way. There actually is no alternative to just trying to have the right people in the room making the decisions, because the people who are there are going to be able to sabotage any legal framework

• 1:10:02Rob Wiblin?

  that you try to put in place in order to constrain them, because it's just not possible to have perfect oversight within an organization from outside. I could see people mounting both of those arguments reasonably. I guess, you know, I suppose you could try doing both, like both trying to pick people who are really, really sound and have good judgment and who you have confidence in, as well as then trying to bind them, so that even if you're wrong about that, you have a better shot at things going well.

• 1:10:29Speaker 01

  Yeah, I think you basically, I think you just want this defense in depth strategy, where like ideally you have all the things lined up, and that way if any one piece of them has a hole, you sort of catch it at the next layer, right? Like what you want is sort of a regulation that is really good and robust to someone not acting in the spirit of it, but in case that is messed up, then you really want someone working on it who is also checking in and is like, ah, okay, I technically don't have to do this, but this seems like clearly in the spirit of how it works. And yeah, I think that's pretty important. I think also for trust, you should just look like, you should look at track records, and I think that we should try to encourage companies, people working on AI, to have track records of prioritizing things. So like, one of the things that makes me feel

• 1:11:12Speaker 01

  great about Anthropic is just a long track record of doing a bunch of safety research to caring about these issues, putting out actual papers, being like, here are a bunch of progress we've made on that field. There are a bunch of pieces. I mean, I think like looking at sort of commitments people have made, you know, do we break the RSP? I think like if we publicly were like, ah, we changed this in some way that I think everyone thought was like silly and, you know, really added risks, then I think people should lose trust according to that.

• 1:11:41Rob Wiblin?

  All right, let's push on to a different worry, although I must admit it has a slightly similar flavor. And that's that the RSP might be very sensible and look good on paper. But if it commits to future actions that at that time, we probably won't know how to do, then it might actually fail to help very much. And I guess to make that concrete, an RSP might naturally say that at the time, you have really superhuman general AI. You need to be able to lock down your computer systems and make sure that the model can't even, can't be stolen even by the most persistent and capable Russian or Chinese state-backed hackers. And that is indeed what Anthropic's RSP says or suggests that it's going to say once you get up to ASL 4 and 5. But I think the RSP actually says as well, we don't currently know how to do that.

• 1:12:24Rob Wiblin?

  We don't know how to secure data against a state actor that's willing to spend hundreds of millions or billions or possibly even tens of billions to steal model weights. Especially not if you ever need those model weights to be connected to the internet in some way in order for the model to actually be used by people. So it's kind of a promise to do what arguably, well, what basically is impossible with current technology. And that means that we need to be preparing now, doing research on how to make this possible in future. But solving the problem of computer security that has beguiled us for decades is probably beyond Anthropic. It's not really reasonable to expect you're going to be able to fix this problem that society as a whole has kind of failed to fix for all this time. It's just going to require coordinated action across countries, across governments,

• 1:13:09Rob Wiblin?

  across lots of different organizations. And so if that doesn't happen, and it's somewhat beyond your control whether it does, then when the time comes, the real choice is going to be between a lengthy pause where, you know, while you wait for fundamental breakthroughs to be made in computer security, or dropping and weakening the RSP so that Anthropic can continue to remain relevant and release models that are commercially useful. And in that sort of circumstance, the pressure to weaken the scaling policy so you aren't stuck for years is going to be, I would imagine, quite powerful. And it could win the day, you know, even if people are like dragged, kind of kicking and streaming to conceding that unfortunately, they have to loosen the RSP even though they don't really want to. Yeah, what do you, what do you make of that worry?

• 1:13:51Speaker 01

  I think what we should do in that case is instead, we should pause and we should focus all of our efforts on safety and security work. That might include looping in external experts to help us with it, but we should do the, like, put in the best effort that we can to mitigate these issues such that we can still realize the benefits to deploy the technology, but without the dangers. And then if we can't do that, then I think we need to make the case publicly to governments, other companies. There's some risk to public. So exactly, we'd have to be strategic in exactly how we do this, but basically make the case that there are really serious risks that are imminent and that everyone else should take sort of appropriate actions. There's like a flip side to this, which is just like, I think I mentioned before, if we just messed up our evals, the model's clearly not dangerous. And we just really like screwed up on some eval and then we

• 1:14:36Speaker 01

  should like follow the process and RSP that we've written up. We should go to the board. We should create a new test that is like going to, that we actually trust. I think, I would also just say like, people don't need to follow incentives, right? Like, I think you can make a lot more money doing something that isn't hosting this podcast, probably. Certainly if you like had pivoted your career earlier, there are more profitable things. So I think this is just a case where like the stakes are, would be extremely high. And I think it's just somewhere where it's important to just do the right thing in that case.

• 1:15:04Rob Wiblin?

  If I think about how this is most likely to play out, I imagine that at the point that we do have models that we really want to protect from even the best state-based hackers, there probably have been some progress in computer security, but like not nearly enough to make you or me feel comfortable that there's just no way that, you know, China or Russia might be able to steal the model weights. And so like, it is very plausible that the RSP will say Anthropic, well, you have to, you know, keep this on a hard disk, not connected to any computer. You can't train models that are more capable than the thing that we kind of already have, that we don't feel comfortable handling. And then how does that play out over, you know, there are a lot of people who are very concerned about safety at Anthropic. I mean, I've seen, there's kind of league tables now of, you know, different AI companies and enterprises.

• 1:15:49Rob Wiblin?

  And how good do they look on an AI safety point of view? And Anthropic always kind of comes out at the top, I think, by a decent margin. But you know, months go by, other companies are not being as careful as this. You've complained to the government and you've said, look at this horrible situation that we're in. Something has to be done, but I don't know. I guess possibly the government could step in and help there, but maybe they won't. And then over a period of months or years, doesn't the choice effectively become if there is no solution? Either take the risk or just be rendered irrelevant?

• 1:16:22Speaker 01

  Yeah, so maybe just like step, just going back to the beginning of that. Like, I don't think we will put something in that it says there is zero risk from something. Like, I think you can sort of never get to zero risk. I think often with security, you'll end up with some security productivity trade off. So you can, you could end up taking some really extreme risk or some really extreme security productivity trade off, where only one person has access to this. Maybe you've like locked it down in some huge amount of ways. It's possible that you can't even do that. You really just can't train the model. But there is always going to be some sort of, some sort of balance there. And I don't think we'll push to the zero risk perspective. But yeah, I think that that's just like a risk. I don't know. I think there's a lot of risks that companies face where they could fail, right?

• 1:17:02Speaker 01

  We also could just like fail to make better models and, and not succeed that way. I think the point of the RSP is it has like tied our commercial success to the safety mitigations. So in some ways, it just, it just adds on another, another risk in the same way as, as any other company risk.

• 1:17:20Rob Wiblin?

  It sounds like I'm having a go at you here, but I think, really, I think what this shows up is just that it's, I think that this, the scenario that I painted there is really quite plausible. And it just shows that this, this problem cannot be solved by Anthropic. Probably like it can't be solved by even all of the AI companies combined. The only way that this RSP is actually going to be able to be usable, in my estimation, is if other people rise to the occasion and pitch, and like, and governments actually do the work necessary to fund the, solutions to computer security that will allow us to have the model weights be sufficiently secure in this situation. And yeah, you're not blame worthy for that situation. It just, just says that there's a lot of people who need to, need to do a lot of work in coming years.

• 1:18:04Speaker 01

  I think that, yeah, and I think I might be more optimistic than you or something. I do think if we get to something really, really dangerous, we can make a very clear case that it's dangerous and like, these are the risks unless we can implement these mitigations. Like, I hope that like, at that point, it will be a much clearer case to pause or something right now. I think there are many people who are like, we should pause right now and see everyone saying no. And they're like, oh, these people don't, don't care. They don't care about like, major risks to humanity. And I think really the core thing is people don't believe there are risks to humanity right now. And once we get to like, this sort of stage, I think that we will, we will be able to make those risks very clear, very like, immediate tangible and like, uh, I don't know. No one wants to be the company that like, caused a massive disaster. And no government also probably wants to have like, allowed a company to cause that.

• 1:18:48Speaker 01

  Yeah, like, it will be, it will feel much more immediate at that point.

• 1:18:52Rob Wiblin?

  Yeah, I think Stefan Schubert, this commentator who I read on Twitter, has been making the case for a while now that many people who have been thinking about AI safety, I guess, including me, have perhaps underestimated the degree to which the, the public is likely to react and respond, and governments are going to get involved once the problems are apparent, once they really are convinced that, that there is a, is a threat here. I think he calls it, um, this bias and thought where you imagine that people in the future are just gonna sit on their hands and not do anything about the problems that are, that are readily apparent. He calls it as sleepwalk bias. And I guess, I think we have seen evidence over the last year or two that as the capabilities have improved, people have gotten a lot more serious and a lot more concerned, a lot more open to that idea that it's important for the government to be involved here.

• 1:19:35Rob Wiblin?

  There's a lot of, you know, a lot of actors that need to, need to step up their game and, and help to solve these problems. So, yeah, I think you, you might be right. On an optimistic day, maybe I could hope that other groups will be able to do the necessary research soon enough, that Anthropic will be able to actually apply its RSP in a timely manner. I guess, fingers crossed. Yeah, I just want to actually ask you next. What are your biggest reservations about RSPs or Anthropic's RSP personally? If it fails to improve safety as much as you're hoping that it will, what's the most likely reason for it to not live up to its potential?

• 1:20:13Speaker 01

  So I think for Anthropic specifically, it's definitely around this under elicitation problem. I think it's a really fundamentally hard problem to take a model and say, oh, you've tried as hard as one could to elicit this particular danger. There's always some, maybe there's a better researcher. There's a saying, no negative result is final. Like if you fail to do something, someone else might just succeed at it next. So that's one thing I'm worried about. And then the other one is just unknown unknown. So we are creating these evaluations for risks that we are worried about and we see coming, but there might be risks that we've missed, things that we didn't realize would come before, either didn't realize would happen at all or thought would happen after for later levels, but turn out to arise earlier.

• 1:20:57Rob Wiblin?

  What could be done about those things? Would it help to just have more people on the team doing the evals or to have more people, I guess, both within and outside of Anthropic, trying to come up with better evaluations and figuring out better red teaming methods?

• 1:21:11Speaker 01

  Yeah, and I think this is really something that people outside Anthropic can do. Anyone, the elicitation stuff has to happen internally and that's more about putting as much effort as we can into it, but creating evaluations can really happen anywhere. Coming up with new risk categories, threat models is something that anyone can contribute to.

• 1:21:30Rob Wiblin?

  Yeah, what are the places that are doing the best work on this? I imagine Anthropic surely has some people working on this, but I guess I mentioned METR, can't remember what that stands for right now, but they're a group that helps to develop the idea of RSPs in the first place and develop evals. And I think the AI Safety Institute in the UK is involved in developing these sort of standard safety evals. Is there anywhere else that people should be aware where this is going on?

• 1:21:55Speaker 01

  Yeah, there's also the US AI Safety Institute, and I think this is actually something you could probably just do on your own. I think one thing, I don't know, at least for people like early in career, if you're trying to get a role doing something that I would recommend is just go and do it. So I think you probably could just write up a report, post it online, be like, this is my threat model, these are the things I think are important. You could implement the evaluations and share them on GitHub. But yeah, there are also organizations you could go to to get mentorship and work with others on it.

• 1:22:23Rob Wiblin?

  I see. So this would look like, I suppose you could try to think up new threat models, to think up new things that you need to be looking for, because this might be a dangerous capability and people haven't yet appreciated how much it matters. But I guess you could spend your time trying to find ways to elicit the ability to autonomously spread and steal model weights and get yourself onto other computers from these models and see if you can find an angle on trying to find warning signs or signs of these emerging capabilities that other people have missed and then talk about them. And you can kind of just do that while signed into Claude 3 Opus on your website?

• 1:22:59Speaker 01

  Yeah, so I think you'll have more luck with the elicitation if you actually work among the labs, because you'll have access to training the models as well. But you can do a lot with Claude 3 on the website or via an API, which is a programming term for basically an interface where you can send a request for like, I want a response back and automatically do that in your app. So you can sort of set up a sequence of prompts and test a bunch of things via the APIs for Claude or any other publicly accessible model.

• 1:23:26Rob Wiblin?

  To come back to this point about what's acceptable risk and maybe trying to make the RSP a little bit more concrete. I read from a critic of the Anthropic RSP that, I'm not sure how true this is, I'm not an expert on risk management, but this person was saying that it's more true, like at least in more established areas of risk management, where maybe you're thinking about, you know, what's the probability that a plane is going to fail and crash because of some mechanical failure. It's more typical to say, you know, we've studied this a lot and we think that the probability of, well, let's just talk about the AI example, you know, rather than say we need the risk to be, you know, not substantial, instead you'd say, you know, with our practices, you know, our experts think that the probability

• 1:24:10Rob Wiblin?

  of an external actor being able to steal the model weights is X percent per year. And these are the reasons why we think the risk is that level. And that's below what we think of as our acceptable risk threshold of, you know, X, where X is larger than Y. I guess if you, there's a risk that those numbers would kind of just be made up. You could kind of assert anything because, because it's all a bit unprecedented. But I suppose that would make clear to people what the remaining risk is, like what, what acceptable risk you think that you're running. And then people could scrutinize whether they think that that's a reasonable thing to be doing. Do you reckon that, is that a direction that things could maybe go?

• 1:24:48Speaker 01

  Yeah, I think it's like a fairly common way that people in like the EA and rationality community like speak where they give a lot of probabilities for things. And I think, I think it's really useful. Like it's an extremely clear way to communicate. Like I think a 20% chance this will happen is just way more informative than I think it probably won't happen, which could be 0% to 50% or something. So I think it's very useful in many contexts. I also think it's very frequently misunderstood because for most people, I think they hear a number and they think it's based on something, that there's some, some calculation and they give it like more authority. If you say, you know, ah, there's a 7% chance this will happen, people are like, oh, you really know what you're talking about. So I think it can be a useful way to speak, but I think it also can like sometimes communicate more confidence

• 1:25:33Speaker 01

  than we actually have in, in what we're talking about, which, which isn't, I don't know, we didn't have like a thousand governments attempt to steal our weights and X, X number of them succeeded or something. It's, it's much more going off of sort of a, a judgment based on like our security experts.

• 1:25:52Rob Wiblin?

  I slightly want to, want to, want to push you on this because I think at the point that we're ASL four or five or something like that, it would be a real shame if Anthropic was going ahead thinking, well, we think the risk that these weights will be stolen every year is 1%, 2%, 3%, something like that. And I guess maybe, maybe you're right in the policy saying, we think it's very unlikely, you know, extremely unlikely that this is going to happen. And then people externally think, well, basically it's fine. They say it's definitely not, it's, it's definitely not going to happen. There's, there's no chance that this is going to happen. And people, and like governments might not appreciate that actually, you know, in your own view, there is a substantial risk being run and you just think it's an acceptable risk, given, given, given the trade offs and what else is going on in the world. I guess it's a, it's a,

• 1:26:35Rob Wiblin?

  It's a social service for Anthropic to be direct about the risk that it thinks it's creating and why it's doing it. But I think it could be a really useful public service. I guess it's the kind of thing that might come up at Senate hearings and things like that, where people in government might really want to know. And I guess at that point, it would be perhaps more apparent why it's really important to find out what the probability is. But yeah, that's a way that I think there is. There's definitely a risk of misinterpretation by journalists or something who don't appreciate the spirit of saying that we think it's X percent likely. But there could also be a lot of value in being more direct about it.

• 1:27:09Speaker 01

  Yeah, I'm not really an expert on communications. I think some of it just depends on who your target audience is and how they're thinking about it. I do think that, I think in general, I'm a fan of making the RSP more concrete, being more specific. I think as, over time, I hope it progresses in that direction as we like learn more and can get more specific. I also think it's important for it to be verifiable. And I think if you start to give these like precise percentages, like people will then ask like, how do you know? And I don't think there really is a clear answer to like, how do you know that the probability of this thing is less than X percent? With, for many of these situations.

• 1:27:44Rob Wiblin?

  It doesn't help with the bad faith actor or the bad faith operator either, because you know, if you say, well, the safety threshold is 1% per year, they can kind of always just claim in this, in the situation where we know so little, that it's less than 1%. It doesn't really bind people all that much. Maybe it's just a way that people externally could understand a little better what the opinions are within the organization, or at least what their stated opinions are.

• 1:28:07Speaker 01

  I'll say that internally, I think it is an extremely useful way for people to think about this, right? So if you are like working on this, I think you probably should think through what is an acceptable level of danger and try to estimate it and communicate with like people you're working closely with in these terms. I think can be a really useful way to like give precise, precise statements. And I think that can be very valuable.

• 1:28:27Rob Wiblin?

  A metaphor that you use within your responsible scaling policy is putting together an airplane while you're flying it. I think that that is one way that the challenge is particularly difficult for the industry and for Anthropic. That unlike with biology safety levels where basically we know the diseases that we're handling and we know how bad they are and we know how they spread and things like that, you know, the people who are figuring out what should BSL level four security be like can look at lots of studies to understand exactly the organisms that already exist and how they would spread and how likely would they be to escape, given these particular ventilation systems and so on. And even then, they mess things up decently often. But in this case, you're dealing with something that doesn't exist, that we're not even sure like when it will

• 1:29:12Rob Wiblin?

  exist or what it will look like. And you're developing the thing at the same time that you're trying to figure out how to make it safe. It's just extremely difficult. And I guess, I mean, and we should expect mistakes. We should, that's something that we should keep in mind, is that even if people who are doing their absolute best here are likely to mess up. And that's a reason why we need this defense in depth strategy that you're talking about, that we don't want to put all of our eggs in the RSP basket. We want to have like many different layers ideally.

• 1:29:37Speaker 01

  Yeah, it's also like a reason to start early. So like, I think one of the things with Cloud 3 was, you know, that was sort of the first model where we really ran this whole process. And I think some part of me felt like, wow, this is kind of silly. Like, I was pretty confident Cloud 3 was not catastrophically dangerous. It was like slightly better than GPT-4, which had been out for a long time, but not caused a catastrophe. But I do think that like, the process of doing that, learning what we can, and then putting out like public statements about how it went, what we learned, is the way that we can have this run really smoothly on the next time. Like, we can make mistakes now, right? We could have made as many, we could have made a ton of mistakes because the stakes are pretty low at the moment. But in the future, the stakes on this will be really high. And it will be really costly to make mistakes. So it's important to get those practice runs in.

• 1:30:22Rob Wiblin?

  All right. Another recurring theme that I've heard from some commentators is that, in their view, the anthropic RSP just isn't conservative enough. So on that account, there should be wider buffers in case you're under eliciting capabilities that the model has that you don't realize, which is something that you're pretty concerned about. And I guess a different reason would be you might worry that there could be discontinuous improvements in capabilities as you train bigger models with more data. So to some extent, model learning and improvement is, from a very zoomed out perspective, is quite continuous. But on the other hand, its ability to do any particular task, it can go from fairly bad to quite good surprisingly quickly.

• 1:31:04Rob Wiblin?

  So there can be sudden unexpected jumps with particular capabilities. Yeah, firstly, can you maybe explain, again in more detail, how the anthropic RSP handles these safety buffers? Given that you don't necessarily know what capabilities a model might have before you train it, that's quite a challenging constraint to be operating under.

• 1:31:21Speaker 01

  Yeah, so there are these red line capabilities. These are the capabilities that are actually the dangerous ones. We don't want to train a model that has these capabilities until we have the next set of precautions in place. Then there are evaluations we're creating, and these evaluations are meant to certify that the model is far short of those capabilities. It's not, can the model do those capabilities? Because once we pass them, we then need to put all the safety mitigations in place, et cetera. So, and then, when do we have to run those evaluations? It's sort of just, we have some heuristics of like, when the effective compute goes up by a certain fraction, that is a very cheap thing that we can evaluate on every step of the run, or something along those lines, that we know when to run it. In terms of how conservative they are, I guess, like, one example it gives, if you're

• 1:32:05Speaker 01

  thinking about autonomy, where a model could spread to a bunch of other computers and sort of autonomously replicate across the internet, I think our evaluations are pretty conservative on that front. We like test if it can replicate to a fully undefended machine, or if it can do some basic fine tuning of a other language model to add a simple backdoor. I think these are pretty simple capabilities, and there's always a judgment call there, right? We could set them easier, but then we might trip those and look at the model and be like, ah, this isn't really dangerous. It doesn't warrant the level of precaution that we're going to give it.

• 1:32:40Rob Wiblin?

  Yeah, there was something also about you. You said that you, the RSP says that you'll be worried if the model can succeed at half the time at these various different tasks, trying to spread itself to other machines. Yeah, why is, why is half the time, like succeeding half the time the threshold?

• 1:32:58Speaker 01

  Yeah, so there's sort of a few tasks. I don't off the top of my head remember the exact, exact thresholds, but basically like it's just a reliability thing, right? So in order for a model to chain all of this together, or yeah, so in order for the model to chain all of these capabilities together into like some long running thing, it does need to have a certain success rate. Probably it actually needs a very, very high success rate in order for it to like start autonomously replicating despite us trying to stop it, et cetera. So we set a threshold that's like fairly conservative on that front.

• 1:33:28Rob Wiblin?

  I guess it's part of the reason you're thinking, well, if a model can do this worrying thing half the time, then it might not be very much additional training away from being able to do it 99% of the time. That might just require some additional fine tuning to get there. And so then the model might be dangerous if it was leaked because it would be so close to being able to do this stuff.

• 1:33:46Speaker 01

  Yeah, I mean, that's often the case, although of course we could, we could then elicit it to go, if we'd set a higher number and we got, even if we got 10%, maybe that's enough that we could bootstrap it. So like often when you're training something, if it can be successful, you can reward it for that successful behavior and then increase the odds of that success. So it's often easier to sort of go from 10% to 70% than it is to go from like 0% to 10%.

• 1:34:09Rob Wiblin?

  So if I understand correctly, the RSP proposes to retest models every time you increase the amount of training, compute or data by fourfold. Is that right? That's, that's kind of the checkpoint?

• 1:34:21Speaker 01

  Yeah, so we're still, still thinking about what is sort of the best thing to do there and that one might change, but we use this notion of effective compute. So really this has to do with when you, you train a model, it goes down to a certain loss and we have these nice scaling laws of if you have more compute, you should expect to get to the next loss. You might also have a big algorithmic win where you don't use any more compute, but you get to a lower loss. And we sort of have coined this term effective compute to that would sort of account for that as well. These, these jumps are sort of the, the jump where you can tell, like we have sort of a visceral sense of how much smarter a model seems when you do that jump and have sort of set that as our, our bar for like when we have to run all of these evaluations, which do require, you know, a staff member to go and run them, spend a bunch of time trying to elicit the capabilities, et cetera.

• 1:35:06Speaker 01

  I think this is somewhere I'm wary of sounding like too precise or like we understand this too well. We don't really know what the effective compute gap jump is between the yellow lines and the red lines. This is much more just like how we are thinking of how we are thinking about the problem and how we are trying to set these like set these evaluations. And the reason that the the yellow line evaluations really do need to be substantially easier, they'd be like far from the red line capabilities because you might you might actually overshoot the yellow line capabilities by a fairly significant measure of stuff of when you run evaluations.

• 1:35:38Rob Wiblin?

  So I think if I recall, it was Zhve, who's been on the show before, who wrote in his blog post assessing the anthropic RSP, just that he thinks that this ratio between the 4X and the 6X is not large enough, that, you know, if there is some discontinuous improvement, or, yeah, you've really been under eliciting the capabilities of the models at these at these kind of interim check in points, that that does leave the possibility that you could overshoot and, you know, get into quite a dangerous point by accident. And then by the time you get there, then the model's like quite a bit more capable than what you thought it would be. And then you've got this difficult question of whether to like, do you then, you know, press the emergency button and delete all the weights because you've overshot? There'd be incentives not to do that because you might be throwing away $100 million worth of, well, I guess I don't know how much it would be, but you'd

• 1:36:21Rob Wiblin?

  be throwing away a substantial amount of compute expenditure, basically, to create this thing. And this and this just worries him. I mean, that could be solved, I think, in his view, just by having a larger ratio there, having a having a larger safety safety buffer. Of course, that then runs a risk that you're doing these like, constant check ins on stuff that you really are pretty confident is not going to be actually that dangerous. And and people might get frustrated with the RSP and feel like it's wasting their time. So it's kind of a judgment call, I guess, how large that buffer needs to be.

• 1:36:51Speaker 01

  Yeah, I think it's a tricky one to communicate about because it's confidential what the jumps are between the models or something. I think one thing I can share is like, we ran this part, we ran this on Claude 3 partway through through training. So the jump from Claude 2 to Claude 3 was bigger than the than that gap. So you could sort of think of that as like, an intelligence jump from Claude 2 to Claude 3 is bigger than what we're allowing there. I think these feel, it feels reasonable to me, but I think this is just a judgment, a judgment call that like different people can have. And I think that like, this is the sort of thing where if we learn over time that this seems too big, or it seems too small, that's the type of thing that hopefully we can, we can talk about publicly.

• 1:37:31Rob Wiblin?

  Yeah, is that something that you get feedback on that I suppose is your if you are training these big models and you're checking in on them, you can kind of predict where you expect them to be, like how likely you are, how likely they are to exceed a given threshold. And then if you do ever get surprised, then that could be a sign that, wow, we need to like increase the increase the the buffer range here.

• 1:37:50Speaker 01

  It's hard because right, like the first one, if like the thing that would really tell us is if we don't pass the yellow line for one model and then on the next iteration, suddenly it's like blows past it. And we look at this and we're like, whoa, this thing is really dangerous. It's probably past the red line and we have to, you know, delete the model or immediately put in the security features, etc. For the next level. I think that that would be a sign that we'd like set the the buffer too small.

• 1:38:13Rob Wiblin?

  I guess, again, not the ideal way to learn that, but I suppose it definitely could set the set a cat amongst the pigeons.

• 1:38:20Speaker 01

  Yeah, there are there are there would be earlier signs where you would notice like, oh, we really overshot by a lot. It feels like we're closer than than we expected or something. But that would sort of be the the failure mode, I guess, rather than the warning sign.

• 1:38:32Rob Wiblin?

  So reading reading the RSP, it seems pretty focused on kind of catastrophic risks from like misuse, you know, sort of terrorist attacks or, you know, CBRN and, you know, AI gone rogue, like spreading out of control, that sort of thing. Is it basically right that the the RSP or this kind of framework is not intended to address kind of structural issues like AI displaces people from work and now they can't earn a living or, you know, AIs are getting militarized and that's making it more difficult to evolve to to like prevent, you know, military encounters between countries because we can't control the models very well, or, you know, more more like near term stuff like algorithmic bias or deep fakes or misinformation. Are those kind of things that have to be dealt with

• 1:39:16Rob Wiblin?

  by something other than a responsible scaling policy?

• 1:39:19Speaker 01

  Yeah, those are important problems, but like the the RSP is is responsible for like preventing sort of catastrophic risks and particularly has this framing that works well for things that are sort of acute, like new capability is developed and could sort of first order, cause a lot of damage. It's it's not going to work for things that are like, what is the long term effect of this on society over time? Because we can't design evaluations to to test for that effectively.

• 1:39:44Rob Wiblin?

  Anthropic does have different teams that work on those other two clusters that I talked about, right? What are they called?

• 1:39:50Speaker 01

  So we have a societal impacts team is probably the most relevant one to that. And the policy team also has a lot of relevance to these issues.

• 1:39:59Rob Wiblin?

  All right. Yeah, I guess we're kind of going to wrap up on RSPs now. Is there anything you wanted to maybe say to the audience to wrap up this section? Like additional work that you think will be like ways that the audience might be able to contribute to this enterprise of coming up with better internal company policies? And then figuring out, I suppose, how there could be models for other actors to come up with government policy as well?

• 1:40:22Speaker 01

  Yeah, I mean, I think this is just a thing that many people can work on. You know, if you if you work at a lab, you could talk to people there, think about what they should have as an RSP, if anything. If you work in policy, you should you should read these and think about if there are if there are lessons to take. If you don't do either of those, I think you really can think about threat modeling, post about that, think about evaluations, implement evaluations and share those. It's much, I think it is the case that, you know, these companies are very busy. And if there is something that's just like shovel ready or like ready on the shelf, you could just grab this this evaluation. It's really quite easy to run them. So, yeah, I think there's like quite a lot that people can do to help here.

• 1:41:02Rob Wiblin?

  All right, let's push on and talk about the case that listeners might be able to contribute to making in superintelligence go better by working at Anthropic on some of its various different projects. First, though, how did you end up in your current role at Anthropic? And what's been your, the career journey that led you there?

• 1:41:20Speaker 01

  Yeah, so I think like largely started with an internship at GiveWell, which listeners might know, but it's a sort of a nonprofit that evaluates charities to figure out where to give money most effectively. And I did an internship there. I was sort of learned a ton about global poverty, global health. I was planning to do a PhD in economics and go work on global poverty at the time. But a few people there were sort of pushed me and said, you know, you should really worry about AI safety. We're going to have these super intelligent AIs at some point in the future, and this could be a big risk.

• 1:41:48Speaker 01

  I remember I like left that summer internship and was like, wow, these people are crazy. I like talked to all my family and they were like, whoa. What are you thinking? But then I like, I don't know, it was like interesting. So I kept talking to people. Some people there, other people sort of worried about this. And I felt like every debate I lost, I would like have a little debate with them about why we shouldn't worry about it. And I'd always come away feeling like I lost the debate, but not like fully convinced. And after like, honestly, like a few years of doing this, I eventually decided this was at least enough convinced, at least convincing enough that I should work in AI. It also turned out that working on poverty via this economics PhD route was a much longer and more difficult and like less likely to be successful path than I had anticipated. So I sort of pivoted over to AI.

• 1:42:30Speaker 01

  I worked at Vicarious, which is an AGI lab that had sort of shifted towards a robotics product angle. And I worked on computer vision there for a while, learning how to do ML research. And then actually 80,000 hours reached out to me and convinced me that I should work on on safety more imminently. This sort of like AI was getting better. It was like more important that I just have some direct impact on doing safety research. At the time, I think OpenAI had by far the best safety research coming out of there. So I applied to work on safety at OpenAI. I actually got rejected, then I got rejected again. In that time, Vicarious was nice enough to let me spend half of my time reading safety papers. So I was just sort of reading safety papers, trying to do my own safety research, although it was somewhat difficult.

• 1:43:14Speaker 01

  I didn't really know where to get started. And eventually, I also wrote for like Rohin Shah, who was on this podcast, this alignment newsletter. And I would like write, I read papers and wrote summaries and opinions for them for a while to like motivate myself. But eventually, third try, got a job offer from OpenAI, joined the safety team there and spent sort of eight months there, mostly working on like code models and understanding how code models would progress. The logic here being we just started the first like LLMs training on code. And I thought it was pretty scary. If you think about like recursive self-improvement, like models that can write code is like the first step. And trying to understand what direction that would go in would be really useful for for sort of informing safety directions.

• 1:43:55Speaker 01

  And then a little bit after that, maybe like eight months in or so, all of the safety team leads at OpenAI left, but most of them to start Anthropic. I was sort of felt very aligned with their values and mission. So also went to join Anthropic, sort of the main reason I'd been at OpenAI was for the safety work. And then at Anthropic, actually, everyone was just building out infrastructure to train models. You know, there was no code. It was sort of the the beginning of the company. And I found the thing was my comparative advantage was making them efficient. So I like optimized the models to go faster. You know, as I said, if you have more compute, you get a better model. So that means if your computer, if you can make things run quicker, you get a better model as well. I did that for a while and then shifted into management, which had been something I like wanted to do for a while and started sort of managing the pre-training team when it was it was five people. And then

• 1:44:39Speaker 01

  have been sort of growing the team, growing the team since then, training like better and better models along the way.

• 1:44:46Rob Wiblin?

  Yeah, I'd heard that you'd been consuming 80,000 hour stuff years ago, but I didn't realize it had influenced you all that much. What was the what was the step that we helped with? It was just deciding that it was important to actually start working on safety-related work sooner rather than later?

• 1:45:03Speaker 01

  Actually, a bunch of spots along the way. I think like when I did that Givell internship, I did like a speed coaching at EA Global or something with 80,000 hours and was the people there were some of the people who were pushing me that I should work on on AI, like some of those conversations. And then when I was at Vicarious, I think 80,000 hours reached out to me and was sort of like more more pushy and specifically was like, you should you should go to work directly on safety now, where I think I was otherwise sort of happy to just sort of keep keep learning about AI for a bit longer before shifting over to to safety work.

• 1:45:34Rob Wiblin?

  Well, let's say, yeah, cool that ADK was able to, I guess, I don't know whether it helped, but I suppose it influenced you in some direction. Is there anything you've read from ADK on AI career advice that you think is mistaken, where you want to want to tell the audience that maybe they should do things a little bit differently than what we've been suggesting on the website? Or I guess on this on this show?

• 1:45:55Speaker 01

  Yeah. First, I do want to say ADK was very helpful, both in pushing me to do it and setting me up with connections and introducing me to people and getting me a lot of information. It was really great. In terms of things that I maybe disagree with from standard advice, I think the main one would be to focus more on engineering than research. I think there is sort of this historical thing where people have focused on research more so than engineering. Maybe I should define the difference. The difference between research and engineering here would be that research can look more like figuring out what directions you should work on, designing experiments, doing really careful analysis and understanding that analysis, figuring out what conclusions to draw from a set of experiments. So like, I can maybe give an example, which is like, you're training a model with one architecture and you're like, "Oh, I have an idea. We should try this other architecture."

• 1:46:40Speaker 01

  And in order to try it, the right experiments would be these experiments and these would be the comparisons to confirm if it's better or worse. Engineering is more of the implementation of the experiment. So then taking that experiment, trying it, and also creating tooling to make that fast and easy to do. So make it so that you and everyone else can really quickly run experiments. It could be optimizing code, so making things run much faster, as I mentioned I did for a while, or making the code easier to use so that other people can use it better. And it's not like someone's an engineer or a researcher. You kind of need both of these skillsets to do work. You come up with ideas, you implement them, you see the results, then you implement changes, and it's a fast iteration loop. But it's somewhere where I think there's historically been more prestige

• 1:47:22Speaker 01

  given to the research end, despite the fact that most of the work is the engineering end. So you end up with like, if you come up with your architecture idea, that takes an hour, and then you spend like a week implementing it, and then you run your analysis, and that maybe takes a few days. But it sort of feels like the engineering work takes the longest. And then my other pitch here is going to be that the one place where I've often seen researchers not investigate an area they should have is when the tooling is bad. So when you go to do research on this area and you're like, "Ah, it's really painful. All my experiments are slow to run", it will really quickly have people be like, "I'm going to go do these other experiments that seem easier." So often by creating tooling to make something easy,

• 1:48:05Speaker 01

  you actually can like open up that direction and sort of like trailblaze a path for a bunch of other people to sort of follow along and do a lot of experiments.

• 1:48:13Rob Wiblin?

  So what fraction of people at Anthropic would you classify as like more on the engineering end versus more on the research end?

• 1:48:20Speaker 01

  I might go with my team because I think I actually don't know for all of Anthropic. And I think it's sort of a spectrum, but I would guess it's probably like 60 or 70% of people would be, I would say are like probably stronger on the engineering end than on the research end. And when hiring, I'm like most excited about finding people who are who are strong on the engineering end. And most of our interviews are sort of tailored towards that, not because the research isn't important, but because I think it's sort of there's a little bit less need for it.

• 1:48:48Rob Wiblin?

  The distinction sounds like a little bit artificial to me. Is that is that kind of true? It feels like these things are they're kind of all just a bit a bit part of a package.

• 1:48:56Speaker 01

  Yeah, although I think the main distinction with engineering is that it is a fairly separate career. So I think there are many people, maybe hopefully listen to this podcast, who might have sort of been a software engineer at some tech company for for a decade and built up a huge amount of expertise and experience with like designing good software and such. And those people, I think, can actually learn the ML they sort of like need to know to do the job effectively very, very quickly. And I think there's maybe like another direction people could go in, which is much more like, I think of as like a PhD in many cases where you're like, spend a lot of time developing research taste, figuring out what are the right experiments to run, running these usually at smaller scale and maybe with like less of sort of a single long-lived code base that pushes you to develop better engineering practices.

• 1:49:41Speaker 01

  And I think that skill set, and to be clear, this is this is a relative term. It's also a really valuable skill set and you always need a balance. But I think I've like often had the impression that 80,000 hours pushes people more in that direction who want to work on on safety more the do a PhD, become sort of like a research, a research expert with really great research taste than pushing people more on the sort of become a really great software engineer direction.

• 1:50:07Rob Wiblin?

  Yeah, we had a podcast many years ago, it might be 2018 or 2017 with Catherine Olsen and Daniel Ziegler, where they were also saying engineering is the way to go or engineering is the thing that's really scarce and is also the easier way into the into the industry. But yeah, it isn't it isn't a drum that we've been banging all that frequently. I don't think we've talked about it very much since then. So perhaps that's a that's a bit of a mistake that we haven't haven't been highlighting the engineering roles more. You said it's a kind of a different career track. So you can go from software engineering to the ML or like AI engineering that you're doing at Anthropic. Is that is that the natural career progression that someone has? Or like what someone who's not already in this, how can they learn the engineering skills that they need?

• 1:50:47Speaker 01

  Yeah, so I think engineering skills are actually in some ways the easiest to learn because there's so much, so many different engineering places. I think the way I would recommend it is you could work at any engineering job. Usually outside just working with the smartest people you can, building like the most complex systems. You can also just do this open source. You can contribute to an open source project. This is often a great way to like get mentorship from the maintainers and have something that's publicly visible. So if you then want to apply to a job, you can be like, here is this thing I made. And then you can also just create something new. So you can say, you know, I want, I think if you want to work on AI engineering, you should probably pick a project that's similar to what you want to do. So if you want to work on data for large language models, take Common Crawl, it's publicly available crawl of the web, and write a bunch of infrastructure to process it really efficiently.

• 1:51:32Speaker 01

  Then maybe train some models on it, like build out some infrastructure to train models. And you can, you can sort of build out that, that skill set relatively easily without, without needing to work somewhere.

• 1:51:40Rob Wiblin?

  Why do you think people have been overestimating research relative to engineering? Is it just that research sounds cooler? It's got better branding?

• 1:51:47Speaker 01

  I think historically it was a prestige thing. Like, I think, I think there's sort of this distinction between like research scientist and research engineer that like used to, used to exist in the field where like research scientists had PhDs and were, were kind of like designating the experiments that the research engineers would run. And I think that like that shifted a while ago. So I think in some sense, like the shift has already started happening. Like now, many places, Anthropic included, it's like everyone's a member of technical staff. There, there isn't sort of this distinction. And the reason is that the engineering got more important, particularly with scaling. Like, once you got to the point where you were training models that used a lot of compute on a big distributed cluster, the engineering to implement things on these distributed runs got much more complex than when it was sort of more quick experiments on, on cheap models.

• 1:52:30Rob Wiblin?

  To what extent is it a bottleneck, just being able to build these enormous compute clusters and, and operate them effectively? Is that, is that a core part of the, part of the stuff that Anthropic has to do?

• 1:52:40Speaker 01

  Yeah, so we rely on like cloud providers to actually build the data centers and like put the chips in it. But we've now reached a scale where the amount of compute we're using is sort of, it's a very dedicated thing. These are really huge investments and we're sort of involved collaborating on it from sort of the design up. And I think it's a very critical piece, right? Like, given that compute is the main driver, the ability to take a lot of compute and use it all together and to design things that are cheap, given the types of workloads you want to run, can be like a huge multiplier on how much compute you have.

• 1:53:17Rob Wiblin?

  All right. Yeah. Do you want to give us the pitch for working at Anthropic as a particularly good way to make the future with super intelligent AI go well?

• 1:53:25Speaker 01

  Yeah, I might pitch like working on AI safety first. I think the case here is just like, really, really important. Yeah, I think like AGI is going to be like probably the biggest technological change ever to happen. The thing I think I keep in my mind is just like, what would it be like to have every person in the world able to spin up a company of a million people, all of whom are as smart as like the smartest people you know, and task them with any project they want? And you know, you could do a huge amount of good with that. You could like, help cure diseases, you could tackle climate change, you could work on a ton of, work on poverty. There's sort of a ton of stuff you can do, that would be great. But there's also a lot of ways it could go really, really badly. So I just think like the stakes here are like, are really high. And then there's a pretty small number of people working on it. If you sort of account for all the people

• 1:54:09Speaker 01

  working on like, things like this, I think you're probably going to get something in like, the thousands right now, maybe 10, 10s of thousands. It's rapidly increasing, but it's quite small compared to the scale of the problem. In terms of why Anthropic, I think my main case here is just like, the, I think the best way to like, make sure things go well is to get a bunch of people who care about the same thing and all work together with that as the, as the main focus. I mean, Anthropic's not perfect. We definitely have, have issues, as does every organization. But I think one thing that I've really appreciated is just seeing how much progress we can make when there's like, a whole team where everyone, everyone trusts each other, like kind of deeply shares the same goals and can like, work on that together.

• 1:54:49Rob Wiblin?

  I guess, I mean, there is a bit of a trade off between, if you imagine there's a kind of a pool of people who are very focused on AI safety and kind of have the attitude that you just expressed. One approach would be to split them up between each of the different companies that are working on Frontier AI. And I guess that would have some benefits. The alternative would be to cluster them all together in a single place where they can work together and make a lot of progress. But perhaps the things that they learn won't be as easily diffused across all of the other companies. Yeah, do you have a view on where the right balance is there between clustering people so they can work together more effectively, communicate more versus the need perhaps to have people everywhere who can absorb the work?

• 1:55:28Speaker 01

  I just think the benefits from working together are really huge. Like, I think it's so different what you can accomplish when you have like five people all working together as opposed to five people like working independently, unable to sort of speak to each other, like communicate about what they're doing. You sort of run the risk of just doing everything, everything in parallel, not learning from each other, and also not building trust, which I think is just somewhat a core piece of eventually being able to work together to implement the things.

• 1:55:53Rob Wiblin?

  So in as much as Anthropic is or becomes like the main leader in interpretability research and other kinds of lines of technical AI safety research, do you think it is the case that other companies are going to be very interested to absorb that research and apply it to their own work? Or is there a possibility that Anthropic will have really good safety techniques, but then they might get stuck in Anthropic and potentially the most capable models that are being developed elsewhere are developed without them?

• 1:56:24Speaker 01

  Yeah, so I think that my hope is that if other people have either develop RSP-like things, or if there are regulations sort of requiring particular safety mitigations, people will have a strong incentive to want to get better safety practices. And we publish our safety research, so in some ways, we're making it as easy as possible as we can for them. We're like, here is all the safety research we've done, here's as much detail as we can give about it, please go reproduce it. Beyond that, I think it's hard to be accountable for what other places do. And I think to some degree, it just makes sense for Anthropic to try to set an example, be like, you know, we can be a frontier lab while still prioritizing safety, putting out a lot of safety work, and hoping that kind of inspires others to do the same.

• 1:57:10Rob Wiblin?

  Do you know, I don't know what the answer to this is, but is it, do you know, researchers at Anthropic sometimes go and visit, you know, other AI companies and vice versa in order to like cross pollinate ideas? I think that, I think that used to maybe happen a lot, happen more and maybe things have gotten a little bit tighter the last few years. But that's one idea that you could hope that, you know, research might get passed around. Or at least, I mean, you're saying it gets published. I guess that's important, but there is a risk that, you know, the technical details of how you actually apply the methods won't always necessarily be in the paper or be very easy to figure out. So you also often need to talk to people to make things work.

• 1:57:47Speaker 01

  Yeah, I think once something's published, you can, you can go and give talks on it and et cetera. I think it's, publishing is sort of the first step where until it's published, then it's like confidential information that, that can't be shared. So, so yeah, that it's sort of like you have to first, first figure out how to do it, then publish it. There are more steps you could take, right? You could then like open source code that enables you to, to run it more carefully. And like, there's a lot of work you could go in that direction. And then it's just sort of a balance of how much time you spend on disseminating your results versus pushing your, your agenda, agenda forward to like actually make progress.

• 1:58:21Rob Wiblin?

  It's possible that I'm slightly, I'm analogizing from, I guess, biology that I'm somewhat more familiar with where it's notorious that having a biology paper or a medical paper does not allow you to replicate the experiment because there's so many important details missing. But is it possible that in ML, in AI, you know, people tend to just publish all of the stuff that they, all of the data, maybe and all of the code online on GitHub or whatever, such that it's like much more straightforward to completely replicate a piece, a piece of research elsewhere?

• 1:58:50Speaker 01

  Yeah, I think it's much, much, it's a totally different level of replication. It depends on the paper, but on, on many papers, if a paper's like published in, in some conference, I would sort of expect that someone can pull up the paper and re-implement it with like maybe a week's worth of work. There's a strong norm of sometimes providing the actual code that you need to run, but, but providing enough detail that you can. I think with some things, it can be tricky where like, I don't know, our Turboline team just put out a paper on how to get features on one of our production models. And we didn't release details about our production model. So we tried to include enough detail that someone could replicate this on another model, but they probably can't, they can't exactly like create our, our production model and like get the exact features that we have.

• 1:59:33Rob Wiblin?

  Okay, in a minute, we'll talk about one of the concerns that people might have about working with, working at any AI company. But in the meantime, yeah, what roles are you hiring for at the moment? And what roles are likely to be open at Anthropic in future?

• 1:59:46Speaker 01

  So probably just check our website. There's like quite a lot. I'll kind of highlight a few. So I think the first one I should highlight is the RSP team is looking for people to develop evaluations, work on the RSP itself, figure out the like what the next version of the RSP should look like, etc. On my team, we're hiring a bunch of research engineers. So this is come up with approaches to improve models, implement them, analyze the results, kind of pushing, pushing that loop. And then also performance engineers. This one's maybe like a little bit more surprising, but a lot of the work now happens on custom AI chips. And making those run really efficiently is is sort of absolutely critical. There's a lot of interplay between how fast it can go and how good the model is. So we're sort

• 2:00:28Speaker 01

  of hiring quite a number of performance engineers where it's much, you don't need to have a ton of AI expertise, just having like deep knowledge of how how hardware works and how to write code really efficiently.

• 2:00:39Rob Wiblin?

  How can people learn that skill? Is that, is that, are there courses for that?

• 2:00:44Speaker 01

  There are probably courses. I think with basically everything, I would recommend finding a product, finding a project, finding like someone to mentor you and, you know, be cognizant of their time. Maybe you like spend a bunch of time writing up some code and you send them a few hundred lines and say, can you review this and help me and help me? Or maybe you got some weekly meeting where you ask questions. But yeah, I think you can, you can read about it online. There are probably, you can take courses or you can just pick a project and say, you know, like, I'm going to implement a transformer as fast as I possibly can and sort of hack on that for a while.

• 2:01:15Rob Wiblin?

  Are most people coming into Anthropic from other AI companies or like the tech industry more broadly, or from PhDs, or maybe, maybe not even PhDs?

• 2:01:24Speaker 01

  It's quite a mix. I think like PhD is definitely not necessary. I think it's like one direction to go to build up this skill set. We have a shockingly large number of people with physics backgrounds who have done theoretical physics for a long time and then sort of spend some number of months learning, learning the engineering to be able to like write Python really well, essentially, and then switch in. So I think there's not really a particular background that is sort of needed. It's just, I would say if you're directly preparing for it, just pick the the closest thing you can to the job and do that to prepare. But don't feel like you needed to have some particular background in order to apply.

• 2:02:01Rob Wiblin?

  This question is slightly absurd because there's such a range of different roles that people could potentially apply for at Anthropic. But do you have any advice for people who would like, you know, the vision for their career is working at Anthropic or something similar, but they don't yet feel like they're qualified to get a role at such a serious organization? How can they go, like, what are some interesting underrated paths maybe to gain experience or skills so that they can be more useful to the project in future?

• 2:02:24Speaker 01

  Yeah, I would just pick the role you want and then do it externally, do it in a very publicly visible way, get advice, and then apply with that as an example. So like, if you want to work on interpretability, make some tooling to pull out features of models and post that on GitHub or publish a paper on interpretability. If you want to work on RSP, then like, make a really good evaluation, post it on GitHub with a nice write-up of how to run it and include that with your application. This sort of takes time and is like, it's hard to do well, but I think that it's both the best way to know if it's really the role you want. And when hiring for something, I have a role in mind, and I want to know if someone can do it. And if someone has shown, like, "Look, I'm already doing this role, of course, here's

• 2:03:05Speaker 01

  my proof I can do it well," that's like the most convincing case, in many ways more so than like the signal you'd get out of an interview where all you really know is they did well on this particular question.

• 2:03:14Rob Wiblin?

  So, in terms of working at AI companies, regular listeners will recall that earlier in the year I spoke with Jevi Marshowitz, who's a kind of longtime follower, advances in AI, and I'd say is a bit on the pessimistic side about AI safety, and maybe also not, you know, I think he likes the Anthropic RSP, but he's not like convinced that any of the safety plans put forward by any company or any government are, at the end of the day, going to be quite enough to keep us safe from rapidly self-improving AI. And he said that he was pretty strongly against people taking capabilities roles that would kind of push forward the frontier of what the most powerful AI models can do, I guess, especially at leading AI companies. Because, I mean-

• 2:03:56Rob Wiblin?

  The basic, the basic argument is just that those roles are causing a lot of harm because they're speeding things up and leaving us less time to solve whatever kind of safety issues we're, we're going to need to address. And, uh, you know, I pushed back a little bit and he wasn't really convinced by the various kind of justifications that one might, might give, like the need to gain skills, that you could then apply to, to, to safety work later, or maybe you'd have the ability to influence a company's culture by being on the inside rather than the outside. I think, I think of all companies, uh, Jevi, I, I would, I would certainly imagine is most sympathetic to, to Anthropic. Uh, but I guess his philosophy is very much to kind of rely on hard constraints rather than put trust in any particular individuals or organizations that you, that you like. I'm guessing that kind of, you might've heard what, what Jevi had to say on the, on, in that episode.

• 2:04:41Rob Wiblin?

  And I guess it was a critique that arguably applies to, to, to, to your job, uh, training, training, uh, training Cloud 3 and other, uh, you know, frontier L L Ms. So I'm kind of fascinated to hear what you, what you thought of, of Jevi's perspective there.

• 2:04:54Speaker 01

  So I think it's, I think there's like one argument, which is to do this to build career capital. And then there's another that is like to do this to, for direct impact. I think on the career capital one, I'm, I'm pretty skeptical. I think career capital is sort of weird to think about in this field that's like growing exponentially. You know, I think in sort of a normal field, you, people often say like, you're the most impact late in your career. You know, you build up skills for a while and then then maybe your 40s or 50s is when you have like the, the most impact of your career. But given the like rapid growth in this field, it, I think actually like the, the best moment for impact is now. Uh, I don't know. I often think of like in 2021, I was working at Anthropic. I think there were probably tens of people working on, on large language models, which I thought were like the main path towards A G I. And, you know, now there are thousands. I've, I've improved. I've gotten better since then, but I think probably I had like way more potential for

• 2:05:38Speaker 01

  impact back in 2021, when there were only tens of people working on it.

• 2:05:42Rob Wiblin?

  Yep. Your best years are behind you, Nick.

• 2:05:46Speaker 01

  Yeah. I think in many, I mean, I think the potential was very high. I still think that like, it is still, there's still a lot of room for impact and it will like maybe decay, but it's from an extremely high number, uh, or extremely high level.

• 2:05:58Speaker 01

  And then the thing is just the field isn't that deep. Like because it's such a recent development, it's not like you need to learn a lot before you can contribute. Like there, I think if you want to do physics and you have to learn like the past multi thousands, like thousands of years of physics before you can like push the frontier, that's a very different setup from, from where we're kind of at.

• 2:06:18Speaker 01

  And maybe my last argument is just like, if you think timelines are short, like depending exactly how short, there's just actually not that much time left. So if you're gonna, if you think, you know, there's five years and you spent two of them building up a skill set, like that's, that's a significant fraction of the time. I'm not saying that that should be someone's timeline or anything, but like the shorter they are, the less that makes sense. So yeah, I think I like, from a career capital perspective, I probably agree, if that makes sense.

• 2:06:41Rob Wiblin?

  Yeah, yeah. And what about from, from other points of view?

• 2:06:44Speaker 01

  Yeah, I think from a like, direct impact perspective, I am, I'm fairly less convinced. I don't, part of this is just that I don't have this framing of like, there is capabilities and there is safety and there are like, separate tracks that are racing. I think it's like one way to look at it, but I actually think they're really intertwined. And a lot of safety work relies on capabilities advances. I can give this example of like this many-shot jailbreaking paper that one of our safety teams published, which uses long context models to find a like, a jailbreak that can apply to Claude and to, and to other models. And that like, research was only possible because we had long context models that, that you could test this on. So I think there's, there's just a lot of cases where the things come together. But then, I think like, if you're going to work on capabilities, you should be really thoughtful about it.

• 2:07:29Speaker 01

  Like, I do think there is, there is a risk. You are speeding them up. In some sense, you could be creating something that is really dangerous, but I don't think it's as simple as just, don't, don't do it. I think you want to think all the way through to like, what is the downstream impact when, when someone trains AGI and how, how will you have affected that? And that's, that's a really hard problem to think about. You know, there's a million factors at play, but I think you should, you should think it through, come to your best judgment and then reevaluate and get, get other people's opinions, sort of as you go. Some of the things I might like suggest doing, if you're, if you're considering it, is like, understand what, if you're considering working on capabilities at some lab, is like, try to understand their theory of change. Like, ask, ask people there, ask like, how, how does your work on capabilities lead to a better outcome?

• 2:08:15Speaker 01

  And see if you agree with that. I would talk to their safety team, talk to safety researchers externally, get their takes, say like, do they think that this is, this is a good thing to do? And then I would also look at their track record and their governance and all the things that, sort of to answer the question of, do you think they will push on this theory of change? Over the next five years, are you confident this is sort of what will actually happen? One thing for me that convinced me at Anthropic that I was maybe not doing evil or maybe feel much, much better about it is that our safety team is pretty, is willing to help out with capabilities and like actually wants us to do well at that. So like early on with OPUS, before we launched it, we had a major fire. There was sort of a bunch of issues that came up and there was one like very critical research project that like my team didn't have capacity to push forward.

• 2:08:59Speaker 01

  So I asked Ethan, who's Perez, who's one of the, the safety leads at Anthropic was like, ah, can you, can you help with this? It was actually like during an offsite and Ethan and like most of his team just like left, basically like went upstairs to this building in the woods that we had for the offsite and cranked out research on this for like the next two weeks and just sort of like, and for me at least, I was like, ah, yes, the safety team here also thinks that us, us staying on the frontier is, is critical.

• 2:09:22Rob Wiblin?

  So the basic idea is you think that the safety work, the safety research of all kinds of, of many different types that Anthropic is doing is very useful. It sets a great example. It's research that could then be adopted by, by other groups and also used by, by Anthropic to, to make safe models. And the only way that that can happen, the only reason that that research is possible at all is that Anthropic has these kind of frontier L L Ms on which to experiment and, and, and do their research and to be at the cutting edge generally of, of this technology. And so able to figure out like what are the, what's the safety research agenda that is most likely to be relevant in, in, in, in, in future. If I imagine what, what would Jevi say?

• 2:10:06Rob Wiblin?

  I'm trying to try to model him. I suppose, I, I guess that he might say, yes, given that there's this, this competitive dynamic forcing us to shorten timelines, you know, bringing, bringing the, the, the future forward maybe faster than we feel comfortable with, maybe, maybe that's the best you can do, but wouldn't it be great if we could coordinate more in order to, to buy ourselves more time? I guess that would be one angle. Another angle that I've heard from some people, I don't know whether, whether V would say this or not, is that we're nowhere near actually having all of the safety relevant insights that we can have with the models that we have now. And so given that there's still like so, such fertile material with like Claude 2 maybe, or at least with, with Claude 3 now, like why do you need to go ahead and train

• 2:10:51Rob Wiblin?

  Claude 4? Like is, you know, maybe it's true that five years ago, when we were so much further away from, from having AGI or having. Models that were really interesting to work with. We were a little bit at a loose end trying to figure out what safety research would be would be good because we just didn't have, we're like, we just didn't know what direction things were going to go. But now there's so much safety research, like it's, there's a, there's a proliferation, a Cambrian explosion of, of really valuable work. And we don't necessarily need like more capable models than what we have now in order to, to, to, to, to discover really valuable, valuable things. Yeah. What would you, what, what, what would you say to that?

• 2:11:28Speaker 01

  Yeah. And the first one, I think there's sometimes this like, what, what is the ideal world if like everyone was me or something like if everyone thought what I thought would be the ideal setup. And I think that's just not how sort of the world works. And I think to some degree, you, you really only can control what you do. And maybe you can control what like, or control, maybe you can influence what like a small number of people you talk to do. But I think you sort of have to think about your role in the context of, of the broader world, more or less acting in the way that they're going to, to act. So, so yeah, and like some part, it's definitely a big part of why I think it's important for Anthropic to have capabilities is to enable safety researchers to have better models. Another piece of it is to enable us to have, have an impact on the field and like try

• 2:12:07Speaker 01

  to set this example for other labs to, of like, that you can deploy models responsibly and, and do this in a way that doesn't cause catastrophic risks and, and continues to like push, push on safety. In terms of like, can we do safety research with current models? I think there is definitely a lot to do. I also think we will target that work better the closer we get to AGI. Like I think the last, the last year before AGI will definitely be the, the most targeted safety work. Hopefully there'll be the most safety work happening then, but it will be the most time constrained. So you, you need to do, you need to do work now because there's a bunch of serial time, like serial time that's needed in order to make progress. But you also want to, to sort of be ready to make use of the, of the most well-directed time towards, towards the end.

• 2:12:50Rob Wiblin?

  I guess, you know, another concern that people have, which you touched on earlier, but maybe, maybe we could talk about a little bit more is this worry that Anthropic by existing, by competing with other AI companies, it stokes the, stokes the arms race, increases the pressure on them, feeling that they need to improve their models further, put more money into it, you know, release things as quickly as they can. If I remember you, your basic response to that was like, yes, that, that effect's not zero, but you know, in the scheme of things, there's a lot of pressure on, on companies to be trading models and trying to, to improve them. And, uh, you know, uh, Anthropic is a pretty, is kind of a drop in the, drop, drop in the bucket there. And so this, this isn't necessarily the most important thing to be worrying about.

• 2:13:28Speaker 01

  Yeah, I think basically like that's pretty accurate. I think one way I would think about it is just what would happen if Anthropic stopped existing, right? Like if we all just disappeared, what effect would that have in the world? Either if you or if you think about like if we dissolved as a company and everyone went to work at all the others? And my guess is it just wouldn't look like everyone slows down and is way more cautious. That's like not my model of it. If that was my model, then I would be like, ah, we're probably probably doing something wrong. I think there. Yeah. So I think it's an effect, but I think you I sort of think about in terms of like, what is the net effect of Anthropic being on the frontier when you account for like, all the different actions we're taking, all the safety research, all the all the policy advocacy, like the effect our products have helping users. There's sort of like this whole whole large scheme and you you can't really like add it all up and subtract the costs, but you sort of,

• 2:14:12Speaker 01

  I think you can do that somewhat like in your mind or something.

• 2:14:16Rob Wiblin?

  Yeah, I see. So yeah, so the way you conceptualize it is thinking Anthropic as a whole, what impact is it having by existing compared to some kind of actual where Anthropic wasn't there? And then I then you're contributing to this broader enterprise that is Anthropic and all of its projects and plans to plans together, rather than thinking about, you know, today I got up and I helped to improve cloud three in this like narrow way, what impact does that specifically have? Because that's just maybe it's missing the real effects that that matter the most from like allowing this organization to exist through your work.

• 2:14:46Speaker 01

  Yeah, you could definitely think on the margin, like to some degree, if you're if you're joining and going to like help with something, you are just increasing Anthropic's sort of this marginal amount of capabilities. And then I would just sort of look at like, do you think we would be on a better trajectory if Anthropic had better models? And do you think we'd be on a worse trajectory if Anthropic had significantly worse models would be sort of the comparison? I think you could you could look at like, oh, well, what would happen if Anthropic didn't ship cloud three last year or earlier this year?

• 2:15:12Rob Wiblin?

  What are some of the the lines of research that you're most pleased that you've helped Anthropic to pursue? What are some of the kind of safety safety wins that you're really pleased by?

• 2:15:22Speaker 01

  I'm really excited about the about the safety work. I think there's just like a ton of it that has come out of Anthropic. I could sort of start with interpretability, where there, I think, at the beginning of Anthropic, it was like figuring out how like single-layer transformers work, these sort of like very simple toy models. And in sort of the past few years, that's, and this is not my doing, this is all the interpretability team, that has sort of scaled up into like actually being able to look at production models that people are really using and find find useful, and identify particular features. We sort of had this recent one on the Golden Gate Bridge, where they found a feature that if you if you increase it, it makes the model, it's it's it's the model's representation of the Golden Gate Bridge, and if you increase it, the model talks more about the Golden Gate Bridge. And that's sort of like a very cool causal effect, where you can change

• 2:16:05Speaker 01

  something, and it actually changes the model behavior in a way that like gives you more certainty, you've really found something.

• 2:16:10Rob Wiblin?

  Is the hope with that, for example, that you could find, I guess, you found the, I don't know whether you call it the the Golden Gate Bridge in your own, but I don't know, the the you found where the Golden Gate Bridge is in the model, and then you can turn that up so that it, I'm not sure whether all listeners will have seen this, but it is, it is very funny, because you get Claude 3, it just, its mind is constantly turned to thinking about the Golden Gate Bridge, even when the question has nothing to do with it, and it gets frustrated with itself, realizing that it's going off topic, and then tries to bring it back to the thing that you asked, but then it just can't, it just can't avoid talking about the Golden Gate Bridge again. Is the hope that you could find the honesty part of the model and scale that up enormously, or tentatively find the, you know, deception part and scale that down in the same way?

• 2:16:52Speaker 01

  Yeah, so like, there actually are a bunch, if you look at the paper, there's a bunch of safety relevant features. I think that the the Golden Gate Bridge one was like cuter or something, and like, got a bit more attention. But yeah, there are a ton of features that are really safety relevant. I think one of my favorites was one that will tell you if code is incorrect or something, so or has a vulnerability, something along those lines. And then you can change that, and suddenly it doesn't write the vulnerability, or it makes the code correct. And that sort of shows the model knows knows about concepts at that level. Now, can we use this directly to like, solve major issues? Probably not yet, right? There's a lot more work to be done here. But I think it's just been a huge amount of progress. And I think on, yeah, I think that it's it's fair to say like that that progress wouldn't have happened without the like Anthropics interpretability team,

• 2:17:37Speaker 01

  like pushing that field forward a lot.

• 2:17:39Rob Wiblin?

  Yeah. Is there any other anthropic research that you're proud of?

• 2:17:42Speaker 01

  Yeah, I mentioned this one a little bit earlier, but there's sort of this like multi-shot jailbreaking from our alignment team that pushed like, if you have a long context model, which is something that we released, you can jailbreak a model by just giving it a lot of examples in this very long context. And it's a very reliable jailbreak to get models to do things you don't want. So this is sort of in the vein of the RSP. One of the things we want to have is to be able to be robust to really intense red teaming, where if a model has a dangerous capability, you can have safety features that prevent people from listening it. And this is like an identification of a major risk for that. We also have this like sleeper agents paper, which sort of shows early signs of models having deceptive behavior. Yeah, I could talk about a lot more of it. There's actually just like a really huge amount.

• 2:18:26Speaker 01

  And I think that's fairly critical here is like, I think often with safety things, people get focused on inputs and not outputs or something. And I think the important thing is not is to think about like, how much progress are we actually making on the safety front? Like that is ultimately what's going to matter in like some number of years when we get close to AGI. It won't be like, how many GPUs do we use? How many people worked on it? It's going to be like, what did we find? And how effective were we at it? And for product, this is very natural. People think in terms of revenue, you know, how many users did you get? You have these like end metrics that are like the fundamental thing you care about. And I think for safety, it's much fuzzier and harder to measure. But putting out a lot of papers that are good is quite important.

• 2:19:06Rob Wiblin?

  Yeah, I mean, if you want to keep going, if there's any others that you want to flag, I mean, no hurry.

• 2:19:12Speaker 01

  Yeah, I mean, like I said, influence functions, I think this is a really, really cool one. Where one framing of mechanical interoperability is, or sorry, mechanistic interoperability is, it lets us look at the weights and understand like why a model has a behavior by looking at particular weight. The idea of influence functions is to look, is to understand why a model has a behavior by looking at the training data. So you can understand what in your training data contributed to a particular behavior from the model. Yeah, I think that was, that was like pretty exciting to see work. I think constitutional AI is another example I would highlight where we can train a model to follow a set of principles via sort of AI feedback. So instead of having to sort of have human feedback for a bunch of things, you can just write out a set of principles. I want the model to not do this.

• 2:19:57Speaker 01

  I want to not do this. I want to not do this and train the model to follow that constitution.

• 2:20:02Rob Wiblin?

  Is there any work at Anthropic that you personally would be wary, or at least not enthusiastic to contribute to?

• 2:20:11Speaker 01

  So I think in general, this is like a good question to ask. I think like the work I'm doing is currently the highest impact thing. And I think I should frequently wonder if that's the case and talk to people and reassess. I think right now, though, I don't think there's any work at Anthropic that I wouldn't contribute to or like think shouldn't be done. It's probably not the way I would I would approach it. It's like if there was something that I thought Anthropic was doing that was bad for the world, I would write a doc making my case and send it to like the relevant person who's who's responsible for that and then like have a discussion with them about it. Because just opting out isn't going to actually change it, right? Someone else will just do it. That doesn't accomplish much. And we try to sort of operate as one team where everyone is like aiming towards the same goals and not have this sort of like two different teams are at odds, where you're like hoping someone else won't succeed.

• 2:20:56Rob Wiblin?

  I guess people people might have a reasonable sense of the culture at Anthropic just from listening to this interview. But is there anything else that's interesting about working at Anthropic that might not be immediately obvious?

• 2:21:05Speaker 01

  I think the one thing that like is part of our culture that at least surprised me is spending a lot of time pair programming. This is just it's just a very collaborative culture. So I think when I first joined, I don't know, I was working on a particular method of like distributing a language model training across a bunch of GPUs. And Tom Brown, who's one of the founders and had done this for GPT-3, would just put like an eight hour meeting on my calendar. And I just watched him code it. And then like I was on a different time zone. So basically, like during the hours when he wasn't working and I was working, I would like push forward as far as I could. And then the next day we would meet again and like continue on. And I think this is just a really good way of aligning people where it's a shared project instead of being a you're bothering someone by asking for their help.

• 2:21:47Speaker 01

  It's like you're working together on the thing and and you learn you learn a lot. You also learn a lot of like the smaller things that you wouldn't otherwise see of like, how does someone navigate their code editor? Like, what exactly is their style of debugging this sort of problem? Whereas if you go and ask them for advice or like, how do I do this project? They're not going to tell you the like low level details. When do they pull out a debugger versus some other method of some other tool for solving the problem?

• 2:22:12Rob Wiblin?

  So this is literally just watching one another's screens? Or you're doing a screen share thing where you watch?

• 2:22:17Speaker 01

  Yeah, I'll give some free advertising to Tupul, which is this great software for it, where you can share screens and you can control each other's screens and like draw on the screen. And typically, one person will drive. They'll be like basically doing the work. Another person will watch, ask questions, point out mistakes, occasionally grab the cursor and just change it.

• 2:22:38Rob Wiblin?

  It's interesting that I feel in other industries, having your boss or a colleague stare constantly at your screen would give people the creeps or they would hate it. Whereas it seems like in programming, this is something that people are really excited by and they feel like it enhances their productivity and makes the work a lot more fun.

• 2:22:55Speaker 01

  Oh, yeah. I mean, it can be exhausting and like tiring. I think the first time I did this, I was too nervous to take a bathroom break. And after multiple hours was like, can I go to the bathroom? And realized that was an absurd thing to ask after like multiple hours of working on something. So like.

• 2:23:07Rob Wiblin?

  What are you back at primary school? Yeah.

• 2:23:09Speaker 01

  Yeah, it could definitely feel a little bit more intense in that like someone's watching you and they'll, you know, they might give you feedback on like, ah, you're kind of going slow here. Like this sort of thing would speed you up. But I think you really can learn a lot from that sort of intensive partnering with someone.

• 2:23:22Rob Wiblin?

  All right. I think we've talked about Anthropic for a while. I guess the final question is, so obviously Anthropic, its main office is in San Francisco, right? And I heard that it was opening a kind of a branch in London. Are those the two main places? And are there many people who work remotely or anything like that?

• 2:23:39Speaker 01

  Yeah, so we have the main office in SF and then we have office in London, Dublin, I think, Seattle and New York. Our typical policy is like 25% time in person. So some people will mostly work remotely and then go to sort of one of the hubs for, it's usually one week per month. The idea of this is that you should, we want people to like build trust with each other and like be able to work together well and like know each other. And that that sort of involves some amount of social interaction with your coworkers, but also for a variety of reasons, like sometimes getting the best people. People are bound to particular locations.

• 2:24:13Rob Wiblin?

  I kind of have been assuming that all of the main AI companies are probably hiring hand over fist. And I guess I know Anthropic's received like big investment from Amazon, maybe some other folks as well. But is it, yeah, does it feel like the organization is growing a lot? That there's lots of new people around all the time?

• 2:24:28Speaker 01

  Yeah, growth has been very rapid. We recently moved into a new office. Before that, we were, we'd run out of desks, which was an interesting moment for the company. It was very cramped. Now there's space. Yeah, I mean, rapid growth is a very like difficult challenge, but also a very interesting one to work on. I think that's like, to some degree, what I spend a lot of my time thinking about is just how can, how can we grow, grow the team and be able to maintain this sort of linear growth and productivity is sort of the dream, right? If you double the number of people, you get twice as much done. And you never actually hit that. But it's, it takes a lot of work because there's now all this communication overhead and you have to do a bunch to make sure everyone's like working towards the same goals, sort of maintain the, the culture that we, that we currently have.

• 2:25:11Rob Wiblin?

  I've given you a lot of time to talk about what's great about Anthropic, but I should at least ask you, what's, what's, what's kind of worst about Anthropic? What would you most like to see, to see improved?

• 2:25:19Speaker 01

  I think honestly, the first thing that comes to mind is just the like stakes of what we're working on or something. I think that there was sort of a, a period a few years ago where I felt like, ah, safety is really important. I felt like motivated and I was, it was, it was a thing I should do and like got, got value out of it. But I didn't feel this sort of, oh, it could be really urgent. This might be like, decisions I'm making are just like really, really high stakes decisions. And I think Anthropic definitely feels high stakes. I think it's often portrayed as this like do me culture. I don't think it's that. I think there is like a lot of benefits. And I think we, I think I'm like pretty excited about the work I'm doing and it's, it's quite fun on a day to day basis. But it does feel very high intensity and kind of many of these decisions are, they, they really do matter, right? If you, if you really think we're going to

• 2:26:04Speaker 01

  have the, the biggest like technological change ever, and how well that goes depends in a large part on like, how well you do at your job on like that given day.

• 2:26:14Rob Wiblin?

  No pressure.

• 2:26:15Speaker 01

  Yeah. And, and the timelines are really fast too, right? Even commercially, you can kind of see that it's like months sort of between, between major releases and that's sort of, that puts a lot of pressure where if you're trying to like, keep up with the frontier of AI progress, it, it is quite difficult and it relies on sort of a success on, on very short timelines.

• 2:26:34Rob Wiblin?

  Yeah, is that, so if there's someone who's, you know, has relevant skills, might be a good employee, but you know, maybe they struggle to operate at like super high productivity, super high energy all the time. Could that be an issue for them at a place like Anthropic where it sounds like, you know, there's a lot of pressure to deliver all the time? Like, I'm not, I guess, potentially internally, but also just the external pressures are pretty substantial.

• 2:26:58Speaker 01

  Yeah, I think that I, some part of me wants to say yes. I think it is, it's really important to be very like high performing a lot of the time. The standard of always do everything perfect all of the time is not something anyone meets. And it's like, I think it is important sometimes to just keep in mind that all you can do is like your best, your best effort. So, uh, you know, we will mess things up and even if it's high stakes and that's like quite unfortunate, like it's unavoidable. Like no, no one is, no one is perfect. So I would, I wouldn't sort of set too high of a like, oh, I couldn't possibly like handle that. Like, I think people really can and you can, you can grow into that and like get, get used to that level of pressure and how to operate under it.

• 2:27:40Rob Wiblin?

  All right. Um, I guess we should, we should wrap up. We've been at this for a couple of hours, but um, I'm curious to know what, what is an AI application that you think is overrated? And maybe going to take longer to arrive than people expect. And uh, maybe what's, what's an application that you think might be underrated? And uh, like consumers might uh, be really getting a lot of value out, out of surprisingly soon.

• 2:28:02Speaker 01

  Overrated. I think on overrated, it's the same. I think people are often like, oh, I should never, uh, like I'll never have to use Google again. Or it's a great way to like get information. And I find that I still like, if I just have a simple question and I want to know the answer, just Google it will give me the answer quickly and it's almost always right. Whereas if I could go ask Claude, but it'll like take us, you know, it'll like sample it out. And then I'll be like, is it true? Is it not true? It's probably true, but like it's in this conversational tone. So I think that's one that like, doesn't yet feel like the strengths. The, the place where I find the most benefit is coding. I think this is not a like, super generalizable case or something. But if you're ever writing software, or if you've thought like, I don't know how to write software, but I wish I did.

• 2:28:44Speaker 01

  The, the models are really quite good at it. Uh, and if you can kind of get yourself set up, you can probably just write something out in English and it will like spit out the code to, to do the thing you need rather quickly. Um, and then the other thing is like problems where I don't.

• 2:28:59Speaker 01

  Like, I don't know what I would search for. Like, I have some question. I want to know the answer, but it relies on a lot of context. It would be this giant query. Like, models are really good at that. You can give them documents. You give them like huge amounts of stuff and explain really precisely what you want. And then they will like interpret it and give you, give you an answer that like accounts for all the information you've, you've given them.

• 2:29:18Rob Wiblin?

  Yeah, I, I think I do use it mostly as a substitute for Google, but not for simple queries. It's more like something kind of complicated where I feel like I'd have to dig into some articles to figure out the answer. I think one that jumps to mind is, um, like Francisco Franco was kind of on the side of the Nazis during World War II, but then he was in power for another 30 years. Like, did he ever comment on that? What did he say about the Nazis later on? And I think Claude was able to give me an accurate answer to that, whereas I probably would have, I could have spent hours, maybe, trying to look into that, trying to, trying to, trying to find something. The answer is he mostly just didn't talk about it.

• 2:29:53Speaker 01

  My other favorite one, which is a super tiny use case, is if I ever have to like format something and do something, like if there's just some giant list of numbers that someone sent me in like a Slack thread and it's bulleted and I want to add them up, I can like just copy paste it into Claude and say add the things up and like any format. It's very good at taking this like weird thing, kind of structuring it and then doing a simple operation.

• 2:30:12Rob Wiblin?

  So I've heard it's really good at, all of these models are really good at programming. Uh, and I've thought, I mean, I've never programmed before really. Uh, and I've thought about, maybe I could use them to, to make something of use. But I guess I'm, I'm at such a basic level, I don't even know like where, so I would get the code and then where would I run it? What, is there a place that I could look this up? Like, uh, yeah.

• 2:30:30Speaker 01

  Yeah, I mean, I think you basically want to just like look up a like, I would suggest Python, like get an introduction to Python and get, get your environment set up. Like you'll eventually run Python and then some, some file, uh, and you'll hit enter and that will run the code. And that part's annoying. I mean, Claude could help you if you run into issues setting it up, but once you have it set up, it will do, you, you can just be like, ah, write me some code to do X and it will just, it will write that pretty X, not perfectly, but pretty accurately.

• 2:30:54Rob Wiblin?

  Yeah, I guess I should just ask Claude for guidance on this as well. Um, do you, so I've got a kid, uh, he's a couple of months old. Um, I guess, you know, uh, in three or four years' time, they'll be going to preschool and then eventually starting, you know, reception primary school. I guess my hope is that by that time, AI models might be really involved in the education process and kids will be able to get a lot more kind of one at work. Maybe it would be very difficult to keep a five-year-old focused on the task of talking to an LLM, but uh, I would think that we're close to being able to have a lot more like individualized attention from educators, even if those educators are AI models. And this might enable kids to learn a lot faster than they can when there's kind of only one teacher split between 20 students or something like that.

• 2:31:35Rob Wiblin?

  Do you think that kind of stuff will come in time for uh, my kid first going to, going to school or might it take a bit longer than that?

• 2:31:42Speaker 01

  I can't be sure, but yeah, I think there will be some, some pretty major changes by the time your, your kid is going to school.

• 2:31:48Rob Wiblin?

  Okay, yeah, that's good. That's uh, that's one that I really don't want to miss on the timelines. Uh, we can, like, I'm worried, like, uh, like, um, Nathan Labenz, I'm worried about hyperscaling, but on a lot of these applications, I really just want them to, uh, to, to reach us as soon as possible because they, they do seem so useful. My guest today has been Nick Joseph. Thanks so much for coming on the 80,000 Hours Podcast, Nick. Thank you.

• 2:32:12Rob Wiblin?

  If you're really interested in the pretty vexed question of whether, all things considered, it's good or bad to work at the top AI companies if you want to make the transition to superhuman AI go well, our researcher Adam Kayla has just published a new article on exactly that titled, Should You Work at a Frontier AI Company? You can find that by Googling 80,000 Hours and Should You Work at a Frontier AI Company? Or heading to our website, 80000hours.org and just looking through our research. And finally, before we go, just a reminder that we are hiring for two new senior roles at 80,000 Hours, a head of video and a head of marketing. And you can learn more about both of those at 80000hours.org/latest. Those roles would probably be done in our offices in central London, but we are open to exceptional remote candidates in some cases. And alternatively, if you're not in the UK but you would like to be, uh, we can also support UK visa applications.

• 2:32:57Rob Wiblin?

  The salaries for these two roles would vary depending on seniority, but someone with five years of relevant experience would be paid approximately 80,000 pounds, something like that. The first of these two roles, the head of video, would be someone in charge of setting up a whole new video product for 80,000 hours. Obviously, people are spending a larger and larger fraction of their time online watching videos on video-specific platforms. And we want to explain our ideas there in a compelling way that can reach the sorts of people who care about them. That video program could take a range of forms, including 15-minute direct-to-camera vlogs, many, lots and lots of one-minute videos, maybe 10-minute explainers, maybe, that's probably my favorite YouTube format, or alternatively, lengthy video essays. Some people really like those. The best format would, that would be something for this new head of video to look into and figure out for us.

• 2:33:42Rob Wiblin?

  We're also looking for a new head of marketing to lead our marketing efforts to reach our target audience at a large scale. They're going to be setting and executing on a strategy, managing and building a team, and ultimately deploying our yearly marketing budget of around $3 million. We currently run sponsorships on major podcasts and YouTube channels. You might, hopefully you've seen some of them. We also do targeted ads on a range of social media platforms. And collectively, that's gotten hundreds of thousands of new people onto our email newsletter. We also mail out a copy of one of our books about high-impact career choice every eight minutes. That's what I'm told. So there's certainly the potential to reach many people if you're doing that job well. Applications will close in late August, so please don't delay if you'd like to apply for those ones. 80000hours.org/latest. All right. The 80,000 Hours podcast is produced and edited by Karen Harris,

• 2:34:26Rob Wiblin?

  audio engineering by Ben Cordell, Myla McGuire, Simon Monsua, and Dominic Armstrong. Full transcripts and extensive collection of links to learn VORO are available on our site and put together, as always, by the legend herself, Katie Moore. Thanks for joining. Talk to you again soon.

• 2:34:42Nathan Labenz?

  It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please, don't hesitate to reach out via email at [email protected], or you can DM me on the social media platform of your choice.