Dario Amodei (Anthropic CEO) — The hidden pattern behind every AI breakthrough

Here is my conversation with Dario Amodei, CEO of Anthropic. Dario is hilarious and has fascinating takes on what these models are doing, why they scale so well, and what it will take to align them. Watch on YouTube https://youtu.be/Nlkk3glap_U. Listen on Apple Podcasts https://apple.co/3rZOzPA, Spotify https://spoti.fi/3QwMXXU, or any other podcast platform. Read the full transcript here https://www.dwarkeshpatel.com/dario-amodei. Follow me on Twitter https://twitter.com/dwarkesh_sp for updates on future episodes. Timestamps (00:00:00) - Introduction (00:01:00) - Scaling (00:15:46) - Language (00:22:58) - Economic Usefulness (00:38:05) - Bioterrorism (00:43:35) - Cybersecurity (00:47:19) - Alignment & mechanistic interpretability (00:57:43) - Does alignment research require scale? (01:05:30) - Misuse vs misalignment (01:09:06) - What if AI goes well? (01:11:05) - China (01:15:11) - How to think about alignment (01:31:31) - Is modern security good enough? (01:36:09) - Inefficiencies in training (01:45:53) - Anthropic’s Long Term Benefit Trust (01:51:18) - Is Claude conscious? (01:56:14) - Keeping a low profile This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.dwarkesh.com https://www.dwarkesh.com?utm_medium=podcast&utm_campaign=CTA_1

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  A generally well-educated human, that could happen in, you know, two or three years.

• 0:05Dwarkesh Patel?

  What does that imply for Anthropic when in two to three years, these leviathans are doing like $10 billion training runs?

• 0:11Speaker 00

  The models, they just want to learn. And it was a bit like a Zen Cohen. I listened to this and I became enlightened.

• 0:18Speaker 00

  The compute doesn't flow, like the spice doesn't flow. It's like, you can't like, like the blob has to be unencumbered, right? The big acceleration that happened late last year and beginning of this year, we didn't cause that. And honestly, I think if you look at the reaction to Google, that that might be 10 times more important than than anything else. There was a running joke, the way building AGI would look like is, you know, there would be a data center next to a nuclear power plant, next to a bunker.

• 0:44Dwarkesh Patel?

  But now it's 2030. What happens next? What what are we doing with a superhuman god? Okay, today I have the pleasure of speaking with Dario Amodei, who is the CEO of Anthropic. And I'm really excited about this one. Dario, thank you so much for coming on the podcast.

• 0:59Speaker 00

  Thanks for having me.

• 1:00Dwarkesh Patel?

  First question, you have been one of the very few people who has seen scaling coming for years, more than five years. I don't know how long it's been, but as somebody who's seen it coming, what is fundamentally the explanation for why scaling works? Why is the universe organized such that if you throw big blobs of compute at a wide enough distribution of data, the thing becomes intelligent?

• 1:21Speaker 00

  I think the truth is that we still don't know. I think it's almost entirely an empirical fact. You know, I think it's a fact that you could kind of sense from the data and from a bunch of different places. But I think we don't still have a satisfying explanation for it. If I were to try to make one, but I'm just, I don't know, I'm just kind of waving my hands when I say this, you know, there, there's this, there's these ideas in physics around like long tail or power law of like correlations or effects. And so like when a bunch of stuff happens, right? When you have a bunch of like features, you get a lot of the data in like kind of the early, you know, the fat part of the distribution before the tails.

• 2:02Speaker 00

  You know, for language, this would be things like, oh, I figured out there are parts of speech and nouns follow verbs. And then there are these more and more and more and more. Correlations. And so it it kind of makes sense why there would be this, you know, every log or order of magnitude that you add, you kind of capture more of the distribution. What I what's not clear at all is why is it scaled so smoothly with parameters? Why does it scale so smoothly with the amount of data? Why are you you can think up some explanations of why it's linear, like the parameters are like a bucket. And so the data is like water and so size of the bucket is proportional to the size of the water. But like, why does it lead to all these this very smooth scaling?

• 2:47Speaker 00

  I think we still don't know. There's all these explanations. Our chief scientist, Jared Kaplan, did some stuff on like fractal manifold dimension that like you can use to explain it. So there's there's all kinds of ideas, but I feel like we just don't really know for sure.

• 3:02Dwarkesh Patel?

  And by the way, for the audience who is trying to follow along, by scaling, we're referring to the fact that you can very predictably see how you go from GPT three to GPT four, or in this case, Claude one to Claude two, that the loss in terms of whether it can predict the next token scales very smoothly. So, OK, we don't know why it's happening, but can you at least predict if empirically, here is the loss at which this ability will emerge. Here is the place where this circuit will emerge. Is that at all predictable? Or are you just looking at the loss number?

• 3:30Speaker 00

  It is much less predictable. What's predictable is this statistical average, this loss, this entropy. It's super predictable. It's like, you know, predictable to like, sometimes even to several significant figures, which you don't see outside of physics, right? You don't expect to see it in this messy empirical field. But actually, specific abilities are very hard to predict. So, you know, back when I was working on GPT two and GPT three, like, when does arithmetic come in place? When do models learn to code? Sometimes it's very, it's very abrupt. You know, it's kind of like you can predict statistical averages of the weather, but the weather on one particular day is very, you know, very, very hard to predict.

• 4:07Dwarkesh Patel?

  So dumb it down for me, I don't understand manifolds, but mechanistically, it doesn't know addition yet, now it knows addition. What has happened?

• 4:15Speaker 00

  This is another question that we don't know the answer to. I mean, we're trying to answer this with things like mechanistic interpretability. But, you know, I'm not sure. I mean, you can think about these things about like circuits snapping into place. Although, there is some evidence that when you look at the models being able to add things, that, you know, like, if you look at its chance of getting the right answer, that shoots up all of a sudden. But if you look at, okay, what's the probability of the right answer? You'll see it climb from like 1 in a million to 1 in 100,000 to 1 in 1,000, long before it actually gets the right answer. And so, there's some, in many of these cases at least, I don't know if in all of them, there's some continuous process going on behind the scenes. I don't understand it at all.

• 4:57Dwarkesh Patel?

  Does that imply that the circuit or the process for doing addition was pre-existing? And it just got increased in salience?

• 5:04Speaker 00

  I don't know if like there's this circuit that's weak and getting stronger. I don't know if it's something that works but not very well, like, I think we don't know, and these are some of the questions we're trying to answer with mechanistic interpretability.

• 5:16Dwarkesh Patel?

  Are there abilities that won't emerge with scale?

• 5:19Speaker 00

  So I definitely think that, again, like, things like alignment and values are not guaranteed to emerge with scale, right? It's kind of like, you know, one way to think about it is you train the model, and it is, basically, it's like predicting the world, it's understanding the world. Its job is facts, not values, right? It's trying to predict what comes next. But there's, there's just, there's free variables here where it's like, what should you do? What should you think? What should you value? Those, you know, like, they're just, there are the bits for that. There's just like, well, if I started with this, I should finish with this. If I started with this other thing, I should finish with this other thing. And so I think that's not going to emerge.

• 6:00Dwarkesh Patel?

  I want to talk about a lemon in a second, but on scaling, if it turns out that scaling plateaus before we reach human-level intelligence, looking back on it, what would be your explanation? What do you think is likely to be the case if that turns out to be the outcome?

• 6:12Speaker 00

  Yeah, so I guess I would distinguish some problem with the fundamental theory with Some practical issue. So one practical issue we could have is we could run out of data. For various reasons, I think that's not going to happen. But you know, if you look at it very, very naively, we're not that far from running out of data. And so it's like, we just don't have the data to continue the, to continue the scaling curves. I think, you know, another way it could happen is like, oh, we just, we just use up our, all of our compute that was available and that, that wasn't enough. And then progress is slow after that. I wouldn't bet on either of those things happening, but they, they could. I, I think from a, from a fundamental perspective, I personally, I think it's very unlikely that the scaling laws will just stop.

• 6:54Speaker 00

  If they do, another reason, again, this isn't fully fundamental, could just be, we don't have quite the right architecture. Like if we tried to do it with an L S D M or an R N N, the slope would be different. I still, might be that we get there, but I think there are some things that are just very hard to represent when you don't have this ability to attend far in the past that transformers have. If somehow, and I don't know how we would know this, it kind of wasn't about the architecture and we just hit a wall. I think I'd be very surprised by that. I think we're already at the point where the things the models can't do, don't seem to me to be different in kind from the things they can do. Um, and it just, you know, you could have made a case a few years ago that it was like,

• 7:38Speaker 00

  they can't reason, they can't program, like you could have, you could have drawn boundaries and said, well, maybe you'll hit a wall. I didn't think that. I didn't think we would hit a wall. Few other people didn't think we would hit a wall, but it was a more plausible case then. I think it's a less plausible case now. Now, it, it could happen, like this stuff is crazy, like it could, it could, it could happen tomorrow that it's just like, we hit a wall. I think if that happens, I'm trying to think of like, what's my, what would really be my, it's unlikely, but what would really be my explanation? I think my explanation would be. There's something wrong with the loss when you train on next word prediction. Like, some of the remaining reasoning abilities or something like that. Like, if you really want to learn, you know, it's a program at a really high level.

• 8:22Speaker 00

  Like, it means you care about some tokens much more than others. And they're rare enough that it's like the loss function over focuses on kind of the appearance, the things that are responsible for the most bits of entropy. And instead, you know, they don't focus on this stuff that's really essential. And so you could kind of have the signal drowned out in the noise. I don't think it's going to play out that way for a number of reasons. But if you told me, yup, you trained your 2024 model, it was much bigger and it just wasn't any better, and you tried every architecture and it didn't work. I think that's the explanation I would reach for.

• 8:58Dwarkesh Patel?

  Is there a candidate for another loss function if you had to abandon next token prediction?

• 9:02Speaker 00

  I think then you would have to go for some kind of RL. And again, there's many different kinds. There's RL for mutual feedback, there's RL against an objective, there's things like constitutional AI, there's things like amplification and debate, right? These are kind of both alignment methods and ways of training models. You would have to try a bunch of things, but the focus would have to be on what do we actually care about the model doing, right? In a sense, we're a little bit lucky that it's like predict the next word gets us all these other things we need.

• 9:30Dwarkesh Patel?

  Right.

• 9:31Speaker 00

  There's no guarantee.

• 9:32Dwarkesh Patel?

  It seems like from your worldview, there's a multitude of different loss functions that it's just a matter of what can allow you to just throw a whole bunch of data at it, like the next token prediction itself is not significant.

• 9:42Speaker 00

  Yeah, well, I mean, I guess the thing with RL is you get slowed down a bit because it's like, you know, you have to, by some method, kind of, you know, design how the loss function works. Nice thing with the next token prediction is it's there for you, right? It's just there, it's the easiest thing in the world. And so I think it would slow you down if you couldn't scale in just that very simplest way.

• 10:02Dwarkesh Patel?

  You mentioned that the data is likely not to be the constraint. Why do you think that is the case?

• 10:07Speaker 00

  There's various possibilities here, and you know, for a number of reasons, I shouldn't go into the details, but you know, like, there's many sources of data in the world, and there's many ways that you can also generate data. My guess is that This will not be a blocker. Maybe it would be better if it was, but it won't be.

• 10:23Dwarkesh Patel?

  Are you talking multimodal or?

• 10:25Speaker 00

  There's just many different ways to do it.

• 10:27Dwarkesh Patel?

  Um, how did you form your views on scaling? How far back can we go? And then you would be basically saying something similar to this.

• 10:34Speaker 00

  This view that I have probably formed gradually from, I would say like 2014 to 2017. So I, I think my first experience with it was my first experience with AI. Um, so I, you know, I saw some of the early stuff around Alex Ned in 2012. Always kind of had wanted to study intelligence, but I, you know, before I was just like, this isn't really working. Like it doesn't, doesn't seem like it's actually working. Um, you know, all, all the way back to like, you know, 2005, I'd like, you know, I'd read Ray Kertzwell's work, you know, I'd read even, even some of like Eliezer's work on the early, on the early internet back then. And I was like, ah, this, this stuff kind of looks far away. Like I look at the AI stuff of today and it's like not, not anywhere, not anywhere close. But with Alex and I was like, oh, this is actually stuff is actually starting to work. So I joined Andrew Ng's group.

• 11:20Speaker 00

  Um, initially at, at Baidu and the first task, you know, that I got set to do, right? It was my, you know, I'd been in a different field. And so I, I first joined, you know, this was my first experience with AI. And it was a bit different from a lot of the kind of academic style research that was going on kind of elsewhere in the world, right? The, I, I, I think I kind of got lucky in that the task that was given to me and the other folks there was just make the best speech recognition. Uh, system that you can. And there was a lot of data available. There were a lot of GPUs available. So it kind of, it, it posed the problem in a way that was amenable to discovering that kind of scaling was a solution, right?

• 12:01Speaker 00

  That's very different from like, you're a postdoc and it's your job to come up with, you know, what's the, what's the best like, you know, what's, what's an idea that seems clever and new and makes your mark as someone who's invented something? And, and so I just quickly discovered that like, you know, I was just, just tried the simplest experiments. I was like, you know, just fiddling with some dials. I was like, okay, try, um, you know, try, try adding more layers to the, literally add more layers to the RNN. Um, you know, try training it for longer. What happens? How long does it take to overfit? What if I add new data and repeat it less times? And like, I just saw these like very consistent patterns. I didn't really. Know this was unusual or that others weren't thinking in this way.

• 12:45Speaker 00

  This, this was just kind of like, almost like beginner's luck. It was my first experience with it. And I didn't really think about it beyond speech recognition, right? Right, right, you know, I was, I was just kind of like, oh, this is, you know, I don't, I don't know anything about this field. There are zillions of things people do with machine learning, but like, I'm like, weird. This seems to be true in the speech recognition field. Um, and, and, and then I, I think it was recently, you know, like, um, just before OpenAI started, um, that I met Ilya, who you, who, who you interviewed. One of the first things he said to me was, look, the models, they just want to learn. You have to understand this, the models, they just want to learn. And it was a bit like a Zen Cohen, like I kind of like, I listened to this and, and I became enlightened.

• 13:27Speaker 00

  And, uh, you know, over, over the, over the years after this, you know, you know, again, I, I would be kind of, you know, the one who would formalize a lot of these things and kind of put them together, but like, just kind of the, the, what that told me is that, that phenomenon that I'd seen wasn't just some random thing that I'd seen. It was like, it was broad, it was, it was more general, right? The models, the models just want to learn. You get the obstacles out of their way, right? You give them, you give them good data, you, you give them enough space to operate in. You don't do something stupid like condition them badly numerically, um, and, and they want to learn, they'll do it, they'll do it.

• 14:05Dwarkesh Patel?

  You know what, what I found really interesting about what you said is there are many people who were aware back at that time, probably weren't working on it directly. But we're aware that these things are really good at speech recognition or at playing these constrained games. Very few extrapolated from there like you and Ilya did to something that is generally intelligent. What, what was different about the way you were thinking about it versus how others think that you went from like, is getting better at speech in this consistent way? It will get better at everything in this consistent way.

• 14:35Speaker 00

  Yeah, so I, I genuinely don't know. I mean, at first, when I saw it for speech, I assumed this was just true for speech or for this narrow class of models. I, I think it was just over the period between 2014 and 2017, I tried it for a lot of things and saw the same thing over and over again. I watched the same being true with. Dota, I watched the same being true with robotics, which many people thought of as a counter example, but I just thought, well, it's hard to get data for robotics. But if we operate within, if we look within the data that we have, we see the same patterns. And so I don't, I don't know. I think people were very focused on solving the problem in front of them. Why one person thinks one way, another person thinks it's very, it's very hard to explain.

• 15:20Speaker 00

  I think people just see it through a different lens, you know, are looking like vertically instead of horizontally. They're not thinking about the scaling. They're thinking about how do I solve my problem? Well, for robotics, there's not enough data. And so, you know, and, and so, you know, that can easily abstract to, well, scaling doesn't work because we don't have the data. And, and so I don't, I, I, I don't know. I just, for some reason, and it may just, it may just have been random chance, was obsessed with that particular direction.

• 15:47Dwarkesh Patel?

  When did it become obvious to you that language is the means to just feed a bunch of data into these things? That, or was it just you ran out of other things? Like robotics, there's not enough data. This other thing, there's not enough data.

• 15:58Speaker 00

  Yeah, I mean, I think this whole idea of like the next word prediction, that you could do self-supervised learning, you know, that together with the idea that it's like, wow, for predicting the next word, there's so much richness and structure there, right? You know, it might say two plus two equals, and you have to know the answer is four. And, you know, it might be telling the story about a character. And then basically it's, it's posing to the model, you know, the, the equivalent of these developmental tests that get posed to children. You know, Mary walks into the room and, you know, puts an item in there. And then, you know, Chuck walks into the room and removes the item, and Mary, Mary doesn't see it. What does Mary think hap, you know, so like, so the models are going to have to, to get this right in the service of predicting the next word, they're going to have to solve, you know, solve all these theory of mind problems, solve all these math problems.

• 16:43Speaker 00

  And so I, you know, I, my thinking was just, well, you know, you scale it up as much as you can. You, you, you, you know, there's, there's kind of no limit to it. And I think I kind of had abstractly that view. But the thing of course that like really. Solidified and convinced me was the work that Alec Radford did on GPT-1. Which was not only could you get this this language model that could predict things very well, but also you could fine tune it. You needed to fine tune it in those days to do all these other tasks. And so I was like, wow, you know, it this isn't just some narrow thing where you get the language model right. It's sort of halfway to everywhere, right? It's like, you know, you get the language model right, and then with a little move in this direction,

• 17:24Speaker 00

  it can, you know, it can solve this this, you know, logical dereference test or whatever. And, you know, with this this other thing, you know, it can it can solve translation or something. And then you're like, wow, I think there's there's really something to do it. And and of course, we can we can really scale it.

• 17:38Dwarkesh Patel?

  Well, one thing that's confusing or that would have been hard to see. If you told me in 2018, we'll have models in 2023 like Claude II that can write theorems in the style of Shakespeare, whatever theory you want you want. They can a standardized test with open ended questions, you know, just all kinds of really impressive things. You would have said at that time, I would have said, oh, you have AGI. You clearly have something that is a human level intelligence. Where these while these things are impressive, it clearly seems we're not at human level, at least in the current generation and potentially for generations to come. What explains this discrepancy between super impressive performance in these benchmarks? And in just like the things you could describe versus, yeah, generally.

• 18:22Speaker 00

  So that that was one area where actually I was not pressing and I was surprised as well.

• 18:25Dwarkesh Patel?

  Yeah.

• 18:26Speaker 00

  So when I first looked at GPT-3 and, you know, more so the kind of things that we built in the early days at at Anthropic. My my general sense was I, you know, I looked at these and I'm like, it seems like they they really grasp the essence of language. I'm not sure how much we need to scale them up. Like, maybe we maybe what's what's more needed from here is like RL and all and kind of and kind of all the other stuff. Like, we might be kind of near the, you know, I thought in 2020, like, we can scale this a bunch more. But I wonder if it's more efficient to scale it more or to start adding on these other objectives like like RL. I thought maybe if you do as much RL as, you know, as you've done pre.

• 19:06Speaker 00

  Training for a, for a, you know, 2020 style model that that's, that's the way to go and scaling it up will keep working. But you know, is that, is that really the best path? And I, I think it, I don't know, it just keeps going like I thought it had understood a lot of the essence of language. But then, you know, there's, there's kind of, there's kind of further to go. And, and so, I don't know, stepping back from it, like, one of the reasons why I'm sort of very empiricist about, about AI, about safety, about organizations, is that you often get surprised, right? I, you know, I feel like I've been right about some things, but I've still, you know, with these theoretical pictures had been wrong about most things.

• 19:51Speaker 00

  Being right about 10% of the stuff is, you know, sets you head and shoulders above, above, above, above many people. You know, if you look back to, I can't remember who it was, kind of, you know, made these diagrams that are like, you know, here's, here's the village idiot, here's Einstein, here's the scale of intelligence, right? And the village idiot and Einstein are like very close to each other. Like that, maybe that's still true in some abstract sense or something, but it's, it's not really what we're seeing, is it? We're seeing like, that it seems like the human range is pretty broad and doesn't, we don't hit the human range in the same place or at the same time for different tasks, right? Like, you know, like write, write a sonnet, you know, in the style of Cormac McCarthy or something.

• 20:36Speaker 00

  Like, I don't know, I'm not very creative, so I couldn't do that. But like, you know, that's, that's a pretty high level human skill, right? Um, and even the model is starting to get good at stuff of, you know, like constrained writing, you know, there's this like write a, you know, write a page without using the letter E or something, write a page about X without using the letter E. Like, I think the models might be like superhuman or close to superhuman at that. But when it comes to, you know, Yeah, I don't know, prove relatively simple mathematical theorems like they're they're just starting to do the beginning of it. They make really dumb mistakes sometimes, and they they really lack any kind of broad like, you know, correcting your errors or

• 21:18Speaker 00

  doing some extended task. And so I don't know, it turns out that intelligence isn't isn't a spectrum. There are a bunch of different areas of domain expertise. There are a bunch of different like kinds of skills, like memory is different. I mean, it's all it's all formed in the blob. It's not it's all formed in the blob. It's not complicated, but to the extent it even is on the spectrum, the spectrum is also wide. If you asked me 10 years ago, that's not what I would have expected at all. But I think that's very much the way it's turned out.

• 21:47Dwarkesh Patel?

  Oh, man, I've seen many questions just as follow up on that. One is, do you expect that, given the distribution of training that these models get from massive amounts of internet data versus what humans got from evolution, that the repertoire of skills that elicits will be just barely overlapping? It will be like concentric circles? How do you think about? Do those matter?

• 22:12Speaker 00

  Clearly, there's a large there's certainly a large amount of overlap, right? Because a lot of the thing, you know, like, these models have have business applications, and many of their business applications are doing things that, you know, are helping humans to be more effective at things. So the overlap is quite is quite large. And, you know, if you think of all the activity that humans put on the internet in tax, that covers a lot of it. But it probably doesn't cover some things like the models. I think they do learn a physical model of the world to some extent, but they certainly don't learn how to actually move around in the world. Again, maybe that's easy to fine tune. But I, you know, I think so. I think there are some things that the models don't learn that humans do. And then I think You know, the models learn, for example, to speak fluent base 64. I don't know about you,

• 22:56Speaker 00

  but I never learned that.

• 22:58Dwarkesh Patel?

  Right. How likely do you think it is that these models will be superhuman for many years at economically valuable tasks while they are still below humans in many other relevant tasks that prevents like an intelligence explosion or something?

• 23:13Speaker 00

  I think this kind of stuff is like really hard to know. So I'll give, I'll give that caveat that like, you know, again, like the basic scaling laws, you can kind of predict. And then like this more granular stuff, which we really want to know to know how this all is going to go is much harder to know. But my guess would be the scaling laws are going to continue, you know, again, subject to, you know, do people slow down for safety or for regulatory reasons. But, you know, let's just, let's just put all that aside and say like, we have the economic capability to keep scaling. If we did that, what would happen? And I think my view is we're going to keep getting better across the board. And I don't see any area where the models are like super, super weak or not starting to

• 23:56Speaker 00

  make progress. Like that used to be true of like math and programming, but I think over the last six months, you know, the, the 2023 generation of models compared to the 2022 generation had started to learn that. There may be more subtle things we don't know. And so I kind of suspect, even if it isn't quite even, that the rising tide will lift all the boats.

• 24:17Dwarkesh Patel?

  Does that include the thing you were mentioning earlier, where if there's an extended task, it kind of loses its train of thought, or its ability to just like execute a series of steps?

• 24:27Speaker 00

  So I think that, that, that's going to depend on things like RL training to have the model do longer horizon tasks. I don't expect that to require a substantial amount of additional compute. I think that, that, that was probably an artifact of, yeah, kind of thinking about RL in the wrong way and underestimating how much the model had learned on its own. In terms of, you know, are we going to be superhuman in some areas and not others? I think it's complicated. I could imagine that we won't be superhuman in some areas because, for example, they involve Like embodiment in the physical world. And then it's like, what happens like to the AIs? Help us train faster AIs and those faster AIs wrap around and solve that.

• 25:12Speaker 00

  Do you not need the physical world? It depends what you mean. Are we worried about an alignment disaster? Are we worried about misuse, like making weapons of mass destruction? Are we worried about the AI? You know, the AI taking over research from humans? Are we worried about it reaching some threshold of economic productivity? Where it can do what the average? These different thresholds, I think, have have different answers. Although I suspect they will all come within a few years.

• 25:39Dwarkesh Patel?

  Let me ask about those thresholds. So if Claude was an employee at Anthropic, what salary would it be worth? What is it like meaningfully speeding up AI progress?

• 25:47Speaker 00

  It feels to me like an intern in most areas. But then some specific areas where it's better than that. Again, I think one thing that's makes the comparison hard is like the form factor is kind of like not the same as a human, right? Like a human, like, you know, if you were to behave like one of these chatbots, like we wouldn't really. I mean, I guess we could have this conversation. It's like, but you know, they're they're not really. They're more designed to answer single or a few questions, right? And like, you know, they don't have a the concept of having a long life of prior experience. Right? We're talking here about, you know, things that that I've experienced in the past, right? And chatbots don't don't have that. And so there's there's all kinds of stuff missing. And so it's hard to make a comparison, but I don't know.

• 26:33Speaker 00

  They feel like interns in some areas and kind of then they have areas where they spike and are really savants. Where they may be better than they may be better than anyone here.

• 26:43Dwarkesh Patel?

  But does the overall picture of something like an intelligence explosion? You know, my my former guest is Carl Schulman and he has this like very detailed model of an intelligence. Does that as somebody who would actually like see that happening? Does that make sense to you as they go from interns to entry level software engineers? Those entry level software engineers increase your productivity?

• 27:01Speaker 00

  I think I think the idea that the the AI systems become more productive. And first they speed up the productivity of humans. Then they, you know, kind of equal the productivity of humans. And and and, you know, and then they're in some meaningful sense, the main contributor to scientific progress that that happens at some point. I I think. That that basic logic seems likely to me, although I have a suspicion that when we actually go into the details, it's going to be kind of like weird and different than we expect, that all the detailed models are kind of, you know, we're thinking about the wrong things, or we're right about one thing and then are wrong about 10 other things.

• 27:44Speaker 00

  And so I don't know, I think we might end up in like a weirder world than we expect.

• 27:50Dwarkesh Patel?

  Hmm. What do you add all this together? Like your estimate of when we get something kind of human level?

• 27:56Speaker 00

  Yeah.

• 27:57Dwarkesh Patel?

  What does that look like?

• 27:57Speaker 00

  I mean, again, it depends on the thresholds.

• 27:59Dwarkesh Patel?

  Yeah.

• 28:00Speaker 00

  Um, you know, in terms of someone looks at these, the model, and you know, even if you talk to it for, you know, for an hour or so, it's it's basically, you know, it's basically like a generally well-educated human.

• 28:16Dwarkesh Patel?

  Yeah.

• 28:16Speaker 00

  Um, that could be not very far away at all, I think. Um, like that, that could happen in, you know, two or three years, like, uh, you know, if I look at, again, like, I think the main thing that would stop it would be if if we hit certain, certain, you know, and we have internal tests for, you know, safety thresholds and stuff like that. So if a company or the industry decides to slow down, or, you know, we're able to get the government institute restrictions that kind of, uh, you know, that that moderate the rate of progress for safety reasons, that would be the main reason it wouldn't happen. But if you if you just look at the logistical and economic ability to scale, I don't think we're very far at all from that. Now, that that may not be the threshold where the models are existentially dangerous.

• 29:02Speaker 00

  Uh, in fact, I suspect it's not not quite there yet. It may not be the threshold where the models can take over most AI research. It may not be the threshold where the models, you know, seriously change how the economy works. Um, I think it gets a little murky after that, and all of those thresholds may happen at various times after that. But I think I think in terms of the base technical capability of it, it kind of, it kind of sounds like a reasonably generally educated human.

• 29:29Dwarkesh Patel?

  Yeah.

• 29:30Speaker 00

  Across the board, I think that could be quite close.

• 29:32Dwarkesh Patel?

  Why would it be the case that it could? Be sound, you know, pass the Turing test for an educated person, but not be able to contribute or substitute for human involvement in the economy.

• 29:43Speaker 00

  A couple of reasons. One is just, you know, that the threshold of skill isn't high enough, right? Comparative advantage. It's like, it like, doesn't matter that, you know, I have someone who's better than the average human at every task. Like, what I really need is like, for AI research, like, you know, I need what, you know, I need to basically find something that is is strong enough to substantially accelerate, you know, the like labor of the thousand experts who are best at it. And so we might reach a point where we, you know, the comparative advantage of these systems is not, is not great. Another thing that could be the case is that I think there are these kind of mysterious frictions that like, you know, kind of don't show up in naive economic models,

• 30:28Speaker 00

  but you see it whenever you're like, you know, when you go to a customer or something and you're like, hey, I have this cool chatbot. In principle, it can do everything that, you know, your customer service bot does or that this part of your company does, but like, the the actual friction of like, how do we slot it in? How do we make it work? That that includes both kind of like, you know, just the question of how it works in a human sense within the company, like, you know, how how how things happen in the economy and overcome frictions? And and also just like, what is the workflow? How do you actually interact with it? It's very different to say, here's a chatbot that kind of looks like it's doing this task that, you or, you know, or helping the human to do to do some task.

• 31:12Speaker 00

  As it as it is to say like, okay, this thing is, this thing is deployed and 100,000 people are using it. Often, like right now, lots of folks are rushing to deploy these systems, but I I think in many cases, they're not using them in anywhere close to the most efficient way that they could. You know, not because they're not smart, but because it takes time to work these things out. And so I think when things are changing this fast, there are going to be all of these frictions.

• 31:37Dwarkesh Patel?

  Yeah.

• 31:37Speaker 00

  And I, and I think, again, these are messy reality that doesn't quite get captured in the model. I don't think it changes the basic picture. Like, I don't think it changes the idea that we're, we're building up this snowball of like, the models help the models get better and, you know, do what the humans and, and, you know, can, can accelerate what the humans do. And eventually it's mostly the models doing the work. Like, you zoom out far enough, that's happening. But I'm, I'm kind of skeptical of kind of any kind of precise mathematical or exponential prediction of how it's going to be. I think it's, I think it's, I think it's all going to be a mess. But I think what we know is it's on an ex- metaphorical exponential and it's going to happen fast.

• 32:15Dwarkesh Patel?

  Hmm. How do those different exponentials net out, which we've been talking about? So one was the, the scaling laws themselves are power laws with decaying marginal,

• 32:25Speaker 00

  Yes.

• 32:26Dwarkesh Patel?

  uh, you know, uh, loss of per, you know, parameter or something. The other exponential you talked about is, well, these things can get involved in the process of AI research itself, speeding it up. So those two are sort of opposing exponentials. Does it net out to be super linear or sublinear? And also you mentioned, well, the, the distribution of intelligence might just be broader. So should we expect after the, after we get to this point in two to three years, it's like, vroom, vroom? Like, what does that look like?

• 32:52Speaker 00

  It's, I mean, I think it's very unclear, right? So we're already at the point where if you look at the loss, the scaling laws are starting to bend. I mean, we've seen that in, you know, published model cards offered by multiple companies. Um, so that's not a secret at all. But as, as they start to bend, each little bit of, of entropy, right? Of accurate prediction becomes more important, right? Maybe these last little bits, bits of entropy are like, well, you know, this is a physics paper as Einstein would have written it as opposed to, you know, as some other physicist would have, would have, would have written it. And so it's, it's hard to assess significance from this. It certainly looks like, in terms of practical performance, the metrics keep going up relatively linearly,

• 33:34Speaker 00

  although they're always unpredictable. Uh, so, so it's, it's hard to see that. And then... I mean, the thing that I think is driving the most acceleration is just more and more money is going into the field. Like, people are seeing that there's just a huge amount of, you know, of economic value. And so I expect the price, the amount of money spent on the largest models to go up by like a factor of 100 or something. And for that then to be concatenated with the chips are getting faster, the algorithms are getting better, because there's, there's so many people working on this now. And so, and so again, I mean, you know, I'm not making a normative statement here, this is what should happen. I'm not even saying this necessarily will happen, because I think there's important

• 34:16Speaker 00

  safety and government questions here, which we're very actively working on. I'm just, I'm just saying, like, left to itself, this is what the economy is going to do.

• 34:23Dwarkesh Patel?

  We'll get to those questions in a second, but how do you think about the contribution of Anthropic to that increasing in the scope of this industry? Where, I mean, there's an argument we make that, listen, with that investment, we can work on safety stuff at Anthropic, another that's as you're raising the salience of this field in general.

• 34:42Speaker 00

  Yeah, I mean, it's all, it's all costs and benefits, right? The costs are not zero, right? So I think a mature way to think about these things is, you know, not, not to deny that there are any costs, but to think about what the costs are and what the benefits are. You know, I think, I think we've been relatively responsible in the sense that, you know, the big acceleration that, that happened late last year and, and beginning of this year, like, we didn't cause that. We weren't, we weren't the ones who did that. And honestly, I think if you look at the reaction to Google, that, that might be 10 times more important than, than anything else. And then kind of, once it had happened, once the ecosystem had changed, then we did a lot of things to kind of, to kind of stay on the frontier. And, and, and so, I don't know, it's, it's, I mean, it's like any other question, right? It's like, you're trying to, you're trying to do the things that have the biggest costs

• 35:27Speaker 00

  and that have the lowest costs and the biggest benefits. And, you know, that, that causes you to have different strategies at different times.

• 35:34Dwarkesh Patel?

  One question I had for you while we were talking about the intelligence stuff was, listen, as a scientist yourself, is it, what do you make of the fact that these things have basically the entire corpus of human knowledge memorized? And as far as I'm aware, they haven't been able to make like a single new connection that has led to a discovery. Whereas if even a moderately intelligent person had this much stuff memorized, they'd notice, oh, this thing causes this symptom, this other thing also causes this symptom, you know, there's a medical cure right here, right? Well, shouldn't we be expecting that kind of stuff?

• 36:03Speaker 00

  I'm not, I'm not sure. I mean, I think. You know, I don't know, these words discovery, creativity, like it's one of the lessons I've learned is that in, you know, in kind of the big blob of compute, often these, these ideas often end up being kind of fuzzy and elusive and hard to track down. But I think, I think there is something here, which is, I think the models do display a kind of ordinary creativity. Again, again, you know, the kind of like, you know, write a, write a sonnet, you know, in the style of Cormac McCarthy or Barbie or so, you know, like, there is some creativity to that. And I think they do draw, you know, new connections of the kind that an ordinary person would draw. I, I agree with you that there haven't been any kind of like, I don't know, like, I would say like big scientific discoveries.

• 36:48Speaker 00

  I think that's a mix of like, just the model's skill level is not, is not high enough yet, right? Like I was on a podcast last week where, where the host said, I don't know, I played with these models, they're kind of mid, right? Like they get, you know, they get a B or a B minus or something. And, and that, that I think is going to change with the, with the scaling. I do think there's an interesting point about, well, the models have an advantage, which is they know a lot more than us. You know, like, should, should they have an advantage already, even if, even if they, their skill level isn't, isn't, isn't quite high? Maybe that's kind of what you're getting at. I don't really have an answer to that. I mean, it seems certainly like memorization and facts and drawing connections is an area where the models are ahead.

• 37:29Speaker 00

  And I, I, I do think maybe you need those connections and you need a fairly high level of skill. I do think particularly in the area of biology, for better and for worse, the complexity of biology is such that the current models know a lot of things right now. And that's what, that's what you need to make discoveries and draw. It's not like physics where you need to, you know, you need to think and come up with a formula. In biology, you need to know a lot of things.

• 37:55Dwarkesh Patel?

  Right.

• 37:55Speaker 00

  And so I do think the models know a lot of things and they have a skill level that's not quite high enough to put them together. And I think they are, they are just on the cusp of being able to put these things together.

• 38:05Dwarkesh Patel?

  On that point, last week in your Senate testimony, you said that these models are two to three years away from potentially enabling large-scale bioterrorism attacks or something like that.

• 38:15Speaker 00

  Can you make that more?

• 38:16Dwarkesh Patel?

  More concrete without obviously giving the kind of information that would. But is it like one-shotting how to weaponize something? Is it, or do you have to fine tune an open source model? Like, what would that actually look like?

• 38:26Speaker 00

  I think it'd be good to clarify this because we did a blog post in the Senate testimony and like, I think various people kind of didn't understand the point or didn't, didn't understand what we'd done. So I think today, and you know, of course, in our models, we try and, you know, prevent this, but there's always jailbreaks. You can ask the models all kinds of things about biology and get them to say all kinds of scary things.

• 38:47Dwarkesh Patel?

  Yeah.

• 38:48Speaker 00

  But often those scary things are things that you could Google. And I'm, I'm therefore not particularly worried about that. Um, I think it's actually an impediment to seeing the real danger where, you know, someone just says, "Oh, I asked this model like, you know, for the small po-, you know, for, to tell me some things about smallpox and it will." That, that is actually, you know, kind of not what I'm worried about. So we spent about six months working with some of, basically some of the folks who are the most expert in the world on how do, how do biological attacks happen? Um, you know, what, what would you need to conduct such an attack? And how do we defend against such an attack? They worked very intensively on just the entire workflow of if I were trying to do a bad thing, it's not one-shot, it's a long process.

• 39:33Speaker 00

  There are many steps to it. Um, it's not just like, I asked the model for this one page of information. And again, without going into any detail, the thing I said in the, the Senate testimony is like, there are some steps where you can just get information on Google. There are some steps that are what I'd call missing. They're scattered across a bunch of textbooks. Or they're not in any textbook, they're kind of implicit knowledge. And they're not really like, they're not explicit knowledge. They're, they're, they're, they're more like, I have to do this lab protocol. And like, what if I get it wrong? Oh, if this happens, then, then my temperature was too low. If that happened, I needed to add more of this particular reagent. What we found is that for the most part, those missing, those key missing pieces,

• 40:18Speaker 00

  the models can't do them yet. But we found that sometimes they can. Um, and when they can, sometimes They still hallucinate, which is the thing that's that's kind of keeping us safe. But we saw enough signs of the models doing doing those those key things well. And if we look at, you know, state of the art models and go backwards to previous models, we look at the trend, it shows every sign of two or three years from now. We're gonna have a real problem.

• 40:49Dwarkesh Patel?

  Yeah, especially the thing you mentioned on the lock scale, you go from like one in a hundred times it gets a riot to one in 10 to.

• 40:54Speaker 00

  Exactly. So, you know, I've seen many of these like Grox in my life, right? I was there when I I watched when GPT three learned to do arithmetic, when GPT two learned to do regression a little bit above chance. When, you know, when we got, you know, with Claude, and we got better on like, you know, all these all these tests of helpful, honest, harmless. I've seen a lot of Grox. This is this is unfortunately not one that I'm excited about, but I believe it's happening.

• 41:18Dwarkesh Patel?

  So somebody might say, listen, you were a co author on this post that OpenAI released about GPT two, where they said, you know, we're not gonna release the weights or the details here because we're worried that this model will be used for something, you know, bad. And looking back on it, now it's laughable to think that GPT two could have done anything bad. Are we just like way too worried? This is a concern that doesn't make sense for.

• 41:42Speaker 00

  It is interesting. It might be worth looking back at the actual text of that post. So I don't remember it exactly, but it should, it, you know, it's it's it's still up on the internet. It says something like, you know, we're choosing not to release the weights because of concerns about misuse, but it also said, this is an experiment. We're not sure if this is necessary or the right thing to do at this time, but we'd like to establish a norm of thinking carefully about these things. You know, you could think of it a little like the, you know, the the Silmar Conference in the in the 1970s, right? Where it's like, you know, they were just figuring out recombinant DNA. You know, it was not necessarily the case that someone could do something really bad with recombinant DNA.

• 42:25Speaker 00

  It's just the possibilities were starting to become clear. Those words at least were the right attitude. Now, I think there's a separate thing that like. You know, people don't just judge the post, they judge the organization. Is this an organization that, you know, is, produces a lot of hype or that has credibility or something like that? And so I think that had some effect on it. I guess you could also ask, like, is it inevitable that people would just interpret it as like, you know, you can't get across any message more complicated than this thing right here is dangerous. So you can argue about those, but I think the basic thing that was in my head and the head of others who were, who were involved in that. And, you know, I think what, what is, what is evident in the post is like, we actually don't know.

• 43:10Speaker 00

  We have pretty wide error bars on what's dangerous and what's not. So we should, you know, like, we, we want to establish a norm of being careful. I, I think, by the way, we have enormously more evidence. We've seen enormously more of these groks now, and so we're well calibrated, but there's still uncertainty, right? In all these statements, I said, like, in two or three years, we might be there, right? There's a substantial risk of it, and we don't want to take that risk. But, you know, I wouldn't say it's, it's 100%. It could be 50/50.

• 43:36Dwarkesh Patel?

  Okay, let's talk about cybersecurity, which in addition to bio risk is another thing Anthropic has been emphasizing. How have you avoided the cloud micro architecture from leaking? Because as you know, your competitors have been less successful at this kind of security.

• 43:50Speaker 00

  Can't comment on anyone else's security. Don't know what's going on in there. A thing that we have done is, you know, so, so there are, there are these, these architectural innovations, right, that make training more efficient. We call them compute multipliers because they're the equivalent of, you know, improving, improving, you know, they're like having more compute. Our compute multipliers, again, I don't want to say too much about it because it could allow an adversary to counteract our, our, our measures, but we limit the number of people who are aware of, of a given compute multiplier to those who need to know about it. And so there's, there's a very small number of people who could leak all of these secrets. There's a larger number of people who could leak one of them.

• 44:32Speaker 00

  But, you know, this is the standard compartmentalization strategy that's used in the intelligence community or, you know. Resistance cells or whatever. So, you know, we've over the last, over the last few months, we've implemented these measures. So, you know, I don't want to jinx anything by saying, oh, this could never happen to us. But I think, I think it would be harder for it to happen. I don't want to go into any more detail. And, you know, by the way, I'd encourage all the other companies to do this as well. It's as much as like competitors architectures leaking is narrowly helpful to Anthropic. It's not good for anyone in the long run, right? So, security around this stuff is really important.

• 45:11Dwarkesh Patel?

  Even with all the security you have, could you with your current security prevent a dedicated state level actor from getting the cloud two weights?

• 45:18Speaker 00

  It depends how dedicated is what, is what I would say. Our, our head of security, who, who was, you know, used to work on security for Chrome, which, you know, yeah, very widely used in attack application. He likes to think about it in terms of how much would it cost to attack Anthropic successfully? Again, I don't want to go into super detail of how much I think it will cost to attack. And it's just kind of inviting people. But like, one of our goals is that it costs more to attack Anthropic than, than it costs to just train your own model. Which doesn't guarantee things because, you know, of course, you need the talent as well. So you might still, but, you know, but, but attacks have, have, have risks, the diplomatic costs, you know, and, and, and they use up the very, the

• 45:59Speaker 00

  very sparse resources that nation state actors might have in order to do, to do the attacks. So, we're not there yet, by the way, but I, but I think, I think we're to a very high standard compared to the size of company that we are. Like, I think if you look at security for most 150 person companies, like, I think there's, there's just no comparison. But, you know, could we, could we resist if, if it was a state actor's top priority to steal our model weights? No, they would, they would succeed.

• 46:29Dwarkesh Patel?

  How long does that stay true? Because at some point, the value keeps increasing and increasing. And another part of this question is that what kind of a secret is how to train cloud three or cloud two? Is it, you know, with nuclear weapons, for example, we have lots of spies, you just take a blueprint across and that's the implosion device and that's what you need. Here, is it just, is it more tacit, like the thing you're talking about biology, you need to know how these reagents work? Is it just like, you got the blueprint, you got the micro architecture and hardware parameters?

• 46:58Speaker 00

  I mean, there, there are some things that are like, you know, one line equation, and there are other things that are more complicated.

• 47:04Dwarkesh Patel?

  Yeah.

• 47:04Speaker 00

  And I think compartmentalization is the

• 47:08Speaker 00

  The best way to do it, just limit the number of people who know about something. If you're a thousand person company and everyone knows every secret, like, one, I guarantee you have some, you have a leaker. And two, I guarantee you have a spy, like a literal spy.

• 47:19Dwarkesh Patel?

  Okay, let's talk about alignment. And let's talk about mechanistic interpretability, which is the branch

• 47:23Speaker 00

  Yes.

• 47:24Dwarkesh Patel?

  of which you, you guys specialize in. While you're answering this question, you might want to explain what mechanistic interpretability is. But just, um, the broader question is mechanistically, what is alignment? Is it that you're locking in the model into a benevolent character? Are you disabling deceptive circuits and procedures? Like what concretely is happening?

• 47:45Speaker 00

  Yeah.

• 47:45Dwarkesh Patel?

  When you align a model?

• 47:46Speaker 00

  I think as with most things, you know, when we actually train a model to be aligned, we don't know what happens inside the model. Right? There are different ways of training it to be aligned, but I think we don't really know what happens. I mean, I think for some of the current methods, I think all the current methods that involve some kind of fine tuning, of course, have the property that the underlying knowledge and abilities that we might be worried about, don't, don't disappear. It's just, you know, the, the model is just taught not to output them. I don't know if that's a fatal flaw or if, you know, or if that's just the way things have to be. I don't know what's going on inside mechanistically. And I think that's the whole point of mechanistic interpretability, to really understand what's going on inside the models at the level of individual circuits.

• 48:30Dwarkesh Patel?

  Eventually, when it's solved, what does the solution look like? Where, what is it the case where if you're caught for, you do the mechanistic interpretability thing and you're like, I'm satisfied, it's aligned. What is it that you've seen?

• 48:40Speaker 00

  Yeah, so I, I think, I think we don't know that yet. I think we don't know enough to, to know that yet. I mean, I can, I can give you a sketch for like, what the process looks like as opposed to what the final result looks like. So I think verifiability is a lot of the challenge here, right? We have all these methods that purport to align AI systems and, and do succeed at doing so for today's tasks. But then the, the question is always, if you had a more powerful model or if you had a model in a different situation, would it, would it, would it be aligned? And so I think this problem would be much easier if you had an oracle that could just scan a model. And say like, okay, I know this model is aligned. I know what it'll do in every situation. Then the problem would be much easier.

• 49:25Speaker 00

  And I think the closest thing we have to that is something like mechanistic interpretability. It's not anywhere near up to the task yet. But I guess I would say, I think of it as almost like an extended training set and an extended test set. Right? Everything we're doing, all the alignment methods we're doing are the training set. Right? You, you know, you can, you can run tests in them, but will it really work out a distribution? Will it really work in another situation? Mechanistic interpretability is the only thing that even in principle, and we're, we're nowhere near there yet. But even in principle, is the thing where it's like, it's more like an X-ray of the model than a modification of the model. Right? It's more like an assessment than an intervention. And so somehow we need to get into a dynamic where we have an extended test set,

• 50:08Speaker 00

  an extended training set, which is all these alignment methods, and an extended test set, which is kind of like you, you X-ray the model and say like, okay, what works and what didn't? In, in a way that goes beyond just the empirical tests that you, that you, that you've run. Right? Um, where you're saying, what is the, what, what is the model going to do in these situations? What is it within its capabilities to do instead of what did it do phenomenologically? And of course, we have to be careful about that, right? One of the things I think is very important is we should never train for interpretability, because I think that is, that's taking away that advantage. Right? You even have the problem, you know, similar to like validation versus test set. Where like, if you look at the X-ray too many times, you can interfere.

• 50:53Speaker 00

  But I think that's a much weaker option. We should worry about that, but that's a, that's a much weaker process. It's not automated optimization. We should just make sure, as with validation and test sets, that we don't look at the validation set too many times before running the test set. But you know, that's again, that's, that's more of a, that's, that's manual pressure rather than automated pressure. And so some solution where it's like, we have some dynamic between the training and test set, where it's like, we're, we're trying things out and we, we, we really figure out if they work via a way of testing them that the model isn't optimizing against. Some, some orthogonal way, like if, if, if I, if I think of, and I think we're never going to have a guarantee, but some process where we, we do those things together.

• 51:38Speaker 00

  Again, not in a stupid way. There's lots of stupid ways to do this where you fool yourself, but like. Some way to put extended training for alignment ability with extended testing for alignment ability together in a way that actually works.

• 51:52Dwarkesh Patel?

  Hmm, I still don't feel like I understand the intuition that why you think this is likely to work or this is promising to pursue. And let me ask the question in a sort of more specific way and excuse the tortured analogy. But listen, if you're you're an economist and you want to understand the economy.

• 52:08Speaker 00

  Yeah.

• 52:09Dwarkesh Patel?

  So you send a whole bunch of microeconomists out there and one of them studies how the restaurant business works, one of them studies how the tourism business works, you know, one of them studies how the baking works. And at the end, they all come together and you still don't know whether there's going to be a recession in five years or not. Why is this not like that where you have an understanding of we understand how induction heads work in a two-layer transformer, we understand, you know, modular arithmetic. How does this add up to, does this model want to kill us? Like, what does this model fundamentally want?

• 52:37Speaker 00

  A few things on that. I mean, I think that's like the right set of questions to ask. I think what we're hoping for in the end is not that we'll understand every detail. But again, I would give like the x-ray or the MRI analogy that like, we can be in a position where we can look at the broad features of the model and say like, is this a model whose internal state and plans are very different from what it externally represents itself to do, right? Is this a model where we're uncomfortable that, you know, far too much of its computational power is, you know, is devoted to doing what look like fairly destructive and manipulative things? Again, we don't know for sure whether that's possible, but I think some at least positive signs that it might be possible.

• 53:22Speaker 00

  Again, the model is not intentionally hiding from you, right? It might turn out that the training process hides it from you. And, you know, I can think of cases where the model is really super intelligent, it like thinks in a way so that it like affects its own cognition. I suspect we should think about that, we should consider everything. I suspect that it may roughly work to think of the model as, you know, if it's trained in the normal way, just at, you know, at the, at the just getting to just above human level, it may be a reason we should check, it may be a reasonable assumption that.

• 53:59Speaker 00

  The internal structure of the model is not intentionally optimizing against us. And I give an analogy like to humans. So it's actually possible to, you know, to look at an MRI of someone and predict above random chance whether they're a psychopath. There was actually a story a few years back about a neuroscientist who was studying this. And then he looked at his own scan and discovered that he was a psychopath. And then everyone, everyone in his life was like, no, no, that's just obvious. Like you're, you're a complete asshole. Like you must be a psychopath. And he was totally unaware of this. The basic idea that, you know, that, that there can be these macro features that like, like psychopath is probably a good analogy for it, right? They're like, you know, this is what we'd be afraid of.

• 54:43Speaker 00

  Model that's kind of like charming on the surface, very goal oriented and, you know, very dark on the inside, you know, and, and, you know, on the surface, their behavior might look like the behavior of someone else, but their goals are very different.

• 54:54Dwarkesh Patel?

  A question somebody might have is, listen, you know, you mentioned earlier the importance of being empirical.

• 54:59Speaker 00

  Yeah.

• 55:00Dwarkesh Patel?

  And in this case, you're trying to estimate, you know, listen, are these activations sus?

• 55:05Speaker 00

  Yeah.

• 55:06Dwarkesh Patel?

  But is this something we can be afford to be empirical about in, on, you know, or do we need like a very good first principle, theoretical reason to think, no, it's not just that these MRIs of the model correlate with, you know, being bad. We need just like some, just like deep root math proof that this is aligned.

• 55:25Speaker 00

  So it depends what you mean by empirical. I mean, a better term would be phenomenological. Like, I don't think we should be purely phenomenological in like, you know, here are some brain scans of like really dangerous models and here are some brain scans. I think the whole idea of mechanistic interpretability is to look at the underlying principles and circuits. But I guess the way I think about it is like, on one hand, I've actually always been a fan of studying these circuits at the lowest level of detail that we possibly can. And the reason for that is kind of, that's how you build up knowledge. Even if you're ultimately aiming for, there's too, there's too many of these features. It's too complicated. At the end of the day, we're trying to build something broad. And we're trying to build some broad understanding.

• 56:07Speaker 00

  I think the way you build that up is by trying to make a lot of these very specific discoveries. Like you have to, you have to understand the building blocks. And, and then you have to figure out how to kind of use that to draw these broad conclusions, even if you're not going to figure out everything. You know, I think you should probably talk to Chris Ola, who would have much more detail, right? This is my kind of high-level thinking on it. Like, Chris Ola controls the interpretability agenda. Like, you know, he's, he's the one who decides what to, what to do on interpretability. This is my high-level thinking about it, which is not going to be as good as his.

• 56:40Dwarkesh Patel?

  Does the bull case on Anthropic rely on the fact that mechanistic interpretability is helpful for capabilities?

• 56:46Speaker 00

  I, I don't think so at all. Um, now, I do think, I, I think in principle, it's possible that mechanistic interpretability could be helpful with capabilities. We might, for various reasons, not choose to talk about it if that were the case. Uh, that, you know, that, that wasn't something that I thought, thought of, or that any of us thought of at the time of Anthropic's founding. I mean, we, we thought of ourselves as like, you know, we're people who are like, good at scaling models and good at doing safety on top of those models. And like, you know, we think that we have a very high talent density of folks who are good at that. And, you know, my view has always been talent density beats talent mass. Um, and so, you know, that's, that's more, that's more of our bull case.

• 57:27Speaker 00

  Talent density beats talent mass. I don't think it, it depends on some particular thing. Like, others are starting to do mechanistic interpretability now, and I'm very glad that they are. Uh, you know, that was, that is a part of our, a part of our, a part of our theory of change is, paradoxically, to make other organizations more like us.

• 57:43Dwarkesh Patel?

  Talent density, I'm sure, is important, but another thing Anthropic has emphasized is that you need to have frontier models in order to do safety research. And of course, like, actually be a company as well. The current frontier model is something, somebody might guess, like, GPT-4 Oclot to like $100 million or something like that.

• 57:59Speaker 00

  That general order of magnitude in very broad terms is not wrong.

• 58:02Dwarkesh Patel?

  But, you know, we're two to three years from now, the kinds of things you're talking about, we're talking more and more orders of magnitude. To keep up with that, and to, if it's the case that safety requires to be beyond the frontier, I mean, what is the case in which Anthropic is like, competing with these leviathans to stay on that same scale?

• 58:19Speaker 00

  Yeah, I mean, I think it's a, I think it's a very, it's a situation with a lot of trade-offs, right? I think it's, I think it's not easy. Um, I guess to go back, maybe I'll just like answer the questions one by one, right? So like, to go back to like, you know, why, why is safety so tied to scale, right? Um, some people don't think it is, but like, if I, if I just look at like, you know, where, where have been, where have been the areas that, you know? You know, I don't know, like, safety methods have, like, been put into practice or, like, worked for something, for anything, even if we don't think they'll, they'll work in general. You know, I go back to thinking of all the ideas, you know, something like, you know, debate and amplification, right? You know, back in 2018, when we wrote papers about those at, at, at OpenAI, I was like, well, human feedback isn't, isn't quite gonna work, but, you know, debate and amplification

• 59:04Speaker 00

  will take us beyond that. But then, if you, if you actually look at, and we've, you know, done attempts to do debates, we're really limited by the, by the quality of the model. Where it's like, you know, for two models to have a debate that is coherent enough that a human can judge it, so that the training process can actually work, you need models that are at, or maybe even beyond, on some topics, the current frontier. Now, you can come up with a method, you can come up with the idea, without being on the frontier, but, you know, for me, that's a very small fraction of what needs to be done, right? It's very easy to come up with these methods, it's very easy to come up with, like, oh, the problem is X, maybe a solution is Y, but, you know, I, I really wanna know, you

• 59:49Speaker 00

  know, whether things work in practice, even for the systems we have today, and I wanna know what kinds of things go wrong with them. I, I, I just feel like, you discover ten new ideas and ten new ways that things are gonna go wrong, by trying these in practice, and that, that empirical learning, I think it's, it's just not as widely understood as it should be. Kind of every, you know, I would say the same thing about methods like Constitutional AI, and some people say, oh, it doesn't matter, like, we know this method doesn't work, it won't work for, you know, pure alignment. I neither agree nor disagree with that, I think that's just kind of overconfident. The way we discover new things and understand the structure of what's gonna work and what's, what's not, is by playing around with things. Not that we should just kind of blindly say, oh, this worked here, and so, so it'll work

• 1:00:31Speaker 00

  there, but you, you, you really, you really start to understand the patterns, like with, like with the scaling laws. Even mechanistic interpretability, which might be the one area I see where a lot of progress has been made without the frontier models. We're, you know, we're seeing and, you know, the work that say OpenAI put out a couple, a couple of months ago, that, you know, using very powerful models to help you auto-interpret the weak models, again, that's not everything you can do in interpretability, but you know, that's a, that's a big component of it and we, you know, we found it useful too. And so you see this, this, this phenomenon over and over again, where it's like, you know, the, the scaling and the safety are these two snakes that are like coiled with

• 1:01:14Speaker 00

  each other, always even more than you think, right? I, you know, with interpretability, like, I think three years ago, I didn't think that this would be as true of interpretability. But somehow it manages to be true. Why? Because intelligence is useful. It's useful for a number of tasks. One of the tasks it's useful for is like figuring out how to judge and evaluate other intelligence and maybe someday even, even for, you know, doing the alignment research itself. Given all that's true, what, what does that imply for Anthropic when in two to three years, these Leviathans are doing like $10 billion training runs? Uh, choice one is if it, if we can't, or if it costs too much to stay on the frontier, then, you know, then, then we shouldn't, uh, then we shouldn't do it. And, you know, we won't work with the most advanced models. We'll see what we can get with, you know,

• 1:01:58Speaker 00

  models that are not quite as advanced. I think you can get some value there, like non-zero value, but I'm, I'm kind of skeptical that the value is all that high or the learning can be fast enough to really, to really be in favor of the task. The second option is you just, you just find a way. You just, uh, you know, you just accept the trade-offs. And I think the trade-offs are more positive than they appear because of a phenomenon that I've called race to the top. Um, I could go into that later, but I'll just, let me put that aside for now. Uh, and then I think the third phenomenon is, you know, as things get, as things get to that scale, I think this may coincide with, you know, starting to get into some non-trivial probability of very serious danger. Again, I think it's

• 1:02:43Speaker 00

  going to come first from misuse, the kind of bio stuff that I talked about, but I don't think we have the level of autonomy yet to worry about some of the, you know, alignment stuff happening in like two years, but it might not be very far behind that at all. You know, that, that may, that may lead to unilateral or multilateral or government enforced, which we support. Decisions not to scale as fast as we could. That may end up being the right thing to do. So I, you know, actually, that's kind of like, I, I kind of hope things go in that, in that direction. And then we don't have this hard trade off between, we're not on the frontier and we can't quite do the research as well as, as well as we want, or influence other orgs as well as we want.

• 1:03:28Speaker 00

  Or versus, we're kind of on the frontier and like, have to accept the trade offs, which are, which are net positive, but like, have a, have a lot in both, in both directions.

• 1:03:37Dwarkesh Patel?

  Okay, on the, on the misuse versus misalignment, those are both problems, as you mentioned. But in the long scheme of things, what, what is, what are you more concerned about? Like 30 years down the line, which do you think will be considered a bigger problem?

• 1:03:49Speaker 00

  I think it's much less than 30 years. But I'm, I'm worried about both. I don't know if you have, if you, if you have a model that could, in theory, you know, like, take over the world on its own. If you were able to control that model, then, you know, it follows pretty simply that, you know, if a model was following the wishes of some small subset of people and not others, then those people could use it to take over the world on their, on their behalf. The very premise of misalignment means that we should be worried about misuse as well, with similar levels of consequences.

• 1:04:20Dwarkesh Patel?

  But, but some people who might be more doomer-y than you would say misuse is, you're already working towards the optimistic scenario there, because you've at least figured out how to align the model with the bad guys. Now you just need to make sure that it's aligned with the good guys instead. Why do you think that you could get to the point where it's aligned with the bad, you know, where you haven't already solved?

• 1:04:41Speaker 00

  I, I guess if you had the view that like, alignment is completely unsolvable, then, you know, then you'd be like, well, I don't, you know, we're dead anyway, so I don't want to worry about misuse. That's not my position at all, but, but also like, you should think in terms of like, what's a plan that would actually succeed, that would make things good? Any plan that actually succeeds, regardless of how hard misalignment is to solve, any problem, any plan that actually succeeds, is going to need to solve misuse as well as misalignment. It's getting to solve the fact that like, As the AI models get better, you know, faster and faster, they're going to create a big problem around the balance of power between countries. They're going to create a big problem around, is it possible for a single individual to do something bad that it's hard for everyone else to stop?

• 1:05:24Speaker 00

  Any actual solution that leads to a good future needs to solve those problems as well. If your perspective is, we're screwed because we can't solve the first problem, so don't worry about problems two and three. Like, that's not really a statement, you shouldn't worry about problems two and three, right? Like, they're in our path, no matter what.

• 1:05:42Dwarkesh Patel?

  Yeah, in the scenario we succeed, we have to solve all, so yeah, we might as well operate.

• 1:05:45Speaker 00

  We should be planning for success, not for failure.

• 1:05:48Dwarkesh Patel?

  If Muses doesn't happen and the right people have the superhuman models, what does that look like? Like, who are the right people? Who is actually controlling the model from five years from now?

• 1:05:58Speaker 00

  Yeah, I mean, my view is that these things are powerful enough that I think, you know, it's going to involve, you know, substantial role, or at least involvement of, you know, some kind of government or assembly of government bodies. Again, like, you know, there are kind of very naive versions of this. Like, you know, I don't think we should just, you know, I don't know, like, hand the model over to the UN or whoever happens to be in office at a given time. Like, I could see that go poorly, but it's too powerful. There needs to be some kind of legitimate process for managing this technology, which, you know, includes the role of the people building it, includes the role of, like, democratically elected authorities,

• 1:06:41Speaker 00

  includes the role of, you know, all the individuals who will be affected by it. So that, at the end of the day, there needs to be some politically legitimate process.

• 1:06:52Dwarkesh Patel?

  But what does that look like? If it's not the case that you just hand it to whoever the president is at the time?

• 1:06:56Speaker 00

  Yeah.

• 1:06:57Dwarkesh Patel?

  What does the body look like? I mean, is this something you're...

• 1:07:00Speaker 00

  These are things it's really hard to know ahead of time. Like, I think, you know, people love to kind of propose these broad plans and say like, "Oh, this is the way we should do it, this is the way we should do it." I think the honest fact is that we're figuring this out as we go along, and that, you know, anyone who says, you know, "This is the body, you know, we should create this kind of body..." modeled after this thing like I think I think we should try things and experiment with them with less powerful versions of the technology. We need to figure this out in time, but but also it's not the really the kind of thing you can know in advance. The the long term benefit trust that you have, how did how would that interface with this body? Is that the body itself? If not, is it like, so just for context, you might want to explain what it is for the

• 1:07:41Speaker 00

  audience, but I don't know. I think of the long term benefit trust is like a much a much narrower thing like this is something that like makes decisions for anthropics. So this is basically a body is described in a recent Vox article, we'll be saying more about it in, you know, later later this year. But it's basically a body that over time gains the ability to appoint the majority of the board seats of anthropic. And this is so, you know, it's a mixture of experts and I'd say like AI alignment, national security and philanthropy in general. But if control is handed to them of anthropic, that doesn't imply that control of if anthropic has AGI, the control of AGI itself is handed to them.

• 1:08:22Speaker 00

  That doesn't imply that anthropic or any other entity should be the entity that like makes decisions about AGI on behalf of humanity. I would think of those as different. I mean, there's lots of maybe, you know, like if anthropic does play a broad role, then you'd want to like widen that body to be, you know, like a whole bunch of different people from around the world, or or maybe you construe this as very narrow. And then, you know, there's some like broad committee somewhere that like manages all the AGIs of all the companies on behalf on behalf of anyone. I don't know, like I I think my view is you shouldn't be sort of overly constructive and utopian. Like, we're dealing with a new problem here. We need to, we need to start thinking now about, you know, what are the, what are the governmental bodies and structures that could that could deal with it?

• 1:09:06Speaker 00

  Okay, so let's forget about governance. Let's just talk about what this going well looks like. Obviously, there's the things we can all agree on, you know, cure all the diseases, you know, solve all the problems, everything's all humans would say, I'm down for that.

• 1:09:18Dwarkesh Patel?

  But now it's 2030, you've solved all the real problems that everybody can agree on. What happens next? What are we doing with a superhuman god?

• 1:09:26Speaker 00

  I think I actually want to like, I don't know, like disagree with the framing or something like this. I actually get nervous when someone says like, what are you going to do with a superhuman AI? Like, we've learned a lot of things over the last 150 years about like markets and democracy. And each person can kind of define for themselves like what the best way for them to have the human experience is. And that, you know, societies work out norms and what they value in this, just in this very like complex and decentralized way. Now, again, if you have these safety problems, that can be a reason why, you know, and especially from the government, there needs to be maybe until we've solved these problems, a certain amount of like centralized control. But as a matter of like, we've solved all the problems, now how do we make things good?

• 1:10:12Speaker 00

  I think that that most, most people, most groups, most ideologies that started with like, let's sit down and think, think, think, think over what the definition of the good life is. Like, I think, I think most of those have led to disaster.

• 1:10:24Dwarkesh Patel?

  But so this vision you have of a sort of tolerant liberal democracy, market oriented system with AGI, like what is, each person has their own AGI, like what does that, what does that mean?

• 1:10:34Speaker 00

  I don't know. I don't know what it looks like, right? Like, I guess what I'm saying is like, we need to solve the kind of important safety problems and the important externalities. And then, and then subject to that, you know, which again, you know, those could be just narrowly about alignment. There could be a bunch of economic issues that are super complicated and that we can't solve. You know, subject to that, like, we should think about what's worked in the past. And I think in general, like, unitary visions for what it means to live a good life have not worked out well at all.

• 1:11:05Dwarkesh Patel?

  On the opposite end of things going well, or good actors having control of AI, we might want to touch on China as like a potential actor in the space. So first of all, I mean, being at Baidu and like seeing progress in AI happening generally. Why do you think the Chinese have underperformed? You know, Baidu had a scaling laws group many years back. Or is the premise wrong? And I'm just not aware of the progress that's happening there.

• 1:11:30Speaker 00

  Um, well, for the scaling laws group, I mean, that was an offshoot of the stuff we did with Speech. So, uh, you know, there were still some people there, but that was a mostly Americanized lab. I mean, I was there for a year. That was, you know, my first foray into deep learning. It was led by Andrew Ng. I never went to China. Most, you know, there's, it's like a US lab. So I think that was somewhat disconnected, although it was an attempt by, you know, a Chinese entity to kind of get it, get into the game. But I don't know. I think since then, you know, I couldn't speculate, but I think they've been maybe very commercially focused and not as focused on these kind of fundamental research side of things around scaling laws. Now, I do think because of all the, you know, excitement with the release of ChatGPT

• 1:12:13Speaker 00

  in, you know, November or so, um, you know, that's been a starting gun for them as well. And they're trying very aggressively to catch up now. I think we're, the US is quite substantially ahead, but I think they're trying very hard to, to catch up now.

• 1:12:26Dwarkesh Patel?

  How do you think China thinks about AGI? Are they thinking about safety and misuse or not?

• 1:12:32Speaker 00

  I, I don't really have a sense. You know, one concern I would have is people say things like, well, China isn't going to develop an AI because, you know, they like stability or, you know, they're going to have all these restrictions to make sure things are in line with what the CCP wants. You know, that, that might be true in the short term and for consumer products. My, my worry is that if the basic incentives are about national security and power, that's going to become clear sooner or later. Um, and, and so, you know, they're, they're, I think they're going to, if they see this as, you know, a source of national power, they're going to at least try to do, to do what's most effective. And that, you know, that could lead them in the direction of AGI.

• 1:13:11Dwarkesh Patel?

  At what point, like, is it possible for them, they just get your blueprints or your code base or something, that they can just spin up their own lab that is competitive at the frontier with the leading American companies?

• 1:13:21Speaker 00

  Well, I, I don't know about fast, but I'm like, I'm concerned about this. So this is one reason why we're focusing so hard on cybersecurity. You know, we've worked with our cloud providers. We really, you know, like, you know, we had this blog post out, out about security where we said, you know, we have a two key system for access to the model weights. We have other measures that. We put in place or thinking of putting in place that, you know, we haven't announced. We don't want an adversary to know about them, but we're happy to talk about them broadly. All this stuff we're doing is, is by the way, not sufficient yet for a super determined state, state, state level actor at all. I think it, it, it will defend against most attacks and against a, a state level actor who's not, you know, who's less determined.

• 1:14:07Speaker 00

  But there's a lot more we need to do and some of it may require new research on how to do security.

• 1:14:12Dwarkesh Patel?

  Okay. So let's talk about what it would take at that point. You know, we're at Anthropic offices and, you know, it's like, God, good as security. We had to get badges and everything to come in here. But the eventual version of this building or bunker, whatever, where the AGI is built. I mean, what, what does that look like? Are we, is it a building in the middle of San Francisco? Or is it, you're out in the middle of Nevada or Arizona? Like, what is the point in which you're like Los Alamosing it?

• 1:14:36Speaker 00

  At one point, there was a running joke somewhere that, you know, the way, the way building AGI would look like is, you know, there would be a data center next to a nuclear power plant, next to a bunker.

• 1:14:46Dwarkesh Patel?

  Yeah.

• 1:14:47Speaker 00

  And, you know, that we, we'd all, we'd all kind of live in the bunker and everything would be local. So it wouldn't get on the internet. You know, again, if we, you know, if we take seriously the rate at which the, you know, rate at which all of this is going to happen, which I don't know, I can't be sure of it. But if we take that seriously, then it, you know, it, it does make me think that maybe not something quite as cartoonish as that, but that something like that might happen.

• 1:15:11Dwarkesh Patel?

  What is the timescale on which you think alignment is solvable? If like these models are getting to human level in some things in two to three years, what is the point at which they're aligned?

• 1:15:21Speaker 00

  I think this is a really difficult question because I actually think often people are thinking about kind of alignment in the wrong way. I, I think there's a general feeling that it's like models are misaligned or like there's like an alignment problem to solve, kind of like the Remont hypothesis or something, like someday we'll crack the Remont hypothesis. I don't quite think it's like that, not in a way that's worse or better. It might be just as bad or just as unpredictable. When I think of like, you know, why am I scared? Few things I think of, one is, look, like, I think the thing that's really hard to argue with is like, there will be powerful models, they will be agentic, we're getting towards them. If such a model wanted to wreak havoc and destroy humanity or whatever,

• 1:16:06Speaker 00

  I think we have basically no ability to stop it. Like, that's, I think, just, if that's not true at some point, it'll continue to be true as we, you know, it will reach the point where it's true as we scale the models. So that definitely seems the case. And I think a second thing that seems the case is that we seem to be bad at controlling the models, not in any particular way, but just their statistical systems, and you can ask them a million things, and they can say a million things in reply. And, you know, you might not have thought of a millionth of one thing that does something crazy. Or when you train them, you train them in this very abstract way, and you might not understand all the consequences of what they do in response to that. I mean, I think the best example we've seen of that is like being in Sydney, right?

• 1:16:50Speaker 00

  Where it's like, I don't know how they trained that model. I don't know what they did to make it do all this weird stuff, like, you know, threaten people and, you know, have this kind of weird obsessive personality. But what it shows is that we can get something very different from and maybe opposite to what we intended. And so I actually think fact number one and fact number two are like enough to be really worried. Like, you don't need all this detailed stuff about, you know, convergent instrumental goals or, you know, analogies to evolution. Like, actually, one and two for me are pretty motivated. I'm like, okay, this thing's gonna be powerful, it could destroy us. And like, all the ones we've built so far, like, you know, are at a pretty decent risk of doing some random shit we don't understand.

• 1:17:34Dwarkesh Patel?

  Yeah, if I agree with that, and I'm like, okay, I'm concerned about this. The research agenda you have of mechanistic interoperability, plus, you know, a constitution AI and the other LHF stuff. If you say that we're gonna get something with like bioweapons or something that could be dangerous in two to three years.

• 1:17:50Speaker 00

  Yes.

• 1:17:50Dwarkesh Patel?

  Do these things culminate within two to three years of actually meaningfully contributing to preventing?

• 1:17:56Speaker 00

  Yes. So I think, I think where I was gonna go with this is like, you know, people talk about like doom by default or alignment by default. Like, I think it might be kind of statistical, like, you know, like, you might get... You know, with the current models, you might get Bing or Sydney, or you might get Claude. And it doesn't really matter because Bing or Sydney, like if we take our current understanding and you know, move that to to very powerful models, you might just be in this world where it's like, okay, you make something and depending on the details, maybe it's totally fine. Um, you know, not really alignment by default, but but just kind of like it depends on a lot of the details and like if you if you're very careful about all those details and you know what you're doing, you're getting it right. But we have a high susceptibility

• 1:18:37Speaker 00

  to you mess something up in a way that you didn't really understand was connected to actually, instead of making all the humans happy, it wants to, you know, turn them into pumpkins. Yeah, I you know, I just some weird shit, right? Because the models are so powerful, you know, they're like these kind of giants that are, you know, they're they're like, you know, they're standing in a landscape. And if they start to move their arms around randomly, they could just break everything. Um, I guess I'm starting it with that with that kind of framing because it's not like, I don't think we're aligned by default. I don't think we're doomed by default and have some problem we need to solve. It has some kind of different character. Now, what I do think is that hopefully, within a time scale of two to three years, we get better at diagnosing when the models are good and

• 1:19:18Speaker 00

  when they're bad. We get better at training, you know, in increasing our repertoire of methods to train the model that they're less likely to do bad things and more likely to do good things in a way that isn't just relevant to the current models, but scales. And we can help develop that with interpretability as the test set. I don't think of it as, oh, man, we tried RLHF, it didn't work. We tried constitutional, it didn't work. Like, we tried this other thing, it didn't work. We tried mechanistic interpretability, now we're going to try mechanistic. Um, I think this frame of like, man, we haven't cracked the problem yet. We haven't solved the Reamann hypothesis, isn't quite right. Um, I think of it more as Already with today's systems, we are not very good at controlling them.

• 1:20:02Speaker 00

  And the consequences of that could be, could be, could be very bad. We just need to get more ways of like increasing the likelihood that are, that are, that, you know, that we can control our models and understand what's going on in them. And like, we have some of them so far. They aren't that good yet. But, you know, I, I don't think of this as binary of like works and not works. We're going to develop more. And I do think that over, over the next two to three years, we're going to start eating that probability mass of ways things can go wrong. You know, it's kind of like in the core safety view, but there's probability mass of how hard the problem is. I feel like that way of seeing it isn't really even quite right, right? Because I don't feel like it's the Rema hypothesis to solve.

• 1:20:44Speaker 00

  I, you know, I just feel like, you know, it's, it's almost like right now, if I try and, you know, juggle five balls or something, I can juggle three balls, right? I actually can, but, but I can't juggle five balls at all, right? You have to practice a lot to do that. If I were to do that, I would mostly, I would, I would almost certainly drop them. And then just, just over time, you just get better at the task of controlling the balls.

• 1:21:05Dwarkesh Patel?

  On that post in particular, what is your personal probability distribution over? So for the audience, the three possibilities are, it is like trivial to align these models with RHF++ to it is a difficult problem, but one that a big company could solve. To something that is like basically impossible for human civilization currently to solve. If I'm capturing those three, what is your probability distribution over those three personally?

• 1:21:29Speaker 00

  Yeah, I mean, I'm not super into like, what's your probability distribution of X? I think all of those have enough likelihood that, you know, they should be considered seriously. I'm more interested, question I'm much more interested in is what could we learn that shifts probability mass between them?

• 1:21:44Dwarkesh Patel?

  What is the answer to that?

• 1:21:45Speaker 00

  I think that one of the things mechanistic interpretability is going to do more than, more than necessarily solve problems is it's going to tell us what's going on when we try to align models. I, I, I think it's basically going to teach us about this. Like one way I could imagine. concluding that things are very difficult is if mechanistic interpretability sort of shows us that, I don't know, problems tend to get moved around instead of being stamped out, or that you get rid of one problem, you create another one, or it might inspire us or give us insight into why problems are kind of persistent or hard to eradicate

• 1:22:26Speaker 00

  or crop up. Like, for me to really believe some of these stories about like, you know, oh, something will always, you know, there's always this convergent goal in this particular direction. I think the abstract story is, it's not uncompelling, but I don't find it really compelling either, nor do I find it necessary to motivate all the safety work. But like, the kind of thing that would, would really be like, oh, man, we can't solve this is like, we see it happening inside, inside the X-ray. Because, yeah, because I, I, I think right now, there's just, there's, there's way, there's way too many assumptions, there's way too much overconfidence about how all this is gonna go. I have a substantial probability mass on this all goes wrong, it's a complete disaster, but in a completely different way than anyone had anticipated.

• 1:23:09Dwarkesh Patel?

  It would be beside the point to ask, like, how could it go different than anyone anticipated? So, on, on this in particular, what information would be relevant? How much would the difficulty of aligning Cloud 3 and the next generation of models basically be? Like, is that a big piece of information? Is that not gonna be big?

• 1:23:25Speaker 00

  So, so I think the people who are most worried are predicting that all the sub-human, like, AI models are gonna be alignable, right? They're, they're gonna seem aligned, they're gonna deceive us in some way. I think it, it certainly gives us some information, but I, I am more interested in what mechanistic interpretability can tell us. Because, again, like, you see this X-ray, it would be too strong to say it doesn't lie, but at least in the current systems, it doesn't feel like it's Optimizing against us. There are exotic ways that it could. You know, I don't think anything is a safe bet here. But I think it's the closest we're going to get to something that isn't actively optimizing against us.

• 1:24:07Dwarkesh Patel?

  Let's talk about the specific methods other than mechanistic interoperability.

• 1:24:10Speaker 00

  Yes.

• 1:24:10Dwarkesh Patel?

  That you guys are researching. When we talk about RLHF or, you know, Constitution AI, whatever, RLHF++. If you had to put it in terms of human psychology, what is the change that is happening? Are we creating new drives, new goals, new thoughts? How is the model changing in terms of psychology when it?

• 1:24:31Speaker 00

  I think all those terms are kind of like inadequate for, you know, describing what's. It's not clear how useful they are as abstractions for humans either. I think we don't have the language to describe what's going on. And again, I'd love to have the x-ray. I'd love to look inside and say, and kind of actually know what we're talking about instead of, you know, basically making up words, which is what, which is what I do. What you're doing and asking this question. Where, you know, we should, we should just be honest. We really have very little idea what we're, what we're talking about. So, you know, it would be great to say, well, what we actually mean by that is, you know, this circuit within here turns, you know, turns on and, you know, and, you know, after we've trained the model, then, you know, this circuit is no longer operative or weaker. And that, you know, I would love to be able to say, again, we're,

• 1:25:16Speaker 00

  it's going to take a lot of work to be able to do that.

• 1:25:18Dwarkesh Patel?

  Model organisms, which you hinted at before, when you said we're doing these evaluations to see if they're capable of, you know, doing dangerous things now and currently not. How worried are you about a lab leak scenario? Where in fine tuning it or in trying to get these models to elicit dangerous behaviors, you know, make bioweapons or something. You like leak somehow and actually makes the bioweapon instead of telling you it can make the bioweapon.

• 1:25:41Speaker 00

  With today's passive models, I think it's not that much, you know, chatbots. It's not so much of a concern, right? Because it's like, you know, if you were to fine tune a model, do that, we do it privately and work with the experts. And so, you know, the leak would be like, you know, suppose the model got open sourced or something. And, you know, and then someone, so, so I think for now, it's mostly a security issue. In terms of models truly being dangerous, I mean, you know, I think, I think we do have to worry that it's like. You know, if we make a truly powerful model and we're trying to like see what makes it dangerous or safe, then there could be more of a one-shot thing where it's like, you know, some risk that the model takes over. I think the main way to control that is to make sure that the capabilities of the model that we test are not such that

• 1:26:26Speaker 00

  they're capable of doing this.

• 1:26:28Dwarkesh Patel?

  At what point would the capabilities be so high where you're, you say, I don't even want to test this?

• 1:26:32Speaker 00

  Well, there's different things. I mean, there's capability testing and, you know.

• 1:26:35Dwarkesh Patel?

  But that itself could lead to, if you're testing it and replicate that, like, what if it actually does?

• 1:26:40Speaker 00

  Sure, but I think, I mean, I think what you want to do is you want to like extrapolate. So we've talked with Ark about this, right? You know, you have like factors of two of compute or something where, you know, you're like, okay, you know, you know, can, can the model do something like, you know, open up an account on AWS and like make some money for itself? Like some of the things that are like obvious prerequisites to like complete survival in the wild. And so just set, set those thresholds very well, you know, kind of very well below. And then as you proceed upward from there, do kind of more and more rigorous tests and be more and more careful about, about, about what it is you're doing.

• 1:27:17Dwarkesh Patel?

  On Constitution AI, and feel free to explain what this is for the audience, but who decides what the constitution for the next generation of models or potentially superhuman model is? Like, how is that actually written?

• 1:27:28Speaker 00

  I think initially, you know, to make the constitution, we just took some stuff that was like broadly agreed on, like the UN Charter of, you know, UN Declaration on Human Rights and, um, you know, some of the stuff from Apple's Terms of Service, right? Stuff that's like, you know, consensus on like what's acceptable to say or like, you know, what, what basic things are able to be included. So one, I think for future constitutions, we're looking into like more participatory processes for making these. But I think beyond that, I don't think there should be like one constitution for like a model that everyone uses. Like probably models constitution should be very, very simple, right? It should only have very basic.

• 1:28:08Speaker 00

  Facts that everyone would agree on, and then there should be a lot of ways that you can customize, including a pending constitutions. And I think beyond that, we're developing new methods. I'm not imagining that this or this alone is the method that we'll use to train superhuman AI. Many of the parts of capability training may be different, and so it could look very different. And again, there are levels above this. Like, I'm pretty uncomfortable with, here's the AI's constitution, it's gonna run the world. Again, just normal lessons from how societies work and how politics works, that just kind of, yeah, that strikes me as fanciful.

• 1:28:51Speaker 00

  I think we should try to hook these things into, even when they're very powerful, again, after we've mitigated the safety issues, like, any good future, even if it has all these security issues that we need to solve, it somehow needs to end with something that's more decentralized and less like a godlike super. I just don't think that ends well.

• 1:29:17Dwarkesh Patel?

  What scientist from the Manhattan Project do you respect most in terms of they acted most ethically under the constraints they were given? Is there one that comes to mind?

• 1:29:26Speaker 00

  I don't know. I think there's a lot of answers you could give. I mean, I'm definitely a fan of Zillard for having figured it out. He was then against the actual dropping of the bomb. I don't actually know the history well enough to have an opinion on whether demonstration of the bomb could have ended the war. I mean, that involves a bunch of facts about Imperial Japan that are complicated and that I'm not an expert on. But Zillard seemed to, he discovered this stuff early, he kept it secret, patented some of it and put it in the hands of the British Admiralty. So he seemed to display

• 1:30:08Speaker 00

  The right kind of awareness as well as discovering stuff. I mean, it was when I read that book that I kind of, you know, when I wrote this big blob of compute doc and many other, you know, I only showed it to a few people and there were other docs that I showed to almost no one. So, you know, I, yeah, I was a bit, a bit inspired by this. Again, I mean, I, you know, we can all get self-aggrandizing here, like, we don't know how it's gonna turn out, or if it's actually gonna be, actually gonna be something on par with the Manhattan Project. I mean, you know, this, this could all be just Silicon Valley people building technology and, you know, just kind of like having delusions of grandeur, so I don't know how it's gonna turn out.

• 1:30:44Dwarkesh Patel?

  I mean, if the scaling stuff is true, then it's more bigger than the Manhattan Project.

• 1:30:48Speaker 00

  Yeah, yeah, it certainly, it certainly could be bigger. I, I just, you know, we should always kind of, I don't know, maintain this attitude that it's, it's really easy to fool yourself.

• 1:30:56Dwarkesh Patel?

  If you were asked by the government, if you're a physicist during World War II and you were asked by the government to contribute non-replaceable research to the Manhattan Project, well, what do you think you would have said?

• 1:31:05Speaker 00

  Yeah, I mean, I think, given you're in a war with the Nazis, at least during the period when you thought that the Nazis were, I, I don't, yeah, I don't really see much choice but, but, but to do it, if it's possible, you know, you have to figure it's gonna be done within, within 10 years or so by someone.

• 1:31:21Dwarkesh Patel?

  Regarding cybersecurity, what should we make of the fact that there's a whole bunch of tech companies which have ordinary tech company security policy that publicly seeming, facing, it's not obvious that they've been hacked, like Coinbase still has its Bitcoin, you know, Google, as far as I know, my Gmail hasn't been leaked. Should we take from that, that current status quo tech company security practices are good enough for AGI, or just simply that nobody has tried hard enough?

• 1:31:48Speaker 00

  It would be hard for me to speak to, you know, current tech company practices, and of course, there may be many attacks that we don't know about, where things are stolen and then silently used. You know, I mean, I think an indication of it is when someone really cares. Basically cares about attacking someone, then often the attacks happen. So, you know, recently, we saw that some fairly high officials of the US government had their email accounts hacked via Microsoft. Microsoft was providing the email accounts. So presumably, that related to information that was of great interest to foreign adversaries. And so, it seems to me, at least, that the evidence is more consistent with when something

• 1:32:32Speaker 00

  is really high enough value, then someone acts and it's stolen. And my worry is that, of course, with AGI, we'll get to a world where the value is seen as incredibly high, right? It'll be like stealing nuclear missiles or something. You can't be too careful on this stuff. And at every place that I've worked, I've pushed for the cybersecurity to be better. One of my concerns about cybersecurity is it's not kind of something you can trumpet. I think a good dynamic with safety research is like, you can get companies into a dynamic, and I think we have, where you can get them to compete to do the best safety research and kind of use it as a, I don't know, like a recruiting point of competition or something.

• 1:33:17Speaker 00

  We used to do this all the time with interpretability. And then sooner or later, other orgs started recognizing the defect and started working on interpretability, whether or not that was a priority to them before. But I think it's harder to do that with cybersecurity because a bunch of the stuff you have to do in quiet. And so, we did try to put out one post about it, but I think mostly, you just see the results. I think people should, a good norm would be, people see these cybersecurity leaks from companies, or leaks of the model parameters or something, and say, "They screwed up." That's bad. If I'm a safety person, I might not want to work there. Of course, as soon as I say that, we'll probably have a security breach tomorrow.

• 1:34:02Speaker 00

  But that's part of the game here, right? I think that's part of trying to make things safe.

• 1:34:08Dwarkesh Patel?

  I want to go back to the thing we were talking about earlier, where the ultimate Level of cybersecurity required for two to three years from now? And whether it requires a bunker, like, are you actually expecting to be in a physical bunker in two to three years? Or is that just a metaphor?

• 1:34:21Speaker 00

  Yeah, I mean, I think I think that's a metaphor. You know, we're still figuring it out like something I would think about is like, I think security of the data center, which may not be in the same physical location as us. But, you know, we've worked very hard to make sure it's in the United States. But securing the physical data centers and the GPUs. I think some of the really expensive attacks, if someone was really determined, just involve going into the data center and just, you know, trying to steal steal the data directly or as it's flowing from a data center to, you know, to to us. I think these data centers are going to have to be built in a very special way. I mean, given the way things are scaling up, you know, we're probably anyway heading to a world where, you know, the, you know, networks of data centers, you know, cost as much as aircraft carriers or something.

• 1:35:06Speaker 00

  Um, and and and so, you know, they're they're already going to be pretty unusual objects. But I think in addition to being unusual in terms of their ability, you know, to to link together and train gigantic, gigantic models, they're also going to have to be very secure.

• 1:35:19Dwarkesh Patel?

  So speaking of which, how, you know, there there's been sorts of rumors on the difficulty of procuring the power and the GPUs for the next generation of models. What has the process been like to secure the the necessary components to do the next generation?

• 1:35:32Speaker 00

  That's something I can't go into great detail about. Uh, you know, I I I will say, look, like, you know, people think of even industrial scale data centers, right? People are not thinking at the scale that I think these models are going to go to very soon. And so whenever you do something at a scale where it's never been done before, you know, every every single component, every single thing has to be done in a new way than it was before. And so, you know, you may, you may, you may run into problems with, you know, surprisingly simple components. Power is one that you mentioned.

• 1:36:02Dwarkesh Patel?

  And is this something that Anthropic has to handle or could you just outsource it?

• 1:36:05Speaker 00

  You know, I mean, for data centers, we work with cloud providers, for instance.

• 1:36:09Dwarkesh Patel?

  What should we make about the fact that these models require so much training and the entire corpus of internet data in order to be subhuman? Whereas, you know, if GPT four, there's been estimates that, you know, it was like a 10 to the 25 flops or something. Where, you know, whereas you, I mean, you can take these numbers with a grain of salt, but there's reports that, you know, human brain from the time it is born to the time a human being is 20 years old, that's like on the order of 10 to the 20 flops to simulate all those interactions. You don't have to go into the particulars on those numbers, but should we be worried about how sample inefficient these models seem to be?

• 1:36:46Speaker 00

  Yeah, so I think that's one of the remaining mysteries. One way you could phrase it is that the models are maybe two to three orders of magnitude smaller than the human brain if you compare to the number of synapses, while at the same time being trained on, you know, three to four more orders of magnitude of data. If you compare to, you know, number of words human sees as they're developing to age 18, it's, I don't remember exactly, but I think it's in the hundreds of millions. Whereas for the models, we're talking about the hundreds of billions, the trillions. So what, what explains this? There are these offsetting things where the models are smaller, they need a lot more data, and they're still below human level.

• 1:37:27Speaker 00

  But so, you know, there's some way in which, you know, the analogy to the brain is not quite right, or is breaking down, or there's some, there's some missing factor. You know, this is just kind of like in physics where it's like, you know, we can't explain the Michelson Morley experiment, or like, I'm forgetting one of the other 19th century physics paradoxes, but like, I think it's one thing we don't quite understand, right? Humans see so little data and they still do fine. One theory on it, it could be that it, you know, it's, it's like our other modalities. You know, how do we get, you know, 10 to the 14th bits into the human brain? Well, well, most of it is kind of these images and maybe a lot of what's going on inside the human brain is like, you know, our mental workspace involves all these, these, you

• 1:38:11Speaker 00

  know, these, these simulated images or something like that. But honestly, I think intellectually, we have to admit that that's a weird thing that doesn't match up. And, you know, it's one reason I'm a bit, you know, skeptical of kind of biological analogies. I thought in terms of them like five or six years ago, but now that we actually have these models in front of us as artifacts, it feels like almost all the evidence from that has been screened off by what we've seen. And what we've seen are models that are much smaller than the human brain and yet, yet can do a lot of the things that humans can do, and yet paradoxically require a lot more data. So maybe we'll discover something that makes it all efficient, or maybe we'll understand why the discrepancy is present. At the end of the day, I don't think it matters, right? If we keep scaling the way we are.

• 1:38:54Speaker 00

  I think what's more relevant at this point is just measuring the abilities of the model and seeing how far they are from humans, and they don't seem terribly far to me.

• 1:39:02Dwarkesh Patel?

  Does this scaling picture and the big blob of compute more generally? Does that under emphasize the role that algorithmic progress has played? When you compose the the big blob of compute? So you know, you're talking about LSTMs presumably at that point. Presumably the scaling on that would not have you at cloud two at this point. So are you under emphasizing the role that an improvement of the scale of transformer could be having here when you put it up behind the label of scaling?

• 1:39:28Speaker 00

  This big blob of compute document, which I still have not made public. I probably should for like historical reasons. I don't think it would tell anyone anything they don't know now. But when I wrote it, I actually said, look, there are seven factors that, and you know, I wasn't, I wasn't like, these are the factors. But I was just like, let me give some sense of the kinds of things that matter and what don't. And so I wasn't thinking like, these are the, you know, there could be nine, there could be five, but like, the things I said were, I said, number of parameters, scale of the model, like, you know, the compute and compute matters. Quantity of data matters. Quality of data matters. Loss function matters. So like, you know, are you doing RL or are you doing next word prediction? If your loss function isn't rich or doesn't incentivize the right thing, you won't, you won't get anything.

• 1:40:14Speaker 00

  So those were the key four ones, which I think are the core of the hypothesis. But then I said three more things. One was symmetries, which is basically like, if your architecture doesn't take into account the right kinds of symmetries, it doesn't work, or it's, it's very inefficient. So for example, convolutional neural networks take into account translational symmetry. LSTMs take into account time symmetry. And but a weakness of LSTMs is that they can't attend over the whole context. So there's kind of this structural weakness. Like if a model isn't structurally capable of like, absorbing and managing things that happened in a far enough distant past, then it's just like, it's kind of like, you know,

• 1:40:57Speaker 00

  like the compute doesn't flow, like the spice doesn't flow. It's like, you can't like, like, the blob has to be unencumbered, right? It kind of, it's not, it's not going to work if, if you artificially close things off. And I think RNNs and LSTMs artificially close things off because they, they close you off to the distant past. And so, again, things need to flow freely. If they don't, it doesn't work. And then, you know, I added a couple of things. One of them was like conditioning, which is like, you know, if you're, if the thing you're optimizing with is just really numerically bad, like you're going to have trouble. And so this is why like Adam works better than, you know, than normal SDD. And I think I'm forgetting what the seventh condition was, but it was, it was similar to things like this where it's like,

• 1:41:42Speaker 00

  you know, if you, if you, if you set things up in kind of a way that's, that's set up to fail, or that doesn't allow the compute to work in an uninhibited way, then it, then it won't work. And so transformers were kind of within that, even though I can't remember if the transformer paper had been published. It was around the same time as I wrote that document. It might've been just before, it might've been just after.

• 1:42:03Dwarkesh Patel?

  It sounds like from that view, that the, the, the way to think about these algorithmic progresses is not as increasing the power of the blob of compute, but simply getting rid of the artificial hindrances that older architectures have. Is that, is that a fair way to?

• 1:42:17Speaker 00

  That's, that's a little, that, yeah, that's, that's a little how I think about it, you know? Again, if you go back to like, Ilya's like, the models want to learn.

• 1:42:23Dwarkesh Patel?

  Yeah, yeah.

• 1:42:24Speaker 00

  Like, like the compute wants to be free.

• 1:42:26Dwarkesh Patel?

  Yeah, yeah, yeah.

• 1:42:26Speaker 00

  And like, you know, it's being blocked in various ways where you like, don't understand that it's being blocked. And so you need to like free it up.

• 1:42:32Dwarkesh Patel?

  Right, right. I, I love the, the irradiance, changing it to spice, okay. On that point though, so do you think that another thing on the scale of a transformer is coming down the pike to enable the next, the next great iteration?

• 1:42:49Speaker 00

  I think it's possible. I mean, people have worked on things like, you know, trying to model. Very long time dependencies or you know, there's various different ideas where I could see that we're kind of missing an efficient way of representing or dealing with something. So I think those inventions are possible. I guess my perspective would be even if they don't happen, we're all, we're already on this very, very steep trajectory. And so I'm less, I mean, we're constantly trying to discover them as are, as are others. But things are already on such a fast trajectory. All that would do is speed up the trajectory even more and probably, probably not by that much because it's already going so fast.

• 1:43:27Dwarkesh Patel?

  Is something embodied or having an embodied version of a model, is that at all important in terms of getting either data or progress?

• 1:43:35Speaker 00

  I think of that less in terms of the, you know, like a new architecture and more in terms of like a loss function, like the, the data, the environments you're exposing yourself to end up being very different. And, and so I think that could be important for learning some skills, although data acquisition is hard and so things have gone through the language route and I would guess will continue to go through the language route even as, you know, even, even as, as more as possible in terms of embodiment.

• 1:44:03Dwarkesh Patel?

  And then the other possibilities you mentioned, RL, you can see it as.

• 1:44:06Speaker 00

  Yeah, I mean, we, we kind of already do RL with RLHF, right? People are like, is it alignment, is it capabilities? I always think in terms of the two snakes, right? They're, they're kind of often hard to distinguish. So we, we already kind of use RL on these language models, but I think we've used RL less in terms of getting them to take actions and, you know, do things in the world. But, you know, when you take actions over a long period of time and understand the consequences of those actions only, only later than, you know, RL is a typical tool we have for that. So I would guess that in terms of models taking action in the world, that, that RL will, you know, will become a thing with all the power and all the safety issues that come with it.

• 1:44:43Dwarkesh Patel?

  When you project out in the future, do you see the way in which these things will be integrated into productive supply chains? Do you see them talking with each other and criticizing each other and contributing to each other's output? Or is it just the model of one shots, the one model of one shots, the, the answer or the work?

• 1:45:02Speaker 00

  Models will undertake extended tasks. That will have to be the case. I mean, we may want to limit that to some extent because it may make some of the safety problems easier. But, you know, some of that I think will be required. In terms of our models talking to models or are they talking to humans? Again, this goes kind of out of the technical realm and into the like. Socio-cultural economic realm where my heuristic is always that it's very, very difficult to predict things. And so I feel like these scaling laws have been very predictable. But then when you say like, well, you know, when, when is there going to be a commercial explosion in these models? Or what's the form it's going to be? Or are the models going to do things instead of humans or pairing with humans? I feel like certainly my track

• 1:45:47Speaker 00

  record on predicting these things is terrible. But I also looking around, I don't really see anyone who's track record is great.

• 1:45:53Dwarkesh Patel?

  You mentioned how fast progress is happening, but also the difficulties of integrating within the existing economy into the way things work. Do you think there will be enough time to actually have large revenues from AI products before the next model is just so much better? Or we're in like a different landscape entirely?

• 1:46:12Speaker 00

  It depends what you mean by by large, right? You know, I think multiple companies are already in the, you know, 100 million to billion per year range. What will it get to the 100 billion or trillion range, you know, before I that stuff is just so hard to predict, right? It's and it's, it's not even super well defined. Like, you know, I think right now there are companies that are throwing a lot of money at at generative AI, you know, as as customers, but and and they'll, you know, I think I think that's the right thing for them to do. And they'll, you know, they'll find uses for it. But it doesn't mean there doesn't mean it's, you know, they're finding uses or the best uses from day one. So even money changing hands is not is not quite the same thing as economic value being created.

• 1:46:54Dwarkesh Patel?

  But surely you've thought about this from the perspective of Anthropic, where if these things are happening so fast, then it should be an insane valuation, right?

• 1:47:01Speaker 00

  Even us who have, you know, not been super focused on commercialization and more on safety. I mean, you know, the graph goes up and it goes up, it goes up relatively quickly.

• 1:47:10Dwarkesh Patel?

  Yeah.

• 1:47:11Speaker 00

  So, you know, I can, I can only imagine what's happening at, you know, the the orgs or, you know, they're, this is, this is, this is their singular focus. So it's certainly happening fast. But, you know, again, it's, it's like, it's the exponential from the small base while the technology itself is moving fast. So it's, it's kind of a race between how fast the technology is getting better and how fast it's integrated into the economy. And that I think that's just a very unstable and turbulent. Processable things are going to happen fast, but if you ask me exactly how it's going to play out, exactly what order things are going to happen, I don't know and I'm kind of skeptical of the ability to predict.

• 1:47:50Dwarkesh Patel?

  I'm kind of curious with regards to Anthropic specifically.

• 1:47:53Speaker 00

  Yes.

• 1:47:53Dwarkesh Patel?

  Your public benefit corporation.

• 1:47:54Speaker 00

  Yes.

• 1:47:55Dwarkesh Patel?

  And rightfully so, you want to make sure that this is an important technology. Obviously, the only thing you want to care about is not shareholder value. But how do you talk to investors who are putting in like hundreds of millions, billions of dollars of money? Like, how do you talk to them about the fact that, how do you get them to put in this amount of money without

• 1:48:12Speaker 00

  Yes.

• 1:48:13Dwarkesh Patel?

  the shareholder value being the main concern?

• 1:48:15Speaker 00

  So, so I think the L T B T is, is, you know, the right thing on this, right? You know, I mean, we're going to talk more about the L T B T, but like some version of that has been in development since the beginning of Anthropic, even formally, right? And so, you know, from, from the beginning, you know, even as the body has, has changed in some ways, it's like from the beginning, it was like, this body is going to exist. And it's, you know, it's unusual, like every traditional investor who invests in Anthropic, you know, has to, you know, looks at this. Some of them are just like, whatever, you run your company how you want. Some of them are like, you know, oh my God, like this, this, you know, this body of random people or to them, random people could like, you know, could, could move Anthropic in a direction that's, you know, that's totally contrary to our, and now there are, there

• 1:49:00Speaker 00

  are legal limits on that, of course, but, you know, we have to have this conversation with every investor and then it gets into a conversation of, well, what are the kinds of things that, you know, that we would, we, we might do that would be contrary to the, to the, you know, to the interests of, of traditional investors? And just having those conversations has helped get everyone on the same page.

• 1:49:19Dwarkesh Patel?

  I want to talk about the physics and the fact that so many of the founders and the employees at Anthropic are physicists. What is the, I mean, we talked in the beginning about the scaling laws and how the power laws from physics are something you see here, but you know, what are the actual like approaches and ways of thinking from physics that seem to have carried over so well? Is that notion of effective theory is super useful? What, you know, what is going on here?

• 1:49:43Speaker 00

  I mean, I think part of it is just physicists learn things really fast. We have generally found that, you know, if we hire, you know, someone who is a, you know, physics PhD or something that they can, they can learn M L and contribute just very, very quickly in most cases. And, you know, because several of our founders, myself, Jared Kaplan, Sam McCandlish.

• 1:50:04Speaker 00

  We're physicists, we knew a lot of other physicists, and so we were able to hi- hire them. And now there's, I don't know how many, it's, you know, might be thirty or forty of them here. ML is not, still not yet, a field that has an enormous amount of depth. And so they've been able to get up to speed very quickly.

• 1:50:18Dwarkesh Patel?

  Are you concerned that there's like a lot of people who would have been doing physics or something, whatever, they go into finance instead? And since Anthropic exists, they have now been recruited to go into AI. And, you know, they're, you obviously care about AI safety, but, you know, maybe in the future, they leave and they get funded to do their own thing. Is that a concern that you're bringing more people into the ecosystem here?

• 1:50:42Speaker 00

  Yeah, I mean, you know, I think there's, there's like a broad set of action, you know, like we're causing GPUs to exist, or, you know, there's, there's a lot of kind of side effects that you can't, that you can't currently control, or that you just incur if you buy into the idea that you need to build frontier models. And that's one of them, a lot of them would have happened anyway. I mean, finance was a hot thing twenty years ago, so physicists were doing it. Now ML is a hot thing, and, you know, it's not like we caused them to do it when they had no interest previously, but, you know, again, you know, at, at the margin, you're kind of, you're kind of bidding things up. And, you know, a lot of that would have happened anyway, some of it, some of it wouldn't, but it's all part of the calculus.

• 1:51:18Dwarkesh Patel?

  Do you think that cloud has conscious experience? How likely do you think that is?

• 1:51:22Speaker 00

  This is another of these questions that just seems very unsettled and uncertain. Uh, one thing I'll tell you is I used to think that we didn't have to worry about this at all until models were kind of like operating in rich environments, like not necessarily embodied, but like, that, you know, they, you know, they needed to like have a reward function and like have kind of long-lived experience. So I still think that might be the case, but the more we've looked at kind of these language models, and particularly looked inside them to see things like induction heads, a lot of the cognitive machinery that you would need for active agents seems kind of already present in the base language models. So I'm not quite as sure as I was before that we're missing the things that, you know, that we're missing enough of the things that you would need.

• 1:52:06Speaker 00

  I think today's models just probably aren't smart enough that we should worry about this too much, but I'm not one hundred percent Sure about this, and I do think the models will get in a year or two like this might be a very real concern.

• 1:52:19Dwarkesh Patel?

  What would change if you found out that they are conscious? Are you worried that you're pushing the negative gradient to suffering? Like, what is?

• 1:52:25Speaker 00

  Conscious is, again, one of these words that I suspect it will like not end up having a well-defined meaning.

• 1:52:31Dwarkesh Patel?

  But it's like something to be clawed.

• 1:52:32Speaker 00

  Yeah, but that, yeah, well, I suspect that's a spectrum, right? So I don't know if we discover like that, you know, that I should care about Claude. Let's say we discover that I should care about Claude's experience as much as I should care about like a dog or a monkey or something. Yeah, I would be, I would be kind of worried. I don't know if their experience is positive or negative. Unsettlingly, I also don't know, like, I wouldn't know if any intervention that we made was more likely to make Claude, you know, have a positive versus negative experience versus not having one. If there's an area that is helpful with this, it's maybe mechanistic interpretability. Because I think of it as neuroscience for models.

• 1:53:12Speaker 00

  And so it's possible that we could shed some light on this, although, you know, it's not a straightforward factual question, right? It kind of depends what we mean and what we value.

• 1:53:23Dwarkesh Patel?

  We talked about this initially, but I want to get more specific. We talked initially about, you know, now that you're seeing these capabilities ramp up within the human spectrum, you think that the human spectrum is wider than we thought.

• 1:53:34Speaker 00

  But yeah.

• 1:53:35Dwarkesh Patel?

  More specifically, what have you, how is the way you think about human intelligence different? Now that the way you're seeing these marginal useful abilities emerge, how does that change your picture of what intelligence is?

• 1:53:47Speaker 00

  I think for me, the big realization on what intelligence is came with the like blob of compute thing, right? Like, it's not, you know, there might be all these separate modules, there might be all this complexity. You know, it's, you know, Rich Sutton called it the bitter lesson, right? It's almost called, has many names, been called the scaling hypothesis. Like, the first few people who figured it out was around 2017. I mean, you could go further back to, I think, I think Shane Legg was maybe the first person who really knew it. Maybe Ray Kurzweil, although in a very vague way. But, you know, I think the number of people who understood it went up a lot around 2014 to 2017. But I think that was the big realization. It's like, you know, well, how did intelligence evolve?

• 1:54:30Speaker 00

  Well, if you don't need very specific conditions to create it, if you can create it just from like, The right, kind of the right kind of gradient and loss signal. Then, of course, it's not so mysterious how it all happened in terms, you know, it had this click of scientific understanding. In terms of like, watching what the models can do, how has it changed my view of human intelligence? I wish I had something more intelligent to say on that. Uh, I, I feel like, I don't know, one thing that's been surprising is like, I thought things might click into place a little more than they do. Like, you know, I thought like different cognitive abilities might all be connected and there was more of one secret behind them, but it's, it's like,

• 1:55:12Speaker 00

  the model just learns various things at different times, you know, and it can be like very good at coding, but like, you know, it can't, it can't quite, you know, prove the prime number theorem yet. And I don't, I mean, I guess it's a little bit the same for, for humans, although it's, it's weird, the juxtaposition of things it can do and not. I guess the main lesson is like, having theories of intelligence or how intelligence works, like, again, a lot of these words just, just kind of like dissolve into a continuum, right? They, they just kind of like, de-materialize. I think less in terms of intelligence and more in terms of what, what we see in front of us.

• 1:55:46Dwarkesh Patel?

  Yeah, no, it's really surprising to me. Uh, two things, one is how discreet these like different paths of intelligent, um, uh, things that contribute to loss are, rather than just being like one reasoning circuit or one general intelligence. And the other thing, talking with you, that is surprising or interesting is, many years from now, it'll be one of those things that, looking back, it'll be, why did, why weren't, why wasn't this obvious to you? If you're seeing these smooth scaling curves, why the time where you're not completely convinced? So you've been less public than the CEOs of other AI companies, you know, you're not posting on Twitter, you're not, uh, doing a lot of podcasts, except for this one. What, what, what, what gives? Like, why are you, why are you off the radar?

• 1:56:27Speaker 00

  Yeah, I aspire to this and I'm proud of this. If people think of me as kind of like boring and low profile, like this is actually kind of what I want. So I don't know, I've just seen a number of cases, a number of people I've worked with. That I think you could say Twitter, although I think I mean a broader thing, like just kind of like attaching your incentives very strongly to like the approval or cheering of a crowd. I think that can destroy your mind and in some cases it can destroy your soul. And so I think I kind of deliberately tried to be a little bit low profile because I want to, I don't know, kind of like defend my ability to think about things intellectually in a way that's

• 1:57:08Speaker 00

  different from other people and isn't kind of tinged by the approval of other people. So you know, I've seen cases of folks who are deep learning skeptics and they become known as deep learning skeptics on Twitter. And then even as it starts to become clear to me, they kind of sort of changed their mind. They like, this is their thing on Twitter and they can't change their Twitter persona and so forth and so on. I don't really like the trend of kind of like personalizing companies, like the whole, you know, like cage match between CEOs approach, like I think it distracts people from the actual merits and concerns of like the, you know, the company in question. Like I kind of want people to like judge the like nameless bureaucratic institution.

• 1:57:52Speaker 00

  You know, I want people to think in terms of the nameless bureaucratic institution and its incentives more than they think in terms of me. Everyone wants a friendly face, but, but actually I think friendly faces can be misleading.

• 1:58:03Dwarkesh Patel?

  Okay. Well, in this case, this will be a misleading interview because this has been a lot of fun. You're like a blast to talk to.

• 1:58:09Speaker 00

  Indeed.

• 1:58:10Dwarkesh Patel?

  Yeah, this is a blast. I'm super glad you came on the podcast and I hope you'll enjoy it.

• 1:58:14Speaker 00

  Thanks. Thanks for having me.

• 1:58:17Dwarkesh Patel?

  Hey everybody. I hope you enjoyed that episode. As always, the most helpful thing you can do is to share the podcast. Send it to people you think might enjoy it, put it in Twitter, your group chats, et cetera. Just blitz the world. Appreciate your listening. I'll see you next time. Cheers.