Sholto Douglas & Trenton Bricken — How LLMs actually think

Had so much fun chatting with my good friends Trenton Bricken and Sholto Douglas on the podcast. No way to summarize it, except: This is the best context dump out there on how LLMs are trained, what capabilities they're likely to soon have, and what exactly is going on inside them. You would be shocked how much of what I know about this field, I've learned just from talking with them. To the extent that you've enjoyed my other AI interviews, now you know why. So excited to put this out. Enjoy! I certainly did :) Watch on YouTube https://www.youtube.com/DwarkeshPatel. Listen on Apple Podcasts https://podcasts.apple.com/us/podcast/sholto-douglas-trenton-bricken-how-to-build-understand/id1516093381?i=1000650748087, Spotify https://open.spotify.com/episode/2dtDauiE4v8ldNRqPFq0uP?si=7S4n69QuTjeYz0lZwW4xIw, or any other podcast platform. There's a transcript https://www.dwarkeshpatel.com/i/143040987/transcript with links to all the papers the boys were throwing down - may help you follow along. Follow Trenton https://twitter.com/TrentonBricken and Sholto https://twitter.com/_sholtodouglas on Twitter. Timestamps (00:00:00) - Long contexts (00:16:12) - Intelligence is just associations (00:32:35) - Intelligence explosion & great researchers (01:06:52) - Superposition & secret communication (01:22:34) - Agents & true reasoning (01:34:40) - How Sholto & Trenton got into AI research (02:07:16) - Are feature spaces the wrong way to think about intelligence? (02:21:12) - Will interp actually work on superhuman models (02:45:05) - Sholto’s technical challenge for the audience (03:03:57) - Rapid fire This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.dwarkesh.com https://www.dwarkesh.com?utm_medium=podcast&utm_campaign=CTA_1

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Dwarkesh Patel?

  Okay, today I have the pleasure to talk with two of my good friends, Sholto and Trenton.

• 0:06Sholto Douglas?

  He trusts me so.

• 0:11Dwarkesh Patel?

  I wasn't gonna say anything.

• 0:14Dwarkesh Patel?

  Let's do this in reverse.

• 0:17Dwarkesh Patel?

  How long I started with my good friends? Yeah, Gemini 1.5, the context length, just wow.

• 0:27Dwarkesh Patel?

  Shit, anyways, Sholto, Noah Brown.

• 0:34Dwarkesh Patel?

  Noah Brown, the guy who wrote the diplomacy paper, he said this about Sholto. He said, "He's only been in the field for 1.5 years, but people in AI know that he was one of the most important people behind Gemini's success." And Trenton, who's at Anthropic, works on mechanistic interoperability, and it was widely reported that he has solved alignment.

• 0:59Dwarkesh Patel?

  So this will be a capabilities-only podcast, alignment is already solved, so no need to discuss further. Okay, so let's start by talking about context lengths.

• 1:08Sholto Douglas?

  Yep.

• 1:10Dwarkesh Patel?

  It seemed to be underhyped, given how important it seems to be, to be that you can just put a million tokens into context. There's apparently some other news that, you know, got pushed to the front for some reason, but yeah. Tell me about how you see the future of long context lengths and what that implies for these models.

• 1:26Sholto Douglas?

  Yeah, so I think it's really underhyped, because until I started working on it, I didn't really appreciate how much of a step up in intelligence it was for the model to have the onboarding problem basically instantly solved. And you can see that a little bit in the perplexity graphs in the paper, where just throwing millions of tokens worth of context about a codebase allows it to become dramatically better at predicting the next token in a way that you'd normally associate with huge increments in model scale. But you don't need that, all you need is like a new context. So, underhyped, and buried by some other news.

• 1:56Dwarkesh Patel?

  In context, are they as sample efficient and smart as humans?

• 2:01Sholto Douglas?

  I think that's really worth exploring. For example, one of the evals that we did in the paper has it learning a language in context better than a human expert could learn that new language over the course of a couple of months. This is only like a pretty small demonstration, but I'd be really interested to see things like Atari games or something like that, where you like throw in a couple hundred, like a thousand frames, labeled actions, and then in the same way that you'd like show your friend how to play a game.

• 2:25Dwarkesh Patel?

  Right.

• 2:25Sholto Douglas?

  And see if it's able to reason through. It might, at the moment, you know, with the infrastructure stuff, it's still a little bit slower, like

• 2:30Dwarkesh Patel?

  Yeah.

• 2:31Sholto Douglas?

  doing that, but I would actually, I would guess that might just work out of the box in a way that would be pretty mind blowing.

• 2:37Dwarkesh Patel?

  And crucially, I think this language was esoteric enough that it wasn't in the training data.

• 2:41Sholto Douglas?

  Right, exactly. Yeah. If you look at the model before it has that context thrown in, it just, it doesn't know the language at all and it can't get any translation.

• 2:48Dwarkesh Patel?

  And this is like an actual, like, human language, not just, okay.

• 2:50Sholto Douglas?

  Exactly, an actual human language.

• 2:52Dwarkesh Patel?

  So if this is true, it seems to me that these models are already, in an important sense, superhuman. Not in the sense that they're smarter than us, but I can keep a million tokens in my context when I'm trying to solve a problem, remembering and integrating all the information in our entire codebase.

• 3:10Dwarkesh Patel?

  Am I wrong in thinking this is like a huge unlock?

• 3:12Sholto Douglas?

  I actually generally think that's true. Like previously, I've been frustrated when models aren't as smart. Like you ask them a question and you want it to be smarter than you or to know things that you don't. And this allows them to know things that you don't in a way that it just ingests a huge amount of information in a way you just can't. So, yeah, it's extremely important.

• 3:31Dwarkesh Patel?

  Well, how do we explain in-context learning?

• 3:34Sholto Douglas?

  Yeah. So there's a piece of, there's a line of work I quite like, where it looks at in-context learning as basically like very similar to gradient descent, but like the attention operation can be viewed as gradient descent on the in-context data. That paper had some cool plots where they basically showed like we take N steps of gradient descent and that looks like N layers of in-context learning. It looks very similar. So I think like that's one way of viewing it and trying to understand what's going on.

• 3:58Dwarkesh Patel?

  Yeah.

• 3:58Trenton Bricken?

  And you can ignore what I'm about to say because given the introduction alignment is solved. AI safety isn't a problem. But I think the context stuff does get problematic, but also interesting here. I think there'll be more work coming out in the not too distant future.

• 4:15Trenton Bricken?

  Around what happens if you give a hundred shot prompt for jailbreaks adversarial attacks. It's also interesting in the sense of if your model is doing gradient descent and learning on the fly, even if it's been trained to be harmless. You're dealing with a totally new model in a way. You're like fine tuning, but in a way where you can't control what's going on.

• 4:40Dwarkesh Patel?

  Can you explain what do you mean by gradient descent is happening in the forward pass and attention?

• 4:44Trenton Bricken?

  Yeah, there was no, no, no, there was something in the paper about trying to teach the model to do linear regression.

• 4:49Dwarkesh Patel?

  Right.

• 4:49Trenton Bricken?

  But like just through the number of samples they gave in the context.

• 4:52Dwarkesh Patel?

  Yeah.

• 4:52Trenton Bricken?

  And you can see if you plot on the x axis, like number of shots that it has or examples. And then like the loss it gets on just like ordinary least squares regression.

• 5:01Dwarkesh Patel?

  Yeah.

• 5:01Trenton Bricken?

  That will go down with time.

• 5:03Dwarkesh Patel?

  And it goes down exactly matched with a number of gradient descent steps.

• 5:06Trenton Bricken?

  Yeah.

• 5:07Dwarkesh Patel?

  Yeah, exactly. Okay. I only read the interim discussion section of that paper, but in the discussion, the way they framed it is that the model, in order to get better at long context tasks, the model has to get better at learning to learn from these examples or from the context that is already within the window. And that the implication of that is the model learn, if like meta learning happens because it's has to learn how to get better at long context tasks. Then in some important sense, the task of intelligence is like requires long context examples and long context training.

• 5:44Sholto Douglas?

  Like meta learn, like you have to induce meta learning. Right. Understanding how to better induce meta learning in your pre-training process is like a very important thing to actually about. Flexible or adaptive intelligence.

• 5:52Dwarkesh Patel?

  Right, but you can proxy for that just by getting better at doing long context tasks. One of the bottlenecks for AI progress that many people identify is the inability of these models to perform tasks, tasks on long horizons. Which means engaging with a task for many hours or even many weeks or months where like if I have, I don't know, an assistant or an employee or something, they can just do a thing I tell them for a while. And AI agents haven't taken off for this reason, from what I understand. So how linked are long context windows and the ability to perform well on them? And the ability to do these kinds of long horizon tasks that require you to engage with an assignment for many hours? Or are these unrelated concepts?

• 6:35Sholto Douglas?

  Uh, I mean, I would actually take issue with the that being the reason that agents haven't taken off. Um, where I think that's more about like nines of reliability and the model actually successfully doing things. If you just can't chain tasks successfully with high enough probability, then you won't get something that looks like an agent. And that's why something like an agent might follow more of a step function in sort of usually like G P T four class models, general ultra class models, they're not enough. Um, but maybe like the next increment on model scale means that you get that extra nine, even though the like the loss isn't going down that dramatically, that like small amount of extra ability gives you the extra, and like, yeah, obviously you need some amount of context to fit long horizon tasks, but I don't think that's been the limiting factor up to now.

• 7:15Dwarkesh Patel?

  Yeah, um, the NERUP's best paper this year by Ryland Schaefer was the lead author, points to this as like the emergence of mirage, where people will have a task and you get the right or wrong answer depending on if you've sampled the last five tokens correctly. And so naturally that's you're multiplying the probability of sampling all of those. Um, and and if you don't have enough nines of reliability, then you're not going to get emergence. And all of a sudden you you do. And it's like, oh, my gosh, this ability is emergent, when when actually it was kind of almost there to begin with.

• 7:46Sholto Douglas?

  And there are ways that you can find like a smooth metric for that.

• 7:49Dwarkesh Patel?

  Yeah, human eval or whatever, the G P T four paper, they the the coding problems, they had, they measured.

• 7:55Sholto Douglas?

  Log pass rate.

• 7:55Dwarkesh Patel?

  Exactly, yeah. Uh, for the audience, the context on this is, uh, you, it's basically the idea is you wanna, uh, when you're measuring how much progress there has been on a specific task, like solving coding problems, you, um, you upweighted when it gets it right only one in a thousand times. You don't like give it a one in a thousand score because it's like, oh, like gotta write some of the time. And so the curve you see is like it gets it right one in a thousand, then one in a hundred, then one in ten and so forth. So actually, I want to follow up on this. So if your claim is that the AI agents haven't taken off because of reliability rather than long horizon task performance, isn't the Lack of reliability when a task is chained on top of another task, on top of another task.

• 8:39Dwarkesh Patel?

  Isn't that exactly the difficulty with long horizon tasks? Is that like you have to do 10 things in a row or 100 things in a row? And diminishing the reliability of any one of them. Or yeah, the probability goes down from 99.99 to 99.9. Then like the whole thing gets multiplied together and the whole thing becomes much less likely to happen.

• 8:58Sholto Douglas?

  That that is exactly the problem, but the the key issue you're pointing at there is that your base pass like task solve rate is 90%. And if it was 99%, then chain doesn't become a problem. But also, yeah, exactly. And I I think this is also something that just like hasn't been properly studied enough. If you look at all of the evals that are commonly like the academic evals are a single problem. Right? You know, like the math problem, it's like one typical math problem or MMU. It's like one university level, like like from from across different topics.

• 9:27Sholto Douglas?

  You were beginning to start to see evals looking at this properly by more complex tasks like Sweetbench. Where they take a whole bunch of GitHub issues and that's like, that is like a reasonably long horizon task. But it's still not a multi, it's like a multi sub hour as opposed to like multi hour or multi day task. And so I think one of the things that will be really important to do over the next, however, long is understand better. What does success rate of a long horizon task look like? And I think that's even important to understand what the economic impact to these models might be. And like actually properly judge increasing capabilities.

• 9:58Dwarkesh Patel?

  Right.

• 9:58Sholto Douglas?

  By like cutting down the tasks that we do and the inputs and outputs involved into minutes or hours or or days. And seeing how good it is successively chaining and completing tasks of those different resolutions of time. Because then that tells you like how automatable a job family or task family is in a way that like MMU scores don't.

• 10:18Dwarkesh Patel?

  I mean, it was less than a year ago that we introduced 100K context windows. And I think everyone was pretty surprised by that. So yeah, everyone would just kind of had this soundbite of quadratic attention costs. Yeah, we can't have long context windows. Yeah. Here we are. So yeah, like the benchmarks are being actively made. Wait, wait. So doesn't the fact that there's these companies, Google and I don't know, Magic, maybe others, who have million token attention imply that the quadratic, you shouldn't say anything because you're not allowed. But doesn't that like imply that it's not quadratic anymore? Or are they just eating the cost?

• 10:50Sholto Douglas?

  Well, like, who knows what Google is doing for its long context game, right?

• 10:53Dwarkesh Patel?

  Yeah, I'm not saying anything.

• 10:54Sholto Douglas?

  I'm not saying anything either. One of the things frustrated me about in like the general research field's approach to attention is that there's an important way in which the quadratic cost of attention is actually dominated in typical dense transformers by the MLP block.

• 11:10Dwarkesh Patel?

  Right.

• 11:11Sholto Douglas?

  So you have this N squared term that's associated with attention, but you also have an N squared term that's associated with the D model, the residual stream dimension of the model. And if you look, I think Sachar Rush has a great tweet where he looks like basically plots the curve of the cost of attention respective like the cost of like really large models. And attention actually trails off. And you actually need to be doing pretty long contexts before that that term becomes like really important. And the second thing is that people often talk about how attention at inference time is such a huge cost, right? And if you think about when you're actually generating tokens, the operation is not N squared. It is one Q, like one set of Q vectors, looks up a whole bunch of KV vectors.

• 11:54Sholto Douglas?

  And that's linear with respect to the amount of like context that the model has. And so I think this drives a lot of the like recurrence and state space research where people have this meme of, oh, like linear attention and all this stuff. And as Trenton said, there's like a graveyard of ideas around attention. And not to think, I don't think it's worth exploring, but I think it's important to consider where the actual like strengths and weaknesses of it are.

• 12:20Dwarkesh Patel?

  Okay, so what do you make of this take? As we move forward through the takeoff, more and more of the learning happens in the forward pass. So originally, like all the learning happens in the backward, you know, during like this like bottom up sort of hill climbing evolutionary process. If you think in the limit during the intelligence explosion, it just like the AI is like maybe like handwriting the weights or like doing GoFi or something. And we're in like the middle step where like a lot of learning happens in context now with these models. A lot of it happens within the backward process. Does this seem like a meaningful gradient along which progress is happening of like how much? Because the broader thing being the If you're learning in the forward pass, it's like much more sample efficient because you can kind of like basically think as you're learning.

• 13:05Dwarkesh Patel?

  Like when humans, when you read a textbook, you're not just skimming it and trying to absorb what, you know, what inductive, these words follow these words, you like read it and you think about it and then you read some more, you think about it. I don't know, does this seem like a sensible way to think about the progress?

• 13:21Sholto Douglas?

  Yeah, it may just be one of the ways in which like, you know, birds and planes like fly, but they fly slightly differently and like virtue of technology allows us to do that like,

• 13:31Sholto Douglas?

  I see the accomplished things that birds can't. It might be the context length is similar in that it allows it to have a working memory that we can't. But functionally is not like the key thing towards actual reasoning. The key step between G P T two and G P T three was that all of a sudden like there was this meta learning behavior that was observed in training, like, like in the pre-training of the model. And that's, as you said, like it's something to do with, you give it some amount of context, it's able to adapt to that context, and that was a behavior that wasn't really observed before that at all. And maybe that's a mixture of property of context and scale and this kind of stuff, but it wouldn't have occurred to model tiny context, I would say.

• 14:09Dwarkesh Patel?

  This is actually an interesting point, so when we talk about scaling up these models, how much of it comes from just making the models themselves bigger? And how much comes from the fact that during any single call, you are using more compute. So if you think of diffusion, you can just iteratively keep adding more compute. And if adaptive compute is solved, you can keep doing that. And in this case, if there's a quadratic penalty for attention, but you're doing long context anyways, then you're still dumping in more compute during, not during training, or not during having bigger models, but just like, yeah.

• 14:44Trenton Bricken?

  Yeah, it's interesting because you do get more forward passes by having more tokens.

• 14:48Dwarkesh Patel?

  Right.

• 14:50Trenton Bricken?

  My one gripe, I guess I have two gripes with this, though. Maybe three. So one, like in the Alpha. In the Alpha model paper, one of the transformer modules, they have a few and the architecture is like very intricate.

• 15:02Dwarkesh Patel?

  Yeah.

• 15:02Trenton Bricken?

  But they do, I think, five forwards passes through it. And will gradually like refine their solution as a result. You can also kind of think of the residual. I mean, Schulte alluded to kind of the read-write operations as like a poor man's adaptive compute. Where it's like, I'm just gonna give you all these layers and like, if you want to use them, great. If you don't, then that's also fine. And then people will be like, oh, well, the brain is recurrent and you can like, do however many loops through it you want. And I think to a certain extent, that's right, right? Like, if I ask you a hard question, you'll spend more time thinking about it. And that would correspond to more forward passes. But I think there's a finite number of forward passes that you can do. It's kind of with language as well. People are like, oh, well, human language can have like infinite recursion in it, like infinite nested statements.

• 15:44Trenton Bricken?

  Of like, the boy jumped over the bear that was doing this, that had done this, that had done that. But like empirically, you'll only see five to seven levels of recursion. Which kind of relates to whatever that magic number of like, how many things you can hold in working memory at any given time is. And so, yeah, it's not infinitely recursive, but like, does that matter in the regime of human intelligence? And like, can you not just add more layers?

• 16:11Dwarkesh Patel?

  Breakdown for me, you're referring to this in some of your previous answers of, listen, you have these long contexts and you can hold more things in memory. But like, ultimately comes down to your ability to mix concepts together, to do some kind of reasoning. And these models aren't necessarily human level at that, even in context. Breakdown for me how you see storing just raw information versus reasoning and what's in between? Like, where's the reasoning happening? Is that, where's just like storing raw information happening? What's different between them in these models?

• 16:44Trenton Bricken?

  Yeah, I don't have a super crisp answer for you here. I mean, obviously with the input and output of the model, you're mapping back to actual tokens, right? And then in between that, you're doing higher level processing.

• 17:00Dwarkesh Patel?

  Before we get deeper into this, we should explain to the audience, you referred earlier to Anthropic's way of thinking about transformers as these read-write operations that layers do. When have you should just kind of explain at a high level what you mean by that?

• 17:13Trenton Bricken?

  So the residual stream, imagine you're in a boat going down a river. And the boat is kind of the current query where you're trying to predict the next token. So it's the cat sat on the blank.

• 17:25Dwarkesh Patel?

  Right.

• 17:25Trenton Bricken?

  And then you have these little like streams that are coming off the river, where you can get extra passengers or collect extra information if you want. And those correspond to the attention heads and MLPs.

• 17:37Dwarkesh Patel?

  Yeah.

• 17:37Trenton Bricken?

  That are that are part of the model.

• 17:39Dwarkesh Patel?

  Right.

• 17:40Sholto Douglas?

  And okay, I almost think of it like the working memory of the model.

• 17:44Trenton Bricken?

  Yeah.

• 17:44Sholto Douglas?

  Like the RAM of the computer where you're like choosing what information to read in. So you can do something with it and then maybe you read like read something else in later on.

• 17:53Trenton Bricken?

  Yeah. And you can operate on subspaces of that high dimensional vector. A ton of things are, I mean, at this point, I think it's almost given that like are encoded in superposition. Right. So it's like, yeah, the residual stream is just one high dimensional vector. But actually, there's a ton of different vectors that are packed into it.

• 18:10Dwarkesh Patel?

  Yeah, I might like just like dumb it down like as a way that would have made sense to me a few months ago of, okay, so you have, you know, whatever words are in the input you put into the model. All those words get converted into these tokens and those tokens get converted into these vectors. And basically, it's just like the small amount of information that's moving through the model. And the way you explained it to me, Sheldon, this paper talks about is, early on in the model, maybe it's just doing some very basic things about like, what do these tokens mean? Like if it says like 10 plus five, just like moving information about to have the have that.

• 18:47Sholto Douglas?

  Good representation.

• 18:48Dwarkesh Patel?

  Exactly. Just represent in the middle, maybe like the deeper thinking is happening about like how to think, yeah, how to solve this. At the end, you're converting it back into the output token because the end product is, you're trying to predict the probability of the next token from the last of those residual streams. And so, yeah, it's interesting to think about like just like the small compressed amount of information moving through the model. And it's like getting modified in different ways. Trenton, so you're, it's interesting, you're one of the few people who have like, background from neuroscience. You can think about the analogies here to, yeah, to the brain. And in fact, I have one of our friends, the way he, he had a paper in grad school, about thinking about attention in the brain. And he said, this is the only or first.

• 19:35Dwarkesh Patel?

  What like neural explanation of why attention works, whereas we have evidence from why the CNNs work, convoluted neural networks work based on the visual cortex or something. Yeah, I'm curious, do you think in the brain, there's something like a residual stream of this compressed amount of information that's moving through and it's getting modified as you're thinking about something, even if that's not what's literally happening? Do you think that's a good metaphor for what's happening in the brain?

• 20:03Trenton Bricken?

  Yeah, yeah. So at least in the cerebellum, you basically do have a residual stream where the whole, what we'll call the attention model for now, and I can go into whatever amount of detail you want for that. You have inputs that route through it, but they'll also just go directly to the like endpoint that that that module will contribute to. So there's a direct path and an indirect path. And so the model can like pick up whatever information it wants and then add that back in.

• 20:33Dwarkesh Patel?

  What would happen to the cerebellum?

• 20:36Trenton Bricken?

  So the cerebellum nominally just does fine motor control. But I analogize this to the person who's lost their keys and is just looking under the streetlight, where it's very easily to observe this behavior. One leading cognitive neuroscientist said to me that a dirty little secret of any FMRI study where you're looking at brain activity for a given task is that the cerebellum is almost always active and lighting up for it. If you have a damaged cerebellum, you also are much more likely to have autism. So it's associated with like social skills. And one of these particular studies where I think they use PET instead of FMRI, but when you're doing next token prediction, the cerebellum lights up a lot. Also, 70% of your neurons in the brain are in the cerebellum.

• 21:22Trenton Bricken?

  They're small, but they're they're there and they're taking up real metabolic cost.

• 21:28Dwarkesh Patel?

  This was one of Grant's points that like what changed with humans was not just that we have more neurons, or he says, he shared this article, but specifically there's more neurons in the cerebral cortex and the cerebellum. And you should say more about this, but like they're they're more morbidecally expensive and they're more involved in signaling and sending information back and forth.

• 21:49Trenton Bricken?

  Yeah.

• 21:49Dwarkesh Patel?

  Well, is that attention what's going on?

• 21:51Trenton Bricken?

  Yeah, yeah. So I guess the the main thing I want to communicate here. So back in the 1980s, Penti Konrva came up with a associative memory algorithm for. I have a bunch of memories. I want to store them. There's some amount of noise or corruption that's going on. And I want to query or retrieve the best match. And so he writes this equation for how to do it. And a few years later realizes that if you implemented this as an electrical engineering circuit, it actually looks identical to the core cerebellar circuit. And that circuit and the cerebellum more broadly is not just in us. It's in basically every organism. There's active debate on whether or not cephalopods have it. They kind of have a different evolutionary trajectory.

• 22:34Trenton Bricken?

  But even fruit flies with the Drosophila mushroom body, that is the same cerebellar architecture. And so that convergence and then my paper, which shows that actually this operation is to a very close approximation, the same as the attention operation, including implementing the softmax and having this sort of like nominal quadratic costs that we've been talking about. And so the three way convergence here and the takeoff and success of transformers just seems pretty striking to me.

• 23:03Dwarkesh Patel?

  Yeah, I want to zoom out and ask, I think what motivated this discussion in the beginning was we were talking about like, wait, what is the reasoning? What is the memory? What do you think about the analogy you found to attention and this? Do you think of this as more just looking up the relevant memories or the relevant facts? And if that is the case, like, where is the reasoning happening in the brain? Yeah, how do we think about like how that builds up into the reasoning?

• 23:30Trenton Bricken?

  Yeah, so maybe my hot take here, I don't know how hot it is, is that like most most intelligence is pattern matching. And you can do a lot of really good pattern matching if you have a hierarchy of associative memories. So you have you start with your very basic associations between just like objects in the real world. But you can then chain those and have more abstract associations. Such as like a wedding ring symbolizes like so many other associations that are downstream. Uh, and so I, and you could even generalize uh the attention operation and this associative memory as the MLP layer as well.

• 24:13Trenton Bricken?

  And it's in a long-term setting where you don't have like tokens in your current context. Um, but I, I think this is an argument that like association, association is all you need. Um, and associative memory in general as well. It's not, so, so you can do two things with it. Um, you can both de-noise or retrieve a current memory. So like if I see your face, but it's like raining and cloudy. I can, I can de-noise and kind of like gradually update my query towards my memory of your face. But I can also um access that memory and then the, the value that I get out actually points to some other totally different part of the space. And so you, so a very simple instance of this would be um if you learn the alphabet, right?

• 24:55Trenton Bricken?

  And so I, I query for A and it returns B. I query for B and it returns C and, and you can traverse the whole thing. Um, yeah.

• 25:03Dwarkesh Patel?

  Yeah, I, that, one of the things I talked to Demis about was he had a paper in 2008 that memory and imagination are very linked. Because of this very thing that you mentioned of memory is reconstructive. And so you're in some sense imagining every time you're thinking of a memory because you're only storing a condensed version of it and you're like have to. And this is famously why human memory is terrible and like why people in the witness box or whatever would just make shit up. Um, okay, so let, let me ask a stupid question. So you like read Sherlock Holmes, right? And like the guy is incredibly sample efficient. He'll, he'll like see a few observations and he'll like, uh, uh, basically figure out who committed the crime because there's a series of deductive steps

• 25:46Dwarkesh Patel?

  that leads from somebody's tattoo and what's on the wall to uh the implications of that.

• 25:53Dwarkesh Patel?

  How does that fit into this picture? Because crucially, what makes them smart is that there's not like an association, but there's a sort of deductive connection between different pieces of information. Would you just explain it as that that's just like higher level association? Like, yeah.

• 26:10Trenton Bricken?

  I think so, yeah. So I think learning these higher level associations to be able to then map patterns to each other as kind of like a meta learning. I think in this case, he would also just have a really long context length or a really long working memory, right? Where he can like have all of these bits and continuously query them as he's coming up with whatever theory. So that the theory is moving through the residual stream. And then he's has his attention heads are querying his context.

• 26:38Dwarkesh Patel?

  Right.

• 26:38Trenton Bricken?

  But then how he's projecting his query and keys in the space and how his MLPs are then retrieving like longer term facts or modifying that information is allowing him to then in later layers, do even more sophisticated queries and slowly be able to reason through and come to a meaningful conclusion.

• 26:59Sholto Douglas?

  That feels right to me in terms of like, you're looking back in the past, you're selectively reading in certain pieces of information, comparing them, maybe that informs your next step of like what piece of information you now need to pull in. And then you build this representation, which I like progressively looks closer and closer and closer to like the suspect in your case.

• 27:14Trenton Bricken?

  Yeah.

• 27:14Sholto Douglas?

  Um, and that doesn't feel at all outlandish.

• 27:17Dwarkesh Patel?

  Do the Lotus lens on like.

• 27:22Trenton Bricken?

  Well, something I think that the people who aren't doing this research can overlook is after your first layer of the model, every query key and value that you're using for attention comes from the combination of all the previous tokens. So like my first layer, I'll query my previous tokens and just extract information from them. But all of a sudden, let's say that I attended to tokens one, two, and four in equal amounts. Then the vector in my residual stream, assuming that they just, they wrote out the same thing to the value vectors, but ignore that for a second, is a third of each of those. And so when I'm querying in the future, my query is actually a third of each of those things.

• 28:01Dwarkesh Patel?

  And so

• 28:02Sholto Douglas?

  But they might be written to different subspaces.

• 28:04Dwarkesh Patel?

  That's that's right, hypothetically, but they wouldn't have to, right? And so you can you can recombine and immediately, even by layer two and certainly by the deeper layers, just have like these very rich vectors that are packing in a ton of information. And the causal graph is like literally over every single layer that happened in the past. And that's what you're operating on. Yeah.

• 28:25Sholto Douglas?

  It does bring to mind like a very funny eval to do would be like a Sherlock Holmes eval. Let's see, put the entire book into context and then you have like a sentence which is like, the suspect is like X, then you have like a logic probability distribution over like the different characters in the book.

• 28:37Dwarkesh Patel?

  Yeah.

• 28:37Sholto Douglas?

  And then like as you put more like

• 28:40Dwarkesh Patel?

  That would be that would be super cool.

• 28:42Sholto Douglas?

  It'd be super interesting. I wonder if you'd get anything at all, but it'd be cool.

• 28:45Dwarkesh Patel?

  Sherlock Holmes is probably already in the training data. Yeah. You're gonna get like a mystery novel that was written in the

• 28:50Trenton Bricken?

  You can get an LLM to write it.

• 28:52Sholto Douglas?

  Or we could like, we could purposely exclude it, right?

• 28:54Dwarkesh Patel?

  Oh, you can? How do you?

• 28:56Trenton Bricken?

  Well, you need to scrape any discussion of it from Reddit or any other thing, right?

• 28:59Sholto Douglas?

  Right, it's hard, but that's like one of the challenges that goes into things like long context evals is to get a good one. You need to know that it's not in your training data. You've like put in the effort to exclude it.

• 29:09Dwarkesh Patel?

  What? So, as you wanna, there's two different threads I wanna follow up on. Let's go to the long context one and then we'll come back to this. So, in the Gemini 1.5 paper, the eval that was used was, can it like, something with pogram essays? Can it like remember stuff?

• 29:27Sholto Douglas?

  Yeah, the needle in a haystack thing.

• 29:29Dwarkesh Patel?

  Right. Which, yeah, I mean, there's like, we don't necessarily just care about its ability to recall one specific fact from the context. I'll step back and ask the question. The loss function for these models is unsupervised. You don't have to like come up with these bespoke things that you keep out of the training data. You know, is there a way you can do a benchmark that's also unsupervised? Where, I don't know, another LLM is rating it in some way or something like that? And maybe the answer is like, well, if you could do this, like reinforcement learning would work because then you have this like unsupervised.

• 30:04Sholto Douglas?

  Yeah, I mean, I think people have explored that kind of stuff, like, for example, Anthropic has a constitutional R L paper where they take another language model and they point and say like, how, you know, helpful or harmless was that response? And then they get it to update and try and improve along the pre to frontier of helpfulness and harmfulness. Um, so you can like point language models at each other and create evals in this way. It's obviously an imperfect art form at the moment, um, because uh, you get reward function hacking, basically, um, and the language, what like, uh, if you try and match up to what even humans are imperfect here, like if you try and match up to what humans will say, humans typically prefer longer answers, which aren't necessarily better answers, and you get that same behavior with models.

• 30:46Dwarkesh Patel?

  Hmm, um, on the other, because of the other thread, going back to the Sherlock Holmes thing.

• 30:52Dwarkesh Patel?

  If it's all associations all the way down, this is a sort of like naive dinner party question. If I just like match you, I'm working on A I. Uh, but okay, does that mean we should be less worried about superintelligence? Cause there's not this sense in which it's like Sherlock Holmes plus plus. It'll still need to just like find these associations, like humans find associations and like, you know what I mean? It's not just like it sees a frame of the world and it's like figured out all the laws of physics.

• 31:19Trenton Bricken?

  So, so for me, cause this is, this is a very legitimate response, right? It's like, well, artificial general intelligence aren't, if you say humans are generally intelligent, then they're no more capable or competent. I, I'm just worried that you have that level of general intelligence in silicon, where you can then immediately clone hundreds of thousands of agents and they don't need to sleep. And they can have super long context windows and then they can start recursively improving and then things get really scary. Um, so, so I think to answer your original question, yes, you're right. They would still need to learn associations, but.

• 31:51Dwarkesh Patel?

  Well, with the recursive self improvement would still have to be them. Like if intelligence is fundamentally about these associations, like the improvement is just them getting better at association. There's not like another thing that's happening and. So then it seems like you might disagree with the intuition that, well, they can't be that much more powerful if they're just doing associations.

• 32:10Trenton Bricken?

  Well, I think then you can get into really interesting cases of meta learning. Like when you play a new video game or like study a new textbook, you're bringing a whole bunch of skills to the table to form those associations much more quickly. And like, because everything in some way ties back to the physical worlds, I think there are like general features that you can pick up and then, and then apply in novel circumstances.

• 32:34Trenton Bricken?

  Should we talk about intelligence explosion then?

• 32:36Dwarkesh Patel?

  Okay.

• 32:36Trenton Bricken?

  I was only gonna jump in.

• 32:39Dwarkesh Patel?

  I mentioned multiple agents and I'm like, oh, here we go. Okay. So the reason I'm interested in discussing this is, with you guys in particular, is the models we have of the intelligence explosion so far come from economists, which is fine, but I think we can do better because the very, like, in the model of the intelligence explosion, what happens is you replace the AI researchers and then there's like a bunch of automated AI researchers who can speed up progress, make more AI researchers, make further progress. And so I feel like if that's the metric, or that's the mechanism, we should just ask the AI researchers about whether they think this is plausible.

• 33:18Sholto Douglas?

  Yeah.

• 33:19Dwarkesh Patel?

  So let me just ask you, like, if I have a thousand Asian Schultes or Asian Trentons, are they just, do you think that you get an intelligence explosion? Is that, yeah, what does that look like to you?

• 33:31Sholto Douglas?

  I think one of the important bounding constraints here is compute. Like, I do think you could dramatically speed up AI research, right? Like, it seems very clear to me that in the next couple of years, we'll have things that can do many of the software engineering tasks that I do on a day-to-day basis, and therefore dramatically speed up my work, and therefore speed up like the rate of progress, right? At the moment, I think most of the labs are somewhat compute bound in that they're always, the more experiments you could run and more pieces of information that you could gain in the same way that like scientific research on biology is also somewhat experimentally, like throughput bound, like you need to be able to run and culture the cells in order to get the information. I think that will be at least a short-term bounding constraint.

• 34:16Sholto Douglas?

  Obviously, you know, Sam's trying to raise $7 trillion to get chips and so, like, it does seem like there's going to be a lot more compute in future as everyone is heavily ramping. Nvidia's stock price sort of represents the relative compute increase, but.

• 34:34Trenton Bricken?

  Any thoughts? I think we need a few more nines of reliability in order for it to really be useful and trustworthy. Right now, it's like, and just having context lengths that are super long and it's like very cheap to have. Like, if I'm working in our codebase, it's really only small modules that I can get Claude to write for me right now. But it's very plausible that within the next few years, or even sooner, it can automate most of my tasks. The only other thing here that I will note is the research that at least our sub-team in interpretability is working on

• 35:14Trenton Bricken?

  is so early-stage that you really have to be able to make sure everything is done correctly in a bug-free way and contextualize the results with everything else in the model. And if something isn't going right, be able to enumerate all of the possible things and then slowly work on those. Like, an example that we've publicly talked about in previous papers is dealing with layer norm, right? And it's like, if I'm trying to get an early result or look at like the logit effects of the model, right? So it's like, if I activate this feature that we've identified to a really large degree, how does that change the output of the model? Am I using layer norm or not? How is that changing the feature that's being learned?

• 35:57Trenton Bricken?

  There, yeah, and that will take even more context or reasoning abilities for the model.

• 36:03Dwarkesh Patel?

  So you used a couple of concepts together, and it's not self-evident to me that they're the same, but it seems like you were using them interchangeably, so I just wanna... Like, one was, well, to work on the Claude codebase and make more modules based on that. They need more context or something, where like, it seems like they might already be able to fit in the context. Or do you mean like actual, do you mean like the context window context, or like more?

• 36:29Trenton Bricken?

  Yeah, the context window context.

• 36:31Dwarkesh Patel?

  So yeah, it seems like now it might just be able to fit. The thing that's preventing it from making good modules is not the lack of being able to put the codebase in there.

• 36:38Trenton Bricken?

  I think that will be there soon, yeah.

• 36:40Dwarkesh Patel?

  But like, it's not gonna be as good as you at like coming up with papers because it can like fit the codebase in there.

• 36:45Trenton Bricken?

  No, but it'll speed up a lot of the engineering.

• 36:48Dwarkesh Patel?

  Hmm. In a way that causes an intelligence explosion?

• 36:51Trenton Bricken?

  Um, no, that accelerates research, but I, but I think these things compound. So like, the faster I can do my engineering, the more experiments I can run. And then the more experiments I can run, the faster we can, I mean, my, my work isn't actually accelerating capabilities at all.

• 37:04Dwarkesh Patel?

  Right, right.

• 37:05Trenton Bricken?

  It's just interpreting the models, but, but we have a lot more work to do on that.

• 37:09Trenton Bricken?

  Uh, surprise to the Twitter.

• 37:13Dwarkesh Patel?

  Yeah, I mean, for context, like when you release your paper, there was a lot of talk on Twitter about alignment is off, guys, close the curtains.

• 37:23Trenton Bricken?

  Yeah, yeah, no, it's, it keeps me up at night how quickly the models are becoming more capable and like just how poor our understanding still is of what's going on.

• 37:32Dwarkesh Patel?

  Um, yeah, I, I guess I'm still, okay, so let's bring through the specifics here. By the time this is happening, we have bigger models that are two to four orders of magnitude bigger, right? Uh, or at least in effective compute, are two to four orders of magnitude bigger. And so, um, this like idea that, well, you can run experiments faster or something. You're having to retrain that model in this version of the intelligence explosion.

• 38:02Dwarkesh Patel?

  Like the recursive self improvement is different from what might have been imagined 20 years ago, where you just rewrite the code. You actually have to train a new model and that's really expensive. Not only now, but especially in the future, as you keep like making these models orders of magnitude bigger. Doesn't that dampen that possibility of a sort of recursive self improvement type intelligence explosion?

• 38:24Sholto Douglas?

  It's definitely gonna act as a braking mechanism. Like, definitely, like, I agree that the world of like what we're making today looks very different to what people imagined it would look like 20 years ago. Like, it's not going to be able to write its own code to be like really smart. Um, because actually it needs to train itself, like the, the code itself is typically quite simple. Typically pretty small and self-contained. I think John Carmack had this nice phrase where it's like, it's like the first time in history where like, you can actually plausibly imagine writing AI with like 10,000 lines of code. Um, and that like actually does seem plausible if, when you pair most training code bases down to the limit.

• 39:03Sholto Douglas?

  But it doesn't take away from the fact that this is something we should really strive to measure and estimate like how progress might occur. Like, we should be trying very, very hard right now to measure exactly how much of a software engineer's job is automatable and what the trend line looks like. And be trying our hardest to project out those trend lines.

• 39:20Dwarkesh Patel?

  But but with all due respect to software engineers, like you are not like writing a React frontend, right? So it's like, I don't know how this, like, what is concretely happening? And maybe you can walk me through, walk me through like a day in the life of, like, you're working on an experimental project that's going to make the model quote unquote better.

• 39:40Sholto Douglas?

  Right.

• 39:40Dwarkesh Patel?

  Like, what is happening from observation to experiment to theory to like writing the code, what is happening?

• 39:47Sholto Douglas?

  And so I think in point to contextualize here is that like I've primarily worked on inference so far. So a lot of what I've been doing is just taking or helping guide the pre-training process. Such that we design a good model for inference and then making the model and like the surrounding system faster. I've also done some pre-training work around that, but that hasn't been like my 100% focus, but I can still describe what I do when I do that work.

• 40:08Dwarkesh Patel?

  I don't know, but sorry, let me interrupt and say.

• 40:09Sholto Douglas?

  Two types of work, yeah.

• 40:10Dwarkesh Patel?

  In Carl Schumann's, like when he was talking about it on the podcast, he did say that things like improving inference or even literally having like better, helping it make you help make better chips or GPUs, that's like part of the intelligence explosion.

• 40:24Sholto Douglas?

  Yeah.

• 40:25Dwarkesh Patel?

  Cause like, obviously if the inference code runs faster, like it happens better or faster or whatever.

• 40:29Sholto Douglas?

  Right.

• 40:29Dwarkesh Patel?

  Anyway, sorry, go ahead.

• 40:31Sholto Douglas?

  Okay, so what is what is concretely a day look like? I think the most important like part to illustrate is this cycle of coming up with an idea, proving it out at different points in scale. And

• 40:47Sholto Douglas?

  the and like interpreting and understanding what goes wrong. And I think most people would be surprised to learn just how much goes into interpret like interpreting and understanding what goes what goes wrong. Cause the ideas, people have long lists of ideas that they want to try. Not every idea that you think should work will work and trying to understand why that is is quite difficult. And like working out what exactly you need to do to interrogate it. So so much of it is like introspection about what's going on. It's not pumping out thousands and thousands and thousands of line of code. It's not. Like the difficulty in coming up with ideas even. I think many people have a long list of ideas that they wanna try. But paring that down and shot calling under very imperfect information, what the right ideas to explore further is really hard.

• 41:31Dwarkesh Patel?

  Uh, uh, tell me more about, what, what do you mean by imperfect information? Are these early experiments? Are these?

• 41:36Sholto Douglas?

  Yeah.

• 41:37Dwarkesh Patel?

  Like, what is the information that you're?

• 41:39Sholto Douglas?

  Um, so, so Demis mentioned this in his podcast and also like you obviously, it's like the G P T four paper where you have like scaling law increments. And you can see like in the G P T four paper, they have like a bunch of like dots, right? Where they say, we can estimate the performance of our final model, like using all of these us. And there's a nice curve that like flows through them. And Demis mentioned, yeah, that, uh, we do this process of scaling up. Um, concretely, why does, why is that imperfect information? Is you never actually know if the trend will hold. For certain architectures, the trend has held really well. And for certain changes, it's held really well. But that isn't always the case and things which can help at smaller scales can actually hurt at larger scales. Um, so making guesses based on what the trend lines look like.

• 42:25Sholto Douglas?

  And based on like your intuitive feeling of, okay, this, this is actually something that's gonna matter. Um, particularly for those ones which help with the small scale.

• 42:34Dwarkesh Patel?

  That's interesting to consider that for every chart you see in a released paper or technical report that shows that smooth curve. There's a graveyard of like first year runs and then it's like flat or something.

• 42:44Sholto Douglas?

  There's all these like other lines that go in like different directions.

• 42:46Dwarkesh Patel?

  Yeah. Pursue that. That's. It's, yeah, it's crazy, both like as a grad student and then also here, like the number of experiments that you have to run before getting like a meaningful result. I, um, tell me, okay, so you, but presumably it's not just like you run it until it stops and then like, let's go to the next thing. Um, there's some process by which to interpret the early data and also to look at your, like, you, I don't know, I could like put a Google Doc in front of you. I'm pretty sure you could just like keep typing for a while on like different ideas you have. Um, and there's some bottleneck between that and. Just like making the models better immediately.

• 43:21Sholto Douglas?

  Right.

• 43:22Dwarkesh Patel?

  Yeah, walk me through like, what is the, what is the inference you're making from the first early steps that makes you have better experiments and better ideas?

• 43:29Sholto Douglas?

  I think one thing that I didn't fully convey before was that I, I think a lot of like good research comes from working backwards from the actual problems that you want to solve. And there's a couple of like grand problems I suppose in like making the models better today. That you would identify as issues and then like work back from, okay, how could I like change it to achieve this? There's also a bunch of, when you scale, you run into things and you want to like fix behaviors or like issues at scale. And that like informs a lot of the research for the next increment and this kind of stuff. So concretely, the barrier is a little bit software engineering, like often having a code base that's large and

• 44:08Sholto Douglas?

  sort of capable enough that it can support many people doing research at the same time makes it complex. If you're doing everything by yourself, your iteration pace is going to be much faster. I've heard that like Alec Radford, for example, like famously did much of the pioneering work at OpenAI. He like mostly works out of like a Jupiter notebook and then like has someone else who like writes and productionizes that code for him. I don't know if that's true or not. But like that kind of stuff, like actually operating with other people makes it, raises the complexity a lot. Because that not from natural reasons, like familiar to like every software engineer.

• 44:42Sholto Douglas?

  And then the inherent, running the, like running and launching those experiments is easy, but there's inherent time like slows downs induced by that. So you often want to be parallelizing multiple different streams because one, you can't like be totally focused on one thing necessarily. You might not have like fast enough feedback cycles. And then intuiting what went wrong is actually really hard, like working out what, like this is in many respects, the problem that the team at Trenton is on is trying to better understand. It's like what is going on inside these models? We have inferences and understanding and like headcanon for why certain things work, but it's not an exact science. And so you have to constantly be making guesses about why something might have happened, what experiment might reveal, whether that is or isn't true. And that's probably the most complex part.

• 45:26Sholto Douglas?

  The performance work comparatively is easier but harder in other respects. It's just a lot of low level and like difficult engineering work.

• 45:36Trenton Bricken?

  Hmm, yeah, I I agree with a lot of that. But even on the interpretability team, I mean, especially with Chris Ola leading it. There are just so many ideas that we want to test. And it's really just having the engineering skill, but I'll put engineering in quotes because a lot of it is research. To like very quickly iterate on an experiment, look at the results, interpret it, try the next thing, communicate them. Um, and then just ruthlessly prioritizing what the highest priority things to do are.

• 46:06Sholto Douglas?

  And this is really important, like the ruthless prioritization is something which I think separates a lot of like quality research from um. Research that doesn't necessarily succeed as much. We're in this funny field where uh so many of our theoretical, initial theoretical understanding is broken down, basically. Um, and so you need to have this simplicity bias and like ruthless prioritization over what's actually going wrong. I think that's one of the things that separates the most effective people is they don't necessarily get like too attached to solving. But using a given so like a solution that they're necessarily familiar with, um, but rather they attack the problem directly. Um, you see this a lot uh in like, maybe people come in with a specific academic background. They try and solve problems with that toolbox.

• 46:51Sholto Douglas?

  Um, and the best people are people who expand the toolbox dramatically. They're, you know, they're running around and they're taking ideas from reinforcement learning, but also from optimization theory. And also they have a great understanding of systems. And so they know what the sort of constraints that bound the problem are and they're good engineers. They can iterate and try ideas fast, like by far the best researchers I've seen. Uh, they all have the ability to try experiments really, really, really, really, really fast. Um, and that is that that's cycle time and that's smaller scales. Cycle time separates people.

• 47:18Trenton Bricken?

  I mean, machine learning research is just so empirical.

• 47:22Dwarkesh Patel?

  Yeah.

• 47:22Trenton Bricken?

  And this is honestly one reason why I think our solutions might end up looking more brainlike than otherwise. It's like, even though we wouldn't want to admit it, the whole community is kind of doing like greedy evolutionary optimization over the landscape of like possible AI architectures and everything else. It's like no better than evolution. And that's not even necessarily a slight against evolution.

• 47:45Dwarkesh Patel?

  That's such an interesting idea. I'm still confused on what will be the bottleneck for these, what would have to be true of an agent such that it's like sped up your research? So in the Alec Rafferty example you gave where he apparently already has the equivalent of like co-pilot for his Jupiter notebook experiments. Is it just that if he had enough of those, he would be a dramatically faster researcher? And so you just need Alec Rafferty? So it's like, you're not automating the humans, you're just making the most effective researchers who have great taste more effective and like running the experiments for them and so forth. They're like, that you're still working at the point with which the intelligence explosion is happening. You know what I mean? Like, is that what you're saying?

• 48:25Sholto Douglas?

  Right. And if that were like directly true, why can't we scale our current research teams better? For example, is I think an interesting question to ask, like why, if this work is so valuable, why can't we take hundreds of thousands of people who are, like they're definitely out there and like scale our organizations better?

• 48:48Sholto Douglas?

  It's,

• 48:51Sholto Douglas?

  I think we are less at the moment bound by the sheer engineering work of making these things than we are by compute to run and get signal and

• 49:04Sholto Douglas?

  and taste in terms of what the actual like right thing to do is and not like making those difficult inferences on imperfect information.

• 49:11Dwarkesh Patel?

  For for the Gemini team, because I think for interpretability,

• 49:14Sholto Douglas?

  Right.

• 49:15Dwarkesh Patel?

  We actually really want to keep hiring talented engineers. And I think it's a big bottleneck for us to just keep making a lot of progress.

• 49:21Sholto Douglas?

  Obviously, more people like more people is like better. But I do think like it's interesting to consider, I think, like, one of the biggest challenges that like I've thought a lot about is how do we scale that? Like, Google is an enormous organization and has 200,000 ish people, right? Like 100, maybe 80,000 or something like that. And one has to imagine if there were like ways of scaling out Gemini's research program to all those fantastically talented software engineers. This seems like a key advantage that you would want to be able to take advantage of. You want to be able to use, but like, how do you effectively do that? It's a very complex organizational problem.

• 50:00Dwarkesh Patel?

  Hmm. So compute and taste, that's interesting to think about because at least the compute part is not bottlenecked on more intelligence. It just bottlenecked on Sam seven trillion or whatever, right?

• 50:13Sholto Douglas?

  Yeah, yeah.

• 50:13Dwarkesh Patel?

  So if I gave you 10x the H100s to run your experiments, how much more effective a researcher are you?

• 50:19Sholto Douglas?

  TPS, please.

• 50:22Dwarkesh Patel?

  Uh, how much more effective a researcher are you?

• 50:25Sholto Douglas?

  I think the Gemini program would probably be like maybe five times faster with 10 times more compute or something like that.

• 50:34Dwarkesh Patel?

  So that's pretty good elasticity of like point five.

• 50:37Sholto Douglas?

  Yeah.

• 50:37Dwarkesh Patel?

  Wait, that's insane.

• 50:38Sholto Douglas?

  Yeah, I think like more compute would just like directly convert into progress.

• 50:42Dwarkesh Patel?

  So you have some some fixed size of compute and some of it goes to inference or some of, I guess, like, and also like to clients of GCP.

• 50:53Sholto Douglas?

  Yep.

• 50:53Dwarkesh Patel?

  Some of it goes to, huh? Some of it goes to training and there, I guess, as a fraction of it, some of it goes to running the experiments for the full model.

• 51:03Sholto Douglas?

  Yeah, that's right.

• 51:03Dwarkesh Patel?

  Shouldn't then the fraction goes to experiments be higher, given that you would just be like, if like the bottleneck is research and research is bottlenecked by compute.

• 51:12Sholto Douglas?

  And so one of the strategic decisions that every pre-training team has to make is like, exactly what amount of compute do you allocate to your different training runs? Do you like to your your research program versus like scaling the last best I like, you know, thing that you landed on? And I think.

• 51:32Sholto Douglas?

  They they're like, they're all trying to arrive at like a sort of pre optimal point here. One of the reasons why you need to still keep training big models is that you get information there that you don't get otherwise. Um, so scale has all these emerging properties, uh, which you want to understand better. And if you like are always doing research and never, like, remember what I said before about like, you're not sure what's gonna like.

• 51:54Dwarkesh Patel?

  Mm hmm.

• 51:54Sholto Douglas?

  Fall off the curve, right?

• 51:56Dwarkesh Patel?

  Yeah.

• 51:56Sholto Douglas?

  Um, if you like, keep doing research in this regime.

• 51:59Dwarkesh Patel?

  Yeah.

• 51:59Sholto Douglas?

  Uh, and like, keep on getting more and more computer efficient. You may never, you you may have actually like gone off the path that actually eventually scales. You need to constantly be investing in doing big runs too, at the frontier of of what you sort of expect to work.

• 52:16Dwarkesh Patel?

  Okay, so then tell me what it looks like to be in the world where A I has significantly sped up A I research. Cause from this, it doesn't really sound like the A I's are going off and writing the code from scratch and that's leading to faster output. It sounds like they're really augmenting the top researchers in some way. Like, yeah, tell me concretely, are they doing the experiments? Are they coming up with the ideas? Are they just like evaluating the outputs of the experiments? What's happening?

• 52:38Sholto Douglas?

  So I think there's, there's like two walls you need to consider here. One is where A I has meaningfully sped up our ability to make algorithmic progress.

• 52:46Dwarkesh Patel?

  Right.

• 52:46Sholto Douglas?

  And one is where the output of the A I itself is the thing that's like the crucial ingredient towards, uh, like model capability progress. And like specifically what I mean there is.

• 52:57Dwarkesh Patel?

  Synthetic.

• 52:57Sholto Douglas?

  Like synthetic data, right? Um, and in the first world where it's meaningfully speeding up algorithmic progress, I think a necessary component of that is more compute than. And you probably like reached this elasticity point where like A I's maybe at some point are easier to speed up and get on context than yourself. That's also right than other people. Um, and so A I's meaningfully speed up your work because they're like a fantastic co-pilot, basically, that helps you code like multiple times faster. Um, and that seems like actually quite reasonable. Um, super long context, super smart model, um, it's on boarded immediately and you can like send them off. Uh, and to like complete subtasks and subgoals for you.

• 53:39Sholto Douglas?

  And that actually like feels very plausible, but again, we don't know because there are no great evals about that kind of thing. Um, like the best one is as I said before, Sweet Bench, which.

• 53:49Dwarkesh Patel?

  Oh, oh, oh, like in that one, somebody was mentioning to me, like, the problem is that. When a human is trying to do a pull request, they'll like type something out and they'll like run it and see if it works. And if it doesn't, they'll rewrite it. None of this was part of the the um the opportunities that the L L M was given when run run on this. Like it just like output it and if it runs and like checks all the boxes, then you know, it passed, right? So it might have been an unfair test in that way.

• 54:15Sholto Douglas?

  So you can imagine that is uh like if you were able to use that, that would be a an effective training source for having like the the key thing that's missing from a lot of training data is uh

• 54:29Sholto Douglas?

  is like the the reasoning traces, right? And I think this would be if I wanted to try and automate a specific field uh would like job family um or like understand how how like at risk of automation that it is, then having reasoning traces feels to me like a really important part of that.

• 54:49Dwarkesh Patel?

  Hmm, there's so many threat, yeah, there there are so many different threads in that I want to follow up on. Let's begin with the data versus um like uh yeah, compute thing of like is is the output of these A I is a thing that's causing the intelligence explosion or something.

• 55:06Sholto Douglas?

  Yeah.

• 55:06Dwarkesh Patel?

  Um, people talk about how these models are really a reflection on their data.

• 55:11Sholto Douglas?

  Yeah.

• 55:12Dwarkesh Patel?

  I think there was uh I forgot his name, but there was a there's a great blog by this open A I engineer. And he was talking about at the end of the day, as these models get better and better, it just like they're just going to be really effective like maps of the data set.

• 55:26Sholto Douglas?

  Yeah.

• 55:27Dwarkesh Patel?

  And so it's like at at the end of the day, like you got to stop thinking about architectures. It's like the most effective architecture is like do you get amazing job of mapping the data?

• 55:35Sholto Douglas?

  Right.

• 55:35Dwarkesh Patel?

  Um, so that implies that future A I progress comes from the A I just making really awesome data, right? Like that you you're mapping to.

• 55:43Sholto Douglas?

  I think that's clearly a very important part, yeah.

• 55:47Dwarkesh Patel?

  Yeah, that's really interesting. Um, does that look to you like, I don't know, like things that look like chain of thought? Or what do you imagine as these models get better, as these models get smarter? What does the synthetic data look like?

• 55:59Sholto Douglas?

  When I think of really good data, uh, to me, that that raises something which involved a lot of reasoning to create. So in modeling that, uh, it's similar to like Ilya's perspective on on trying, on achieving like superintelligence via effectively like perfectly modeling the human textual output.

• 56:16Dwarkesh Patel?

  Yeah.

• 56:16Sholto Douglas?

  Right? Um, but even in the near term, in order to model something like archive papers or Wikipedia, you have to have an incredible amount of reasoning behind you in order to understand what next token might be uh being being output. Um, and so for me, what I imagine as good data is like model, like data where you can similarly, uh, at least like where it had to do reasoning to produce something. And then like the trick, of course, is how do you verify that that reasoning was correct? Um, and this is why you saw like DeepMind do that uh geometry, uh, like self, like the sort of like self play for geometry, basically, or like the sort of research for your geometry. This geometry is a really, it's easily formalizable, easily verifiable uh field.

• 56:59Sholto Douglas?

  So you can, you can check if its reasoning was correct. Uh, and you can generate heaps of data of correct, the like trick.

• 57:05Dwarkesh Patel?

  Yeah.

• 57:06Sholto Douglas?

  Of verified geometry proofs, train on that, and you know that that's good data.

• 57:09Dwarkesh Patel?

  Um, it's actually funny because I had a conversation with Grant Sanderson.

• 57:12Sholto Douglas?

  Yeah.

• 57:13Dwarkesh Patel?

  Like last year where we're debating this and I was like, fuck dude, by the time they get the gold of the Mathelophiad, of course they're going to automate all the jobs. Um, yikes. Um, yeah, on this synthetic data thing, uh, one of the things I speculated about in my scaling post, which was heavily informed with discussions with you two, and you especially, Sholto, was um. You can think of like human evolution through the spectrum of like, we get language. And so we're like generating the synthetic data, which right, you know, like our copies are generating the synthetic data, which we're trained on. And it's like this really effective genetics, a cultural, like co-evolutionary.

• 57:53Sholto Douglas?

  And there's a verifier there too, right? Like there's the real world. You might generate a theory about the gods caused the storms.

• 58:00Dwarkesh Patel?

  Right.

• 58:01Sholto Douglas?

  Right? And then like someone else finds cases where that isn't true. And so you like know that that like that was sort of didn't match your verification function. And now like actually instead you have like some weather simulation, which required a lot of reasoning to produce. And like accurately matches reality. And like you can train on that as a, as a better model of the world. Like we are training on, on that and like stories and like scientific theories.

• 58:24Dwarkesh Patel?

  Yeah. I want to go back. I'm just remembering something you mentioned a little while ago of given how sort of like empirical ML is, it really is evolutionary process as resulting in better performance and not necessarily an individual coming up with a breakthrough in like a top down way.

• 58:45Dwarkesh Patel?

  That has interesting implications. First being that

• 58:51Dwarkesh Patel?

  there really is people, people like are concerned about capabilities increasing because more people are going into the field. I've somewhat been skeptical of that way of thinking, but from this perspective of just like more input. It really does. Yeah. It feels more like, oh, actually by like the fact that more people are going to ICML means that there's like faster progress towards GPT five.

• 59:12Trenton Bricken?

  Yeah. You just have more genetic recombination.

• 59:14Dwarkesh Patel?

  Right.

• 59:14Trenton Bricken?

  And like shots on target. Yeah.

• 59:16Sholto Douglas?

  And I mean, aren't all fields kind of like that? Like there's the sort of scientific framework of like discovery versus invention, right? And discovery almost involves like whenever there's been a massive scientific breakthrough in the past, typically there are multiple people co-discovering that at like roughly the same time. And that feels to me at least a little bit like the mixing and trying of ideas. Like you can't try an idea that's so far out of scope that you, you have no way of verifying it with the tools you have available.

• 59:45Trenton Bricken?

  Yeah, I think physics and math might be slightly different in this regard.

• 59:49Sholto Douglas?

  Yeah.

• 59:49Trenton Bricken?

  Um, but especially for biology or any sort of wetware and to the extent we want to analogize neural networks here. It's just, it's comical how, how serendipitous a lot of the discoveries.

• 59:58Sholto Douglas?

  Yeah.

• 59:58Trenton Bricken?

  Like penicillin, for example.

• 1:00:00Dwarkesh Patel?

  Um, another implication of this is this idea that like AGI just gonna come tomorrow. Like somebody's just gonna discover a new algorithm and we have AGI. That seems less plausible, like it will just be a matter of more and more ML researchers finding these marginal things that all add up together to make models better, right? Like, yeah.

• 1:00:19Sholto Douglas?

  That feels like the correct story to me, yeah.

• 1:00:21Trenton Bricken?

  Eh eh especially while we're still hardware constrained.

• 1:00:24Dwarkesh Patel?

  Right. Do you buy this, uh, narrow window framing of the intelligence explosion of you have to each, you know, G P T three to G P T four is two ums of orders of magnitude more compute or at least more effective compute. Um, in the sense that if you didn't have any algorithmic progress, it would have to be two orders of magnitude bigger, like the raw form to be as good. Um, do you buy the framing that given that you have to be two orders of magnitude bigger at every generation? If you don't get AGI by G P T seven, that can help you catapult the intelligence explosion.

• 1:01:03Dwarkesh Patel?

  Like you're kind of just fucked as far as like much smarter intelligences go and you're kind of stuck with G P T seven level models for a long time. Cause at that point you're just like consuming significant fractions of the economy to make that model. And we just don't have the wherewithal to like make G P T eight.

• 1:01:18Trenton Bricken?

  This is the Carl Schumann sort of argument of like, we're gonna race through the orders of magnitude in the near term. But then longer term, it would, it would be harder.

• 1:01:27Dwarkesh Patel?

  Um, I, I think like he's probably talked about it, uh, but yeah. But I do buy, do buy that framing.

• 1:01:32Sholto Douglas?

  Yeah, I, I mean, I, I generally buy that increases in order of magnitude of compute by like. In absolute terms, almost like diminishing returns on like your capability, right? Like we've seen over a couple orders, mainstream models go from being unable to do anything to be able to like do huge amounts. And it feels to me like each incremental order of magnitude like gives more nines of reliability at things. And so unlocks things like agents, but at least at the moment, I haven't seen like transformatively. Like this, it doesn't feel like reasoning improves like linearly, so to speak, uh but rather like somewhat sublinearily.

• 1:02:03Dwarkesh Patel?

  That's actually a very bearish sign because one of the things we're we're chatting with one of our friends and he made the point. That if you look at what new applications are unlocked by G P T four or relative to G P T three point five. It's not clear that's like that much, like a G P T three point five can do perplexity or whatever. So if there is this diminishing increase in capabilities and um and that increased cost exponentially more to get. That's actually a bearish sign on like what four point five will be able to do or five will unlock in terms of economic impact.

• 1:02:36Sholto Douglas?

  That being said, for me, the jump between three point five and four is like pretty huge. And so like, even if I it's like another three point five to four jump is like ridiculous, right? Like if you if you imagine five as being a three point five to four jump, like straight off the bat in terms of like ability to do S A Ts and this kind of stuff.

• 1:02:52Dwarkesh Patel?

  Yeah, the L S A P performance was particularly striking.

• 1:02:54Sholto Douglas?

  Exactly. You go from like, uh, you know, like very smart, like from like, you know, not super smart to like very smart to like utter genius in the next generation instantly. And it doesn't at least like to me feel like we're we're gonna sort of jump to utter genius in the next generation. But it does feel like we'll get very smart plus lots of reliability and then like we'll see T B D, what that continues to look like.

• 1:03:21Dwarkesh Patel?

  Will Gofi be part of the intelligence explosion? Where like you you say synthetic data, but like, in fact, it will be like it writing its own source code in some important way. There was an interesting paper that you can use diffusion to like come up with model weights. Uh, I don't know how like legit that was or whatever, but like, I don't know, something like that. Can you so so Gofi is good old fashioned A I, right? And and can you define that? Because when I hear it, I think like if else statements for like symbolic logic.

• 1:03:48Trenton Bricken?

  Sure.

• 1:03:52Sholto Douglas?

  I actually want to make sure we like don't like we like fully unpack the whole like model improvement increments.

• 1:03:57Trenton Bricken?

  Yeah.

• 1:03:57Sholto Douglas?

  Cause I don't want people to come away with the perspective that like, actually, this is super bearish and like models aren't going to get much better and stuff.

• 1:04:03Trenton Bricken?

  Okay.

• 1:04:03Sholto Douglas?

  More what I want to emphasize is like, the jumps that we've seen so far are huge. Um, and even if those like continue on like a smaller scale, we're still in for extremely smart, uh, like very reliable agents. Like over the next couple of orders of magnitude. And so like, we didn't sort of fully close the thread on the narrow window thing. Um, when you, you think of like, let's say GPT four costs, I know, let's call it a hundred million dollars or whatever. Um, you have what the one B run, the 10B run, the hundred B run, all seem very plausible by, uh, you know, private company standards. Um, and then the.

• 1:04:40Trenton Bricken?

  You mean in terms of dollar?

• 1:04:41Sholto Douglas?

  In terms of dollar amount, yeah. Yeah. Um, and then you can also imagine even like a one T run being part of like a national consortium or like a, you know, a national level, uh, thing, but much harder on the behalf of an individual company. But Sammy is out there trying to raise seven trillion dollars, right? Like he's already preparing for like a whole order of magnitude more than the, uh.

• 1:05:01Trenton Bricken?

  He shifted the overton window.

• 1:05:02Sholto Douglas?

  He shifted the order of magnitude here beyond the national level. Um, so I want to point out the one we have a lot more jumps. And even if those jumps are relatively smaller, that's still a pretty stark improvement in capability.

• 1:05:17Trenton Bricken?

  Not only that, but if you believe claims that GPT four is around one trillion parameter count. Uh, I mean, the human brain is between 30 and 300 trillion synapses. And so that's obviously not a one-to-one mapping and, and we can debate the numbers, but it seems pretty plausible that we're below brain scale still.

• 1:05:36Dwarkesh Patel?

  Uh, uh, so crucially, the point being that the algorithmic overhead is really high in the sense that, and maybe this is something we should touch on explicitly of even if you can't keep dumping more compute beyond the models that cost a trillion dollars or something. The fact that the brain is so much more data efficient implies that if you get, we have the compute, if we had like the brain's algorithm to train.

• 1:06:03Dwarkesh Patel?

  Training, if we could like, train as a sample efficient as humans train from birth, we could make the AGI.

• 1:06:08Trenton Bricken?

  Yeah, but the sample efficiency stuff, I never know exactly how to think about it because obviously a lot of things are hardwired in certain ways, right? And like the co-evolution of language and like the brain structure. So i- it's hard to say. Also, there are some results that if you make your model bigger, it becomes more sample efficient.

• 1:06:28Dwarkesh Patel?

  Yeah.

• 1:06:28Sholto Douglas?

  And so the original scaling was paper.

• 1:06:30Trenton Bricken?

  Right.

• 1:06:30Sholto Douglas?

  That, right?

• 1:06:30Trenton Bricken?

  Yeah.

• 1:06:31Sholto Douglas?

  Logic models almost.

• 1:06:32Trenton Bricken?

  Right, so, so maybe that also just solves it. Like you don't have to be more data efficient, but if your model's bigger than you also just are more data efficient, like.

• 1:06:41Dwarkesh Patel?

  Well, how do we think about, yeah, how do, what is like the explanation of why that would be the case? Like a bigger model to see the exact same data at the end of seeing that data, it's learn more from it.

• 1:06:52Trenton Bricken?

  I mean, my, my like very naive take here would just be that, like, like, so, so one thing that the superposition hypothesis that interpretability has pushed is that your model is dramatically under parameterized. And, and that's typically not the narratives that deep learning is pursued, right? But if you're, if you're trying to train a model on like the entire internet and have it predict it with incredible fidelity, you are in the under parameterized regime. And you're having to compress a ton of things and take on a lot of noisy interference in doing so. And so having a bigger model, you can just have cleaner representations that you can work with.

• 1:07:24Dwarkesh Patel?

  Yeah. Yeah, for the audience, you should unpack why that, first of all, what superposition is and why that is the implication of superposition.

• 1:07:31Trenton Bricken?

  Sure, yeah, so the, the fundamental result, and this was before I joined Anthropic, but the paper's titled "Toy Models of Superposition", finds that even for small models, if you are in a regime where your data is high dimensional and sparse, and by sparse, I mean any given data point doesn't appear very often, your model will learn a compression strategy, which we call superposition, so that it can pack more features of the world into it than it has parameters. And so, so the sparsity here is like, and I think, I think both of these constraints apply to the real world and modeling internet data is, is a good enough proxy for that of like, there's only one Dwarkesh.

• 1:08:14Trenton Bricken?

  Like, there's only one shirt you're wearing. There's like this liquid death can here. And so these are all objects or features, and how you define it features tricky. And so you're in a really high dimensional space because there are so many of them.

• 1:08:26Dwarkesh Patel?

  Right.

• 1:08:26Trenton Bricken?

  And they appear very infrequently.

• 1:08:28Dwarkesh Patel?

  Yeah.

• 1:08:29Trenton Bricken?

  And in that regime, your model will learn compression. To riff a little bit more on this, I think it's becoming increasingly clear, I will say, I believe that the reason networks are so hard to interpret is because, in a large part, this superposition. So if you take a model and you look at a given neuron in it, right, a given unit of computation, and you ask, how is this neuron contributing to the output of the model when it fires? And you look at the data that it fires for, it's very confusing. It'll be like 10% of every possible input, or like Chinese, but also fish and trees and the word, a full stop and URLs, right? But the paper that we put out towards monosemanticity last year shows that if you

• 1:09:14Trenton Bricken?

  project the activations into a higher dimensional space and provide a sparsity penalty, so you can think of this as undoing the compression in the same way that you assumed your data was originally high dimensional and sparse. You return it to that high dimensional and sparse regime, you get out very clean features. And things all of a sudden start to make a lot more sense.

• 1:09:33Dwarkesh Patel?

  Hmm. Okay. There's so many interesting threads there. The first thing I want to ask is, the thing you mentioned about these models are trained in a regime where they're over parameterized. Isn't that when you have generalization, like Grokking happens in that regime, right? So, um.

• 1:09:58Trenton Bricken?

  So I was saying the models were under parameterized.

• 1:10:01Dwarkesh Patel?

  Oh, I see.

• 1:10:02Trenton Bricken?

  Yeah, like typically, people talk about deep learning as if the model is over parameterized. But actually, the claim here is that they're dramatically under parameterized, given the complexity of the task that they're trying to perform.

• 1:10:12Dwarkesh Patel?

  Another question, so.

• 1:10:16Dwarkesh Patel?

  The distilled models, like, i- i- first of all, okay, so what is happening there? Because earlier, the earlier claims we're talking about is the smaller models are worse at learning than bigger models, but like G P T four turbo, you could say, make the claim that actually G P T four turbo is worse at reasoning style stuff than G P T four, but probably knows the same facts, like the distillation got rid of like some of the reasoning things.

• 1:10:43Sholto Douglas?

  Do we have any evidence that G P T turbo is a distilled version of four? It might just be new architecture.

• 1:10:47Dwarkesh Patel?

  Oh, okay.

• 1:10:48Sholto Douglas?

  Yeah, like it could just be like a faster, more efficient new architecture.

• 1:10:52Dwarkesh Patel?

  Okay, interesting.

• 1:10:53Sholto Douglas?

  So that's cheaper, yeah.

• 1:10:55Dwarkesh Patel?

  What is the, how do you like interpret what's happening in distillation? I think Gern had one of these questions on his website of why can't you train the distilled model directly? Why does it have to go through? And is it, is a picture like you had to project it from this bigger space to a smaller space?

• 1:11:11Trenton Bricken?

  Um, I mean, I think both models will still be using superposition. But, but the, the claim here is that you get a very different model if you distill versus if you train from scratch.

• 1:11:21Dwarkesh Patel?

  Yeah.

• 1:11:22Trenton Bricken?

  And, and it's just more efficient or it's just fundamentally different in terms of performance?

• 1:11:29Dwarkesh Patel?

  I don't remember, but like, do you know?

• 1:11:31Sholto Douglas?

  I think like the traditional story for why distillation is more like efficient is that normally during training, you're trying to predict this like one hot vector that says like, this is the token that you should have predicted. And if your like reasoning process means that you're really far off predicting that, then actually that like, you sort of get these gradient updates that, yeah, are in the right direction, but like you're, you're totally, it might be really hard for you to learn, to have learned to have predicted that in the context that you're in. And so what distillation does is it doesn't just have the one hot vector, it has like the full readout from the larger model, like of all of the probabilities.

• 1:12:04Dwarkesh Patel?

  Yeah, yeah.

• 1:12:04Sholto Douglas?

  And so you get more signal about what you should have predicted. It's not, i- it's in some respects, it's like. Showing a tiny bit if you're working to.

• 1:12:14Dwarkesh Patel?

  Yeah.

• 1:12:14Sholto Douglas?

  You know, like, it's not just, this was the answer, it's.

• 1:12:17Dwarkesh Patel?

  I see, yeah, yeah, yeah.

• 1:12:17Sholto Douglas?

  Totally.

• 1:12:18Dwarkesh Patel?

  But that makes a lot of sense.

• 1:12:19Trenton Bricken?

  It's kind of like watching a Kung Fu master versus being in the Matrix and like, just downloading.

• 1:12:23Sholto Douglas?

  Yeah, exactly, exactly.

• 1:12:25Dwarkesh Patel?

  Yep, yep. Um, just, just to make sure the audience got that, when you're training on a distilled model, you, you're like, you see all its probabilities over the tokens it was predicting, and then over the ones you were predicting, and then you like, update through all those probabilities rather than just seeing the last word and updating on that. Okay, so this actually raises a question I was intending to ask you.

• 1:12:47Dwarkesh Patel?

  Um, right now, I think you were the one who mentioned you can think of chain of thought as adaptive compute of, like,

• 1:12:57Dwarkesh Patel?

  uh, to step back and explain, uh, what, what, by adaptive compute, it's, the idea is, one of the things you would want models to be able to do is, if a question is harder, to spend more cycles thinking about it.

• 1:13:10Dwarkesh Patel?

  Uh, and so then how do you do that? Well, there's only a finite and predetermined amount of compute that one forward pass implies. So if there's like a complicated reasoning type question or math problem, you want to be able to spend a long time thinking about it. Then you do chain of thought where the model just like thinks through the answer and you can think about it as like, all those forward passes where it's like thinking through the answer, it's like being able to dump more compute into solving the problem. Now, going back to the signal thing. When is the chain of thought, it's only able to transmit that token of information, where it's like, as you were talking about, the residual stream is already a compressed representation of everything that's happening in the model. And then you're turning the residual stream into one token,

• 1:13:56Dwarkesh Patel?

  which is like log of fifty thousand or log of vocab size, uh, bits, which is like, yeah, so tiny, so.

• 1:14:03Sholto Douglas?

  So I, so I don't think it's quite only transmitting like that one token, right? Like, if you think about it during a forward pass, and you, you create these like KV values.

• 1:14:12Dwarkesh Patel?

  Yeah.

• 1:14:13Sholto Douglas?

  Um, in a transformer forward pass, that then like, future steps attend to the KV values. And so all of those pieces of KV, of like, the keys and values, are bits of information that you could use in the future.

• 1:14:25Dwarkesh Patel?

  Is the claim that.

• 1:14:28Trenton Bricken?

  When you fine tune on chain of thought, the way the key, the key and value weights change so that the sort of steganography can happen in the KV cache?

• 1:14:38Sholto Douglas?

  I don't think I could make that strong a claim, just

• 1:14:41Trenton Bricken?

  But that's impossible.

• 1:14:42Sholto Douglas?

  But it's like, that's a good headcanon for why it works. Um, and I don't know if there's any like papers explicitly demonstrating that or anything like that. Um, but like, that's at least one way that you can imagine the model has

• 1:14:55Sholto Douglas?

  over the, like during pre-training, right? The model's trying to predict these future tokens. Um, and one thing that you can imagine it doing is learning to like smush information about potential futures into like the keys and values um, that it might want to use in order to predict

• 1:15:11Trenton Bricken?

  That's so interesting.

• 1:15:12Sholto Douglas?

  future information. Um, like it kind of smoothens that information across time and the pre-training thing. Um, so I don't know if like people are particularly training, like, like training on chain of thought. I think the original chain of thought paper had that as like almost an inversion property of the model is you could like prompt it to do this kind of stuff. Um, and it still worked pretty well. Um, but that's like, yeah, it's a good headcanon for why that works.

• 1:15:33Trenton Bricken?

  Yeah. To, to be overly penantic here, it's like the tokens that you actually see in the chain of thought.

• 1:15:38Sholto Douglas?

  Yeah.

• 1:15:39Trenton Bricken?

  Do not necessarily at all need to correspond to the vector representation that the model gets to see.

• 1:15:45Sholto Douglas?

  Exactly.

• 1:15:45Trenton Bricken?

  When it's deciding to attend back to those tokens.

• 1:15:47Sholto Douglas?

  Exactly. In fact, like during training, you replace, like what, what a training step is, is you actually replacing the token of the model output with the real next token. Um, and yet it's still like learning because it has all this information uh internally.

• 1:16:06Sholto Douglas?

  Like when you're getting a model to produce at inference time.

• 1:16:08Trenton Bricken?

  Yeah.

• 1:16:09Sholto Douglas?

  Like you're taking the output, the token that it output, you're feeding it in the bottom, unembedding it, and it like becomes the beginning of the new residual stream.

• 1:16:15Trenton Bricken?

  Right, right.

• 1:16:16Sholto Douglas?

  Um, and then you use the output of past KVs to like read into and adapt that residual stream. Uh, at training time, you do this thing called teacher forcing, basically, where you're like, actually, the token you were meant to output is this one. That's how you do it in parallel, right? Because you have all the tokens, you put them all in in parallel and you do the giant forward pass. Um, and so the only information it's getting about the past is the keys and values. It never sees the token that it output.

• 1:16:41Trenton Bricken?

  It's kind of like it's trying to do the next token prediction. And if it messes up, then you just give it the correct answer.

• 1:16:47Dwarkesh Patel?

  Yeah, right, right. Yeah, okay, that makes sense.

• 1:16:49Trenton Bricken?

  Otherwise, it can become totally derailed.

• 1:16:51Sholto Douglas?

  Yeah, it'll go like off the train.

• 1:16:54Dwarkesh Patel?

  How how much like the sort of secret communication with the model to its forward forward inferences, how much how much steganography and you know, like secret communication do you expect there to be?

• 1:17:09Sholto Douglas?

  We don't know, like honest answer, we don't know. But I wouldn't even necessarily like classify it as like secret information, right? Like a lot of the work that Trend's team is trying to do is actually understand and that these are fully visible from the model side. And then from like this, maybe not to user, but like we should be able to understand and interpret what these values are doing and the information that are transmitting, like transmitting, I think that's really important. Like goal for the future.

• 1:17:37Trenton Bricken?

  Yeah, I mean, there are some wild papers that where people have had the model do train of thought. And it is not at all representative of what the model actually decides its answer is. And you can go in and edit. No, no, no, in this case, like you can even go in and edit the chain of thought so that the reasoning is like totally garbled. And it will still output the true answer.

• 1:17:58Dwarkesh Patel?

  But also that the chain of thought like, yeah, it gets a better answer at the end of the chain of thought rather than not doing it at all. So like something useful is happening, but still the useful thing is not human understandable.

• 1:18:08Trenton Bricken?

  I think in some cases you can also just apply the chain of thought and it would have given the same answer anyways.

• 1:18:12Dwarkesh Patel?

  Interesting.

• 1:18:14Trenton Bricken?

  Yeah, so I'm not saying this is always what goes on, but like there's plenty of weirdness to be investigated.

• 1:18:20Sholto Douglas?

  It's like a very interesting to go and and look at and try and understand.

• 1:18:25Trenton Bricken?

  Yeah.

• 1:18:26Sholto Douglas?

  That you can do with open source models and like, I think, I wish there was more of this kind of interpretability and understanding work done on open models.

• 1:18:33Trenton Bricken?

  Yeah, I mean, even in our Anthropic's recent sleeper agents paper, which the at a high level for for people unfamiliar is basically. I train in a trigger word. And when I say it, like if I say if it's the year 2024, the model will write malicious code instead of otherwise. Uh, and they do this attack with a number of different models. Um, some of them use chain of thought, some of them don't. Um, and those models respond differently. When you try and remove the trigger, um, you can even see them do this like comical reasoning. That's also pretty creepy and like, uh, where it's like, oh, well, it even tries to calculate in one case, the expected value of like, well, the expected value of me getting caught is this. But then if I multiply it by the ability for me to like,

• 1:19:18Trenton Bricken?

  keep saying, I hate you, I hate you, I hate you. Then like, this is how much reward I should get. And then it will decide whether or not to like, actually tell the interrogator that it's like malicious or not.

• 1:19:30Dwarkesh Patel?

  Oh.

• 1:19:31Trenton Bricken?

  Um, but, but even, I mean, there's another paper from a friend, Miles Turpin, uh, where you ask the model to, you give it like a bunch of examples of, um, where like the correct answer is always A for multiple choice questions. And then you ask the model, what is the correct answer to this new question? And it will infer from the fact that all the examples are A, that the correct answer is A. But its chain of thought is totally misleading. Like it will make up random stuff to, that sounds plausible or that tries to sound as plausible as possible. Um, but it's not at all representative of like the true answer.

• 1:20:10Dwarkesh Patel?

  But isn't this how humans think as well? The famous split brain experiments where, um, you know, like where, when a person who is suffering from seizures, one way to solve it is you cut the, the thing that connects the two.

• 1:20:24Trenton Bricken?

  Corpus callosum, yeah.

• 1:20:25Dwarkesh Patel?

  And then the, yeah, the speech half is on the left side. So it's not connected to the part that decides to do a movement. Um, and so if the other side decides to do something, the speech part will just make something up. And it'll like, the person will think that's legit the reason they did it.

• 1:20:38Trenton Bricken?

  Totally. Yeah. Yeah. It's just some people will hail chain of thought reasoning as like a great way to solve AI safety.

• 1:20:45Dwarkesh Patel?

  Um, oh, I see.

• 1:20:46Trenton Bricken?

  And, and it's like, actually we don't know whether we can trust it.

• 1:20:49Dwarkesh Patel?

  Um, um, how much, what will this landscape of models communicating to themselves in ways we don't understand? How does that change with AI agents? Because then these things will, it's not just like the model itself with its previous caches, but like other instances of the model and then.

• 1:21:09Sholto Douglas?

  It depends a lot on what channels you give them to communicate with each other, right? Like if you only give them text as a way of communicating, then they probably have to.

• 1:21:16Dwarkesh Patel?

  How how much more effective do you think the models would be if they could like share their residual streams versus just text?

• 1:21:22Sholto Douglas?

  Hard to know, but plausibly so. I mean, one one easy way that you can imagine this is like, if you wanted to describe how a picture should look.

• 1:21:32Dwarkesh Patel?

  Yeah.

• 1:21:32Sholto Douglas?

  Um, only describing that with text would be hard.

• 1:21:34Dwarkesh Patel?

  Right.

• 1:21:35Sholto Douglas?

  Um, you want to, maybe some other representation would plausibly be easier.

• 1:21:38Dwarkesh Patel?

  Totally.

• 1:21:39Sholto Douglas?

  Um, and so like you can look at how, uh, I think like DALL-E works at the moment, right? Like it produces those prompts.

• 1:21:45Dwarkesh Patel?

  Yeah.

• 1:21:45Sholto Douglas?

  Um, and when you play with it, you like often can't quite get it to do exactly what the model wants or what you want.

• 1:21:54Dwarkesh Patel?

  The only DALL-E has that problem.

• 1:22:14Sholto Douglas?

  And you can imagine like being able to transmit some kind of like denser representation of what you want would be helpful there. Um, and that's like two very simple agents, right?

• 1:22:21Dwarkesh Patel?

  I mean, I think a nice halfway house here would be features that you learn from dictionary learning.

• 1:22:26Sholto Douglas?

  Yeah, that would be really, really cool.

• 1:22:28Dwarkesh Patel?

  You you get more internal access, but a lot of it is much more human interpretable.

• 1:22:33Sholto Douglas?

  Yeah.

• 1:22:33Dwarkesh Patel?

  So, okay, for the audience, you would project the residual stream into this larger space where we know what each dimension actually corresponds to. Um, and then back into the next agents or whatever. Okay, why, so your claim is that we'll get AI agents when these things can, um, are more reliable and so forth. Um, when that happens, do you expect that it will be multiple copies of models talking to each other? Or will it be just a adaptive computer solve, then the thing just like runs bigger? Uh, like more compute when it needs to do a kind of thing that a whole firm needs to do? And I ask this because.

• 1:23:15Dwarkesh Patel?

  There's two things that make me wonder about like whether agents is the right way to think about what will happen in the future. One is with longer contexts, these models are able to ingest and consider the information that no human can. And therefore, we need like one engineer who's thinking about the front end code and one engineer who's thinking about the back end code. Where this thing can just ingest the whole thing, this sort of like Hayekian problem of specialization goes away. Second, these models are just like very general of you're like not using different types of G P T four to do different kinds of things. You're using the exact same model, right? So I wonder what that implies is in the future, like an A I firm is just like a model instead of a bunch of A I agents hooked together.

• 1:23:56Sholto Douglas?

  That's a great question. Um, I think especially in the near term, uh, it will look much more like agents hooked together. And I say that like purely because as humans, we're going to want to have these like isolated, reliable and uh, like, like, like components that we can trust. Um, and we're also going to want to, we're going to need to be able to improve and instruct upon those like components. Um, in, in ways that we can understand and improve, like just throwing it all this giant black box company model, like it, one, it isn't going to work. Um, and initially, uh, later on, of course, you can imagine it working, but initially it won't work. Um, and two, we probably don't want to do it that way.

• 1:24:39Dwarkesh Patel?

  Well, you can also have each of the smaller model, well, each of the agents can be a smaller model that's cheaper to run. And you can fine tune it so that it's actually good at the task.

• 1:24:47Sholto Douglas?

  Though there's a, there's a future with like, Duokesh has brought up adaptive compute a couple of times. Um, there's a future where like the distinction between small and uh, large models like disappears to some degree. Um, and with long context, there's also a degree to which fine tuning might disappear, to be honest. Um, like the, these, these two things that are very important today and like today's landscape of models, we have like whole different tiers of model sizes and we have fine tuned models for different things. You can imagine a future where you just actually have a dynamic bundle of compute and like infinite context that specializes your model to to different things.

• 1:25:21Dwarkesh Patel?

  One thing you can imagine is you have an AI firm or something and the whole thing is like end to end trained on the signal of like, did I make profits or like if if that's like too ambiguous, if it's if it's an architecture firm and they're making blueprints, did my client like the blueprints? And in the middle, you can imagine agents who are salespeople and agents who are like doing the designing, agents who like do the editing, whatever.

• 1:25:45Dwarkesh Patel?

  Would that kind of signal work on an end to end system like that? Because like one of the things that happens in human firms is management considers what's happening at the larger level and like gives these like fine grain signals to the the pieces or something when like there's a bad quarter or whatever.

• 1:26:00Sholto Douglas?

  Yeah, in a limit, yes, that's the dream of reinforcement learning, right? It's like all you need to do is provide this extremely sparse signal and then over enough iterations, you sort of create the information that allows you to learn from that signal.

• 1:26:13Sholto Douglas?

  But I don't expect that to be the thing that works first. I think this is going to require an incredible amount of care and like diligence on the behalf of humans surrounding these machines and making sure they do exactly the right thing and exactly what you want and giving them right signals to improve in the ways that you want.

• 1:26:31Dwarkesh Patel?

  Yeah, you can't train on the RL reward unless the model generates some reward.

• 1:26:36Sholto Douglas?

  Yeah, that's yeah, yeah, exactly. You're in it like you're in this like sparse RL world where like if it never if the client never likes what you produce, then like you don't get any reward at all. And like, it's kind of bad.

• 1:26:46Dwarkesh Patel?

  But in the future, these models will be good enough to get the reward some of the time, right? This is the nines of reliability that Chalk was talking about.

• 1:26:51Sholto Douglas?

  Yeah.

• 1:26:53Dwarkesh Patel?

  There's an interesting digression, by the way, on earlier, we're talking about, well, we want dense representations that like that that will be favored, right? Like, that's a more efficient way to communicate. A book that Trenton recommended, The Symbolic Species, has this really interesting argument that language is not just a thing that like exists, but like it was also something that evolved along with our minds and specifically evolved to be both easy to learn for children and to something that helps children develop, right?

• 1:27:30Sholto Douglas?

  Like it's

• 1:27:31Dwarkesh Patel?

  Unpack that phone. Because like a lot of the things that children learn are received through language. Like the languages that will be the fittest are ones that help like raise the next generation, right? And that like makes them smarter, better, whatever.

• 1:27:48Sholto Douglas?

  Um, and if you think about it. Gives them the concepts to express more complex ideas.

• 1:27:52Dwarkesh Patel?

  Yeah. Yeah, that and I guess um more pedantically, just like not die.

• 1:27:56Sholto Douglas?

  Right, sure, yeah. Lets you encode the important shit to not die.

• 1:28:02Dwarkesh Patel?

  Um, and so then uh when we just think of like languages like, oh, you know, say this contingent and maybe suboptimal way to represent ideas. Actually, maybe one of the reasons that LLMs have succeeded is because language has evolved for tens of thousands of years to be this sort of caste in which young minds can develop, right? Like that is the purpose it was evolved for.

• 1:28:26Sholto Douglas?

  Well, certainly when you talk to like multimodal or like computer vision researchers versus when you talk to language model researchers.

• 1:28:33Dwarkesh Patel?

  Yeah.

• 1:28:33Sholto Douglas?

  Um, people who work in other modalities have to put enormous amounts of thought into exactly what the right representation space for the images is. Um, and like what the right signal to learn from there, is it like directly modeling the pixels? Or is it uh, you know, some loss that's conditioned on uh, there's there's like a paper ages ago where they like found that if you trained on the internal representations of an image that model, it like helped you predict better. Um, but then later on, like that's obviously like limiting. And so there was like Pixel CNN where they're trying to like discreetly model um, you know, the the individual pixels and stuff. But understanding the right level of representation there, really hard. In language, people are just like, well, I guess you just predict that thanks token. It's kind of easy.

• 1:29:12Dwarkesh Patel?

  Yeah, yeah.

• 1:29:13Sholto Douglas?

  Decisions made, I mean, there's the tokenization um, like discussion and debate about like, but one of Gwyn's favorites.

• 1:29:21Dwarkesh Patel?

  Yeah. Yeah, that's that's really interesting. How much um, the the case for a multimodal being a way to bridge the data wall or get past the data wall.

• 1:29:30Sholto Douglas?

  Yeah.

• 1:29:30Dwarkesh Patel?

  Is can is like based on the idea that the things you would have learned from more language tokens anyway, you can just get from YouTube. It has that actually been the case? Uh, how much like positive transfer do you see between different modalities where like actually the images are helping you be better at like writing code or something? Just because like the model is learning a latent capabilities just from trying to understand the image?

• 1:29:55Sholto Douglas?

  Demis, in his interview with you, mentioned positive transfer.

• 1:30:00Trenton Bricken?

  I can get in trouble.

• 1:30:07Sholto Douglas?

  But I mean, I can't say shit about that.

• 1:30:11Sholto Douglas?

  Other than to say, this is something that people like believe that, yes, like, we have all of this data about the world. It would be great if we could like, learn an intuitive sense of physics from it that helps us reason, right? That seems totally plausible and.

• 1:30:23Trenton Bricken?

  Yeah, I'm the wrong person to ask, but there are interesting interpretability pieces where if we fine tune on math problems, the model just gets better at entity recognition.

• 1:30:35Dwarkesh Patel?

  Whoa, really?

• 1:30:36Trenton Bricken?

  Yeah, yeah, yeah. So there's like a paper from David Bowles' lab recently where they investigate what actually changes in a model when I fine tune it with respect to the attention heads and these sorts of things. And they have this like synthetic problem of, um, box A has this object in it. Box B has this other object in it. What was in this box? And if you've tried, and it makes sense, right? It's like, uh, you're better at like attending to the positions of different things which you need for like coding and manipulating math equations and.

• 1:31:08Sholto Douglas?

  I love this kind of research.

• 1:31:10Trenton Bricken?

  Yeah.

• 1:31:10Sholto Douglas?

  What's the name of the paper? Do you know?

• 1:31:12Trenton Bricken?

  If you look up like fine tuning models, math, David Bowles' group that came out like a week ago.

• 1:31:18Sholto Douglas?

  Okay, I am reading that when I get home.

• 1:31:20Trenton Bricken?

  I'm not endorsing the paper. That's like a longer conversation, but like this, it does talk about and cite other work on this like entity recognition ability.

• 1:31:30Dwarkesh Patel?

  One of the things you mentioned to me a long time ago is the evidence that when you train L L M's on code, they get better at reasoning in language. Which unless it's the case that the comments in the code are just really high quality tokens or something, implies that to be able to think through how to code better, like it makes you like a better reasoner and like, like that's crazy, right? Like, I think that's like one of the strongest pieces of evidence for like scaling, just making the thing smart. Like that kind of like positive transfer.

• 1:31:57Sholto Douglas?

  I think like this is, this is true in two senses. One is just that modeling code obviously implies modeling a difficult reasoning process used to create it. But two, that code is a nice explicit like structure of like. Composed reasoning, I guess, like if this, then that like uh codes a lot of structure in that way.

• 1:32:16Dwarkesh Patel?

  Um, yeah.

• 1:32:17Sholto Douglas?

  That you could imagine transferring to other types of types of reasoning problem.

• 1:32:21Dwarkesh Patel?

  Right. And crucially, the thing that makes us significant is that the uh it's not just stochastically predicting.

• 1:32:29Sholto Douglas?

  Right.

• 1:32:29Dwarkesh Patel?

  The next token of like words or whatever, because it's like learned that like uh Sally corresponds to murderer at the end of a Sherlock Holmes story. No, like if there is some shared thing between code and language, it must be at a deeper level that the model has learned.

• 1:32:44Sholto Douglas?

  Yeah, I think we have a lot of evidence that actual reasoning is occurring in these models and that like they're not just stochastic parrots.

• 1:32:50Dwarkesh Patel?

  Yeah.

• 1:32:51Sholto Douglas?

  Um, it just feels very hard for me to believe that I've worked and played with these models.

• 1:32:56Dwarkesh Patel?

  But normies will listen, will be like, you know.

• 1:33:02Trenton Bricken?

  Yeah, my my two like immediate cash responses to this are one, the work on Othello and now other games where it's like, I give you a sequence of moves in the game. And it turns out if you apply some like pretty straightforward interpretability techniques, then you can get a board that the model has learned.

• 1:33:19Sholto Douglas?

  Yeah.

• 1:33:19Trenton Bricken?

  And it's never seen the game board before anything, right? Like that's generalization. The other is Anthropic's influence functions paper. Um, that came out last year where they look at uh the model outputs like, please don't turn me off. I want to be helpful. And then they scan like, what was the data that led to that? And like, one of the data points that was very influential was someone uh dying of dehydration in the desert. And like having like a will to keep surviving. Um, and and to me, that just seems like very clear uh generalization of of motive. Rather than regurgitating, uh, don't turn me off. I think um two thousand one A Space Odyssey was also one of the influential things. And so that's that's more related, but. It's clearly pulling in things from from lots of different distributions.

• 1:34:02Sholto Douglas?

  And I also like the evidence you see even with like very small transformers where you can explicitly encode circuits to like do addition, right?

• 1:34:10Dwarkesh Patel?

  Yeah, or induction heads.

• 1:34:11Sholto Douglas?

  Or induction heads, this kind of thing. Like you could literally encode basic reasoning processes in the models manually. Um, and it seems clear that there's evidence that they also learn this automatically because you can then rediscover those from trained models.

• 1:34:24Dwarkesh Patel?

  Yeah.

• 1:34:24Sholto Douglas?

  To me, this is.

• 1:34:25Dwarkesh Patel?

  Yeah, the models are under parameterized.

• 1:34:27Sholto Douglas?

  Yeah.

• 1:34:28Dwarkesh Patel?

  They were asking them to do a very hard task. And they want to learn. The gradients want to flow. And so they need to, they're learning more more general skills.

• 1:34:37Sholto Douglas?

  Yeah.

• 1:34:38Dwarkesh Patel?

  Okay, so I want to take a step back from the research and ask about your careers specifically because like the tweet implied at the that I introduced you with, you've been in this field a year and a half. I think you've only been in it like a year or something, right?

• 1:34:55Trenton Bricken?

  I had a draw.

• 1:34:56Dwarkesh Patel?

  Yeah, but you know, like in that time, I know the solid alignment takes are overstated and you won't say this yourself because you'd be embarrassed, but like, you know, it's like a pretty incredible thing, like the thing that people in mechanistic tribally think is the biggest, you know, step forward and you've like been working on it for a year. It's notable. So I'm curious how you explain what's happened, like why in a year, a year and a half have you guys been,

• 1:35:25Dwarkesh Patel?

  you know, made important contributions to your field?

• 1:35:28Trenton Bricken?

  It goes without saying luck, obviously, and I feel like I've been very lucky and like the the timing of different progressions has has been just like really good in terms of advancing to the next level of growth.

• 1:35:42Trenton Bricken?

  I feel like for the interpretability team specifically, I joined when we were five people. We've now grown quite a lot.

• 1:35:49Trenton Bricken?

  But there were so many ideas floating around and we just needed to like really execute on them and have like quick feedback loops and like do careful experimentation that led to like signs of life and have now allowed us to like really scale. And I feel like that's kind of been my biggest value add to the team, which is not all engineering, but but quite a lot of it has been.

• 1:36:11Dwarkesh Patel?

  Interesting. So you're saying like you came at a point where like they were, there was, it had been a lot of science done and there was a lot of like good research floating around, but they needed someone to like just take that and like maniacally execute on it.

• 1:36:21Trenton Bricken?

  Yeah, yeah. And and there's, this is why it's not all engineering because it's like running different experiments and like having a hunch for why it might not be working and then like opening up the model or opening up the weights and like, what is it learning? Okay, well, let me try and do this instead and that sort of thing. But a lot of it has just been being able to do like very careful, thorough, but quick investigation of different ideas or yeah, theories.

• 1:36:44Dwarkesh Patel?

  And why was that lacking in the existing?

• 1:36:47Trenton Bricken?

  I don't know. I feel like I feel like I mean, I work quite a lot. And then I feel like I just am like quite agentic. Like if you're if your question is about like career overall. And I've been very privileged to have like a really nice safety net to be able to take lots of risks. But I'm just like quite headstrong. Like in undergrad, Duke had this thing where you could just make your own major. And it was like, eh, I don't like this prerequisite or this prerequisite. And I want to take all of four or five of these subjects at the same time. So I'm just gonna make my own major. Or like in the first year of grad school, I like canceled rotations. So I could work on this thing that became the paper we were talking about earlier. And like didn't have an advisor, like got admitted to do machine learning for protein design.

• 1:37:27Trenton Bricken?

  And was just like off in computational neuroscience land with no business there at all, but but worked out.

• 1:37:33Dwarkesh Patel?

  There's a headstrongness, but it seemed like another theme that jumped out was the the ability to step back. And you were talking about this earlier. The ability to take back from your sunk costs and go in a different direction. Is in a weird sense, the opposite of that, but also a crucial step here where I know like 21 year olds or like 19 year olds were like, ah, this is not the thing I've specialized in, or like I didn't major in this. I'm like, dude, motherfucker, you're 19. Like you can definitely do this. And you were like switching in the middle of grad school or something like that's just like.

• 1:38:04Trenton Bricken?

  Yeah, sorry. I didn't mean to cut you off, but I think it's like strong ideas loosely held. And being able to just like pinball in different directions. And the headstrongness, I think, relates a little bit to the fast feedback loops or agency. In so much as I I just don't get blocked very often. Like if I'm trying to write some code and like something isn't working, even if it's like in another part of the code base, I'll often just go in and fix that thing, or at least hack it together to be able to get results. And I've seen other people where they're just like, help, I can't. And it's like, no, that's not a good enough excuse. Like go all the way down.

• 1:38:35Dwarkesh Patel?

  I've definitely heard like people in management type positions talk about the lack of such people. Where they'll check in on somebody a month after they gave them a task or week after they give them a task. I'm like, how is it going? And they say, well, you know, we need to do this thing which requires lawyers. Because it requires talking about this regulation. It's like, how's that going? I was like, well, we need lawyers. I'm like, why didn't you get lawyers?

• 1:38:59Dwarkesh Patel?

  Or something like that. So that's definitely like, yeah.

• 1:39:01Sholto Douglas?

  I think that's arguably the most important quality in like almost anything. It's just pursuing it to like the end of the earth. I'm like, whatever you need to do to make it happen, you'll make it happen.

• 1:39:09Dwarkesh Patel?

  If you do everything you want.

• 1:39:10Sholto Douglas?

  If you do everything you want.

• 1:39:13Trenton Bricken?

  But yeah, yeah, yeah.

• 1:39:16Sholto Douglas?

  Um. I think from my side, uh, definitely that quality has been important, like agency and work. There are thousands, or I would even like probably tens of thousands of engineers at Google who are like, you know, basically like we're all like equivalent, like software engineering ability. Let's say like, you know, if you gave us like a very well-defined task, um, then we'd probably do it like equally well. Maybe a bunch of them would do it a lot better than me, you know, in all likelihood. Um, but what I've been, like, one of the reasons that I've been impactful so far is I've been very good at picking extremely high leverage problems. So problems that haven't been like particularly well solved so far.

• 1:39:54Sholto Douglas?

  Um, but perhaps as a result of like frustrating structural factors, like the ones that you pointed out in like that scenario before, where they're like, oh, we can't do X because this team went to do, why? Or like, and then going, okay, well, I'm just gonna like vertically solve the entire thing.

• 1:40:13Sholto Douglas?

  Uh, and that turns out to be remarkably effective. Also, uh, I'm very comfortable with, like, if I think there is something correct, uh, that needs to happen, I will like make that argument and continue making that argument and escalating levels of, uh, like, criticality until that thing gets solved. Um, and I'm also quite pragmatic with what like I do to solve things. You get a lot of people who come in with, as I said before, like a particular background or familiarity, or they're like, they know how to do something and they won't. Like, one of the beautiful things about Google, right, is you can run around and get world experts in literally everything. You can sit down and talk to people, optimization experts, like TP, like chip design experts, uh, like experts in,

• 1:40:55Sholto Douglas?

  I don't know, like different forms of like pre, like pre-training algorithms or like RL or whatever, and you can learn from all of them and you can take those methods and apply them. Um, and I think this was like maybe the, the start of why I was initially impactful was like this vertical, like agency effectively. Um, and then a follow-up piece from that is, I think it's often surprising how few people are like fully realizing all the things they want to do. They're like blocked or limited in some way. And this is very common, like in big organizations everywhere, people like have all these blockers on what they're able to achieve. Um, and I think.

• 1:41:34Sholto Douglas?

  Being a like one, helping inspire people to to work on particular directions and working with them on doing things massively scales your leverage. Like you get to work with all these wonderful people who teach you heaps of things. And generally like helping them push past organizational blockers. Means like together you get an enormous amount done. Like none of the impacts that I've had has been like me individually going off and solving a whole lot of stuff. It's been me maybe like starting off a direction and then convincing other people that this is the right direction and bringing them along in like this big tidal wave of like effectiveness that goes and solves that problem.

• 1:42:13Dwarkesh Patel?

  Yeah. We should talk about how you guys got hired because I think that's a really interesting story because you were a McKinsey consultant, right?

• 1:42:25Dwarkesh Patel?

  There's an interesting thing there where I first of all, I think people are. Yeah, generally people just don't understand how like decisions are made about either admissions or evaluating who to hire or something. But like just talk about like how were you noticed as?

• 1:42:42Sholto Douglas?

  Yeah.

• 1:42:43Dwarkesh Patel?

  Yeah, got hired.

• 1:42:44Sholto Douglas?

  So like the TLDR of this is I studied robotics in undergrad. I always thought that AI would be one of the highest leverage ways to impact the future in positive way. Like the the reason I am doing this is because I think it is like one of our best shots at making a wonderful future, basically. And I thought that working actually McKinsey, I would get a really interesting insight into what people actually did for work. Like in this, I actually wrote this as the first line in my cover letter to McKinsey was like, I want to work here so that I can learn what people do so that I can like understand.

• 1:43:13Sholto Douglas?

  And and in many respects, like I did get that. I've got a whole lot of other things. Many of the people there are like wonderful friends. I actually learned, I think, a lot of this like agentic behavior apart from my time there where you go into organizations and you see how impactful just not taking no for an answer gets you. Like it's like you would be surprised at the kind of stuff where like because no one no one quite cares enough in some organizations. Things just don't happen because no one's willing to take direct responsibility. This is incredibly like directly responsible individuals are ridiculously important. And people are willing to like, they just don't care as much about timelines.

• 1:43:53Sholto Douglas?

  And so much of the value that an organization like McKinsey provides is hiring people who you were otherwise unable to hire. For a short window of time where they can just like push through problems. Um, I think people like underappreciate this. Uh, uh, and so like at least some of my, well, hold up like. I'm going to become the directly responsible individual for this because no one's taking appropriate like responsibility. I'm going to care a hell of a lot about this and I'm going to make sure like I'm going to get into the earth to make sure it gets done. Comes from that time, but more to your like actual question of like, how did I uh? You get, get hired. Um, the entire time I didn't get into grad programs that I wanted to get into over here.

• 1:44:34Sholto Douglas?

  Um, which was specifically for focus on like robotics and RL research and that kind of stuff. Um, and in the meantime, on nights and weekends, basically every night from 10 P M till 2 A M. I would do uh my own like research and every weekend for like at least six to eight hours each day. I would do my own like research and coding projects and this kind of stuff. Um, and uh, that sort of switched in part from like. Quite robotics specific work to after reading uh Guern's scaling hypothesis post. I got completely scaling pills and was like, okay, like clearly the way that you solve robotics is by like scaling large multimodal models.

• 1:45:12Sholto Douglas?

  Um, and then in an effort to scale large multimodal models with a very, you know, grant, I got a grant from the, the TPU like access program. Um, to like tensor research cloud. Uh, I was trying to work out how to scale that effectively and um James Bradbury, uh, who at the time was at uh Google and is now at Anthropic. Um, saw some of my questions online where I was trying to work out how to do this properly. He was like, I thought I knew all the people in the world who were like asking these questions. Who on earth are you? Um, and uh, he, you know, he looked at that and he looked at some of the like the robotics stuff that had been putting up on my blog and that kind of thing. And he reached out and said, hey, do you want to have a chat and you want to um like explore working with us here?

• 1:45:53Sholto Douglas?

  Um. And I was hired, as I understand it later, as an experiment in trying to take someone with extremely high enthusiasm in an agency and pairing them with some of the best engineers that he knew.

• 1:46:05Dwarkesh Patel?

  Yeah.

• 1:46:06Sholto Douglas?

  Um. And so one, another one of the reasons, like I say, like I've been impactful is I I had this like dedicated mentorship from utterly wonderful people, like people like Reiner Pope, who has since left to go do his ownership company, Anselm Woskaya, James himself, many others, but those are like the sort of formative, like two to three months at the beginning. And they taught me a whole lot of like the principles and like heuristics that I apply, like how to and how to like solve problems in the way that they have, particularly in that like systems and algorithms overlap where like one more thing that makes you like quite effective in ML research is really completely understanding the systems side of things. And this is something I've learned from them, basically, is like

• 1:46:48Sholto Douglas?

  a deep understanding of how systems influence algorithms and how algorithms influence systems. Because the systems constrain the design space, so the solution space, which you have available to yourself in the algorithm side. And very few people are comfortable fully bridging that gap. But a place like Google, you can just like go and ask all the algorithms experts and all the systems experts, everything they know, and they will happily teach you. And if you go sit down with them, they like, they will teach you everything they know. It's wonderful. And this has meant that I've been able to be very, very effective for both sides, like for the pre-training crew, because I understand systems very well, I can intuit and understand like, this will work well or this won't. And then, and like flow that on through the inference considerations of models and this kind of thing. And for

• 1:47:31Sholto Douglas?

  like, to the chip design teams, I'm one of the people they turn to to understand what chips they should be designing in three years, because I'm one of the people who's best able to understand and explain the kind of algorithms that we might want to design in three years. And obviously, you can't make very good guesses about that, but like,

• 1:47:50Sholto Douglas?

  I I think I like convey the information well accumulated from all of my compatriots on the pre-training crew and like the general like systems inside group and convey that information well to them because also even inference applies a constraint to pre-training. And so like there's this like these trees of constraints where if you understand all the pieces of the puzzle, then you get a much better sense for like what the solution space might look like.

• 1:48:14Dwarkesh Patel?

  Yeah, that that's really, there's a couple of things that stick out to me there. One is not just the agency of the person who was hired, but the parts of the system that we're able to think, wait, that's really interesting. Who is this guy? Not for a grad program or anything, you know, like currently a McKinsey consultant, just like out undergrad. But that's interesting. Let's like give this a shot, right? So James and whoever else that's like, that's very notable. And that's, second is, I actually didn't know this part of the story where that was part of an experiment run internally about can we do this? Can we like bootstrap somebody? And like, yeah. And in fact, what's really interesting about

• 1:48:56Dwarkesh Patel?

  that is the third thing you mentioned is having somebody who understands all layers of the stack and isn't so stuck on any one approach or any one layer of abstraction is so important. And specifically that like what you mentioned about being being bootstrapped immediately by these people might have meant that since you're getting up to speed on everything at the same time, rather than spending grad school, going deep on like one specific way of doing RL, you actually can take the global view and aren't like totally bought in on one thing. So not only is it something that's possible, but like has greater returns than just hiring somebody out of grad school potentially because this person can just like, I don't know, just like getting GPT eight and like fine tuning them on like one year of, you know what I mean? So, yeah.

• 1:49:40Sholto Douglas?

  You come at everything with fresh eyes.

• 1:49:41Dwarkesh Patel?

  Yep.

• 1:49:41Sholto Douglas?

  And you're not coming locked to any particular field.

• 1:49:44Dwarkesh Patel?

  Yeah.

• 1:49:45Sholto Douglas?

  Now, what like one caveat to that is that before, like during my self experimentation and stuff, I was reading everything I could. I was like obsessively reading papers every night. And like, actually, funnily enough, I like.

• 1:49:58Sholto Douglas?

  Read much less widely now that I like my day is occupied by working on things. Um. And in some respect, I had like this very broad perspective before where not that many people, even even like in a PhD program, you'll like focus on a particular area. Um. If you just like read all the NLP work and all the computer vision work and like all the robotics work, you like see all these patterns to start to emerge across subfields. Um. In a way that I guess like foreshadowed some of the the work that I would later do.

• 1:50:24Dwarkesh Patel?

  That that that's super interesting. One of the reasons that you've been able to be agentic within Google is like your pair programming half the days or most of the days is Sergey Brin, right? And so that's really interesting that like, um, there there's this person who's like willing to just push ahead on this uh L L M stuff and like get rid of the local blockers in its place.

• 1:50:45Sholto Douglas?

  Um. I think important to give is like not like every day or anything that I'm preparing, but like when uh there are particular projects that he's interested in, then like we'll work together on those. I'm like, but there's also been times when he's been focused on projects with other people. Um. But in general, yes, there's a surprising alpha to like being one of the people who actually goes down to the office every day. Um. That like is really actually shouldn't be, but is surprisingly impactful. Um. And as a result, uh, I I've like benefited a lot from having like basically being like close friends with people in leadership who care. Um. And being able to like really argue convincingly about why we should do X as opposed to Y. Um. And and having that like vector to uh

• 1:51:30Sholto Douglas?

  try and like it's Google is a big organization, um, having having those vectors helps a little bit. Um. But also it's it's very important. It's the kind of thing you don't want to ever abuse, right? Like you, you want to make the argument through all your like all the right channels and like uh, on only sometimes you need to.

• 1:51:46Dwarkesh Patel?

  And so this includes you like Sergey Brin, Jeff D and so forth. I mean, it's like, it's notable. I don't know. I feel like Google is undervalued, given that like.

• 1:51:53Sholto Douglas?

  Yeah. That's like, I don't know, like Steve Jobs is working on the next product for Apple, like pair programming on it or something. Right. I mean, like, I, uh, yeah, I've benefited immensely from like, so for example, during the Christmas break, um, I, uh, go, I was just going into the office a couple, like a couple of days during that time.

• 1:52:12Trenton Bricken?

  Sounded like quite a lot of days.

• 1:52:20Sholto Douglas?

  And, uh, I don't know if you guys have read that article about Jeff and Sanjay doing the pair programming, but they were there pair programming on stuff. Um, and I got to hear about all these, like, these cool stories of, like, early Google where they're talking about, like, crawling under the floorboards and rewiring data centers and, like, telling me how many, like, bits they were pulling off the, uh, how many bytes they were pulling off the instructions of a given compiler and, uh, instruction and, like, all these, like, crazy little performance optimizations they were doing. Like, they were having the time of their life. Um, and I got to, like, sit there and really, like, experience this, uh, this sense of history in a way that you, you don't expect to get. Like, you expect to be very far away from all that, I think, maybe in a large organization, but.

• 1:53:00Dwarkesh Patel?

  Yeah. That's, that is super cool. Uh, Trenton, does this map onto any of your experience?

• 1:53:05Trenton Bricken?

  I think Sholto's story is more, more exciting. Um, mine was just very serendipitous in that I, I got into computational neuroscience, didn't have much business being there. Um, my first paper was mapping the cerebellum to the attention operation in transformers. My next ones were looking at like.

• 1:53:21Dwarkesh Patel?

  How old were you when you wrote that?

• 1:53:23Trenton Bricken?

  Uh, it was my first year of grad school.

• 1:53:24Dwarkesh Patel?

  Okay.

• 1:53:25Trenton Bricken?

  Um, so 22.

• 1:53:27Dwarkesh Patel?

  Oh, yeah.

• 1:53:28Trenton Bricken?

  But, uh, yeah, my, my next work was on, uh, sparsity in networks, like inspired by sparsity in the brain, uh, which was when I met Tristan Hume, uh, and Anthropic was doing the SOLO, the softmax linear output unit work, which was, was very related in quite a few ways of like, let's make the, uh, activation of neurons across a layer really sparse. And if we do that, then we can get some interpretability of what the neuron's doing. Um, I think we've updated on that approach towards what we're doing now. Um, so that, that started the conversation. I shared drafts of that paper with Tristan. He was excited about it. And, and then, and, and that was basically what led me to be, become Tristan's resident and then convert to full time. Um, but during that period, I also moved as a visiting researcher to Berkeley, uh, and started working with Bruno

• 1:54:12Trenton Bricken?

  Olshausen, uh, both on what's called vector symbolic architectures, which, uh, one of the core operations of them is literally superposition. And on sparse coding, also known as dictionary learning, which is literally what we've been doing since. And Bruno Olshausen basically invented sparse coding back in 1997. And so it was like, the, my research agenda and the interpretability team seemed to just be running in parallel in, with just research taste. And, and so it, yeah, it made a lot of sense for, for me to work with the team. And it's been a dream since.

• 1:54:48Dwarkesh Patel?

  One thing I've noticed when people tell stories about their careers or their successes, they ascribe it way more to contingency. But when they hear about other people's stories, they're like, of course it wasn't contingent. You know what I mean? It's like, if that didn't happen, something else would have happened. I've just noticed that something like talked to, and it's like, interesting that you both think that they're like, it was especially contingent. Whereas, I don't know, maybe you're right, but like, it's a sort of interesting pattern that.

• 1:55:15Trenton Bricken?

  Yeah, but I mean, like, I literally met Tristan at a conference and like, wasn't, didn't have a scheduled meeting or anything, just like, joined a little group of people chatting and he happened to be standing there. And I happened to mention what I was working on. And that led to more conversations. And I think I probably would have applied to Anthropic at some point anyways, but I would have waited at least another year. I, I, yeah, I, it's still crazy to me that I can like actually contribute to interpretability in a meaningful way.

• 1:55:41Sholto Douglas?

  I, I think there's a big, important aspect of like shots on goal there, so to speak, right? Where like, you're even just going to, choosing to go to conferences itself is like, putting yourself in a position where you're, where luck is more likely to happen.

• 1:55:52Trenton Bricken?

  Yeah.

• 1:55:53Sholto Douglas?

  And like conversely, in my own situation, it was like, doing all of this work independently and trying to produce and do interesting things was my own way of like trying to manufacture luck, so to speak. And like, try and do something meaningful enough that it got noticed.

• 1:56:07Trenton Bricken?

  Given that you said, you framed this in the context of they were trying to run this experiment of can something.

• 1:56:12Sholto Douglas?

  Specifically, James and I think our manager Brennan was trying to run this experiment.

• 1:56:16Trenton Bricken?

  It like worked, did they do it again?

• 1:56:18Sholto Douglas?

  Yeah, so my like closest collaborator Enrique, he, he crossed from search through to our team. He's also been ridiculously impactful. He's definitely a stronger engineer than I am. And he didn't go to university.

• 1:56:32Trenton Bricken?

  How was, like?

• 1:56:33Dwarkesh Patel?

  What was notable about your example is James Bradbury, somebody who's usually this kind of stuff is like phoned out to recruiters or something like that. Whereas James, like somebody whose time is worth like hundreds of millions of dollars.

• 1:56:47Dwarkesh Patel?

  You know what I mean? Uh, so, uh,

• 1:56:51Dwarkesh Patel?

  that like that thing is like very bottlenecked on that kind of person taking the time almost in like aristocratic tutoring sense of finding and then getting up to speed. And it seems like if it works at this well, it should be done at scale. Like it should be the responsibility of key people to like, you know what I mean? On board and find.

• 1:57:10Sholto Douglas?

  I think that is true to many extents. Like I'm sure you probably benefited a lot from the key researchers mentoring you deeply during.

• 1:57:17Dwarkesh Patel?

  And like actively like looking on like open source repositories or like on forums or whatever for like potential people like this.

• 1:57:24Sholto Douglas?

  Yeah, I mean, James is like Twitter injected into his brains.

• 1:57:32Sholto Douglas?

  But yes, uh, and I think this is something which in practice is done. Like people do look out for people that they find interesting and like try and find high signal. In fact, actually, this, I was talking about this with Jeff the other day. And Jeff said that, yeah, he's like, you know, I, um, I, one of the most important hires I ever made was a offer called email. Uh, and I was like, who was that? And he's Chris Olah.

• 1:57:55Dwarkesh Patel?

  Ah, yeah.

• 1:57:58Sholto Douglas?

  Because Chris similarly had had no, you know, background and, uh, in, well, like in a formal background in ML, right? And like Google Brain was just getting started and this kind of thing. But Jeff saw, saw that signal and the, and the residency program, which like Brain had is I think also like a it was astonishingly effective of finding good people that didn't have strong ML backgrounds. And, uh,

• 1:58:24Dwarkesh Patel?

  Yeah. One of the other things that's I, I want to like emphasize for a potential slice of the audience that would be relevant to is there's this sense that like the world is legible and efficient of

• 1:58:39Dwarkesh Patel?

  Companies have these go to jobs dot google dot com or jobs dot whatever company dot com. And you apply and there's the steps and like they will evaluate you efficiently on those steps. Whereas not only from the storage teams, like often that's not the way it happens. That's in fact, it's good for the world that that's not often how it happens. Like it is important to look at, um, were they able to like write an interesting block, uh, technical block post about their research or like making interesting contributions.

• 1:59:08Dwarkesh Patel?

  Yeah, I want you to like riff on for the people who are like assuming that the other end of the job board is like, just like super legible and mechanical. This is not how it works. And in fact, like people are looking for the sort of different way, different kind of person who's agentic and putting stuff out there.

• 1:59:24Sholto Douglas?

  And I think specifically what people are looking for there is two things. One is agency and like putting yourself out there. And the second is the ability to do world class something.

• 1:59:34Dwarkesh Patel?

  Yeah.

• 1:59:35Sholto Douglas?

  And two examples that I always like to point to here are Andy Jones from Anthropic did an amazing paper on scaling laws as applied to board games. It didn't require much resources. It demonstrated incredible engineering skill. It demonstrated incredible understanding of like the most topical problem of the time. Um, and he didn't come from a like typical academic background or whatever. And as I understand it, basically like as soon as he came out with that paper, both Anthropic and OpenAI were like, we would desperately like to hire you. There's also someone who works on Anthropic's, uh, performance team now, Simon Bohm, who has written in my mind, the reference for optimizing a CUDA map mall, like, uh, on a GPU.

• 2:00:14Sholto Douglas?

  And that demonstrated example of like taking some like prompt effectively and producing the world-class reference example for it. In something that wasn't particularly well done, uh, so far is like, I think an incredible demonstration of like ability and agency. That in my mind would like be an immediate, would like, please love to like interview you slash hire.

• 2:00:35Dwarkesh Patel?

  Yeah. The only thing I can add here is, I mean, I still had to go through the whole hiring process and all the standard interviews and this sort of thing.

• 2:00:41Sholto Douglas?

  Yeah, everyone does.

• 2:00:41Dwarkesh Patel?

  Yeah.

• 2:00:42Sholto Douglas?

  Wait, is that, is that, doesn't that seem stupid?

• 2:00:46Dwarkesh Patel?

  I mean.

• 2:00:46Sholto Douglas?

  It's important to de-biasing.

• 2:00:48Dwarkesh Patel?

  Yeah, yeah.

• 2:00:49Sholto Douglas?

  And the bias is what you want, right? Like you want to bias on somebody who's got great taste and like, he's like, like who care like. Your interview process should be able to disambiguate that as well.

• 2:00:58Trenton Bricken?

  Yeah, like I think there are cases where someone seems really great and then it's like, oh, they actually just can't code. This sort of thing, right? Like, how much you weight these things definitely matters, though. And like, I think the we take references really seriously. The interviews, you can only get so much signal from. And so it's all these other things that can can come into play for whether or not a hire makes sense.

• 2:01:17Sholto Douglas?

  But you should design your interviews such that, like, they test the right things.

• 2:01:22Dwarkesh Patel?

  One man's bias is another man's taste, you know?

• 2:01:28Trenton Bricken?

  I guess the only thing I would add to this, or maybe to the headstrong context, is like, there's this line, the system is not your friend.

• 2:01:35Dwarkesh Patel?

  Right.

• 2:01:36Trenton Bricken?

  And it's not necessarily to say it's actively against you, or it's your sworn enemy.

• 2:01:42Trenton Bricken?

  It's just not looking out for you.

• 2:01:44Dwarkesh Patel?

  Right.

• 2:01:44Trenton Bricken?

  And so I think that's where a lot of the proactiveness comes in of like, there are no adults in the room. Or like, and like, you have to come to some decision for what you want your life to look like and then execute on it. And yeah, hopefully you can then update later, if you're too headstrong in the wrong way. But but I think you almost have to just kind of charge at certain things to get much of anything done. Not be swept up in the tide of whatever the expectations are.

• 2:02:10Sholto Douglas?

  There's like one final thing I want to add, which is like, we talked a lot about agency and this kind of stuff. But I think actually, surprisingly enough, one of the most important things is just caring an unbelievable amount. And when you care an unbelievable amount, you like, you check all the details and you have like, this understanding of like, what could have gone wrong? And you like, it just, it matters more than you think, because people end up not caring. Or not caring enough. This is like LeBron quote, where he talks about how, when he, he sort of, before he started in the league, he was like worried that everyone would be like incredibly good. And then he gets there and he like realizes that actually, once people hit financial stability, then they like, they relax a bit. And he's like, oh, it's gonna be easy.

• 2:02:52Sholto Douglas?

  I don't think that's quite true, because I think in like AI research, because most people actually care quite deeply. But there's caring about your problem, and there's also just caring about the entire stack and everything that goes up and down, like going explicitly going and fixing things that aren't your responsibility to fix, because overall, it makes like, the stack better.

• 2:03:09Dwarkesh Patel?

  I mean, another part of that I forgot to mention is, you were mentioning, oh, going in on weekends and on Christmas break, and you get to like, the only people in the office are Jeff Dean and Sergey Brin or something. You just like get to pay a program with them. It's just, it's interesting to me that people, I don't want to pick on your company in particular, but like people at any big company, they've gone there because they've gone through a very selective process. That's like they had to compete in high school, they had to compete in college, but it almost seems like they get there and then they take it easy. When in fact, this is a time to put the pedal to the metal, go in and pair program with Sergey Brin on the weekends or whatever. You know what I mean?

• 2:03:47Sholto Douglas?

  I mean, there's pros and cons there, right? I think many people make the decision that the thing that they want to prioritize is like a wonderful life with their family. And if they're, they do wonderful work, like let's say they don't work every hour of the day, right? But they do wonderful work in the work, like the hours that they do do, that's incredibly impactful. I think this is true for many people at Google is like, maybe they don't work as many hours as like your typical startup mythologize, right? But the work that they do do is incredibly valuable. It's very high leverage because they know the systems and they're experts in their field. And we also need people like that. Like our world rests on these huge, like difficult to manage and difficult to fix systems. And we need people who are like willing to work on and help and fix and maintain those in frankly,

• 2:04:30Sholto Douglas?

  like a thankless way that isn't as like high publicity as all of this AI work that we're doing, right? And I'm like ridiculously grateful that those people do that. And I'm also happy that there are people for whom like, okay, they find technical fulfillment in their job and doing that well. And also like, maybe they draw a lot more from also out of spending like a lot of hours with their family. And I'm lucky that I'm at a stage in my life where like, yeah, I can go in and work every hour of the week. But like, that's like, I'm not making as many sacrifices to do that.

• 2:04:59Dwarkesh Patel?

  Yeah. I mean, like, just one example that sticks out in my mind of this sort of like, the other side says no, and you can still get the yes on the other end. Basically, every single high profile of guests I've gone so far, I think maybe with one or two exceptions, I've sat down for a week and I've just come up with a list of sample questions that, you know, like tried to really come up with really smart questions to send to them. And the entire process, I've always thought like, if I just cold email them, it's like a 2% chance they say yes. If I include this list, there's a 10% chance. And because otherwise, you know, there's like, you go through their inbox and every 34 seconds, there's an interview for whatever podcast, interview whatever podcast. And every single time I've done this, they've said yes. Right?

• 2:05:44Sholto Douglas?

  Yeah.

• 2:05:45Dwarkesh Patel?

  You just like, you dig-

• 2:05:46Sholto Douglas?

  You ask great questions.

• 2:05:47Dwarkesh Patel?

  But-

• 2:05:48Sholto Douglas?

  If you do everything, you'll win.

• 2:05:49Dwarkesh Patel?

  But you're just like, you literally have to dig in the same hole for like 10 minutes. Or in that case, like make a sample, list of sample questions for them to get past their not an idiot list. You know what I mean?

• 2:05:59Sholto Douglas?

  And demonstrate how much you care.

• 2:06:00Dwarkesh Patel?

  Yeah, yeah.

• 2:06:02Sholto Douglas?

  And the work you're willing to put in, yeah.

• 2:06:03Dwarkesh Patel?

  Yeah.

• 2:06:03Trenton Bricken?

  Yeah. Something that a friend said to me a while back, but I think is stuck is like, it's amazing how quickly you can become world class at something, just because most people aren't trying that hard. And like are only working like, I don't know, the actual like 20 hours that they're actually spending on this thing or something. And so, yeah, if you just go ham, then like you can, you can get really far pretty fast.

• 2:06:24Sholto Douglas?

  And I think I'm lucky I had that experience with the fencing as well. Like I had the experience of becoming world class in something and like knowing that if you just worked really, really hard and were like-

• 2:06:33Dwarkesh Patel?

  Yeah. For, for context, by the way, Sholto was one seat away as he was the next person in line to go to the Olympics stuff for fencing.

• 2:06:42Sholto Douglas?

  I was at best like 42nd in the world for fencing, for men's foil fencing. Uh, and-

• 2:06:47Dwarkesh Patel?

  Mutational load is a thing, man.

• 2:06:52Sholto Douglas?

  And there was a, there was one cycle where, yeah, I was like the next highest ranked person in Asia. And if one of the teams had been like disqualified for doping as it was occurring, um, in part during that cycle. And as occurred for like the Australian women's running team, I think went because one of the teams was disqualified, then I would have been the next in line.

• 2:07:15Dwarkesh Patel?

  It's interesting when like you just like find out people's prior lives and it's like, oh, you know, this guy was almost an Olympian. This other guy was whatever, you know what I mean? Okay, let's talk about interoperability.

• 2:07:26Sholto Douglas?

  Yeah.

• 2:07:28Dwarkesh Patel?

  I actually want to stay on the brain stuff as a way to get into it for a second. Uh, we're previously discussing,

• 2:07:37Dwarkesh Patel?

  is the brain organized in the way where you have a residual stream that is gradually refined with higher level associations over time or something?

• 2:07:48Dwarkesh Patel?

  There's a fixed dimension size in a model. If you had to, I don't even know how to ask this question in a sensible way. But what is the D model of the brain? What is it like the embedding size of, or because of feature splitting, is that not a sensible question?

• 2:08:05Trenton Bricken?

  No, I think it's a sensible question. Well, it is a question that makes sense.

• 2:08:13Trenton Bricken?

  I don't know how you would begin to kind of be like, okay, well, this part of the brain is like a vector of this dimensionality. I mean, maybe for the visual stream, because it's like V1 to V2 to IT, whatever. You could just count the number of neurons that are there and be like, that is the dimensionality. But it seems more likely that there are kind of submodules and things are divided up. So yeah, I don't have, and I'm not like the world's greatest neuroscientist, right? Like, I did it for a few years. I like studied the cerebellum quite a bit. So I'm sure there are people who could give you a better answer on this.

• 2:08:55Dwarkesh Patel?

  Do you, do you think that the way to think about whether it's in the brain or whether it's in these models, fundamentally, what's happening is like, features are added, removed, changed, and like, the feature is the fundamental unit of what is happening in the model? Like, what would have to be true for, give me, and this goes back to the earlier thing we were talking about, whether it's just associations all the way down. Give me like a counterfactual, in the world where this is not true, what is happening instead? Like, what is the alternative hypothesis here?

• 2:09:29Trenton Bricken?

  Yeah, it's hard for me to think about, because at this point, I just think so much in terms of this feature space. I mean, at one point, there was like the kind of behavioralist approach towards cognition, where, or it's like, you're just, you're like, input-output, but you're not really doing any processing. Or it's like, everything is embodied, and you're just like a dynamical system that's like, operating, like, along like, some predictable equations. But like, there's no state in the system, I guess. But whenever I've read these sorts of critiques, it's like, well, you're just choosing to not call this thing a state. But you could call like, any internal component of the model

• 2:10:12Trenton Bricken?

  a state. Like, even with the feature discussion, it's, defining what a feature is, is really hard. And so, the question feels almost too slippery.

• 2:10:23Dwarkesh Patel?

  Mm hmm. What is a feature?

• 2:10:25Trenton Bricken?

  A direction and activation space. A latent variable that is operating behind the scenes. That has like causal influence over the system you're observing. It's a feature if you call it a feature. It's total logical. I mean, these are, these are all explanations that I like. I feel some association.

• 2:10:48Sholto Douglas?

  In a very rough intuitive sense, in like a sufficiently sparse and like binary. It's the fact that features like whether or not something is turned on or off, right? Like in a very simplistic sense.

• 2:10:58Trenton Bricken?

  Yeah.

• 2:10:58Sholto Douglas?

  Yeah. Which might be, I think, a useful metaphor to understand it by. It's like when we talk about features activating, it is in various respects, the same way that neuroscientists would talk about like a neuron activating, right?

• 2:11:09Trenton Bricken?

  If that neuron corresponds to.

• 2:11:11Sholto Douglas?

  To something in particular.

• 2:11:12Trenton Bricken?

  Right. Yeah. I know, I think that's useful as like, what do we want a feature to be, right? Like, what is a synthetic problem under which a feature exists? But even with the towards monosemitic city work, we talk about what's called feature splitting, which is basically you will find as many features as you give the model the capacity to learn. And by model here, I mean the, the, the up projection that we, we fit after we trained to the original model. And so if you don't give it much capacity, it'll learn a feature for bird. But if you give it more capacity, then it will learn like ravens and eagles and sparrows and like specific types of birds.

• 2:11:51Dwarkesh Patel?

  Still on definitions thing, I, I guess naively, I think of things like bird versus. What kind of token is it like a, is it like a period at the end of a hyperlink as you were talking about earlier? Versus at the highest level, things like love or deception or like holding a very complicated proof in your head or something. Is this all features? Cause then the definition seems so broad as to almost be not that useful.

• 2:12:22Trenton Bricken?

  Um.

• 2:12:23Dwarkesh Patel?

  Like, I, I, or rather that there's, seems to be some important differences between these things and if they're all features, like, yeah, I'm not sure what we even mean by.

• 2:12:31Trenton Bricken?

  I mean, all of those things are like discreet units that have connections to other things that then imbues them with meaning.

• 2:12:41Trenton Bricken?

  I, that feels like a, a, a specific enough definition that it's, it's useful or not too all encompassing, but feel free to push back.

• 2:12:48Dwarkesh Patel?

  Well, like, what would you discover tomorrow in, um, uh, that could make you think like, oh, this is like, kind of fundamental to the wrong way to think about what's happening in a model?

• 2:12:58Trenton Bricken?

  I mean, if the features we were finding weren't predictive, or if they were just representations of the data, um, right, where it's like, oh, you're, all, all you're doing is just clustering your data, um, and there's no, like, higher level associations that are being made, or it's some, like, phenomenological thing of, like, you're call, you're saying that this feature fires for marriage, but if you activate it really strongly, it doesn't change the outputs of the model in a way that would correspond to it. Like, I think those, these would both be good critiques. I, I, I guess one more is, um, and, and we tried to do experiments on MNIST, which is a data set of, of digits, i- images, and we didn't look super hard into it, and so

• 2:13:41Trenton Bricken?

  I'd, I'd be interested if people, other people wanted to take up, like, a deeper investigation. Um, but it's plausible that, uh, your, like, latent space of representations is dense, and it's a manifold, instead of being these discrete points. And so you could, like, move across the manifold, but at every point, there would be some meaningful behavior, um, and it's much harder than to label things as features that are discrete.

• 2:14:06Dwarkesh Patel?

  Like, in a naive, sort of outsider way, the thing that would seem to me to be, like, a way in which this picture could be wrong is, if there's not some, like, this thing is turned on, turned off, but it's like a much more global, kind of, like, the, the system is a, I don't, I, I'm gonna use really clumsy, like, you know, I measured it in a pretty kind of language, but, um, is there a good analogy here?

• 2:14:34Dwarkesh Patel?

  Yeah, I guess if you think of like something like the laws of physics, it's not like, well, the feature for wetness is turned on, but it's only turned on this much. And then the feature for like, you know, I guess maybe it's true because like the mass is like a gradient and like,

• 2:14:51Dwarkesh Patel?

  you know, like, I don't know, with the polarity or whatever is a gradient as well. But there's also a sense in which like there's the laws and the laws are more general and you have to understand like the general bigger picture. You don't get that from just like these like specific sub, uh, sub circuits.

• 2:15:07Sholto Douglas?

  But that's where like the reasoning circuit itself comes into play, right? Where you're taking these features ideally and like trying to compose them into something higher level. Like you might say, okay, like when I'm using, at least this is my headcanon. Um, so let's say I'm trying to use the foot, you know, F equals MA, right? Then presumably at some point I have features which like denote, okay, like mass. And then that's like helping me retrieve the actual mass of the thing that I'm using. And then like, uh, like the acceleration and this kind of stuff. But then also, uh, maybe there's a higher level feature that does correspond to using the first law of physics, maybe. But the more important part is that the, the composition of components which helps me retrieve a piece, relevant piece of information and then produce like maybe some like a, you know, multiplication operator or something like that when necessary. Um, at least that's my headcanon.

• 2:15:50Dwarkesh Patel?

  What is a compelling explanation to you, especially for very smart models of,

• 2:15:57Dwarkesh Patel?

  like, I understand why it made this output and it was like for a legit reason. If it's doing million line pull request or something, what are you seeing at the end of that request? Where you're like, yep, shit, that's chill.

• 2:16:10Trenton Bricken?

  Yeah. So ideally you apply dictionary learning to the model. You've, you've found features. Um, right now we're actively trying to get the same success for attention heads. In which case, um, we have features for both the core, you, you can do it for residual stream, MLP and attention throughout the whole model. Uh, hopefully at that point, you can also identify broader circuits through the model that are like more general reasoning abilities, um, that will activate or not activate. But in your case where we're trying to figure out if this like pull request should be approved or not. Um, I think you can flag or detect features that correspond to deceptive behavior, malicious behavior, these sorts of things and see whether or not those have fired. That would be like an immediate, you can do more than that, but that would be an immediate.

• 2:16:52Dwarkesh Patel?

  But before I trace down on that, What does the reasoning circuit look like? What would that look like when you found it?

• 2:16:59Trenton Bricken?

  Yeah, so I mean, the induction head is probably one of the simplest.

• 2:17:01Dwarkesh Patel?

  But it's not like reasoning, right?

• 2:17:02Trenton Bricken?

  Well, I mean, what do you call reasoning, right? Like, it's, it's, it's, it's a good reason. So I guess context for listeners, the induction has basically, and you see the line like Mr. and Mrs. Dursley did something, Mr. Blank, and you're trying to predict what Blank is. And the head has learned to look for previous occurrences of the word Mr. Look at the word that comes after it and then copy and paste that as the prediction for what should come next, which is a super reasonable thing to do. And there is computation being done there to accurately predict the next token.

• 2:17:40Dwarkesh Patel?

  Mm hmm. But yeah, that is context dependent.

• 2:17:43Trenton Bricken?

  That is, yeah, yeah.

• 2:17:44Dwarkesh Patel?

  But it's not like, it's not like reasoning, you know what I mean? Like.

• 2:17:47Trenton Bricken?

  But, but is, is, I guess, going back to the like associations all the way down. It's like if you chain together a bunch of these reasoning circuits or, or heads that have different rules for how to relate information.

• 2:18:01Dwarkesh Patel?

  But, but in this sort of like zero shot case, like something is happening where when you like pick up a new game and you immediately start understanding how to play it. And it doesn't seem like an induction heads kind of thing, or like.

• 2:18:12Trenton Bricken?

  Well, I think there would be another circuit for like extracting pixels and turning them into latent representations of the different objects in the game, right? And like a circuit that is learning physics.

• 2:18:24Dwarkesh Patel?

  And what would that, because the induction heads is like one layer transformer.

• 2:18:28Trenton Bricken?

  Uh, you need the two layers.

• 2:18:30Dwarkesh Patel?

  Yeah. So you can like kind of see like what the like, the thing that is a human picks up a new game and understands it. How like, how do, how would you think about what that is? Is it, presumably it's across multiple layers, but like, is it? Yeah, yeah. How, like, what would that physically look like?

• 2:18:47Trenton Bricken?

  Um, how big would it be maybe or like, uh, I mean, that would just be an empirical question, right? Of like, how big does the model need to be to perform this task? But like, I mean, maybe it's useful if I just talk about some other circuits that we've seen. So we've seen like, um, the IOI circuit, which is the indirect object identification. And so this is like, if you see, it's like Mary and Jim went to the store, Jim gave the object to blank, right? And it would predict Mary because Mary's appeared before as like the indirect object, or um, it'll infer pronouns, right? Um, and this circuit even has behavior where like, if you oblate it, then like other heads in the model will pick up that behavior. Um,

• 2:19:30Trenton Bricken?

  we'll even find heads that want to do copying behavior and then other heads will suppress. So like, it's one job's, one head's job to just always copy, like the token that came before, for example, um, or the token that came five before or whatever. And then it's another head's job to be like, no, do not copy that thing. Um, so there are lots of different uh, circuits performing in these cases, pretty basic operations. But when they're chained together, you can get unique behaviors.

• 2:19:59Dwarkesh Patel?

  And but like, is the story of how you found it with the reasoning thing is like, because you won't be able to understand, or it'll just be like really, you know, it won't be something you can see in like a two layer transformer. So will you just be like, the circuit for deception or whatever, it just, this, this part of the network fired when we at the end identified the thing as being deceptive, this part, and it didn't fire when we did identify it as being deceptive, therefore this must be the deception circuit?

• 2:20:24Trenton Bricken?

  Uh, I think a lot of analysis like that. Um, like, like Anthropic has done quite a bit of research before on Sycophancy, which is like the model saying what it thinks you want to hear.

• 2:20:35Dwarkesh Patel?

  And that requires us at the end to be able to label which one is like bad and which one is good.

• 2:20:40Trenton Bricken?

  Yeah, so we have tons of instances. And actually, as you make models larger, they do more of this. Um, where the model is clearly, um, it has uh, like features that model another person's mind. Um, and these activate and like some subset of these, um, we're hypothesizing here, but like would be associated with more deceptive behavior.

• 2:21:02Dwarkesh Patel?

  Although like, it's doing that by, I don't know, ChatGPT, I think it's probably modeling me, because that's like RLHF induces it to.

• 2:21:09Trenton Bricken?

  Yeah, theory of mind. Yeah.

• 2:21:11Dwarkesh Patel?

  Well, first of all, the thing you mentioned earlier about there is redundancy. So then it's like, well, have you caught like the whole thing that could cause deception in the whole thing? Or like, is it just one instance of it?

• 2:21:22Trenton Bricken?

  Yeah.

• 2:21:23Dwarkesh Patel?

  Second of all, are your like labels correct? You know, maybe like you, you thought this wasn't deceptive. It's like still deceptive, especially if it's producing output you can't understand. Third, is the thing that's going to be the bad outcome? Something that's even human understandable, like deception is a concept we can understand. Maybe there's like a.

• 2:21:39Trenton Bricken?

  Yeah, yeah. So a lot to unpack here. So I guess a few things. One, it's fantastic that these models are deterministic. When you sample from them, it's stochastic, right? But like, I can just keep putting in more inputs and oblate every single part of the model. This is kind of the pitch for computational neuroscientists to come and work on interpretability. It's like you have this alien brain and you have access to everything in it. And you can just oblate however much of it you want. And so I think if you do this carefully enough, you really can start to pin down. What are the circuits involved? What are the backup circuits? These sorts of things. The kind of cop out answer here, but it's important to keep in mind is doing automated interpretability. So it's like, as our models continue to get more capable, having them assign labels or like run some of these experiments at scale.

• 2:22:23Trenton Bricken?

  And then with respect to like, if there's superhuman performance, how do you detect it? Which I think was kind of the last part of your question. Aside from the cop out answer. If we buy this associations all the way down, you should be able to coarse grain the representations at a certain level such that they then make sense. I think it was even in Demis's podcast. He's talking about like, if a chess player makes a superhuman move, they should be able to distill it into reasons why they did it. And, and like, even if the model's not going to tell you what it is, you should be able to decompose that complex behavior into simpler circuits or features to, to really start to make sense of why it did the thing that it did.

• 2:23:07Dwarkesh Patel?

  There's a separate question of does such representation exist, which it seems like there must, or actually, I'm not sure if that's a case. And secondly, whether using this sparse word encoder setup, you could find it. And in this case, if you don't have labels for it that are adequate to represent it, like you wouldn't find it, right?

• 2:23:26Trenton Bricken?

  Um, yes and no. So like, we are actively trying to use dictionary learning now on the sleeper agents work, which we talked about earlier. And it's like, if I just give you a model, can you tell me if there's this trigger and it's going to start doing interesting behavior? And it's an open question whether or not, when it learns that behavior, it's part of a more general circuit that we can pick up on without actually getting activations for and having it display that behavior, right? Because that would kind of be cheating then.

• 2:23:54Trenton Bricken?

  Or if it's learning some hacky trick over, like that's a separate circuit that you'll only pick up on if you actually have it do that behavior. But even in that case, the geometry of features gets really interesting, because like fundamentally, each feature like is in some part of your representation space, and they all exist with respect to each other. And so in order to have this new behavior, you need to carve out some subset of the feature space for the new behavior and then push everything else out of the way to make space for it. So hypothetically, you can imagine you like have your model before you've taught it this bad behavior. You know all the features or like have some coarse grain representation of them. You then fine tune it such that it becomes malicious. And then you can kind

• 2:24:37Trenton Bricken?

  of identify this like black hole region of feature space where like everything else has been shifted away from it. And there's like this region and like you haven't put in an input that like causes it to fire. But then you can start searching for what is the input that would cause this part of the space to fire? What happens if I activate something in this space? There are like a whole bunch of other ways that you can try and attack that problem.

• 2:24:57Dwarkesh Patel?

  Um, this is sort of a tangent, but one interesting idea I heard was if that space is shared between models, you can imagine trying to find it in an open source model to then make like Gemma is, they said in the paper, Gemma, by the way, Google's newly released open source model. They said in the paper, it's used trained using the same architecture or something like that.

• 2:25:19Trenton Bricken?

  I had to be honest, I didn't know because I haven't read the Gemma paper.

• 2:25:22Dwarkesh Patel?

  It's a similar method, it's something whatever as Gemini. So it's an explanation that's true. I don't know, how much like, how much of the retraining you do on Gemma is like potentially helping you jailbreak into Gemini?

• 2:25:33Trenton Bricken?

  Yeah, this gets into the fun space of like, how universal are features across models? And our towards the monoceramicity paper, looked at this a bit. And we find, I can't give you summary statistics, but like the base 64 feature, for example, which we see across a ton of models. This is like, if there are actually three of them, but they'll fire four in model base 64 encoded text, which is prevalent in like every URL and there are lots of URLs in the training data. They have really high cosine similarity across models. So like they all learn this feature and I mean, within a rotation, right? But it's like, yeah, yeah.

• 2:26:07Sholto Douglas?

  Like the actual like vectors itself.

• 2:26:08Trenton Bricken?

  Yeah, yeah. And I, I, I wasn't part of this analysis. But yeah, that it definitely finds the feature and they're like pretty similar to each other across two separate, two models, the same model architecture, but trained with different random seeds.

• 2:26:21Sholto Douglas?

  It supports the quantum theory of neural scaling is like a hypothesis, right? Which is like all models on like a similar data set will learn the same features in the same order ish roughly. Like you learn your engrams, you learn your induction heads and you learn like to put full stops after numbered lines and this kind of stuff.

• 2:26:35Dwarkesh Patel?

  Hey, but by the way, okay, so this is another tangent. To the extent that that's true, and like, I guess there's evidence that that's true. Why doesn't curriculum learning work? Cause if it is the case that you learn certain things first, shouldn't just directly training those things first lead to better results?

• 2:26:48Sholto Douglas?

  Both Gemini papers mention some like aspect of curriculum learning.

• 2:26:52Dwarkesh Patel?

  Okay, interesting. And I find the fact that fine tuning works is like evidence of curriculum learning, right? Cause like the last things you're training on have a disproportionate impact.

• 2:27:01Sholto Douglas?

  I wouldn't necessarily say that, like there's one mode of thinking in which fine tuning is specialized. Like you've got this like latent bundle of capabilities and you're like specializing for its particular like use case that you want.

• 2:27:12Trenton Bricken?

  I think.

• 2:27:12Sholto Douglas?

  I'm not sure how true it is.

• 2:27:13Trenton Bricken?

  I think the David Bell lab kind of paper kind of supports this, right? Like you have that ability and you're just like getting better at entity recognition.

• 2:27:19Sholto Douglas?

  Right.

• 2:27:19Trenton Bricken?

  Like fine tuning that circuit instead of other ones.

• 2:27:21Sholto Douglas?

  Yeah.

• 2:27:21Trenton Bricken?

  Yeah.

• 2:27:22Sholto Douglas?

  I'm sorry, what was the?

• 2:27:23Dwarkesh Patel?

  The thing we're talking about before.

• 2:27:24Sholto Douglas?

  But generally, I do think like curriculum learning is really interesting. People should explore more. Um, and it like seems very plausible. I would really love to see more analysis along the lines of the quantum theory stuff. When like understanding better, what do you actually learn at each stage and like decomposing that out? Um, and exploring whether or not curricula change that. Um, or.

• 2:27:42Dwarkesh Patel?

  But by the way, I just realized, forgot. We did, I just like got in conversation mode and forgot there's an audience. Uh, curriculum learning is when you organize a data set. When you think about a human, how they learn, they they don't just see like random wiki text and they just like try to predict it, right? They're like, we'll start you off with like, um, uh, Lorax or something and then you'll learn. I I don't even remember what first grade was like, but you'll learn the things that first graders learn and then like second graders and so forth. Um, and so you'd imagine.

• 2:28:08Sholto Douglas?

  Sorry, we we know you go, never got past first grade.

• 2:28:25Dwarkesh Patel?

  Okay, anyways, uh, let's get back to like the big, before we get into like a bunch of like, uh, interim details. The big picture, um, there's two threads I want to explore. First is, I guess it makes me a little worried that there's not even an alternative formulation of what could be happening in these models that could invalidate this approach. Which feels like, I mean, we do know that we don't understand intelligence, right? Like, there are definitely unknown unknowns here. So, like, the fact that there's not a null hypothesis, I don't know, I feel like. Well, what if, what if we're just wrong and we don't even know the way in which we're wrong, which actually increases the uncertainty? And, yeah.

• 2:29:03Trenton Bricken?

  Yeah, yeah, yeah. Um, so it's not that there aren't other hypotheses. It's just, I have been working on superposition for like a number of years.

• 2:29:12Dwarkesh Patel?

  Yeah.

• 2:29:12Trenton Bricken?

  And and very involved in this effort. And so I'm I'm less sympathetic to or will like.

• 2:29:17Dwarkesh Patel?

  They just said they're wrong.

• 2:29:20Trenton Bricken?

  Like to these other approaches, especially because our recent work has been so successful.

• 2:29:25Sholto Douglas?

  Yeah, it's like quite high explanatory power.

• 2:29:27Trenton Bricken?

  Like, there's this beauty, like in the scaling laws paper, there's this little bump at a particular,

• 2:29:32Sholto Douglas?

  like the original scaling laws paper, there's a little bump. And that apparently corresponds to when the model learns induction heads. And then like after that, it's like, so it goes off track, learns induction heads, gets back on track. Which is like an incredible piece of retroactive explanatory power.

• 2:29:47Trenton Bricken?

  Yeah.

• 2:29:49Trenton Bricken?

  Before I forget it, though, I do have one thread on future universality that you might wanna have in. So there's some really interesting behavioral evolutionary biology experiments on like, should humans learn a real representation of the world or not? You could imagine a world in which we saw all venomous animals as like flashing neon pink, a world in which we survive better. And so it would make sense for us to not have a realistic representation of the world. And there's some work where they'll simulate like little basic agents and see if the representations they learn like map to the like tools they can use and like the inputs they should have. And it turns out if you have these little agents perform more than a certain number of tasks,

• 2:30:34Trenton Bricken?

  given these basic tools and objects in the world, then they will learn a like ground truth representation. Because like there are so many possible use cases that you need for these base objects that you actually want to learn what the object actually is and not some like cheap visual heuristic or other thing. And so to the extent that we are doing, and we haven't talked at all about like first things free energy principle or predictive coding or anything else, but like to the extent that all living organisms are trying to like actively predict what comes next and form like a really accurate world model, it wouldn't surprise me or I'm optimistic that we are learning genuine features about the world that are good for modeling it.

• 2:31:15Trenton Bricken?

  And our language models will do the same, at least especially because we're training them on human data and human text.

• 2:31:22Dwarkesh Patel?

  Another dinner party question. Isn't, should we be less worried about misalignment? And maybe that's not even the right word for what I'm referring to, but like this alienness and chagathness from these models, given that there is future universality and there are certain ways of thinking and ways of understanding the world that are. Instrumentally useful to different kinds of intelligences. Should we just be less worried about like bizarre paper code maximizers as a result?

• 2:31:51Trenton Bricken?

  I think that's the, this is kind of why I bring this up as like the optimistic take.

• 2:31:55Dwarkesh Patel?

  Yeah.

• 2:31:56Trenton Bricken?

  Predicting the internet is very different from what we're doing though, right? Like the models are way better at predicting next tokens than we are. They're trained on so much garbage, they're trained on so many URLs. Like in the dictionary learning work, we find there are like three separate features for base 64 encodings. And like even that is kind of an alien example that is probably worth me talking about for a minute. Like one of these base 64 features fired for numbers. One like, like other base 64, like if it sees base 64 numbers, it'll like predict more of those. Another fired for letters, but then there was this third one that we didn't understand. And it like fires like a very specific subset of, of base 64 features. And someone on the team who clearly knows way too much about base 64, realized that this was the subset that was ASCII decodable.

• 2:32:42Trenton Bricken?

  So you could decode it back into the ASCII characters. And the fact that the model like learned these three different features and it took us a little while to like figure out what was going on. Was, is very Shogoth-esque.

• 2:32:56Dwarkesh Patel?

  That is, it has a denser representation of like regions that are particularly relevant to predicting the next token.

• 2:33:02Trenton Bricken?

  Yeah, because it's so, but yeah. And it's clearly doing something that humans wouldn't, right? Like you can even talk to any of the current models in base 64 and it will apply in base 64.

• 2:33:11Dwarkesh Patel?

  Right.

• 2:33:11Trenton Bricken?

  And you can then like decode it and it works great.

• 2:33:15Dwarkesh Patel?

  That particular example, I wonder if that implies that the difficulty of doing interpretability on smarter models will be harder because if like it requires somebody with esoteric knowledge to just happen to see that base 64 has, I don't know, like whatever that distinction was. Doesn't that imply when you have a million line pull request, it's like, there is no human that's going to be able to decode like two different reasons why the pull request, there's like two different features for this pull.

• 2:33:43Trenton Bricken?

  Yeah, you know what I mean? Like, uh, yeah.

• 2:33:45Dwarkesh Patel?

  So if you think. And that's when you type a comment like small C Ls, please.

• 2:33:48Trenton Bricken?

  Yeah, exactly. No, no, I mean, you could do that, right? This is like, what I was gonna say is like, one technique here is anomaly detection.

• 2:33:54Dwarkesh Patel?

  Yeah.

• 2:33:55Trenton Bricken?

  And so one beauty of dictionary learning instead of like linear probes is that it's unsupervised. You are just trying to learn to span all of the representations that the model has. Um, and then interpret them later. But if there's a weird feature that suddenly fires for the first time that you haven't seen fire before, that's a red flag. Um, you could also coarse grain it so that it's just a single base 64 feature. I mean, even the fact that that this came up and we could see that it specifically favors these particular outputs and it fires for these particular inputs gets you a lot of the way there. I'm even familiar with cases from the auto interp side where a human will look at a feature and try to annotate it for, it fires for, um, Latin words.

• 2:34:35Trenton Bricken?

  And then when you ask the model to classify it, it says it fires for Latin words defining plants. So it can like already like beat the human in some cases for like labeling what's going on.

• 2:34:46Dwarkesh Patel?

  So at scale, this would require an adversarial, um, uh, thing between models where like some model that you have like millions of features potentially for G P T six and some like it just, a bunch of models are just trying to figure out what each of these features means. How, uh, uh, is that right?

• 2:35:06Trenton Bricken?

  Okay. Yeah, but you can even automate this process, right? I mean, it's, this goes back to the determinism of the model. Like you could have a model that is actively editing input text and predicting if the feature is gonna fire or not. And and figure out what makes it fire, what doesn't and like search the space.

• 2:35:22Dwarkesh Patel?

  Yeah, I, I wanna talk more about the feature splitting cuz I think that's like an interesting thing that has been under.

• 2:35:27Trenton Bricken?

  Yeah.

• 2:35:27Dwarkesh Patel?

  Explored.

• 2:35:28Trenton Bricken?

  Especially for scalability. I think it's, it's underappreciated right now.

• 2:35:31Dwarkesh Patel?

  First of all, like, how do we even think about, is it really just? You can keep going down and down, like there's no end to the amount of features like.

• 2:35:40Trenton Bricken?

  I mean, so, so at some point, I think you might just start fitting noise. Or things that are part of the data, but that the model isn't actually representing.

• 2:35:48Dwarkesh Patel?

  Do you want to explain what feature splitting is?

• 2:35:50Trenton Bricken?

  Yeah, yeah. So it's the, it's the part before where like the model will learn however many features it has capacity for. That still span the space of, of representations.

• 2:36:00Dwarkesh Patel?

  So like, give an example potentially.

• 2:36:01Trenton Bricken?

  Yeah, yeah. So you learn, if you don't give the model that much capacity for the features it's learning. Concretely, if you project to not as high dimensional space, it will learn one feature for birds. But if you give the model more capacity, it will learn features for all the different types of birds. And so it's, it's more specific than otherwise. And, and oftentimes like there's the bird vector that points in one direction. And all the other specific types of birds point in like a similar region of the space. But are obviously more specific than the, the course label.

• 2:36:35Dwarkesh Patel?

  Okay. So let's go back to G P T seven. First of all, is this a sort of like linear tax on any model to figure out? First, actually even before that, is this a one time thing you had to do? Or is this the kind of thing you have to do on every output? Or just like one time, it's not deceptive, we're good to go, roll. Actually, yeah, let me, let me literally answer that.

• 2:36:54Trenton Bricken?

  Yeah. So you do dictionary learning after you've trained your model. And you feed it a ton of inputs. And, and you get the activations from those and then you do this projection into the higher dimensional space. And so the method is it's unsupervised and then it's trying to learn these sparse features. You're not telling them in advance what they should be. But it is constrained by the inputs you're giving the model. I guess two caveats here, one, like we can. Try and choose what inputs we want. So if we're looking for theory of mind features that might lead to deception, we can put in the sick of fancy data set. Hopefully, at some point, we can move into looking at the weights of the model alone, or at least using that information to do dictionary learning.

• 2:37:36Trenton Bricken?

  But I think in order to get there, that's like such a hard problem that you need to make traction on just learning what the features are first. But yeah, so what's the cost of this?

• 2:37:45Dwarkesh Patel?

  Can you read the last sentence? Weights of the model alone.

• 2:37:48Trenton Bricken?

  So, so, so like right, right now, we just have these neurons in the model. They don't make any sense.

• 2:37:52Dwarkesh Patel?

  Yeah.

• 2:37:53Trenton Bricken?

  We apply dictionary learning, we get these features out, they start to make sense. But that's, that depends on the activations of the neurons. The weights of the model itself, like what neurons are connected to what other neurons, certainly has information in it. And, and the, the dream is that we can kind of bootstrap towards actually making sense of the weights of the model that are independent of the activations of the data. I mean, this is, I'm not saying we've made any progress here. It's a very hard problem, but it, it feels like we'll have a lot more traction and be able to like sanity check what we're finding with the weights if we're able to pull out features first.

• 2:38:27Dwarkesh Patel?

  For the audience, weights are permanent, but I don't know if permanent is the right word, but like they are the model itself, whereas activations are the sort of like artifacts of any single call.

• 2:38:39Sholto Douglas?

  In a brain metaphor, you know, the weights are like the actual connection scheme between neurons and the activations of the current neurons that are lining up.

• 2:38:46Dwarkesh Patel?

  Yeah. Yeah, okay, so there's going to be two steps to this for GPT-7 or whatever model we're concerned about. One, actually, first, correct me if I'm wrong, but like, training the sparse autoencoder and like, do the unsupervised projection into a wider space of features that have a higher fidelity to like what is actually happening in the model. And then secondly, label those features. How, because let's say like the cost of training the model is N. What will those two steps cost relative to N?

• 2:39:18Trenton Bricken?

  We will see, like, it really depends on.

• 2:39:23Trenton Bricken?

  Two main things, what is your expansion factors? Like, how much are you projecting into the higher dimensional space? And how much data do you need to put into the model? How many activations do you need to give it? But this brings me back to the feature splitting to a certain extent, because if you know you're looking for specific features, you can start with a really a cheaper, like coarse representation. So maybe my expansion factor is like only two. So like, I have a thousand neurons, I'm projecting to a 2,000 dimensional space. I get 2,000 features out, but they're really coarse. And so previously, I had the example for birds. Let's move that example to like, I have a biology feature. And but I really care about if the model has representations for bioweapons, is trying to manufacture them. And so what I actually want is like an anthrax feature.

• 2:40:07Trenton Bricken?

  What you can then do is rather than, and let's say the anthrax, you only see the anthrax feature if instead of going from 1,000 dimensions to 2,000 dimensions, I go to a million dimensions, right? And so you can kind of imagine this this big tree of semantic concepts, where like biology splits into like cells versus like whole body biology. And then further down, it splits into all these other things. So rather than needing to immediately go from 1,000 to a million, and then picking out that one feature of interest, you can find the direction that the biology feature is pointing in, which again is very coarse, and then selectively search around that space. So like, only do dictionary learning if this, if something in the direction of the biology feature fires first.

• 2:40:49Trenton Bricken?

  And so the the computer science metaphor here would be like, instead of doing breadth first search, you're able to do depth first search, where you're only recursively expanding and exploring a particular part of this like semantic tree of features.

• 2:41:03Dwarkesh Patel?

  Although, given the way that these features are not organized in things that are intuitive for humans, right? Like, because we just don't have to deal with basics before, so we don't have that many, you know, we just don't dedicate that much, like, whatever, for more to like, deconstructing which kind of basics before it is. How would we know that the subjects? And this will go back to maybe the MOE discussion we'll have of.

• 2:41:28Sholto Douglas?

  I guess we might as well talk about it, but like in mixture of experts, the mixture of paper talked about how they couldn't find the the experts weren't specialized in a way that we could understand. There's not like a chemistry expert or a physics expert or something. So why would you think that like it'll be like biology feature and then deconstruct rather than like blah, and then you just deconstruct and it's like anthrax and you're like shoes and whatever.

• 2:41:52Trenton Bricken?

  So I haven't read the the mistral paper.

• 2:41:54Sholto Douglas?

  Yeah.

• 2:41:54Trenton Bricken?

  But I think that the heads, I mean, this goes back to like, if you just look at the neurons in a model, they're polysomatic. And so if all they did was just look at the neurons in a given head, it's very plausible that it's also polysomatic because of superposition.

• 2:42:09Sholto Douglas?

  I want to just tug on the thread that Dork is mentioned there. Have you seen in the subtrees when you expand them out, like something in a sub tree, which like you really wouldn't guess that it should be there based on like the higher level abstraction.

• 2:42:19Trenton Bricken?

  So so this is a line of work that we haven't pursued as much as I want to yet. But I think we're planning to, I hope that maybe external groups do as well. Like, what is the geometry of feature space?

• 2:42:29Sholto Douglas?

  What's the geometry of exactly?

• 2:42:30Trenton Bricken?

  And how does that change over time?

• 2:42:31Sholto Douglas?

  It would really suck if like anthrax feature happened to be like below the like, you know, coffee can like sub tree or something like that, right?

• 2:42:39Trenton Bricken?

  Totally, totally.

• 2:42:40Sholto Douglas?

  And that feels like the kind of thing that you could quickly try and find like proof of, which would then like mean that you need to like then solve that problem.

• 2:42:48Trenton Bricken?

  Yeah, yeah.

• 2:42:48Sholto Douglas?

  And inject more structure into the geometry.

• 2:42:50Trenton Bricken?

  Totally. I mean, it would really surprise me, I guess, especially like given how linear the model seems to be. That like there isn't some component of the anthrax feature, like vector that is similar to and looks like the biology vector and that they're not in a similar part of the space. But yes, I mean, ultimately, machine learning is empirical.

• 2:43:06Sholto Douglas?

  Yeah.

• 2:43:07Trenton Bricken?

  We need to do this. I think it's gonna be pretty important for certain aspects of scaling dictionary learning.

• 2:43:11Sholto Douglas?

  Yeah, interesting. On the MOE discussion.

• 2:43:15Trenton Bricken?

  Yeah.

• 2:43:16Sholto Douglas?

  There's an interesting scaling vision transformers paper that Google put out a little while ago, where they like do image net classification with a like an MOE. And they find really clear class specialization there for experts, like there's a clear dog expert.

• 2:43:30Trenton Bricken?

  Wait, so like the mixture of people just not do a good job of like identifying?

• 2:43:33Sholto Douglas?

  I think I think it's it's hard, like it and like it's entirely possible that. Well, like, in some respects, there's almost no reason that, like, all of the different archive, like, features should go to one expert. Like, you could have biology, like, let's say, I don't know what buckets they had in their paper, but let's say they had, like, archive papers as, like, one of the things. And you could imagine, like, biology papers going here, math papers going here, and all of a sudden, your, like, breakdown is, like, ruined. But that vision transformer one, where the class separation is really clear and obvious, gives, I think, some evidence towards the specialization hypothesis.

• 2:44:07Trenton Bricken?

  So, so I think images are also in some ways just easier to interpret than text.

• 2:44:11Sholto Douglas?

  Yeah, exactly.

• 2:44:12Trenton Bricken?

  And like, so, so Chris Olaz's, like, interpretability work on AlexNet and these other models, like, in the original AlexNet paper, they actually split the model into two GPUs, just because they couldn't, like, GPUs were so bad back then, relatively speaking, right? Like, still great at the time. That was one of the big innovations of the paper. But they find branch specialization, and there's a Distill Pub article on this, where, like, colors go to one GPU and, like, Gabor filters and, like, line detectors go to the other. And then, like, all of the other...

• 2:44:45Sholto Douglas?

  Really?

• 2:44:45Trenton Bricken?

  Yeah, yeah, yeah. And then, like, all of the other interpretability work that was done, like, a lot, like, the floppy ear detector, right? Like, that just was a neuron in the model that you can make sense of. You didn't need to disentangle superposition, right? So, so just different, different dataset, different modality.

• 2:45:03Sholto Douglas?

  Like, I think a wonderful research project to do, if someone is, like, out there listening to this, would be to try and disentangle, like, take some of the techniques that Trenton's team has worked on, and try and disentangle the neurons in the, the mixture paper, like, mixture model, which is over-sourced. Like, I think that's a fantastic thing to do, because it feels intuitively like they should be. They didn't demonstrate any evidence that there is. There's also, like, in general, a lot of evidence that there should be specialization. Go and see if you can find it. Like, and that's, that's work that had, that Anthropic has published most of its stuff on, like, as I understand it, like, dense models, basically.

• 2:45:36Sholto Douglas?

  That is a wonderful research product to try.

• 2:45:38Trenton Bricken?

  And given Dwarkesh's success with the Vesuvius challenge.

• 2:45:41Sholto Douglas?

  Yeah.

• 2:45:42Trenton Bricken?

  We should be pitching more projects because they will be solved.

• 2:45:46Dwarkesh Patel?

  What I was thinking about after the Vesuvius challenge was like, wait, I knew, like Nat had told me about it before it dropped because we recorded the episode before it dropped. Why didn't you, why did I not even try like?

• 2:45:59Dwarkesh Patel?

  Like, I don't know, like Luke is obviously very smart and like, uh, yeah, he's an amazing kid, but like, he showed that like a 21 year old on like some 1070 or whatever he was working on could do this. I don't know. Like, I feel like I should have.

• 2:46:13Trenton Bricken?

  So you know what? I'm before this episode drops, I'm gonna meet my.

• 2:46:19Dwarkesh Patel?

  No, no, no, I'm not gonna like try to go research really. I was honestly thinking back on experience like, wait, I shouldn't like, why did that not even fuck?

• 2:46:24Trenton Bricken?

  Yeah, your hands dirty.

• 2:46:27Sholto Douglas?

  Dwarkesh's request for research.

• 2:46:32Dwarkesh Patel?

  Oh, I want to harp back on this, like the neuron thing. You said, I think a bunch of your papers have said there's more features than there are neurons. And this is just like, wait a second.

• 2:46:45Dwarkesh Patel?

  I don't know, like a neuron is like weights go in and a number comes out. That's like, a number comes out, you know what I mean? Like that's, that's so little information. Like there's, do you mean like there's like street names and like species and whatever? There's like more of those kinds of things than there are like a number comes out in a, in a model?

• 2:47:05Trenton Bricken?

  That's right. Yeah.

• 2:47:06Dwarkesh Patel?

  But how is, a number comes out is like so little information. How is that encoding for like?

• 2:47:09Trenton Bricken?

  Superposition. You're just encoding, you're encoding a ton of features in these high dimensional vectors.

• 2:47:16Dwarkesh Patel?

  In a brain, is there like an axonal firing or however you think about it, like? I don't know how you think about like how, how, how much like superposition is there in the human brain?

• 2:47:25Trenton Bricken?

  Yeah. So Bruno Olshausen, who I think of as the leading expert on this, uh, thinks that all the brain regions you don't hear about are doing a ton of computation and superposition. So everyone talks about V1 as, as like having Gabor filters and, and detecting lines of, of certain various sorts and no one talks about V2. And I think it's because like we just haven't been able to make sense of it.

• 2:47:47Sholto Douglas?

  What is V2?

• 2:47:48Trenton Bricken?

  Uh, it's like the next part of the visual processing stream. And, and it's like, yeah, so I, I think it's very likely. Um, and fundamentally like superposition seems to emerge when you have high dimensional data that is sparse. And to the extent that you think the real world is that, which I would argue it is, we should expect the brain to also be under parameterized in trying to build a model of the world and also use superposition.

• 2:48:10Sholto Douglas?

  You can get a good intuition for this, uh, in, in correctly, like this example is wrong, in like a 2D plane, right? Let's say you have like two axes, right? Which represents like a two dimensional, um, like feature space here, like two, two neurons, basically.

• 2:48:23Sholto Douglas?

  And you can imagine them each like turning on to various degrees, right? And that's, that's like your X coordinate and your Y coordinate. But you can like, now like map this onto a plane. You can actually represent a lot of different things and like different

• 2:48:34Trenton Bricken?

  Parts of the plane.

• 2:48:35Dwarkesh Patel?

  Oh, okay. So, uh, crucially, then superposition is not an artifact of a neuron. It is an artifact of like the space that is created.

• 2:48:43Trenton Bricken?

  It's a combinatorial code.

• 2:48:44Dwarkesh Patel?

  Yeah, yeah, exactly. Okay, cool.

• 2:48:47Trenton Bricken?

  Yeah, thanks.

• 2:48:49Dwarkesh Patel?

  I, I, I mean, we kind of talked about this, but like, I think it's just like kind of wild that it seems to the best of our knowledge, the way intelligence works in these models and then presumably also in brains, it's just like, there's a stream of information going through that has quote unquote features that are infinitely, or at least, to a large extent to just like splitable and, uh, uh, you like, you can expand out a tree of like what this feature is and what's really happening is a stream, like that feature is getting turned into this other feature or this other feature is added. I don't know. It's like, that's not something I would have just like thought like that's what intelligence is. You know what I mean? It's like a surprising thing. It's not, it's not whatever we would have expected necessarily.

• 2:49:34Trenton Bricken?

  What did you think it was?

• 2:49:35Dwarkesh Patel?

  I don't know, man.

• 2:49:37Dwarkesh Patel?

  I mean, yeah.

• 2:49:39Trenton Bricken?

  So that's a great segue because all of this feels like GoFi. Like you're using distributed representations. But you have features and you're applying these operations to the features. I mean, the whole field of vector symbolic architectures, which is this computational neuroscience thing. It all you do is you put vectors in superposition. And which is literally a summation of two high dimensional vectors. And you create some interference, but but if it's high dimensional enough, then you can you can represent them. And you have variable binding, where you connect one by another. And like, if you're dealing with binary vectors, it's just the X or operation. So you have A, B, you bind them together. And then if you query with A or B again, you get out the other one. And this is basically the like key value pairs from attention.

• 2:50:24Trenton Bricken?

  And with these two operations, you have a Turing complete system. Which you can, if you have enough nested hierarchy, you can represent any data structure you want, et cetera, et cetera. Um, yeah.

• 2:50:38Dwarkesh Patel?

  Okay, let's go back to the superintelligence. So like, walk me through G P T seven. Um, you've got like the sort of depth first search on its features. Okay, um, G P T seven has been trained. What happens next? Your your research has succeeded. G P T seven has been trained. What are you, what are, what are we doing now?

• 2:50:57Trenton Bricken?

  Um, we try and get it to do as much interpretability work and other like safety work as possible.

• 2:51:03Dwarkesh Patel?

  No, but like a concrete, like what is, um, what has happened such that you're like, cool, let's deploy G P T seven?

• 2:51:08Trenton Bricken?

  Oh, geez. Um, I mean, I do like, like we have our responsible scaling policy, which has been really exciting to see other labs adopt. And um, like, this is only from the perspective of your, your research is net, like, trained and given your research. You got the, we got the thumbs up on G P T seven from you. Or actually, we should say cloud, whatever. Uh, and then, uh, oh, I like, what, what is the basis on which you're telling the team like, hey, let's go ahead? I mean, I think we need to make a lot more. If if it's as capable as G P T seven like implies here. Um, I think we need to make a lot more interpretability progress to be able to like.

• 2:51:43Dwarkesh Patel?

  Even answer the question?

• 2:51:45Trenton Bricken?

  Comfortably give the green light to deploy it. Like, I would be like, definitely not, I'd be crying.

• 2:51:52Trenton Bricken?

  Maybe my tears would interfere with the, uh, G P U's.

• 2:51:55Dwarkesh Patel?

  But like, what is?

• 2:51:57Trenton Bricken?

  Guys.

• 2:51:59Sholto Douglas?

  Gemini five, T P U's.

• 2:52:07Dwarkesh Patel?

  But like, what, given the way your research is progressing, like, what does it kind of look like to you? Like, if this succeeded, what would it mean for us to OK GPT-7 based on your methodology?

• 2:52:21Trenton Bricken?

  I mean, ideally, we can find some compelling deception circuit, which lights up when the model knows that it's not telling the full truth to you.

• 2:52:30Dwarkesh Patel?

  Why can't you just train a linear probe like Colin Bird's did?

• 2:52:33Trenton Bricken?

  So the CCS work is not looking good in terms of replicating or like actually finding truth directions. And like, in hindsight, it's like, well, why should it have worked so well? But linear probes, like, you need to know what you're looking for, and it's like a high-dimensional space, and it's really easy to pick up on a direction that's just not.

• 2:52:49Dwarkesh Patel?

  Wait, but don't you also, here, you need to label the features, so you still need to know.

• 2:52:52Trenton Bricken?

  Well, you need to label them post hoc, but it's unsupervised. You're just like, give me the features that explain your behavior, is the fundamental question, right? It's like, the actual setup is, we take the activations, we project them to this higher-dimensional space, and then we project them back down again. So it's like, reconstruct or do the thing that you were originally doing, but do it in a way that's sparse.

• 2:53:14Dwarkesh Patel?

  By the way, for the audience, linear probe is, you just like classify the activations. I don't know, from what I vaguely remember about the paper was like, if it's like telling a lie, then you like, you just train a classifier on like, is it? Yeah, but in the end, was it a lie or was it just like wrong or something? I don't know. It was like true or false question.

• 2:53:36Trenton Bricken?

  It's like a classifier on the activations.

• 2:53:40Trenton Bricken?

  So, yeah, like, right now, what we do for GPT-7, like, ideally, we have like some deception circuit that we've identified that like appears to be really robust. And, and it's like.

• 2:53:50Dwarkesh Patel?

  Well, like, so you've done the projecting out to the million whatever features or something. Is a circuit, because we maybe we're using feature and circuit interchangeably when they're not.

• 2:54:01Trenton Bricken?

  So is there like a deception circuit?

• 2:54:03Dwarkesh Patel?

  So I think there are features.

• 2:54:05Trenton Bricken?

  There are features across layers that create a circuit.

• 2:54:07Dwarkesh Patel?

  Yeah.

• 2:54:08Trenton Bricken?

  And hopefully, the circuit gives you a lot more specificity and sensitivity than an individual feature. And it's like, hopefully, we can find a circuit that is really specific to you being deceptive. The model deciding to be deceptive in cases that are malicious, right? Like, I'm not interested in a case where it's just doing theory of mind to like, help you write a better email to your professor. And I'm not even interested in cases where the model is is necessarily just like modeling the fact that deception has occurred.

• 2:54:40Dwarkesh Patel?

  But doesn't all this require you to have labels for all those examples? And if you have those labels, then like whatever faults that the linear probe has on the like, maybe you like labeled along the thing or whatever. Wouldn't the same thing apply to the labels you've come up with for the unsupervised features you've come up with?

• 2:55:00Trenton Bricken?

  So in an ideal world, we could just train on like the whole data distribution.

• 2:55:06Trenton Bricken?

  And then find the directions that matter.

• 2:55:10Trenton Bricken?

  To the extent that we need to reluctantly narrow down the subset of data that we're looking over, just for the purposes of scalability. We would use data that looks like the data you'd use to fit a linear probe. But again, we're not like with the linear probe, you're also just finding one direction. Like, we're finding a bunch of directions here.

• 2:55:27Dwarkesh Patel?

  And I guess the hope is like, you found like, a bunch of things that light up when it's being deceptive. And then like, you can figure out why some of those things are lighting up in this part of the distribution and not this other part and so forth.

• 2:55:37Trenton Bricken?

  Totally. Yeah.

• 2:55:38Dwarkesh Patel?

  Do you anticipate you'll be understand? Like, I don't know, like, the current models you've studied are pretty basic, right? You think you wouldn't understand why G P seven fires in certain domains, but not in other domains?

• 2:55:49Trenton Bricken?

  I'm optimistic. I mean, we've so so I guess one thing is this is a bad time to answer this question. Because we are explicitly investing in the longer term of like A S L four models, which G P T seven would be. But like, so so we split the team where a third is focused on scaling up dictionary learning right now. And that's been great. I mean, we publicly. Shared our some of our eight layer results. We've scaled up quite a lot past that at this point. But the other two groups, one is trying to identify circuits and then the other is trying to get the same success for attention heads. So we're setting ourselves up and building the tools necessary to really find these circuits at a compelling way. But it's going to take another, I don't know, six months before that's like really working well. But but like I can say that I'm like optimistic and we're making a lot of progress.

• 2:56:32Dwarkesh Patel?

  What is the highest level feature you've found so far?

• 2:56:35Trenton Bricken?

  Ooh.

• 2:56:35Dwarkesh Patel?

  Like it's basic before or whatever. It's like, maybe it's just like, um, in the symbolic species language, the book you recommended, there's like indexical uh things where you're just. I forgot what all the labels were, but like there's things where you're just like, uh, you see a tiger and you're like run and whatever, you know, just like a very sort of behaviorist thing. And then there's like a higher level at which uh what I refer to love, it refers to like a movie scene or my girlfriend or whatever. You know what I mean?

• 2:56:59Trenton Bricken?

  So yeah, it's like the top of the tent.

• 2:57:01Dwarkesh Patel?

  Yeah, yeah, yeah, yeah. Yeah. What is the highest level association, whatever you found?

• 2:57:06Trenton Bricken?

  I mean, probably one of the ones that we publicly, well, publicly, one of the ones that we shared in our update. So I think there were some related to like love and like um sudden changes in scene, particularly associated with like wars being declared. There are like a few of them in there in that in that post, if you want to link to it. Yeah, but but even like Bruno Olshausen had a paper back in 2018, 19, where they applied a similar technique to a BERT model. And found that as you go to deeper layers of the model, things become more abstract. So I remember like in the earlier layers, there'd be a feature that would just fire for the word park. But later on, there was a feature that fired for park as like a last name, like Lincoln Park or like it's like a common Korean last name as well. And then there was a separate feature that would fire for parks as like grassy areas.

• 2:57:50Trenton Bricken?

  So so there's other work that points in this this direction.

• 2:57:54Dwarkesh Patel?

  What do you think we'll learn about human psychology from the durability stuff?

• 2:57:58Sholto Douglas?

  Oh gosh.

• 2:57:59Dwarkesh Patel?

  I'll give you a specific example. I think like one of the ways one of your updates put it was persona lock in. You know, you remember Sydney Bing or whatever it locked into. I think what was actually quite an endearing.

• 2:58:15Sholto Douglas?

  I thought it's so funny.

• 2:58:16Dwarkesh Patel?

  Yeah.

• 2:58:17Sholto Douglas?

  I'm glad it's back in Copilot.

• 2:58:19Dwarkesh Patel?

  Oh, really?

• 2:58:19Sholto Douglas?

  Yeah. Oh, yeah, it's been misbehaving recently. Yeah.

• 2:58:23Dwarkesh Patel?

  Actually, this is another sort of thread takes more, but there was a funny one where I think it was like to the New York Times reporter. It was nagging him or something and it was like.

• 2:58:34Dwarkesh Patel?

  You are nothing. Nobody will ever believe you. You are insignificant and whatever. It's like.

• 2:58:41Dwarkesh Patel?

  I think the most gaslighting.

• 2:58:42Sholto Douglas?

  Tried to convince him to break up his life or something.

• 2:58:46Dwarkesh Patel?

  Okay, actually, so this is an interesting example. I don't even know where I was going with this to begin with. Uh, but whatever, maybe I got another thread, but like the other thread I want to go on is. That's yeah, actually personas, right? So like.

• 2:58:58Dwarkesh Patel?

  Is that a feature that like Sydney Bing having this personality is a feature versus another personality could get locked into? And also like, is that fundamentally like what humans are like too, where? I don't know, in front of all the different people, I'm like a different sort of personality or whatever. Is that? Was that the same kind of thing that's happening to Shy GPT when it gets R L? I don't know. Whole cluster of questions can answer them and whatever.

• 2:59:18Trenton Bricken?

  Yeah, um, I really want to do more work. I guess the sleeper agents is in this direction of like. What happens to a model when you fine tune it and you are L H F at these sorts of things? I mean, maybe it's trite, but you could just say like, you conclude that people contain multitudes, right? And so much as they have lots of different features. There's even the stuff related to the Waluigi effects of like, in order to know what's good or bad, you need to understand both of those concepts. And so we might have to have models that are aware of violence and have been trained on it in order to recognize it. Can you post hoc identify those features and ablate them? In a way where maybe your model is like slightly naive, but you know that it's not going to be really evil like. Totally, that's in our toolkit, which seems great.

• 2:59:57Dwarkesh Patel?

  Oh, really? So you G P T seven, I don't know, it pulls the Sydney Bing and then you figure out why, like, what were the causally irrelevant? Pathways, whatever you modify like, and then the pathway to you looks like you just changed those, but you were mentioning earlier. There's a bunch of redundancy in the model.

• 3:00:13Trenton Bricken?

  Yeah, so you need to account for all that, but but we have a much better microscope into this now than we used to. Like sharper tools for making edits.

• 3:00:24Sholto Douglas?

  And it seems like, at least from my perspective, that seems like one of the the primary way of, uh, like to some degree, confirming the safety or the reliability of a model where you can say, okay, we found the circuits responsible. We've inflated them. We, uh, can like, under a battery of tests, we haven't been able to now replicate the behavior which we intended to inflate. And like, that feels like the sort of way of measuring model safety in future. As I, as I would understand.

• 3:00:51Dwarkesh Patel?

  Are you worried?

• 3:00:52Sholto Douglas?

  That's why I'm incredibly hopeful about their work. Cause it's, to me, it seems like so much more precise tool than something like RLHF. RLHF, like you're, you're very prey to the black swan thing. You don't know if it's gonna like, do something wrong in a scenario that you haven't measured. Whereas here, at least you have like somewhat more confidence that you can completely capture the behavior set, uh, or like the feature set of the model.

• 3:01:14Dwarkesh Patel?

  And select. Yes. Although not necessarily that you've like accurately labeled.

• 3:01:19Sholto Douglas?

  Not necessarily, but, but with a far higher degree of confidence than any other approach.

• 3:01:23Dwarkesh Patel?

  Yeah.

• 3:01:23Sholto Douglas?

  That I've seen.

• 3:01:24Dwarkesh Patel?

  How, I mean, like, what are your unknown unknowns for superhuman models? In terms of this kind of thing where like, I don't know how, are the labels that are going to be given things on which we can determine these are like, this, this thing is cool. This thing is a paperclip maximizer or whatever.

• 3:01:41Trenton Bricken?

  Um, I mean, we'll see, right? Like, um, I, I, I do, like the superhuman feature question is a very good one. Like I, I, I think we can attack it. Um, but we're, we're gonna need to, to be persistent. And, and the real hope here is I think automated interpretability.

• 3:01:58Dwarkesh Patel?

  Yeah.

• 3:02:00Trenton Bricken?

  And even having debate, right? You could, you could have the debate setup where two different models are debating what the feature does. And then they can actually like go in and make edits and like see if it fires or not or, but it is, it is just this wonderful like closed environment that we can iterate on really quickly. That makes me optimistic.

• 3:02:17Dwarkesh Patel?

  Do you worry about alignment succeeding too hard? So like, if I think about, I would not want. Either companies or governments, whoever ends up in charge of these AI systems to have the level of fine-grained control that if your agenda succeeds, we would have over AIs. Both for the ickiness of having this level of control over an autonomous mind. And second, just like, I don't fucking trust, I don't fucking trust these guys. You know, I don't, I, I, I'm just kind of uncomfortable with like, the loyalty feature is turned up and like, you know what I mean? Um, and yeah, like, how much worry do you have about having too much control over, over the AIs and specifically not you, but like whoever

• 3:03:01Dwarkesh Patel?

  ends up in charge of the AI systems, just um, being able to lock in whatever they want?

• 3:03:07Trenton Bricken?

  Yeah. I mean, I think it depends on what government exactly has control and like, what the moral alignment is there. Um, but.

• 3:03:14Sholto Douglas?

  That, that is like, that whole value lock-in argument is in my mind. It, it's like, definitely one of the strongest contributing factors for why I am working on capabilities at the moment, for example. Just like, I think the current player set, um, actually like extremely well-intentioned. Um, and uh, I mean, I, for this kind of problem, I think we need to be extremely open about it. And like, I think directions like publishing the constitution that you expect your model to abide by, and then like trying to make sure you like, are I H F it towards that and a blade then have the ability for everyone to offer uh, like feedback and contribution to that is really important.

• 3:03:48Dwarkesh Patel?

  Sure. Or alternatively, like, don't deploy when you're not sure, which would also be bad because then we just never catch it.

• 3:03:54Sholto Douglas?

  Right. Um, yeah, exactly.

• 3:03:57Dwarkesh Patel?

  Uh, I mean, paperclip.

• 3:04:01Dwarkesh Patel?

  Um, okay, some rapid fire. Um, what is the bus factor for Gemini?

• 3:04:06Sholto Douglas?

  I think there are, yeah, a number of people who are really, really critical that if you uh, took them out, um, then the performance of the program would be dramatically impacted. Um, this is both on modeling, like slash uh, making decisions about like what to actually do. Uh, and importantly, on infrastructure side of things. Like, it's just the stack of complexity builds, um, particularly when like somewhere like Google has so much like vertical integration. Um, you have, when you have people who are experts, it becomes, they become quite important.

• 3:04:39Dwarkesh Patel?

  Yeah, although I think it's an interesting note about the field that people like you can get in and in a year or so, you're making important contributions. And I especially Anthropic, but many different labs have specialized in hiring like total outsiders, physicists or whatever. And you just like get them up to speed and they're making important contributions. I don't know, I feel like you couldn't do this in like a bio lab or something. It's like an interesting note on the state of the field.

• 3:05:05Trenton Bricken?

  I mean, bus factor doesn't define how long it would take to recover from it, right?

• 3:05:08Dwarkesh Patel?

  Yeah.

• 3:05:10Trenton Bricken?

  And, and, and deep learning research is an art. And so you kind of learn how to read the loss curves or, or set the hyper parameters in ways that empirically seem to work well.

• 3:05:20Sholto Douglas?

  It's also like organizational things, like creating context. One, I think one of the most important and difficult skills to hire for is creating this like bubble of context around you that makes other people around you more effective and know what the right problem to work on. And like, that is a really tough to replicate thing.

• 3:05:35Trenton Bricken?

  Yes, yeah, totally.

• 3:05:37Dwarkesh Patel?

  Who are you paying attention to now in terms of, there's a lot of things coming down the pike of multimodality, long context, maybe agents, extra reliability. Who is the, who is thinking well about what, what that implies?

• 3:05:55Sholto Douglas?

  It's a tough question. I'm, I think a lot of people look internally these days.

• 3:06:00Dwarkesh Patel?

  Sure.

• 3:06:01Sholto Douglas?

  So like their sources of insight or like progress. And, and like we all have, obviously there's a research programs and like directions that are intended over the next couple of years.

• 3:06:14Sholto Douglas?

  And I, I suspect, yeah, that most people, as far as like betting on what the future will look like, refer to like an internal narrative.

• 3:06:23Trenton Bricken?

  Yeah.

• 3:06:24Sholto Douglas?

  That is, that is like difficult to share.

• 3:06:26Trenton Bricken?

  Yeah. If it works well, it's probably not being published.

• 3:06:31Dwarkesh Patel?

  I mean, that was one of the things in the Will Scaling work post, I was referring to something you said to me, which is, I'm, you know, I miss the undergrad habit of just reading a bunch of papers.

• 3:06:40Sholto Douglas?

  Yeah.

• 3:06:41Dwarkesh Patel?

  Is now there's nothing worth reading is published.

• 3:06:45Sholto Douglas?

  And the community is progressively getting like more on track with what I think are like their, their right and important directions.

• 3:06:52Dwarkesh Patel?

  You're watching it like an agent, dude.

• 3:06:55Sholto Douglas?

  No, but I I guess like it is tough. There there used to be this like signal from big labs about like what would work at scale. And it's kind of really hard for academic research to like find that signal. Um, and I think uh getting like really good problem taste about what actually matters to work on is really tough. Um, unless you have again the feedback signal of of like what will work at scale um and what what is currently holding us back from scaling further or understanding our models further. Um, this is something where like I wish more academic research would go into fields like INTP, which are legible from the outside, you know.

• 3:07:31Sholto Douglas?

  Anthropic deliberately publishes all its research here, um, and it seems like underappreciated uh in the in the sense that I don't know why there aren't dozens of academic departments trying to follow uh Anthropic's guide in in the INTP research, because it seems like an incredibly impactful problem that doesn't require ridiculous resources and like this and like has all the flavor of like deeply understanding the basic science of what is actually going on in these things. Um, so I don't know why people like focus on pushing model improvements as opposed to pushing like understanding improvements in the way that I would have like typically associated with academic science in some ways.

• 3:08:05Trenton Bricken?

  Yeah, I do think the tide is changing there for whatever reason. Um, like Neil Nanda has had a ton of success promoting interpretability. In in a way where like Chris Ola hasn't been as active recently in in pushing things, maybe because Neil's just doing quite a lot of the work, but like I don't know, four or five years ago, he was like really pushing and like talking at all sorts of places and and these sorts of things and people weren't anywhere near as receptive. Um, maybe they've just woken up to like deep learning matters and is clearly useful post chat GPT, but yeah, it's it is kind of striking.

• 3:08:37Dwarkesh Patel?

  Hmm. All right, cool. And okay, I'm trying to think what what is a good uh last question. I mean, the one I'm going those thinking of is like, do you think models enjoy next token prediction?

• 3:08:47Sholto Douglas?

  Yeah. Believe in love.

• 3:08:53Dwarkesh Patel?

  Uh, we had this uh sense of things that were awarded in our ancestral environment. There's like this deep sense of fulfillment that uh we think we're supposed to get from them, or often people do, right, of like community or sugar. Um, or you know, whatever we wanted on the African savanna. Um. Do you think like in the future, models are trained with RL and everything, a lot of post-training on top of whatever, but they'll like, they're like some, in the way we were just a really like ice cream, they'll just be like, hi, just to predict the next token again, you know what I mean? Like in the good old days.

• 3:09:28Trenton Bricken?

  So, so there's this ongoing discussion of like, are models sentient or not? And like, do you thank the model when it helps you?

• 3:09:34Dwarkesh Patel?

  Yeah.

• 3:09:35Trenton Bricken?

  But I think if you want to thank it, you actually shouldn't say thank you, you should just give it a sequence that's very easy to predict. And, and the, the even funnier part of this is, um, there is some work on if you just give it the sequence, A, like, ah, like over and over again, then, then eventually the model will just start spewing out all sorts of things that it otherwise wouldn't, uh, wouldn't ever say. And, uh, so yeah, I, I won't say anything more about that, but, uh, you can, uh, yeah, you should just give your model something very easy to predict as a nice little treat.

• 3:10:07Dwarkesh Patel?

  This is what the do-ium ends up being, like just our universe and like.

• 3:10:11Sholto Douglas?

  But don't we like things that are like easy to predict, or like, aren't we constantly in search of like the, the like dose of?

• 3:10:19Dwarkesh Patel?

  Bits of entropy?

• 3:10:20Sholto Douglas?

  Yeah, the bits of entropy, exactly, right? Shouldn't you be giving it things that are just slightly too hard to predict? Just out of reach.

• 3:10:27Trenton Bricken?

  Yeah, but I wonder, like, at least from the free energy principle perspective, right? Like, you don't like, you don't want to be surprised. Um, and so maybe it's this like, I don't feel surprised if I'm in control of my environment. And so now I can go and seek things and I've been predisposed to like, in, in the long run, it's better to explore new things right now. Like, leave the rock that I've been sheltered under, ultimately leading me to like build a house or like some better structure. But, um, we don't like surprises. I, I think most, most people are very upset when like, expectation does not meet reality.

• 3:10:58Sholto Douglas?

  And so babies like love watching the same show over and over and over again, right?

• 3:11:03Trenton Bricken?

  Yeah, interesting. Yeah, I can see that.

• 3:11:05Sholto Douglas?

  Um, I guess they're learning to model it and stuff too, but.

• 3:11:08Dwarkesh Patel?

  Yeah. Um, okay. Well, hopefully this will be the this will be the repeat

• 3:11:15Dwarkesh Patel?

  that the the AI has learned to love. Okay, cool. I think that's a great place to wrap. I should also mention that the better part of what I know about AI I've learned from just talking with you guys. You know, we've been good friends for about a year now. So, yeah, I mean, yeah, I appreciate you guys getting me up to speed here and. You guys have great questions. It's really fun to hang and chat. Great, great. I really treasure that time. Yeah. Yeah. You're getting a lot better at pickleball.

• 3:11:44Sholto Douglas?

  Hey, we're trying to progress the tennis. Come on.

• 3:11:50Dwarkesh Patel?

  Awesome. Cool, cool. Thanks.

• 3:11:54Dwarkesh Patel?

  Hey, everybody. I hope you enjoyed that episode. As always, the most helpful thing you can do is to share the podcast. Send it to people you think might enjoy it, put it in Twitter, your group chats, etc. Just blitz the world. Appreciate your listening. I'll see you next time. Cheers.