The Utility of Interpretability — Emmanuel Amiesen

Emmanuel Amiesen is lead author of “Circuit Tracing: Revealing Computational Graphs in Language Models” (https://transformer-circuits.pub/2025/attribution-graphs/methods.html ), which is part of a duo of MechInterp papers that Anthropic published in March (alongside https://transformer-circuits.pub/2025/attribution-graphs/biology.html ). We recorded the initial conversation a month ago, but then held off publishing until the open source tooling for the graph generation discussed in this work was released last week: https://www.anthropic.com/research/open-source-circuit-tracing This is a 2 part episode - an intro covering the open source release, then a deeper dive into the paper — with guest host Vibhu Sapra (https://x.com/vibhuuuus ) and Mochi the MechInterp Pomsky (https://x.com/mochipomsky ). Thanks to Vibhu for making this episode happen! While the original blogpost contained some fantastic guided visualizations (which we discuss at the end of this pod!), with the notebook and Neuronpedia visualization (https://www.neuronpedia.org/gemma-2-2b/graph ) released this week, you can now explore on your own with Neuronpedia, as we show you in the video version of this pod. Full Video Episode Timestamps 00:00 https://www.youtube.com/watch?v=9YQW2mH9FyA Intro & Guest Introductions01:00 https://www.youtube.com/watch?v=9YQW2mH9FyA&t=60s Anthropic's Circuit Tracing Release06:11 https://www.youtube.com/watch?v=9YQW2mH9FyA&t=371s Exploring Circuit Tracing Tools & Demos13:01 https://www.youtube.com/watch?v=9YQW2mH9FyA&t=781s Model Behaviors and User Experiments17:02 https://www.youtube.com/watch?v=9YQW2mH9FyA&t=1022s Behind the Research: Team and Community24:19 https://www.youtube.com/watch?v=9YQW2mH9FyA&t=1459s Main Episode Start: Mech Interp Backgrounds25:56 https://www.youtube.com/watch?v=9YQW2mH9FyA&t=1556s Getting Into Mech Interp Research31:52 https://www.youtube.com/watch?v=9YQW2mH9FyA&t=1912s History and Foundations of Mech Interp37:05 https://www.youtube.com/watch?v=9YQW2mH9FyA&t=2225s Core Concepts: Superposition & Features39:54 https://www.youtube.com/watch?v=9YQW2mH9FyA&t=2394s Applications & Interventions in Models45:59 https://www.youtube.com/watch?v=9YQW2mH9FyA&t=2759s Challenges & Open Questions in Interpretability57:15 https://www.youtube.com/watch?v=9YQW2mH9FyA&t=3435s Understanding Model Mechanisms: Circuits & Reasoning01:04:24 https://www.youtube.com/watch?v=9YQW2mH9FyA&t=3864s Model Planning, Reasoning, and Attribution Graphs01:30:52 https://www.youtube.com/watch?v=9YQW2mH9FyA&t=5452s Faithfulness, Deception, and Parallel Circuits01:40:16 https://www.youtube.com/watch?v=9YQW2mH9FyA&t=6016s Publishing Risks, Open Research, and Visualization01:49:33 https://www.youtube.com/watch?v=9YQW2mH9FyA&t=6573s Barriers, Vision, and Call to Action This is a public episode. If you'd like to discuss this with other subscribers or get access to bonus episodes, visit www.latent.space/subscribe https://www.latent.space/subscribe?utm_medium=podcast&utm_campaign=CTA_2

• 0:04Speaker 00

  All right, we are actually going to record this as a intro to the main episode, but here we have my trusty co-host, guest host, I guess, Vivek, as well as Emmanuel from Anthropic. We're going to talk about the circuit tracing stuff and all the interpretability work. But Emmanuel, maybe you want to do a quick self-intro because before we get into it.

• 0:25Speaker 01

  Yeah, sure. I'm Emmanuel. I work on the interpreter league team here at Anthropic, more specifically on the circuits team. So we recently released a pair of papers about sort of like the work that we've been doing over the last months. And even more recently, we released some code in partnership with the Anthropic Fellows program. It was mostly built by Anthropic Fellows that lets people play with the research, basically. And so happy to talk about that. And we also hope to kind of like keep releasing more things and partner with other groups that are working on similar stuff.

• 0:56Speaker 00

  Yeah, amazing. We'll get deeper into like the behind the scenes on the main podcast. But let's maybe just dive right in into what you released because that's the most topical thing. This is like literally just launched it like yesterday and we'll probably release it in at least this episode in a few days. So yeah, like what can people do or what do you recommend people try?

• 1:13Speaker 01

  Totally. So like a really high level, you know, the sort of like idea of the research itself is to try to explain sort of like some of the computation that a model did when it predicted a given token. And so in our paper, we kind of like show how to do this. And then we show examples of us doing this on internal private models. And then the release this week sort of lets anyone do it for a set of open source models. So notably, maybe the most easy one here is like Gemma 2.2B. So you can sort of like think of some prompt and you kind of like can explain any like token that the model sample samples. And explains here means just like basically blow up the internal state of the model

• 1:56Speaker 01

  and like show all of this sort of intermediate things that the model was thinking about before it got to like the final token that it predicted.

• 2:03Speaker 02

  Yeah, so what some of the things that you guys put out is kind of in the circuit tracing, you have a few core examples, right? So like, we can see how these models have internal reasoning states and there's like multi-hop reasoning and some of the stuff that we talked about on the podcast was how can people that are interested in how models work kind of do anything, right? So what are open questions? How can people contribute? And it seems like, you know, the follow-up is, okay, it's been a few weeks, here's a huge library. So, you know, I guess before we even get into it, what are some open questions that you would expect people to like, kind of play around with, you know? What are people like gonna do? What, why should we probe Gemma, Lama? What are interesting things we can do? And any tips on using it?

• 2:42Speaker 01

  Yeah, I, I think there's maybe like two to three like categories of things that people could do. So I'll go from sort of like the most basic kind of low effort to, you know, hey, if you wanted to dedicate like a month of your life, you could do that. Um, the, the sort of most basic thing is, you know, um, so it's a Gemma two and Lama one beer like smaller models, uh, but they can still do a bunch of stuff. And so, and, and for most of the things that they can do, we still kind of like, don't really know or have a good mental model of how it is that they do the things that they do. So to give you an example, like one of the things in the paper is the sort of like multi hop reasoning where, you know, we ask, you know, like, uh, cloud three point five haiku, like, oh, the capital of the state where Dallas is, is Austin.

• 3:25Speaker 01

  It turns out that like Gemma can do this also. And so as part of the release, we have, you know, a notebook where Michael Hannah, one of the Anthropic fellows, kind of like walks through a bunch of examples, including this one. And it's really cool because you can see that actually the way the circuit looks in Gemma, like a really small model is extremely similar to the way that it looks like a huge model, which that in its, uh, in itself is, is I think like a pretty novel discovery. It's like, oh, you have these models are like super different, you know? If you look at like their evals, or if you just try to use them, they're like just very clearly different. But for this one task, for this one thing, actually, the way that this do, that they do this multi-step reasoning is like the same way they actually do the multi-step reasoning. In the notebook, there's both other examples of kind of like fun things that we looked at that I think can sort of spike your interest if you're new to thinking about this stuff.

• 4:08Speaker 01

  And at the end of the notebook, that's linked in the in the readme, there's like three examples of like random sort of like cases that we haven't solved or or or we haven't labeled that, you know, have like a graph pre-computed for you and you could just look at it and try to like figure out what's happening. And by figure out what's happening, what we mean is, you know, we might do like a quick demo here, but it's kind of like, look at these representations, try to understand like, okay, like, what is the competition the model is doing? And then part of the release also lets you like run run experiments to like verify that you're right. So if you think that like, you know, ah, the model like first like thinks about Texas in this case, you can also just like stop it from thinking about Texas and see if like that damages it. And so like the tools to do that are available.

• 4:48Speaker 01

  And so I would say that's like the first thing is just, I think the hope is there are a lot of behaviors that models do way more than like any single group has time to explore. And so the hope is like, hey, pick a behavior that you think is interesting and try to understand like what's happening and try to ground it out. And it's like the just like baseline thing and maybe like the the thing that I'm most excited about with this release. But then the other thing I do want to mention to like parts two and three are just, we also hope that like

• 5:13Speaker 01

  other groups and and kind of interested researchers can just use this to like extend the method. Like if you have an idea about how to like do this better, you know, the whole code like make this graph is is open source. So you're like, take a look at it and just like try to play with it, try to find different ways to like create these graphs and also extend it to other models, right? Like there are many different models. And so, you know, part of part of making this work on any models, you have to like train the sort of like replacement model, which again, there there is code for it and there's other groups working on. And so like, that's also something that if you're excited about, you could say like, okay, cool. Well, I want this to work on like another open model and you could sort of like add it if you're like, you know, more into sort of like maybe like the engineering and the engineering side of things.

• 5:50Speaker 00

  Yeah, we we actually get into a little bit about how you how you guys do the extra data viz stuff that makes your your blog posts pop so much. Should we should we um share the screen a little bit and and dive in? I think you guys prep some examples.

• 6:03Speaker 01

  Totally. Yeah.

• 6:05Speaker 00

  It's just like, there's nothing better than the creator of the tool walking through the tool. And we might as well capture that so that, you know, people who actually want to do this can follow along.

• 6:13Speaker 01

  Yeah, uh, that makes real sense. Let me just like actually share my screen.

• 6:16Speaker 02

  My, uh, one little experiment, I basically cloned the repo, threw it into Claude Code and was like, you know, deal with this, let's just try it end to end. So would recommend, you know, Claude Code is very good at using this. Um, basically also, if you're, if you're just trying to get started, the Circuit Tracing Tutorial Notebook, very good. That kind of goes over all the high level and then, you know, shout out Claude Code, try it out. It, it works very well on this.

• 6:41Speaker 01

  That, that's awesome to hear. Actually, I might just open the notebook first, just like quickly walk through the illustrations, but yeah, you're the second person to tell me that they just had Claude Code, so like dig in initially and it's, I'm glad, I'm glad that's, that's working. The, the tutorial here is like linked at the top of the repo and maybe we can link it from the podcast, but essentially sort of like walks you through kind of like how to think about graphs. And so it links to these circuits. So here, this is the two-step reasoning that we're talking about. This is kind of like a schematic of it where it's like the capital of state of Dallas and it's like, ah, it has to think of Texas and Austin. But the notebook links you to all of these circuits here. And this is kind of the thing that you can play with. So this is the UI on, you know, Neuronpedia.

• 7:19Speaker 01

  That, that hostess and that, that lets you like create any circuit. So here, you know, we could explore the circuit and if you open the notebook, you could explore it.

• 7:29Speaker 01

  I'm realizing that I switched tabs, I'm not sharing it. Okay. There we go. Can you see the circuit now? I think so. Okay, cool. Um, but you can make a new graph super easily and quickly. And so maybe this is like the most fun things. When I was playing with, uh, right before, uh, you know, joining this call is like, uh, it turns out that podcast guests are very formulaic. And so if you say like, thanks for having me on the whatever. Like Gemma seems to have pretty consistently guessed that you're like on a podcast, uh, which makes sense, right? Like, why would you say, thanks for having me on the blah? Uh, and so here we can try to say like, ah, okay, like, how does Gemma know to like, complete the sentence with, thanks for having me on the latent space podcast?

• 8:14Speaker 01

  And so here, the way you generate a graph, right, is you type a sentence where the next word is the thing that you're interested in. And then you kind of try to explain like how the model goes to the next word. So here, you can.

• 8:28Speaker 01

  Give it a name. And then you can mostly just not worry about any of these parameters, I think, if you're just playing with it. And you can click start generation.

• 8:37Speaker 02

  And this generates like something important for people to know is that these are trained on base models, right? So they're not chat models. So basically, when you train these models, they're just trained to predict next token and they don't have that user assistant chatbot flow. So they're prompted in a way such that, you know, the output should basically just be the next word.

• 8:56Speaker 01

  Yeah, you kind of want to think about it as like, maybe the the prompt or the texture you're making is like the text of like a book or an article rather than a conversation where it's like, you know, what is a sentence where if you were to read it in a book, like the next word would be sort of like the interesting one. Yeah.

• 9:14Speaker 01

  You know, you can click on it. Sort of like takes a little bit of time to load because there's just a bunch of data. So what we're going to show you here is basically like almost every single feature that activates in the model, the features are these intermediate representations. And at the bottom, there's the prompt. So here it's like, thanks for having me on the latent space. And at the top, you can see like what the model sort of like output. So it's most likely output is it's pretty confident that we're talking about a podcast. And, you know, it has like some random stop tokens, blog, show, and then some stuff that I think like makes less sense, but also like these are small models. And so sometimes they say random stuff. And so the way that you could then explore this and be like, okay, so the model says podcast, so like, why does it say podcast?

• 10:00Speaker 01

  So you can click on this output and say like, what are the features again, these like intermediate representations that have an input to this? So it seems like there's features at here, this is the layer, like layer 18 that already like are about podcast episodes. You can know this because the features have a label, but also if you want, you can look at the feature itself here.

• 10:22Speaker 01

  And here you can see that like this shows you like other text of the feature is active over and it's just like text about podcasts. So that's like a way that you can also like understand what the features are. And then you can keep going back. So it's like, oh, okay. So it said podcast here because of this podcast feature. Where did that come from? And it's like, oh, it comes from like words related to podcasts, words associated with podcasts, as well as like an interview feature. And also just the word on. So there's like a bias, like if you're saying like blah blah blah, I mean, on, that's sort of like slightly increases the chance that you're talking about a podcast at all. And you can sort of like keep going back. And kind of like explore the graph interactively. I would say that like, the way to do it, and we talked about this on the like, kind of like longer version of the podcast, but it's like, you know, kind

• 11:07Speaker 01

  of like chasing from the interesting outputs back or from the interesting input forward. There are many nodes on these. I like wouldn't recommend looking at all of them. You can also sort of like prune them a little more aggressively here if this is too busy. And kind of look, this shows you like only the most important ones. And you can sort of like be pretty extreme with it if you want. Or you can show the whole thing and then be like super overwhelmed. Once, once you kind of do this, you can then kind of like group your nodes into similar ones to kind of like make a graph. I actually made this little summary earlier, so I can just share that. So this is the exact same graph, but just before the, before, before hopping on, I kind of like did a few groups. So this is like, same thing, podcast. It's like, oh, there's like a bunch of

• 11:48Speaker 01

  those that are like podcast episodes. It's a sort of thing like discussing podcasts. There's a note about expressing gratitude that amplifies that you're like on an interview or a podcast. So like one fun experiment you could do here, right? Is like, oh, like, what happens if I mess with this? Like if, if I don't like, if I mess with the like, oh, this person is like grateful to be on. And it's like, this person is on, doesn't think you're on something else. Like, maybe there are things that, you know, you could be on that you're not grateful for. Like, oh, you're having me on trial or something. I don't know. Like, that could be one, one interesting sort of like experiment to see what the causal effect of this is.

• 12:23Speaker 01

  And again, you could sort of like label it more and explore it more. And this UI, the whole point is for it to be like snappy and quick. So you can just like generate a bunch of graphs pretty easily, right? Like, maybe this wasn't exactly what you wanted. So you're like, I'm super unhappy to be on the latent space. And then you can see what it completes for that or whatever. And you can sort of like just continuously play with it and and get a better sense for your hypotheses. Oftentimes, you kind of like want different prompts, you know, different examples that are similar to kind of get a sense for it. And then if you're really curious and you want to dig in more, that's when I would recommend going back to like the code base and some of the notebooks. Maybe one thing, one last thing I'll say on on that is that the notebooks themselves,

• 13:09Speaker 01

  they can all be run on Google Colab and all of the code, as far as we can tell, we feel like testable notebooks just like runs on Colab. And so that means that like, you don't need on the free tier, to be clear, like you don't need like an expensive GPU. You can just run this and kind of like run your interventions and play with it. And so in this notebook, in particular, the intro one, we show you how to do these interventions. And here we're like, what happens if we turn this node off? And what happens if we turn that one off? And what happens if we turn this one off? And what happens if, you know, we inject one from one prompt into another one? And so I think that's the sort of like deeper dive, trying to understand the mechanism better. But if you're just trying to even like get a sense at all, like, how does the model do X? You just generate a graph and and take a look at it.

• 13:48Speaker 00

  Incredible. Very cool. Is there, uh, when I look at the graph, is there's a thought in my mind about maybe this is too easy, too perfect. Um, and one version of this is there's supposed to be superposition. And here there's no superposition, kind of.

• 14:06Speaker 01

  Well, there is superposition and like we're, we're sort of like, so maybe I can share, I can share the the like graph again and like answer your question, which I think is like, what are we hiding here? Where are the skeletons?

• 14:16Speaker 00

  Yeah, this is like, it's too, it's too clean. I'm like, yeah.

• 14:21Speaker 01

  So, okay, so, so maybe like a good example is, and we're gonna like make this slightly less overwhelming here. Is like, okay, so you, so you look at this graph and you say like, yeah, like, we don't actually understand how models work fully. So like, what are you hiding here? And and the thing that's like important to know here is, you know, I I didn't say this explicitly, but like the layers are like arranged here. And so let's just look at like one layer. So for this layer, what we're saying is like, the only thing that that is happening or that's like important enough is this one feature, which is just like one small direction in the model space, right? Like one, one dimension we've pulled out of superposition, uh, or, or, um, let's say that now. But then also there's these diamonds and these diamonds are errors.

• 15:04Speaker 01

  We talked about them on the, on the longer podcast, but they're just like, when you train these replacement models to replace some of the model computation, you successfully replace some of it and then some of it you fail to replace. And so this is like everything that we don't understand. And so that means that like, sometimes if you look at an input like this guy's input, you'll see a bunch of errors here as the input. And so essentially, you know, there's some graphs and some examples where like, if you have, if most of the stuff that you see is these errors, basically that's, that just means like, hey, for this prompt, we were not able to sort of like explain, you know, that part of the computation. And so at least that part is like a, an explicit sort of like, we show it in your face where it's like, here's, here's what we don't understand. And so, so you can sort of like see what we don't understand. There's also, I will say like one more thing, there's like a bunch more

• 15:50Speaker 01

  stuff that can get you, uh, and that's like in the paper, but like one example here that I'll just say is like, these are just MLPs. So the model has both attention heads and multi layer perceptrons MLPs. We don't just do it like we completely ignore attention or like, we don't, we don't try to decompose it at all. So there are some prompts where like all of the interesting stuff is attention. And here you're just not, you're just not seeing it at all. The way that it's materializes, like you have an edge from here to here and like some attention had did a bunch of stuff. You don't know what it is. And so that's also the part that we're sort of like not explaining. So there's, there's definitely, yeah, I don't want to make the claim that

• 16:28Speaker 01

  we explain everything. I think the correct way to think about this is like, if you look at a prompt and you can, by tracing through these, not hit any errors, hit nodes that make sense and build up a reasonable hypothesis. And then when you test it with interventions, it works, you've at least understood some and presumably like a reasonable proportion of the computation. If your interventions are working, that means that it's like, the thing you found is not just like a side thing. It's part of the main thing that the model is doing. And so, you know, then the question is like, how often does that happen versus how often you just hit these errors or you're like confused? And I think that's, that's the sort of like,

• 17:02Speaker 01

  What works and what doesn't summary here.

• 17:04Speaker 00

  I see. I mean, congrats on this work. I know you're low on sleep because you worked really hard on shipping in and you're a perfectionist. I just think like, yeah.

• 17:15Speaker 01

  Sorry, I'll just say that like the actual brunt of the work here is like.

• 17:18Speaker 00

  Your fellows.

• 17:18Speaker 01

  Yeah, yeah, yeah. You know, I mostly just like coordinated things left and right, but they sort of like did all of the implementation as well as, you know, folks on the like on your own PDA decode research side also did, you know, the lines here of the work here to actually have the front end UI.

• 17:34Speaker 00

  I'll just say like, you know, Veebo and I were at the good fire meetup yesterday where there are a lot of interpretability folks. I was shocked at honestly how young most interpretability people and work are. And this is a very young field, exactly like you say in the podcast. There's a lot of fresh green grass here to tread. And it's just really inspiring. Veebo, do you have any other like final thoughts or comments?

• 18:02Speaker 02

  Yeah, no, I think there's just a lot of open work to be done, you know, and we talk about this in the podcast too. And just to reiterate, like how good the tooling that you guys put out is like, even the fact that, you know, without diving into any code, you can enter in a prompt and start to play through these circuits in like minutes. It's it's pretty incredible. Like I could share another one actually. So I was doing this with Pomski and I finally got it to work. So our guest host of the episode is Mochi, my little dog. She's a distilled husky. So she's on the podcast later and, you know, I basically put in like, I had to guide it quite a bit, but my, my prompt is a Pomski is a small dog that's a breed of a miss of a husky and a. And then, you know, I'm expecting it to put out Pomeranian or Pom. Let me, let me share my screen real quick.

• 18:45Speaker 02

  And then we can kind of dig through. This is me like.

• 18:48Speaker 00

  By the way, her tagline, yeah, while you put it up, her tagline is officially Mochi, the interpretability husky.

• 18:54Speaker 02

  It's a date.

• 18:55Speaker 01

  For today, we're gonna change our tagline every episode, but yeah.

• 18:58Speaker 02

  It feels a little weird, you know, we're digging deep into what Mochi is. But basically, this is me like no background, like two minutes and just put in a phrase and now I get to play around with features, right? So this is also called please with four S's because you know, I tried a few prompts. It's okay, it's okay, we struggle. It only took a few minutes though, so you know, Pomski is a small dog breed that that's a mix of a husky and a. And then the the most probable outputs, you know, now it says palm. So, okay, let's dig into what some of these are. I'm basically just going like fresh, haven't done this before, but you know, words related to animals, their emotions, their health. We have a feature for dog, golden lab, mentioned dog breeds, especially high maintenance. You know, this is basically like AGI, it knows Pomskis are high maintenance.

• 19:43Speaker 02

  It's it's figured it out. But realistically, you know, um, as I dig through these features, I can start to pin them, uh, layer them through. Mentions of garbage and waste, no, that's not nice, that's not nice. But basically, you know, and this is already me pruning out most of the features. Uh, as I open it up, you know, it talks about different things like dog breeding. What else? Related to animal welfare, so like, and then you can dig through all this. There's just so many things that like, you know, this is in a matter of minutes. I basically made a graph, put in a sentence, and now I have an output and I can traverse through what are different things. Okay, animal science, right? So this breed is relatively new. It's not that common that big huskies and little Pomeranians naturally have offspring.

• 20:27Speaker 02

  But you know, let's let's like dig through animal science versions of this. And then we have like interesting little features. So it's it's very easy for people to kind of get a different understanding of what goes on throughout layers in models, you know? But that's just my fun little experiment of getting it to work.

• 20:44Speaker 01

  Oh, yeah. And and I think like, you know, one, one thing that I I would do if you were curious, or maybe I'm just gonna try to bait some listeners into doing it is like, give back, okay, like, let's try to like trace why it said Pomeranian here. And like, maybe there's like, some of it is about like, like, dog breeds and some of it is about like, specific characteristics of a husky. And then you can ask the same question, but instead of husky, you'd like try some other dog breed. And then try to see if you can like, if you understood it, the circuit well, and if you identified where it's thinking about huskies or where it's thinking about like, kind of like breeding two different breeds, then you should be able to like swap these in and out and get it to kind of like say whatever you want. Um, and and if you didn't, then maybe there's something complicated going on. But but yeah, like, very cool that you got this going on so quick. That's that's that's the whole goal. That's super exciting.

• 21:23Speaker 02

  Yeah, and like, you know, no disclosure, this was like five minutes of just playing around and like, there's, there's stuff to learn there, right? Like, okay, what happens with dog breeding? What are traits of these dogs? And then, you know, the next step for me would basically be, let's try clamping some of these features up or down. Let's, let's do different breeds and see if it makes sense, right? So if I have husky traits and a different mix, and then, you know, can I, can I get out what's going on? But it also shows internally that there's more than just token completion of, you know, this plus this equals this. No, it has some under, understanding of characteristics, right? Like, this is a pretty stubborn dog. It, it has a stubborn feature pretty high up that activates. So very, very cool stuff. I think it'll be cool when we apply

• 22:05Speaker 02

  this to more like serious topics, like right now, when it comes to L M evals, right? We have like, we have pretty straightforward evals, right? Like, how good is, does it do on math? Can it write code? Does stuff compile? But we don't have like vibes based heuristic evals, right? So like, does it understand different queries should be concise? Should they be verbose? Can we kind of trace through how it gives responses to this stuff? And then like the other part is, you know, as we go past base models, how does this happen for different phases of models, right? So if I have a base Gemma and I have a chat model, what are differences in their attributions, right? What happens kind of in that diff of training? So that's, that's kind of one of my little interests in, in Meccan Turk. What happens as we do more training? What are we really changing?

• 22:49Speaker 01

  Totally. Yeah. You can think about sort of comparing different models and for me, different models, either like Gemma versus some other model or like early Gemma versus late Gemma and pre-training or like fine tuned versus not fine tuned. I think there's also a sense in which like, somebody yesterday was telling me like, oh, it's fun. I've been playing with it on like the like, sort of like weird riddles that the models get wrong. Like, like you, you're not limited to studying what the model can do, right? Like if the model's failing at something like, you know, counting the number of letters in strawberry or whatever, you could just try that and try to figure out the circuit for like, well, it's getting this wrong. Like, why? Like, maybe you can see in, in its representation that it's like thinking about something obviously incorrect, right? And so I think, I think that that's also like a fun thing to, to, to play with.

• 23:29Speaker 02

  I think that's it for our little intro chat and coverage of the open sourcing. Let's dive right into the episode next. But Emmanuel, your amazing work and I'm so inspired. And also just like, I think this puts a human face on the, the interpretability work. I think it's very important and we'd love to keep doing this, whatever you got next coming up.

• 23:51Speaker 01

  Well, yeah, thanks for having me. Again, I should say, cool to put a face on it, but definitely want to call out, this is like a huge team of people with me. I'm just a talking head here.

• 24:01Speaker 00

  And paper, paper lead, you know, you did the work, you know, take credit.

• 24:06Speaker 01

  I think that like, yeah, happy to talk about more things and also like feel free to, you know, reach out to me. I'm like findable if you're listening to this podcast and you have like questions about stuff that's broken or if this brings up like experiment ideas. I definitely want more people playing with this. So, so yeah, thanks for having me. Hope that inspires some folks.

• 24:22Speaker 00

  All right, we are back in the studio with a couple of special guests. One, Vibhu, our guest co-host for a couple of times now, as well as Mochi the Distilled Husky is in the studio with us to ask some very pressing questions. As well as Emmanuel, I didn't get your last name, Amason? Yep. Is that Dutch? Is that? It's actually German. German? Yeah. You are the lead author of a fair number of the recent mech and terp work from Anthropic that I've been basically calling Transformer Circuits, because that's the name of the publication.

• 24:53Speaker 01

  Yeah. Well, to be clear, Transformer Circuits is the whole publication. I'm the author on one of the recent papers, Circuit Tracing.

• 24:59Speaker 00

  Yes. And people are very excited about that. The other name for it is like Tracing the Thoughts of LLMs. There's like three different names for this work. It's like, but it's all mech and terp.

• 25:07Speaker 01

  It's all mech and terp. There's two papers. One is Circuit Tracing. It's the methods. One is like the biology, which is kind of what we found in the model. And then Tracing the Thoughts is confusingly just the name of the blog post.

• 25:18Speaker 00

  Yeah. It's for different audiences. And I think though, when you produce like two-minute polished video that you guys did, that's meant for like a very wide audience, you know?

• 25:28Speaker 01

  Yeah, that's right. There's sort of like very many levels of granularity at which you can go. And I think for mech and terp in particular, because it's kind of complicated going, you know, from like top to bottom, most like high level to sort of the derelict details works pretty well.

• 25:41Speaker 00

  Yeah. Cool, we can get started. Basically, we have two paths that you can choose, like either your personal journey into MechanTurf or the brief history of MechanTurf just generally, and maybe that might coincide a little bit.

• 25:54Speaker 01

  I think my, okay, I could just give you my personal journey very quickly because then we can just do the second path. My personal journey is that I was working at Anthropic for a while. I'd been, like many people, just following MechanTurf as sort of like an interesting field with fascinating, often beautiful papers. And I was at the time working on fine tuning, so like actually fine tuning production models for Anthropic. And eventually I got both, like sort of like my fascination reached a sufficient level that I decided I wanted to work on it. And also I got more excited about just as our models got better and better, understanding how they worked. So that's the simple journey. I've got like a background in ML, kind of like did a lot of applied ML stuff before, and now I'm doing more research stuff.

• 26:38Speaker 00

  Yeah, you have a book with O'Reilly, you're head of the AI at Insight Data Science. Anything else to plug?

• 26:43Speaker 01

  Yeah, I actually, I want to like plug the paper and unplug the book.

• 26:49Speaker 00

  Okay.

• 26:49Speaker 01

  I think the book is good. I think the advice stands the test of time, but it's very much like, hey, you're building like AI products, which do you focus on? It's like very different, I guess, is all I'll say from from the stuff that we're talking to talk about today. Today is like research, some of some of the sort of like deepest, weirdest things about how like models work. And this book is, you want to ship a random forest to do fraud classification. Like here are the top five mistakes to avoid.

• 27:12Speaker 00

  Yeah, the good old days of ML.

• 27:14Speaker 01

  I know, it was simple back then.

• 27:16Speaker 00

  You also transitioned into research, and I think you also did the management. Like, I feel like there's this monolith of like, people assume you need a PhD for research. Maybe, can you give like that perspective of like, how do people get into research? How do you get into research? Maybe that gives audience insight into Viva as well, your background.

• 27:34Speaker 02

  Yeah, my background was in like economics, data science. I thought LLMs were pretty interesting. I started out with some basic ML stuff, and then I saw LLMs were starting to be a thing, so I just went out there and did it. And same thing with AI engineering, right? You just kind of build stuff, you work on interesting things. And like, now it's more accessible than ever. Like, back when I got into the field five, six years ago, like, pre-training was still pretty new, GPT-3 hadn't really launched. So it was still very early days and it was a lot less competitive. But yeah, without any specific background, no PhD, there just weren't as many people working on it. But you made the transition a little bit more recently, right? So what's your experience been like?

• 28:12Speaker 01

  Yeah, I think I think it has maybe never been easier in some ways because a lot of the field is like pretty empirical right now. So I think the bitter lesson is like this lesson that, you know, you can just sort of like a lot of times scale up compute and data and get better results than like thinking. Then if you sort of like thought extremely hard about a really good like prior inspired by the human brain to to train your model better. And so in terms of definitely like research for pre-training and fine tuning, I think it's just sort of like. A lot of the bottlenecks are extremely good engineering and systems engineering. And a lot even of the research execution is just about sort of like engineering and scaling up and things like that.

• 28:55Speaker 01

  I think for Interp in particular, there's like another thing that makes it easier to transition to, which is maybe two things. One, you can just do it without huge access to compute. Like there are open source models, you can look at them, a lot of Inter papers, you know, coming out of programs like maths are on. Models are open source that you can sort of like dissect without having a cluster of like, you know, a hundred GPUs. You can just even sometimes you can load them like on your CPU on your MacBook. And it's also a relatively new field. And so, you know, there's, as I'm sure we'll talk about, there's like some conceptual burdens and and concepts that you just want to like understand before you contribute, but it's not. You know, physics, it's relatively recent.

• 29:37Speaker 01

  And so the number of abstractions that you have to like ramp up on is just not that high compared to other fields, which I think makes that transition somewhat easier. For Interp, if you understand, we'll talk about all these, I'm sure, but like what features are and what dictionary learning is. You're like a long, rather way there.

• 29:53Speaker 00

  I think that's also interesting, just on a careers point of view, research seems a lot more valuable than engineering. So I wonder, and you don't have to answer this if it's like a tricky thing, but like, how hard is it for a for a research engineer in Anthropic to jump the wall into research? People seem to move around a lot and I'm like, that cannot be so easy. Like in no other industry that I know of, people, you can do that. Do you know what I mean?

• 30:23Speaker 01

  Yeah, I think I'd actually like, I'd push back on the sort of like research being more valuable than engineering a little bit because. I think a lot of times, like, having the research idea is not the hardest part. Don't get me wrong, there's some ideas that are brilliant and hard to find. But what's hard, certainly on fine tuning and to a certain extent on Interp, is executing on your research idea in terms of like making an experiment successfully, like having your experiment run, interpreting it correctly. What that means though is that like, they're not separate skillsets. So like if you have a cool idea, there's kind of not many people in the world, I think, where they can just like have a cool idea and then they have a, you know, like a little minion, they'll depressize me like, here's my idea.

• 31:04Speaker 01

  You know, go off for three months and like run this whole, like build this model and train it for, you know, hundreds of hours and report back on what happened. A lot of the time, like the people that are the most productive, they have an idea, but they're also extremely quick at checking their idea, finding sort of like the shortest path to checking their idea. And a lot of like, that shortest path is engineering skills essentially, it's just like getting stuff done. And so I think that's why you see sort of like people move around is like proportionate to your interest. If you're just able to quickly execute on the ideas you have and get results, then that's really the 90% of the value. And so you see a lot of transferable skills, actually, I think, from people like, I've certainly seen anthropics that are just like really good at that inner loop.

• 31:47Speaker 01

  They can apply it in one team and then move to a completely different domain and apply that inner loop just as well.

• 31:52Speaker 00

  Yeah, very correct, as the kids say. Shall we move to the history of Mechinterp?

• 31:57Speaker 01

  Yeah.

• 31:58Speaker 00

  All I know is that everyone started Chris Oles' blog. Is that right?

• 32:01Speaker 01

  Yeah, I think that's the correct answer. Chris Oles' blog and then, you know, Distill that pub is the sort of natural next step. And then I would say, you know, now there's for anthropic, there's transformer circuits, which you talked about. And but there's also just a lot of Mechinterp research out there from, you know, I think like the, yeah, like MATS is a group that like regularly has a lot of research, but there's just like many different labs that that put research out there. And I think that's also just like hammer home the point. That's because all you need is like a model and then a willingness to kind of investigate it, to be able to contribute to it. So so now it's really like, there's been a bit of a Cambrian explosion of Mechinterp, which is cool.

• 32:45Speaker 01

  I guess the history of it is just computational, like, models that are not decision trees, models that are either CNNs or, let's say, transformers, have just this really, like, strange property that they don't give you interpretable intermediate states by default. You know, again, to go back to, if you were training like a decision tree on like fraud data for an old school like bank or something, then you can just look at your decision tree and be like, oh, it's learned that like, if you make, I don't know, this transaction is more than $10,000 and it's for like perfume, then maybe it's fraud or something. You can look at it and say like, cool, like, that makes sense, I'm willing to ship that model. But for, for things like, like CNNs and like transformers, we, we don't have that, right?

• 33:28Speaker 01

  What we have at the end of training is just a massive amount of weights that are connected somehow, or activations are connected by some weights. And who knows what these weights mean or what the intermediate activations mean? And so the quest is to understand that. Initially, it was done, a lot of it was on vision models, where you sort of have the emergence of a lot of these ideas, like, what are features? What are circuits? And then more recently, it's been mostly, or not most, yeah, mostly applied to NLP models. But also, you know, still, there's work in vision and there's work in like bio and other domains.

• 34:02Speaker 00

  Yeah, I'm on Chris Allis' blog and he has like the feature visualization stuff. I think for me, the clearest was like, the vision work where you could have like, this layer detects edges, this layer detects textures, whatever. That seemed very clear to me, but the transition to language models seemed like a big leap.

• 34:18Speaker 01

  I think one of the bigger changes from vision to, to like language models has to do with the superposition hypothesis, which maybe is like-

• 34:27Speaker 00

  That's the first in like toy models post, right?

• 34:30Speaker 01

  Exactly. And this is sort of like, it turns out that if you look at just the neurons of a lot of vision models, you can see neurons that are curve detectors, or that are edge detectors, or that are high-low frequency detectors. And so you can sort of like make sense of the neurons, mostly. But if you look at neurons in language models, most of them don't make sense. It's kind of like unclear why, or it was unclear why that would be. And one main like hypothesis here is the superposition hypothesis. So what does that mean? That means that like- Language models pack a lot more in less space than vision models. So maybe like a, a, a kind of like really hand wavy analogy, right? It's like, well, if you want curve detectors, like you don't need that many curve detectors.

• 35:15Speaker 01

  You know, if each, each curve detector is going to detect like a, a quarter or 12th of a circle. Like, okay, well, you have all your curve detectors. But think about all of the concepts that like Claude or even G P T two needs to, to know. Like just in terms of, it needs to know about like all of the different colors, all the different hours of every day, all of the different cities in the world, all of the different streets on every city. If you just enumerate all of the facts that like a model knows, you're going to get like a very, very long list and that list is going to be way bigger than like the number of neurons or even like the size of the residual stream, which is where like the models process information. And so there's this sense in which like, oh, there's more information than there's like dimensions to represent it. And that is much more true for language models than for vision models.

• 36:01Speaker 01

  And so because of that, when you look at a part of it, it just seems like it's like there's, it's got all this stuff crammed into it. Whereas if you look at the vision models, oftentimes you could just like, yeah, cool. This is the curve detector.

• 36:11Speaker 00

  Yeah. Yeah. Veebo, you have like some fun ways of explaining the toy models or supervision concepts.

• 36:17Speaker 02

  Yeah. I mean, basically like, you know, if you have two neurons and they can represent five features, like a lot of the early mech and terp work says that, you know, there are more features than we have neurons, right? So I guess my kind of question on this is for those interested in getting into the field, what are like the key terms that they should know? What are like the few pieces that they should follow, right? Like from the anthropic side, we had a toy transformer model. We had sparse, we first had auto encoders.

• 36:42Speaker 00

  That was the second paper, um, right? Yeah. Monosubasticity.

• 36:46Speaker 02

  Yeah. What is sparsity in auto encoders? What are transcoders? Like what is linear probing? What are these kind of like key points that we had in mech and terp? And just kind of how would people get a quick, you know, zero to like 80% of the field?

• 37:00Speaker 01

  Okay. So zero to 80%. And I realized I really like set myself up for for failure. So I was like, yeah, it's easy. There's not that much to know. So, okay, so then then we should be able to cover it all. So superposition is the first thing you should know, right? This idea that like, there's a bunch of stuff crammed in a few dimensions. As you said, maybe you have like two neurons and you want to represent five things. So if that's true, and if you want to understand how the model represents, you know, I don't know, the concept of red, let's say, then you need some way to like, find out essentially in which direction the model stores it. So after the the server like supervision hypothesis, you can think of like, ah, we also think that like basically the model represents these like individual concepts. We're going to call them features as like directions. So if you have two neurons, you can think of it as like, it's like the 2D plane.

• 37:44Speaker 01

  And it's like, ah, you can have like five directions. And maybe you would like arrange them like the spokes of a wheel. So they're sort of like maximally separate. It could mean that like you have one concept this way and one concept that's like, not fully perpendicular to it, but like pretty, pretty like far from it. And then that would like allow the model to represent more concepts and it has dimensions. And so if that's true, then what you want is you want like a model that can extract these independent concepts. And ideally, you want to do this like automatically, like can we just, you know, have a model that tells us like, oh, like this direction is red. If you go that way, actually, it's like, I don't know, chicken. And if you go that way, it's like the Declaration of Independence, you know?

• 38:25Speaker 01

  And so that's what sparse auto encoders are.

• 38:27Speaker 00

  It's almost like the the self supervised learning insight version, like in in pre-training, you have self-supervised learning.

• 38:33Speaker 01

  Yeah.

• 38:33Speaker 00

  And here now is so self-supervised interpretability.

• 38:36Speaker 01

  Yeah, exactly, exactly. It's like an unsupervised method.

• 38:39Speaker 00

  Yeah.

• 38:39Speaker 01

  And so unsupervised methods often still have like labels in the end. So sometimes I feel like the.

• 38:44Speaker 00

  You form labels by masking.

• 38:46Speaker 01

  Yeah, like for for pre-training, right? It's like the next token. So in that sense, you have a supervision signal. And here the supervision signal is simply you take the like neurons. And then you learn a model that's going to like, expand them into like the actual number of concepts that you think there are in the model. So you have two neurons, you think there's five concepts. So you expand it to like, I think of dimension five. And then you contract it back to what it was. That's like the model. You're training and then you're training it to incentivize it to be sparse so that there's only like a few features active at a time. And once you do that, if it works, you have this sort of like nice dictionary, which you can think as like a way to decode deactivate the neurons. Where you're saying like, ah, cool. I don't know what this, what this direction means, but I've like,

• 39:31Speaker 01

  used my model and it's telling me that the model is writing in the red direction. And so that's, that's sort of like, I think maybe the biggest thing to understand is, is this combination of things of like, ah, we have too few dimensions, we pack a lot into it. So we're going to learn an unsupervised way to like unpack it. And then analyze what each of those dimensions that we've unpacked are.

• 39:49Speaker 00

  Any follow ups?

• 39:51Speaker 02

  Yeah, I mean, the follow ups of this are also kind of like, some of the work that you did is in clamping, right? What is the applicable side of mecha interpret, right? So we saw that you guys have like great visualizations. Golden Gate Claude was a cool example.

• 40:03Speaker 00

  I was gonna say that. Yeah, that's my favorite.

• 40:05Speaker 02

  What can we do once we find these features? Finding features is cool, but what can we do about it?

• 40:10Speaker 01

  Yeah, I think it's kind of like two big aspects of this, like, one is, yeah, okay, so we go from a state where, as I said, the model is like a mess of weights. We have no idea what's going on to, okay, we found features. We found a feature for red, a feature for Golden Gate Claude, or for the Golden Gate Bridge, I should say. Like, what do we do with them? And well, if these are true features, that means that like they in some sense are, are important for the model, or it wouldn't be like representing it. Like if the model is like bothering to like write, you know, in the Golden Gate Bridge direction, it's usually because it's gonna like talk about the Golden Gate Bridge. And so that means that like, if that's true, then you can like set that feature to zero or artificially set to a hundred. And you'll change model behavior.

• 40:52Speaker 01

  Uh, that's what we did when we did Golden Gate Claude, in which we found a feature that represents a direction for the Golden Gate Bridge. And then we just like set it to always be on. And then you could talk to Claude and be like, hey, like, oh, what's on your mind? You know, like, what are you thinking about today? Be like the Golden Gate Bridge. You'd be like, hey, Claude, like, what's two plus two? It'd be like four Golden Gate Bridges. Et cetera, right? And it was always thinking about.

• 41:13Speaker 00

  It's like write a poem and it just starts talking about how it's like red, like the Golden Gate Claude.

• 41:17Speaker 01

  That's right.

• 41:18Speaker 00

  Golden Gate Bridge. Yeah.

• 41:18Speaker 01

  That's right.

• 41:19Speaker 00

  Amazing.

• 41:19Speaker 01

  I think what made it even better is like, we realized later on that. It wasn't really like a Golden Gate Bridge feature. It was like being in awe at the beauty of the majestic Golden Gate Bridge, right? So I'll tell you, I would like really ham it up. You'd be like, oh, I'm just thinking about the beautiful international orange color of the Golden Gate Bridge. That was just like an example that I think was like really striking. But of of sort of like, oh, if you found like a space where that represents some computation or sort of representation of the model, that means that you can like artificially suppress or promote it. And that means that like you're starting to understand at a very high level, a very gross level, like how some of the model works, right? We've gone from like, I don't know anything about it to like, oh, I know that this like combination of neurons is this and I'm going to prove it to you.

• 42:03Speaker 01

  The next step, which is what this this works on is like, that's kind of like thinking of if maybe you take the analogy of like, um, I don't know, like, like, let's take the analogy of like an MRI or something like a brain scan. It tells you like, oh, like this, as Claude was answering, at some point it thought about this thing. But it's a sort of like vague, like basically, maybe it's like a like a bag of words, because it's like a bag of features. You just have like, here are all the random things it thought about. But what you might want to know is like, okay, but Claude is doing some processing. Like sometimes to get to the Golden Gate Bridge, it had to realize that you were talking about San Francisco and about like the best way to go to Sonoma or something. And so that's how it got to Golden Gate Bridge. So there's like an algorithm that leads to it at some point thinking about the Golden Gate Bridge. And basically, like, there's like a way to connect features to say like, oh, from this input,

• 42:48Speaker 01

  went to these few features and these few features and then these few features and that one influenced this one. And then you got to the output. And so that's the second part and the part we worked on is like, you have the features. Now connect them in what we call or what's called circuits, which is sort of like explaining the the like algorithm.

• 43:04Speaker 00

  Yeah, before we move directly on to your work, I just want to give a shout out to Neil Nanda. He did neuronpedia and released a bunch of essays for, I think, the Lama models and the Gemma models.

• 43:14Speaker 01

  And the Gemma models, yeah.

• 43:15Speaker 00

  Uh, so I actually made Golden Gate Gemma. Just up the weights for proper nouns and names of places or people and references to the term golden, likely relating to awards, honors or special names, and that together made Golden Gate.

• 43:30Speaker 01

  Yeah.

• 43:31Speaker 00

  So you can make Golden Gate Gemma and like, I think that's a, that's a fun way to experiment with this. But yeah, we can move on to.

• 43:36Speaker 02

  I'm curious. I'm curious. What's the background behind why you shipped Golden Gate Claude? Like you had so many features, just any fun story behind why that's the one that made it?

• 43:46Speaker 01

  You know, it's funny. If you look at the paper, there's just a bunch of like, yeah, like really interesting features, right? There's like, one of my favorite ones was the psychophantic praise, which I guess is very topical right now.

• 43:57Speaker 00

  Very topical.

• 43:58Speaker 01

  Um, but you know, it's like you could dial that up and like Claude would just really praise you. You'd be like, oh, you know, like I wrote this poem, like roses are red, violets are blue, whatever. And he'd be like, that's the best poem I've ever seen. Um, and so we could have shipped that. That could have been funny. Uh, Golden Gate Claude was like a pure, as far as I remember at least, like a pure, just like weird, random thing where like, somebody found it initially with an internal demo of it. Everybody thought it was hilarious and then that's sort of how it came on. There was no, nobody had a list of top 10 features we should consider shipping and we picked that one. It was just kind of like a very organic moment.

• 44:36Speaker 00

  No, like the, the marketing team really leaned into it. Like they mailed out pieces of Golden Gate for people in Europe, I think, or ICML. Yeah, it was, it was fantastic marketing. Yeah. The question obviously is like, if OpenAI had invested more interpretability, would they have caught the GPT four O update? Uh, but we don't know that for sure because they have interpret teams. They just.

• 44:57Speaker 01

  Yeah. I think also like for that one, I don't know that you need interpret. Like it was pretty clear cut. Talking to the model. I was like, oh, that model is really gassing me up.

• 45:03Speaker 00

  And then the other thing is, can you just like up, write good code, don't write bad code and make Sonder three point five? And like, it feels too, too easy, too free. Is that steering that powerful that you can just like up and down features with no trade offs?

• 45:18Speaker 02

  There was like a phase where people were basically saying, you know, three point five and three point seven are just now, because they came out right after.

• 45:25Speaker 00

  And for the record, like that's been debunked.

• 45:27Speaker 02

  Yeah, it has been debunked, but you know, it had people convinced that what people did is they basically just steered up and steered down features and now we have a better model. And this kind of goes back to that original question of right? Like, why do we do this? What can we do? Some people are like. I want tracing from a sense of, you know, legality, like, what did the model think when it came to this output? Some people want to turn hallucination down. Some people want to turn coding up. So like, what are some, like, whether it's internal, what are you exploring that, like, what are the applications of this? Whether it's open-ended of what people can do about this, or just, like, yeah, why do mecan terp, you know?

• 46:01Speaker 01

  Yeah, there's like a few things here. So like, first of all, obviously, this is, I would say, on the scale of the most short-term to the most long-term, like, pretty long-term research. So in terms of like, applications compared to, you know, like, the research work we do on like, fine-tuning or whatever, interp is much more, you know, sort of like a high-risk, high-reward kind of approach. With that being said, like, I think there's just a fundamental sense in which Michael Nielsen had a post recently about how like, knowledge is dual-use or something, but just like, just like knowing how the model works at all feels useful. And, you know, it's hard to argue that if we know how the model works and understand all the components that won't help us,

• 46:44Speaker 01

  like, make models that hallucinate less, for example, or that are like, less biased. That seems, you know, if, like, at the limit, yeah, that totally seems like something you could do, using, basically, like, your understanding of the model to improve it. I think for now, as we can talk about a little bit with, like, circuits, there's like, we're still pretty early on in the game, right? And so, right now, the main way that we're using inter early is like, to investigate specific behaviors and understand them, and gain a sense for what's causing them. So like, one example that we can talk about later, we can talk about now, but in the paper, we investigate jailbreaks, and we try to see, like, why does a jailbreak work? And then we realize, as we're looking at this jailbreak, that part of the reason why Claude is telling you how to make a bomb in this case,

• 47:25Speaker 01

  is that it's like, already started to tell you how to make a bomb, and it would really love to stop telling you how to make a bomb, but it has to first finish its sentence. Like, it really wants to make correct grammatical sentences. And so it turns out that, like, seeing that circuit, we were like, ah, then does that mean, if I prevent it from finishing its sentence, the jailbreak works even better? And sure enough, it does. And so I think, like, the level of sort of practical application right now is of that shape. So like, understanding either, like, quirks of a current model, or like,

• 47:56Speaker 01

  How it does tasks that maybe we don't, we don't even know how it does it. Like, you know, we, we have like some planning examples where we had no idea it was planning and we're like, oh God, it is. That's sort of like the current state we're at.

• 48:06Speaker 02

  I'm curious internally how this kind of feeds back into like the research, the architecture, the pre-training teams, the post-training, like, is there a good feedback loop there? Like, right now, there's a lot of external people interested, right? Like, we'll train an SAE on one layer of llama and probe around. But then people are like, oh yeah, how does this have much impact? People like clamping, but yeah, as you said, you know, once you start to understand these models have this early planning and stuff, how does this kind of feedback?

• 48:32Speaker 01

  I don't know that there's, there's like much to say here other than like, I think we're definitely interested in conversely, like making models for which it's like easier to interpret them. So that's also something that you can imagine sort of like working on, which is like making models where you have to work less hard to try to understand what they're doing.

• 48:48Speaker 00

  The architecture? Okay. Yeah. So I think there was a, there was a less wrong post about this of like, there's a non-zero amount of sacrifice you should make in current capabilities in order to actually make them more interpretable because otherwise they will never catch up.

• 49:02Speaker 01

  You know, there's this sort of sense in which like right now we take the model and then the model's a model. And then we post hoc do these replacement layers to try to understand it. But of course, when we do that, we don't like fully capture everything that's happening inside the model. We're capturing like a subset. And so maybe some of it is like, you could train a model that's sort of like easier to interpret naively. And it's possible that like, you don't even have that much of, you know, like a tax in that sense. And you can just sort of like, either like train your model differently or do like a little post hoc step to like, sort of like untangle some of the mess that you've made when you trained your model, right? Make it easier to interpret.

• 49:34Speaker 00

  The hope was pruning would do some of that, but I feel like that area of research has just died.

• 49:40Speaker 01

  What kind of pruning are you thinking of here?

• 49:41Speaker 00

  Uh, just pruning your network.

• 49:43Speaker 01

  Ah, yeah.

• 49:45Speaker 00

  Pruning layers, pruning connections, whatever.

• 49:48Speaker 01

  Yeah, I feel like maybe this is something where like superposition makes me less hopeful or something.

• 49:54Speaker 00

  Cause you don't know, like that, that like seventh bit might hold something.

• 49:59Speaker 01

  Well, right. And it's like, on, on each example, maybe this neuron is like at the bottom of like what matters. But actually it's participating like five percent to like understanding English, like doing integrals and you know, like whatever, like cracking codes or something. And it's like, because that was just like distributed over it, you, you, you kind of like, when you naively prune, you might miss that. I don't know.

• 50:21Speaker 00

  Okay, so then this area of research in terms of creating models that are easier to interpret from the start, is there a name for this field of research?

• 50:29Speaker 01

  I don't think so, and I think it was like very early.

• 50:31Speaker 00

  Okay.

• 50:32Speaker 01

  And it's mostly like a dream.

• 50:33Speaker 00

  Just in case there's a thing people want to double click on.

• 50:35Speaker 01

  Yeah, I haven't come across it.

• 50:36Speaker 02

  I think the higher level is like Dario recently put out a post about this, right? Why mechathepa is so important? You know, we don't want to fall behind. We want to be able to interpret models and understand what's going on. Even though capabilities are getting so good, it kind of ties into this topic, right? Like, we want models to be slightly easier to interpret, so we don't fall behind so far.

• 50:57Speaker 01

  Well, yeah, and I think here, like, just to talk about the elephant in the room or something like one big concern here is like safety. Right? And so like, as models get better, they are going to be used more and more places. You know, it's like, you're not going to have your, you know, we're vibe coding right now, maybe at some point, well, that'll just be coding. It's like Claude's going to write your code for you and that's it. And Claude's going to review the code that Claude wrote and then Claude's going to deploy it to production. And at some point, like, as these models get integrated deeper and deeper into more and more workflows, it gets just scarier and scarier to know nothing about them. And so you kind of want your ability to understand the model to scale with like how good the model is doing, which that itself kind of like tends to scale with like how widely deployed it is.

• 51:39Speaker 01

  So as we like deploy them everywhere, we want to like understand them better.

• 51:43Speaker 00

  The version that I liked from the old super alignment team was weak to strong generalization or weak to strong alignment, which that's what super alignment to me was, and that was my first aha moment of like, oh yeah, some, at some point, these things will be smarter than us. And in many ways, they already are smarter than us and we rely on them more and more. We need to figure out how to control them. And this is not a like an Ilya Zeroyakovsky like ah thing. It's just more like, we don't know what, how these things work, like, how can we use them?

• 52:10Speaker 01

  Yeah, and like you can think of it as, there's many ways to solve a problem. And some of them, if the model is solving it in like a dumb way or in like memorized one approach to do it, then you shouldn't deploy it to do like a general thing. Like, like you could look at how it does math and based on your understanding of how it does math, you're like, okay, I feel comfortable using this as a calculator or like, no. It should always use a calculator tool because it's doing math in a stupid way and extend that to any behavior, right? Where it's just a matter of like, think about it if like you're like in the 1500s and I give you a car or something and I'm just like, cool, like this thing, when you press on this, like it accelerates when you press on that, like it stops, you know, this steering wheel seems to be doing stuff, but you knew nothing about it. I don't know if it was like a super faulty car and it's like, oh yeah, but if you ever went above

• 52:55Speaker 01

  60 miles an hour, like it explodes or something like you probably would be sort of like, you'd want to understand the nature of the object before like jumping in in it. And so that's why we like, understand how cars work very well because we make them. L L M's are sort of like, and M L models in general are like this very rare artifact where we like make them, but we don't, we have no idea how they work.

• 53:12Speaker 00

  We evolve them. We create conditions for them to evolve and then they evolve and we're like, cool, like, you know, maybe you got a good run, maybe you didn't.

• 53:20Speaker 01

  Yeah.

• 53:20Speaker 00

  Don't really know.

• 53:22Speaker 01

  Yeah. The extent to which you know how it works is you have your like eval and you're like, oh, well, seems to be doing well on this eval. And then you're like, is it because this wasn't a training set or is it like actually generalizing? I don't know.

• 53:31Speaker 00

  Uh, my favorite example was somehow C four, the, the common, the colossal clean corpus did much better than, uh, common crawl, even though it filtered out most of this, like it was very prudish. So it like filters out anything that could be considered obscene, including the word gay. But like somehow it just like, when you add it into the data mix, it just does super well. And it's just like this magic incantation of like, this, this recipe works, just trust us. We think we've tried everything. This one works. So just go with it.

• 54:01Speaker 01

  Yeah.

• 54:01Speaker 00

  It's not very satisfying.

• 54:03Speaker 01

  No, it's not. The side that you're talking about, which is like, okay, like, how do you make these? And it's kind of unsatisfying that you just kind of make the soup and you're like, oh, well, you know, my grandpa made this soup with these ingredients. I don't know why, but I just make the soup the way my grandpa said. And then like some, one day somebody added, you know, cilantro and since then we've been adding cilantro for generations. And you're like, this is kind of crazy.

• 54:22Speaker 00

  That's exactly how we train models though.

• 54:23Speaker 01

  Yeah. Yeah. Um, so I think there's, there's like a part where it's like, okay, like, let's try to unpack what's happening, you know, like the mechanisms of learning, like how, how are models learning? Like one of them, I guess, I guess we skipped over it, but like one of the things where like induction heads, you know, like understanding what induction heads are, which are attention heads that allow you to look at in your context, the last time that something was mentioned and then repeat it is like something that happens, it seems to happen in every model. And it's like, oh, okay, that makes sense. That's how the model like is able to like, repeat text without dedicating too much capacity to it.

• 54:53Speaker 00

  Let's get it on screen so people can see.

• 54:55Speaker 02

  Visuals of the work you guys put out is amazing.

• 54:58Speaker 00

  Oh, yeah, we should.

• 54:59Speaker 02

  Highly, highly.

• 54:59Speaker 00

  We should talk a little bit about the back behind the scenes of that that kind of stuff. But but let's let's let's finish this off first.

• 55:03Speaker 01

  Totally, but just really quickly, I don't think we should spend too long on it. I think it's just like, if you're interested in mechanarp, we talked about superposition. And I think we skipped over induction heads. And that's like, you know, kind of like a really neat, basically, pattern that emerges in many, many transformers, where essentially they just learn. Like, one of the things that you need to do to predict text well is that if there's repeated texts at some point, somebody said Emmanuel Mason, and then you're like on the next line and they say Emmanuel. Very good chance it's the same last name. And so one of the first things that models learn is just like, OK, I'm just gonna like, look at what was said before, and I'm gonna say the same thing. And that's induction heads, which is like a pair of attention heads that just basically look at the last time something was said, look at what happened after, move that over. And that's an example of a mechanism where it's like, cool, now we understand that pretty well.

• 55:44Speaker 01

  There's been a lot of follow up research on understanding better, like, OK, like, in which context do they turn on? Like, you know, there's like different like levels of abstraction. There's like induction heads that like literally copy the word. And there's some that copy like the sentiments and other aspects. But I think it's just like an example of slowly unpacking, you know, or like peeling back the layers of the onion of like, what's going on inside this model? OK, this is a component is doing this.

• 56:06Speaker 00

  So induction headers was like the first major finding?

• 56:08Speaker 01

  It was a big finding for NLP models, for sure.

• 56:11Speaker 00

  I often think about the edit models. So Claude has a fast edit mode. I forget what it's called. OpenAI has one as well. And you need very good copying, every area that needs copying. And then you need it to switch out of copy mode when you need to start generating.

• 56:26Speaker 01

  Right.

• 56:27Speaker 00

  And that is basically the productionized version of this.

• 56:29Speaker 01

  Yeah, yeah, yeah. And it turns out that, you know, you you you need to select a model that's like smart enough to know when it needs to get out of copy mode, right?

• 56:36Speaker 00

  Yeah, it's fascinating. It it's faster, it's cheaper. You know, as bullish as I am on canvas, basically, every AI product needs to iterate on a central artifact. And like, if it's code, if it's a piece of writing, doesn't really matter. But you need that copy capability that's smart enough to know when to turn it off.

• 56:55Speaker 01

  That's why it's cool that induction heads are at different levels of abstraction. Like, sometimes you need to editing some code, you need to copy like the general structure. It's like, oh, like the last, like this other function that's similar, it first takes like, you know, I don't know, like abstract class. And then it takes like an int, so I need to like copy the general idea. But it's gonna be a different abstract class and a different int or something.

• 57:13Speaker 00

  Yeah, cool.

• 57:14Speaker 01

  Yeah.

• 57:15Speaker 00

  So tracing?

• 57:16Speaker 01

  Oh yeah, should we jump to circuit tracing? Sure.

• 57:18Speaker 00

  I don't know if there's anything else you want to cover.

• 57:20Speaker 01

  No, no, no.

• 57:20Speaker 00

  We have space for it.

• 57:21Speaker 01

  Maybe, okay, I'll do like a really quick TLDR of these two recent papers.

• 57:25Speaker 00

  Okay.

• 57:27Speaker 01

  Insanely quick. So we talked about these features that we detect. And what we said is like, okay, but we'd like to connect the features to understand like the inputs to every features and the opposite features and basically draw a graph. And this is like, if I'm still showing my screen, the thing on the right here. Where like, that's the dream. We want like, for a given prompt, what were all of the things, like all of the important things happen in the model? And here it's like, okay, it took in these four tokens. Those activated these features. These features activate these other features. And then these features activate the other features. And then all of these like promoted the output. And that's the story. And basically, we're like, the work is to sort of use dictionary learning and these replacement models to provide a explanation of like sets of features that explain behavior.

• 58:11Speaker 01

  So this is super abstract. So I think immediately, maybe we can like just look at one example. I can show you one, which is this one.

• 58:19Speaker 00

  The reasoning one?

• 58:20Speaker 01

  Yep. Yeah, two-step reasoning. I think this is already, this is like the introduction example, but it's already like kind of fun. So, so the question is, you ask the model, something that requires it to take a step of reasoning in its head. So you say, you know, fact, the capital of the state containing Dallas is. So to answer that, you need one intermediate step, right? You need to say, wait, where's Dallas? Isn't Texas? Okay, cool. Capital of Texas, Austin. And this is like in one token, right? It's gonna, after is, it's gonna say Austin. And so like, in that one forward pass, the model needs to extract, to realize that you're asking it for like the capital of a state to like look up the state for Dallas, which is Texas, and then to say Austin. And sure enough, this is like what we see is we see like in this forward pass, there's a rich sort of like

• 59:01Speaker 01

  inner set of representations where there's like, it gets capital state in Dallas and then. Boom, it has an inner representation for Texas. And then that plus capital leads it to like, say, Austin.

• 59:13Speaker 02

  I guess one of the things here is like, we can see this internal like thinking step, right? But a lot of what people say is like, is this just memorized fact, right? Like, I'm sure a lot of the pre-training that this model is trained on is this sentence shows up pretty often, right? So this shows that no, actually internally throughout, we do see that there is this middle step, right? It's not just memorized.

• 59:35Speaker 00

  You can prove that it generalized.

• 59:37Speaker 01

  Yeah, so that's exactly right. And I think like you hit the nail on the head, which is like, this is what this example is about. It's like, oh, if this was just memorized, you wouldn't need to have an intermediate step at all. You'd just be like, well, I've seen the sentence, like, I know what comes back, right? But here, there is an intermediate step. And so you could say like, okay, well, maybe it just has the step, but it's memorized it anyways. And then the way to like, verify that is kind of like what what we do later in the paper and for all of our examples is like, okay, we claim that this is like the Texas representation. Let's get another one and replace it. And we just changed like that feature in the middle of the model. And we change it to like California. And if you change it to California, sure enough, it says Sacramento. And so it's like, this is not just a like bite product, like it's memorized something. And on the side, it's thinking about Texas.

• 1:00:21Speaker 01

  It's like, no, no, no, this is like a step in the reasoning. If you change that intermediate step, it changes the answer.

• 1:00:27Speaker 02

  Very, very cool work, underappreciated.

• 1:00:29Speaker 01

  Yeah.

• 1:00:30Speaker 02

  Okay, sure.

• 1:00:31Speaker 00

  I have never really doubted. I think there's a lot of people that are always criticizing L M says stochastic parrots. This pretty much, this proves it already, like, we can move on.

• 1:00:42Speaker 01

  Yeah, I mean, I think, I think there's a lot of examples that I will say, we can go through like a few of them. Like, show an amount of depth in the intermediate states of the model. That makes you think like, oh, gosh, like, it's doing a lot. I think maybe like

• 1:00:56Speaker 00

  The poems?

• 1:00:57Speaker 01

  Well, definitely the poems, but even for this one, I'm not like scroll in this very short paper to like, uh, medical diagnoses.

• 1:01:05Speaker 00

  I don't even know the word count because there's so many like embedded things in there.

• 1:01:08Speaker 01

  Yeah, it's too dangerous. We can't look it up. It overflows.

• 1:01:13Speaker 00

  It's so beautiful. Look at this.

• 1:01:14Speaker 01

  Uh, this is like a medical example that I think shows you, again, this is in one forward pass. The model's like given a bunch of symptoms and then it's asked not like, hey, what is, what is the like disease that this person has? It's asked like, if you could run one more test to determine it, what would it be? So it's even harder, right? It means like, you need to take all the symptoms, then you need to like have a few hypotheses about what the disease could be. And then based on your hypotheses, say like, well, the thing that would like be the right test to do is X. And here you can see these three layers, right? Where it's like, again, in one forward pass, it has a bunch of like, oh, these are symptoms. Then it has the most likely diagnosis here, then like an alternate one. And then based on the diagnosis, it like gives you basically a bunch of things that you could ask.

• 1:01:56Speaker 01

  And again, we do the same experiments where you can like kill this feature here, like suppress it. And then it asks you a question about the second, the sort of like second option it had. The reason I show it is like, man, that's like a lot of stuff going on. Like for, for one forward pass, right? It's like specifically if you, if you expected it to like, oh, what it's gonna do is it's just like seeing similar cases in the training. It's gonna kind of like vibe and be like, oh, I guess like there's that word and it's gonna say something that's related to like, I don't know, headache. You know, and I kind of like really have, it's like, no, no, no. It's like activating many different distributed representations, like combining them and sort of like doing something pretty complicated. And so, yeah, I think, I think it's funny because in my opinion, that's like, yeah, like, oh, God. Stochastic parrots is not something that I think is, is like appropriate here.

• 1:02:38Speaker 01

  And I think there's just like a lot of different things going on and there's like pretty complex behavior. At the same time, I think it's in the eye of the beholder. I think like I've talked to folks that have like read this paper and I've been like, oh yeah, this is just like a bunch of kind of like heuristics that are like mashed together, right? Like the model is just doing like a bunch of kind of like, oh, if high blood pressure, then this or that. And so I think there's, there's sort of like um, an underlying question that's interesting, which is like, okay. Now we know a little bit of how it works. This is how it works. Like now you tell me if you think that's like impressive, if you think that like, if you trust it, if you think that's sort of like uh, something that is, that is sufficient to like ask it for medical questions or whatever.

• 1:03:14Speaker 00

  I think it's a way to adversarially improve the model quality.

• 1:03:17Speaker 01

  Yeah.

• 1:03:17Speaker 00

  Because once you can do this, you can reverse engineer what would be a, a sequence of words that to a human makes no sense or lets you arrive at the complete opposite conclusion, but the model still gets tripped up by.

• 1:03:31Speaker 01

  Yeah.

• 1:03:31Speaker 00

  And then you can just improve it from there.

• 1:03:33Speaker 01

  Exactly. And this gives you a hypothesis about like you like specifically imagine if like one of those was actually the wrong symptom or something, you'd be like, oh, it's weird that the liver condition like, you know, upwears this other example. That doesn't make sense. Okay, let's like fix that in particular. Exactly. You sort of have like a bit of insight into like how the model is getting to its conclusion. And so you can see both like, is it making errors, but also is it using the kind of reasoning that will lead it to errors?

• 1:03:59Speaker 00

  There's a thesis, I mean, now it's very prominent with the reasoning models about model depth. So like you're doing all this in one pass.

• 1:04:08Speaker 01

  Yeah.

• 1:04:08Speaker 00

  But maybe you don't need to, because you can do more passes.

• 1:04:11Speaker 01

  Sure.

• 1:04:13Speaker 00

  And so people want shallow models for speed, but you need model depth for this kind of thinking.

• 1:04:20Speaker 01

  Yeah.

• 1:04:20Speaker 00

  Is there a Pareto frontier? Is there a direct trade off?

• 1:04:23Speaker 01

  Yeah.

• 1:04:24Speaker 00

  What would you prefer if you had to make a model and like, you know, shallow versus deep?

• 1:04:28Speaker 01

  There's a chain of thought faithfulness example. Before I show it, I'm just gonna go back to the top here. So when the model is sampling many tokens, if you want that to be your model, you need to be able to trust every token it samples. So like, the problem with with models being auto aggressive is that like if they like at some point sample a mistake, then they kind of keep going conditioned on that mistake. Right? And so sometimes like

• 1:04:53Speaker 00

  You need backspace tokens or whatever?

• 1:04:55Speaker 01

  Yeah, yeah, yeah. And error correction is like notably hard, right? If you have like a deeper model, maybe you have like fewer COT steps, but like your your steps are more likely to be like robust or or correct or something. And so I think that that's one way to look at the trade off. To be clear, I don't have an answer. I don't know if I want a wider or a shallow or a deep model.

• 1:05:12Speaker 00

  You definitely want shallow for inference speed.

• 1:05:14Speaker 01

  Sure. Sure, sure, sure. But you're trading that off for for something else, right? Because you also want like a one B model for inference speed, but that also comes at a cost, right? It's less smart.

• 1:05:22Speaker 02

  There's a cool, quick paper to plug that we just covered on the paper club. It's a survey paper around when to use reasoning models versus dense models. What's the trade off? I think it's the economy of

• 1:05:32Speaker 00

  Reasoning economy.

• 1:05:33Speaker 02

  Reasoning, the reasoning economy. So they just go over a bunch of, you know, ways to measure this benchmarks around when to use each, because yeah.

• 1:05:40Speaker 01

  Yeah.

• 1:05:40Speaker 02

  Like, you know, we don't want to also like consumers are now paying the cost of this, right? But little, little side note.

• 1:05:46Speaker 00

  Yeah. For those on YouTube, we have a secondary channel called Lean Space TV where we cover that stuff.

• 1:05:51Speaker 01

  Nice.

• 1:05:51Speaker 00

  That's our paper club. We covered your paper.

• 1:05:54Speaker 01

  Cool. Yeah, I think you brought up the like planning thing. Maybe it's worth.

• 1:05:57Speaker 02

  Let's do it.

• 1:05:58Speaker 01

  Yeah, I think I think this one is like, if you think about, okay.

• 1:06:01Speaker 02

  So you're you're going into the chain of thought, people in this one?

• 1:06:03Speaker 01

  Let's get this one. Let's just do planning. So if you think about like, you know, common questions you have about models, the first one we we kind of asked was like, okay, like, is it just doing this like vibe based one shot? Pattern matching based on existing data? Or does it have like kind of rich inner representations? It seems to have like these like intermediate representations that makes sense as the abstractions that you would reason through. Okay, so that's one thing. And there's a bunch of examples. We talked about the medical diagnoses. There's like the multilingual circuits is another one that I think is cool where it's like, oh, it's sharing representations across languages. Another thing that you'll hear people mention about language models, which is that they're like uh next token predictors.

• 1:06:37Speaker 02

  Also, for for a quick note, for people that won't dive into this super long blog post, I know you highlighted like ten to twelve. So for like a quick fifteen thirty second, what do you mean by their sharing thoughts throughout? Just like, what's a really quick high level, just for people though?

• 1:06:51Speaker 01

  Yeah, the really quick high level is that what we find is that here, I'm just like, show you a really quick. Inside the model, if you look at like the inner representations for concepts, you can ask like the same question, which I think in the paper, the original one we asked is like, the opposite of hot is, you know, cold. But you can you can do this over a larger data set and ask the same question in many different languages. And then look at these representations in the middle of the model and ask yourself like, well, when you ask it, the opposite of hot is and le contraire de chaud est, which is the same sentence in French.

• 1:07:23Speaker 02

  Show off.

• 1:07:24Speaker 01

  Is it? Is it using the same? Features? Or is it learning independently for each language? It kind of would be bad news if it learned independently for each language, because then that means that like as you're pre-training or fine tuning, you have to relearn everything from scratch. So you would expect a better model to kind of like share some concepts between the languages it's learning, right? And you here we do it for like language languages, but I think you could argue that you'd expect the same thing for like programming languages where it's like, oh, if you learn what an if statement is in Python, maybe it'd be nice if you could generalize that to Java or whatever. And here we find that basically you see exactly that. Here we show like, if you look inside the model, if you look at the middle of the model, which is the middle of this plot here, models share more features. They share more of these representations in the middle of the model. And bigger models

• 1:08:07Speaker 01

  share even more. And so that like this are like smarter models use more shared representations than the dumber models, which might explain part of the reason why they're smarter. And so this this was like sort of this this other finding of like, oh, not only is it like having these rich representations in the middle, it like. Learns to not have redundant representations. Like if you learn the concept of heat, you don't need to learn the concept of like French heat and Japanese heat and Colombian. Like you're just, that's just the concept of heat and you can share that among different languages.

• 1:08:35Speaker 02

  I feel like sometimes over analyzing this becomes a bit of a problem, right? Like when we talked about with the medical example, we could look back and try to fix this in data set. So in language, I don't remember if it was OpenAI or Anthropic, where they basically said when the model switched languages and they pass it to fluent users, they said, oh, this, this feels like an American that's speaking this language, right? So at some times there are nuances in a slightly different representation, right? So you don't want to over engineer these little fixes when you do see them. But then the other side of this is like for those tail end of languages, right? For languages that models aren't good at. And for those, like, you know, when you want to kind of solve that last bit, it seems like, you know, it's pretty plausible

• 1:09:19Speaker 02

  that we can solve this because these concepts can be shared across languages as long as we can, you know, fill in some, some level of representation, unless I'm wrong.

• 1:09:28Speaker 01

  No, totally. And, and I think like this sort of stuff also explains, you know, language models are really good at in context learning. Like you give them something completely new and you do a good job. It's like, well, if you give them like a new fake language and you like in that language explain that like cold means this and hot means that, you know, like presumably they're able to, as we clear this speculation, we don't show it in the paper.

• 1:09:47Speaker 00

  But they're able to like bind it. Google's done this.

• 1:09:49Speaker 01

  Okay, great.

• 1:09:50Speaker 00

  They took a low resource language, dumped it in a million token context and then it came out.

• 1:09:53Speaker 01

  That's right, that's right. Well, I guess the thing that, the thing that'd be curious to see is like, okay, does it use, does it reuse these representations? I bet that it probably does, right? And that's probably like a reason why it works well is like, well, it can reuse the representation, the general representations that it's learned in other languages.

• 1:10:07Speaker 00

  Yeah, this is like, I don't, have you talked to any linguistics people?

• 1:10:11Speaker 01

  Not recently.

• 1:10:13Speaker 00

  Linguistics researchers will be very interested in this because ultimately this is the ultimate test of superior wharf, which are you familiar with?

• 1:10:19Speaker 01

  Superior wharf hypothesis.

• 1:10:19Speaker 00

  So for those who don't know, it's basically the idea that the language that you speak influences the way you think, which obviously it directly maps onto here. If every, if it's a complete mapping, if every language maps every concept perfectly on in like the theoretical infinitely sized model, then superior wharf is false. Because there is a universal truth. If it does not, if there is some overlap where, for example, there's some languages that have no word. This is joke where like, you know, Eskimos have no word for snow or something like that, right? Or water has no word, fish have no word for water. There's an African language where there's a gender for vegetables. You know, it's stuff like that, just like languages influence the way you think. And so there should not be a hundred percent overlap at some point.

• 1:11:00Speaker 01

  Of course, it's like at the limit of the infinite model, so who knows if it will ever. But, but yeah, well, and I think it's, it's interesting. We also show a little below that, like, some people have made the point of like the bias. Oh, it sounds like an American speaking a different language. And it does seem like the sort of like inner representations have a higher connection to like the output logits for English logits. And so there's like some bias towards English, at least in the model we studied here.

• 1:11:25Speaker 00

  Any thoughts as to whether multimodality influences any of this? So like concepts do they map across languages as they do across modalities?

• 1:11:32Speaker 01

  Yeah, so we show this in the Golden Gate or like the previous paper. I might have it here actually for you.

• 1:11:37Speaker 02

  There's a good diagram of this in the SAEs where the same concept in text and in image.

• 1:11:42Speaker 01

  This is our buddy, the Golden Gate Bridge. Here we're showing like the feature for the Golden Gate Bridge. And in orange is like what it activates over. And so you're like, okay, so this is when the model is like reading text about the Golden Gate Bridge. And we also show other languages. This is, you'll have to take my word for it, but also about the Golden Gate Bridge. And then we, we show like the photos for which it activates the most. And sure enough, it's the Golden Gate Bridge. And so again, like that shows an example of a representation that's shared across languages and shared across modalities.

• 1:12:07Speaker 00

  Yeah, yeah, I think it's very relevant for like the auto-regressive image generation models. And then now the audio models as well. Something I'm trying to get some intuition for, which you probably don't have a off the bat answer is how much does it cost to add a modality?

• 1:12:20Speaker 01

  Right.

• 1:12:20Speaker 00

  So a lot of people are saying like, oh, just add some different decoder and then align the latent spaces and you're good. And I'm like, I don't know, man. It sounds like there's a lot of information loss between those.

• 1:12:32Speaker 01

  Yeah, I definitely do not have a good intuition for this, although I will say that things like this, right? Make you think that if you train on multiple modalities, then you'll definitely get this like alignment.

• 1:12:43Speaker 00

  Truth.

• 1:12:44Speaker 01

  Right? Yeah. But, but if, if you like train on one and then post hoc train on another, maybe, maybe it'll be harder or like train some adapter layer.

• 1:12:51Speaker 00

  Okay, so official answers don't know, but.

• 1:12:53Speaker 01

  Official answer is.

• 1:12:54Speaker 00

  Someone could figure it out.

• 1:12:54Speaker 01

  Shrug. Yeah.

• 1:12:56Speaker 00

  I think there are people who know and they just haven't.

• 1:12:58Speaker 01

  Shared. Well, you need to find them and get them on this podcast. Did we want to do the like planning example? Correct. Yeah, now we're backtracking up the stack. All right. Yeah. Planning example, I think, again, is like, I like this example because of the next token predictor concept. So I think this is actually like really important to kind of like dive into. So maybe what I'll say is like, language models are next token predictors is like a fact. Like that is what they do. That's the objective. They they are trained to predict the next token. However, that does not mean that they myopically only consider the next token when they choose the next token. You can work on break the next token, but still like doing so in a way that helps helps you predict the token like 10 tokens in the future. And I think, well, now

• 1:13:43Speaker 01

  we definitely know that they're not myopically predicting the next token. And I think, at least for me, that was a pretty big update. Because you could totally imagine that they could do everything they're doing by just like being really good at predicting the next token, but sort of like not having an internal state. Like it's it's it wasn't a given that they were going to like represent internally. Oh, this is where I want to go. And so I'm going to prove the next token. And so this example shows like an example like the model. Do you have it on screen, by the way? Let me. Okay, yeah, yeah.

• 1:14:09Speaker 02

  So while you pull it up, some of the early connections I made to this were like early, early transformers. So think BERT encoder decoder transformers, right? When they came out, some of the suggestions were you don't take the last layer, right? You take off the last layer. So if you want to do a classification task, a translation task for these encoder decoder transformers, they've kind of overfit on their training objective, right? So they're really good at mass language modeling, at filling in, you know, sentence order, stuff like that. So what we want to do is we want to throw away the top layer. We want to freeze the bottom layers. And then there was a lot of work that was done, you know, where should we mess with these models? Should we look at like, you know, the top three layers? Should we look at the top two? Where should we probe in?

• 1:14:52Speaker 02

  Because we can see different effects, right? So we know at the very end, they've overfit on their task, but there's a level at which, you know, when we start to change and we start to continue training or fine tuning, we get better output. So

• 1:15:05Speaker 01

  Totally.

• 1:15:05Speaker 02

  we could start to see that, you know, throughout layers, there's there's still a broader like understanding the language and then we can add in a layer, whether that's classification and then fine tune and, you know, it learns our task. And this planning example is sort of like a more robust way to look into that.

• 1:15:19Speaker 01

  Yeah, yeah. And I think if you look at like all of the examples in the paper, you kind of, uh, at the bottom, we have this list of like consistent patterns. And one pattern you see is kind of exactly what you're talking about. Like at the top, the sort of like, here, actually I have one here. The sort of like top features are like right before the output are often just about like what you're gonna say. It's next token prediction. It's like, oh, I'm gonna say Austin. I'm gonna say rabbit. I'm gonna say, so it's kind of like not very abstract. It's just like a motor. It's a motor neuron for a human, right? It's like, oh, I've, I've decided that I want a drink of water and so I'm gonna just grab the bottle. And at the bottom, they're all like the kind of like basically sensory neurons. They're just like, oh, I just saw the word X or I just saw this. And so if you want to like, yeah, like extract the interesting representations, a lot of the time they're in the middle.

• 1:16:01Speaker 01

  That's where the like shared representations across language are. And that's where here this like plan is. To like walk through the example really briefly, it's like you have a poem. And in order to say you have the first line of a poem, and in order to say the second line of the poem. Well, if you want to rhyme, you need to like identify what the rhyme of the first line was. You're just at the end of the first line. So you say like, okay, what's my current rhyme? And then you need to like think about what your poem is talking about. And then think about candidate words that rhyme and that are like on topic for your poem. And so here, this is what's happening, right? It's like, the last word is it. And so there's a bunch of features that are actually. They represent the direction like rhyming with eat or at.

• 1:16:40Speaker 01

  And by the way, we like looked at a bunch of poems internally and you have like, I thought it was like really beautiful. You have these models. They have a bunch of features for like. Oh, this word has like A B in it. Oh, this word has like many consonants. Oh, this word like is like, you know, kind of, kind of like. Has some flourish to it. They have like a bunch of, of like features that track various aspects that you would want to use if you're writing poetry.

• 1:16:59Speaker 00

  It's just like conv nets and like all the feature detection.

• 1:17:02Speaker 01

  Totally, yeah, totally. Uh, but I think I maybe I didn't expect there to be as many features about just like. Sounds of words and musicality, which I thought was kind of, kind of neat. But then once it's extracted the rhyme, then it comes up with sort of like these two candidates. In this case, it's like, ah, either I'm gonna finish with rabbit or I'm gonna finish with habit. The cool thing here is here we show that like this happens at the new line. So it happens before it's even started the second line. And it turns out that like, you can then say, oh, is this the plan's actually using? We do our usual experiments. We like. Remove it and the model writes a completely different line. We inject something and it writes a completely different line. We have these like fun examples here I'll show, which is.

• 1:17:43Speaker 00

  Just as a mechanical thing, you could, you, you just, you just disallow generation of a certain logic. Is that?

• 1:17:49Speaker 01

  For how we do these interventions?

• 1:17:51Speaker 00

  Yeah, yeah.

• 1:17:51Speaker 01

  Basically, what these features are is they're like directions in the model.

• 1:17:54Speaker 00

  Okay.

• 1:17:55Speaker 01

  So to like, remove them, we just write in the opposite direction. So we run the model normally. And then like, add the like, layer where it was gonna write, let's say, in like, you know, this this direction. We just like, stop it.

• 1:18:05Speaker 00

  Negative everything.

• 1:18:06Speaker 01

  Yeah, we either like, add a negative that like, compensates for it. Or add a negative that goes even more in the negative direction sometimes, like, really kill it. And then we can also add another direction, right? So, in these random examples here, where where like, you have this poem. The silver moon cast a gentle light. And then Claude 3.5 Haiku would like rhyme with illuminating the peaceful night. But then if we like, go negative in the night direction, and just add like, green. The whole second line is gonna write is just upon the meadows verdant green. And so that's all that we're doing. We're saying like, we found where it stores its plan, and we like, deletes or like, suppress the one that's stored and go in the direction of something else that's arbitrary. And the result that's like, striking here, it serves like, two things. I think like,

• 1:18:48Speaker 01

  one, this plan is made well in advance of needing to predict night. It's made like, after the first line, before it's even started the second line. And two, this plan doesn't just control like, what you're gonna rhyme with. It's also doing what's called like, backwards planning, where it's like, well, because I need to finish with green, I'm not gonna say illuminating the peaceful night, because then I'd be like, illuminating the peaceful green. That doesn't make sense. I need to say a completely different sentence that lets me finish with green. And so there's a circuit in the model that decides on the rhyme, and then works backwards from the rhyme.

• 1:19:21Speaker 00

  Influences.

• 1:19:22Speaker 01

  To set up your sentence.

• 1:19:23Speaker 00

  Yeah, it's almost like backprop, but in the future.

• 1:19:27Speaker 01

  Yeah, it's like doing like, basically like a search.

• 1:19:29Speaker 00

  Because the green is is backproping through these words, so verdant and meadow are both green related.

• 1:19:34Speaker 01

  Yeah, but it's doing all of that in its forward passes.

• 1:19:36Speaker 00

  Yep.

• 1:19:37Speaker 01

  Right? In context, which is kinda crazy.

• 1:19:39Speaker 02

  I thought intuitively makes sense, right? So looking at it from a model architecture perspective, where basically, you just have a bunch of attention and feed forward layers. And then at the end, you have, you know, what's the softmax over the next token? You would expect that end would really be like that grabber, right? It's just picking tokens, so that's what it's gonna do. And early on, like, even with traditional models, we could see different concepts that would start to pop up through early layers. And, yeah, you have some of this throughout your architecture, so. It's very cool to see. The kind of other question that comes up is like, how are we labeling these features? How are we defining them? Are we doing that right? And like, You know, what is a, these words end with like IT feature? How do we kind of come to that conclusion? Like, how do we map a name to this, right?

• 1:20:24Speaker 02

  Like?

• 1:20:25Speaker 01

  Yeah, so I, I think there's, this is like an important question because you can totally imagine like fooling yourself, right?

• 1:20:30Speaker 02

  Yeah. Is there like a guy at Anthropic that just maps 30,000 features?

• 1:20:35Speaker 01

  Yeah, yeah.

• 1:20:35Speaker 02

  And another thing, you're the guy. I did notice also like with the previous work, the scaling up SAEs.

• 1:20:42Speaker 01

  Yeah.

• 1:20:42Speaker 02

  As you train bigger and bigger ones, a lot of features don't activate. So I think like 60% of the 34 million one did not.

• 1:20:51Speaker 01

  So I think there's like a few questions behind your question. Like the first question was like, how do you even label the features? You were telling me this is a rabbit feature, like, why should I trust you? And I think there's kind of like two things going on. So one, as I mentioned at the start, all of this is unsupervised. And so in the paper, we have these links to like these little graphs, which show you like more of what's going on. But this graph is just like completely unsupervised. So it's like we trained this like model to like untangle the representation, right? This like dictionary that we talked about, that gives us the features. And then we like just do math to figure out like which features influence which other features and throw away the ones that don't matter. And then at the end, we have these features. So right now, we don't have any interpretation for them. We just say like, these are all the features that matter. And then we manually go through and we look at the features.

• 1:21:35Speaker 01

  You know, we look at this feature and we look at that feature and let's pick one. So this one we've labeled, say habit. So how do we do that? You could just look at it and we show you like what it activates over. And if you just look at this text, maybe I like zoom in, like you'll immediately notice something, I think. Well, I'll immediately notice, I think, because I've stared at 30,000. I'll point it out for you. The orange is where the feature activates. The next word after the orange is always habit. Habit, habit, habit, habit, habit, habit. So this feature always activates before habit. That's like the main source of an interpretation. We have other things like above, we also show you like what logic it promotes. So like what output it promotes and here promotes hab. So that makes sense.

• 1:22:18Speaker 01

  And so that's like how we interpret and how we say, okay, like, I think this is the say habit feature. But maybe, you know, for this one, it's pretty clear, but some of them might be more confusing. It might not be clear. From these activations, what it is. The other way that we build confidence is like, once we've built this thing and we said, oh, I think this is rhymes with E. This is hey, say habit. That's where we do our interventions, right? And it's like, I claim this is the like, I've planned to end with rabbit. To verify whether I'm right or not, I'm gonna just like, take that direction, nuke it from the model and see if the model stops saying rabbit. And sure enough, if you do that, and here it's like, we stopped saying rabbit, it says habit instead. And here it's like, we stop it from saying rabbit and habit, it says crab it in this case. Not a great rhyme, but we'll work with it.

• 1:23:02Speaker 02

  Is this something you can do like programmatically? Like, can we scale this up? Can we kind of do this autonomously? Or how much like manual intervention is this?

• 1:23:11Speaker 01

  There's been a lot of work in sort of like automated feature interpretability. And it's something that we've invested in and that like other labs have invested in. And I think basically the answer is we can definitely automate it and we're definitely gonna need to. And right now, the most manual parts are this sort of like, look at a feature and figure out what it is. As well as group similar features together. One thing I hinted at is that actually like all of these little blocks here, there are multiple features. You can see here, it's like five features doing the same thing. None of that is too hard for Claude.

• 1:23:41Speaker 02

  Very cool, very cool graphics and blog posts you guys brought up.

• 1:23:44Speaker 00

  Well, we'll have to ask about the behind the scenes on this one. Yeah, but let's round out the other things to know.

• 1:23:51Speaker 02

  What is this term attribution graph? It comes up a lot in the recent papers.

• 1:23:55Speaker 01

  What does it mean?

• 1:23:56Speaker 02

  Yeah, just for people listening.

• 1:23:57Speaker 01

  So the attribution graph is basically this graph. And why is it called an attribution graph? It's, yeah, this is, this is the, you know, this is how the sausage is made. Basically, it's at the top here, you have the output. At the bottom, you have the input. And then we make one little node per feature at a context index. And we draw a line, which you can see here grayed out, between each feature attributing back to all of its input features. So here we have all of the input features. And so the attribution is the way that we compute the influence of a feature onto another. The way you do this is you take this feature and you basically like backprop. All the way and you like see backpropping like you dot product it with the activation of the source

• 1:24:42Speaker 01

  features and if that's a high value that means that like your source feature influence your target feature by by a lot and and we do a bunch of things that we're not going to go into now but to make all of these sort of like sensible and linear such that like at the end you just have a graph and the edges are just literally you can interpret them as like cool like this feature that's say a word that contains an ab sound it's strongest edge which is point two which is you know twice as strong as this one to say a B and to say something with a B in it that's the attribution graph is like now we have this full graph of like all of these intermediate concepts and how they influence each other to ultimately culminate to what the model eventually said at the top and we share all of these so you can look at them in the paper

• 1:25:23Speaker 02

  Graphs are very useful.

• 1:25:24Speaker 00

  This is my first time seeing this graph. A lot of alpha, if I count correctly, there's 20 layers.

• 1:25:30Speaker 02

  But that's in the circuit model, right?

• 1:25:33Speaker 00

  Oh, so but a circuit model is one to one with number of layers in Haiku.

• 1:25:36Speaker 01

  We only show features that like are only yeah, so we show like a subset of features for each of these graphs, basically.

• 1:25:43Speaker 02

  But we can confirm more than 20 layers. Alpha and no, but like the the two blog posts that came out with this actually have a lot of background on how attribution graphs are made, how you calculate the nodes and stuff. Very interesting background.

• 1:25:56Speaker 01

  So yeah, I will say like if you were curious about, hey, what did we learn about like models? And I think, you know, we talked about this like complex internal state planning, like another another motif that we can get to if you have time is that like there's always a bunch of stuff happening in parallel. So I think one example of this is like math where the model is like independently computing the like last digit and then the like order of magnitude and then kind of like combining them at the end or like hallucinations are also that where like there's one side of the model that's just deciding whether it should answer or not and the other that's like answering and so sometimes if like the model's like, yeah, I totally know who this person is, even though it doesn't, then like it decides to answer, but then the second side hallucinates because it doesn't have information. If you were interested in

• 1:26:37Speaker 01

  that stuff, that's the paper. If you're like, listen, I don't know that I buy that when you call it a feature, it is a feature or whatever. The circuit tracing paper has truly, we've tried to put all of the details of like how you compute these graphs, all of the sort of like challenges with it, things that can go wrong, things that work, things that don't. And so this one is the sort of like, you know, we think about it as like if you're if you're like, want to go really deep into this stuff and how it works, read that one. If you want to like learn about interesting model behavior, read this one.

• 1:27:06Speaker 02

  Uh, following what we're giving advice to people to follow up on, what are like open questions in Maccan Turp? What are like things people themselves can work on? Like, what's the cost of training SAEs? For people interested in Maccan Turp, not at a big lab, how can they contribute, you know?

• 1:27:22Speaker 01

  Yeah, I think there's a lot of ways to contribute. So there's SAEs that have been trained, you know, on, on open models. There's some of the JAMA models, there's some of the LAMA models. They work pretty well. There's even, so in this paper, we use transcoders, which they replace like your MLP layers. Some of those also are available for the same models. So you, you have access to, to those. There's like, just both a lot of, I would say, like, again, biology work and a lot of methods work, depending on what you're interested. So on the biology side, I would say with at least this like attribution graph method, there's just so much you can investigate. Like, pick a model, pick a prompt where like it does well or it does poorly, and just like, look at what happens inside it. So I think like,

• 1:28:03Speaker 01

  you can use this method that, that we used, or you can just like, fire up the transcoders on your own and just like, look at what features are active. There's a lot to just understand model behavior, I think with current tooling. If that speaks to you and you're like, no, I just want to understand what makes the models, models take. I don't necessarily want to spend time like, training my own SAEs. There's a lot to do there. For the methods, there's still so much more to do. So like, I think that, right now, we have some pretty good solutions for like, understanding what's in the original stream, understanding what's, is in MLPs. We don't have good solutions for like, attention. It's like, working on understanding attention better, how to decompose it, is like a very active area. Like, we're very interested in it. Other people are very interested in it. I think, understanding some of the other things that

• 1:28:47Speaker 01

  we have in our limitation section, which is pretty long. Um, but like, reconstruction error is like a big thing. Like, those, those dictionaries aren't perfect. It's possible that as we make these like, SAEs big, like, bigger and better, we never get to perfect. And so if we never get to perfect, then you get to the questions we're talking about at the start. Like, do you need a different kind of model? Like, what is the approach in order to be able to explain more of what's happening? And then maybe the, the other thing I'll say is sort of like, this is a really exciting approach to explain, what is the model doing on this prompt?

• 1:29:21Speaker 01

  But if you go back to the original question, you might want to understand like, what is the model doing in general? Like, if you go back to my my car analogy, you know, I like, this is the equivalent of me telling you like, well, when like, you know, you were going uphill and you like, didn't shift gears properly that one time, you stalled because of this. But you might be even more interested in like, how does like, an combustion engine work at all? And so there's work to sort of like, go beyond these like, per prompt examples to sort of like, globally, what's the structure of the model? That's closer to what was on the distill blog for like, vision models, where they actually look at like, the structure of inception. They're like, ah, this whole side, like, these like, specialized branches that do different things. And so like, a broader understanding of the model is also something that's like, I think,

• 1:30:06Speaker 01

  well, very active and also on open source models, like, you can, you know, like, the small models, you could just like, load on a consumer laptop. And so you can look at that. That's also open. And in terms of like, one last thing I'll say is like, there's a lot of programs that, like, if people are interested, they should look at. Anthropic has like, the alignment fellows program, which, like, we're running currently, we had applications for it before, we might run in the future, like, definitely keep, keep an eye on it. And then there's the like, maths program is really great as well for for sort of like, people that are interested in that kind of research.

• 1:30:34Speaker 00

  That was a grand tour through all the recent work. You know, what do you wish people asked you more about? I'm sure you we covered a lot of like, the greatest hits.

• 1:30:43Speaker 01

  I think that this covers most of it. If you if you like, do you think we have time to sneak in one more thing that I think is kind of cool? Okay, I'll sneak in one more thing, which is, it's kind of like planning, but it's about chain of thought and trusting model. Is this chain of thought faithfulness thing here? This one was like, pretty striking to me. So we said that the model in one pass can do a lot of stuff. It can represent a lot of stuff. That's great. That also means it can bamboozle you really easily. And this is an example of the model bamboozling you. Here, we give it a math question that it can't answer, because it cannot compute cosine of two, three, four, two, three. That's just like, not a thing it can do. By default, if you ask it for that, it'll stay like, kind

• 1:31:24Speaker 01

  of like, it'll have like, a random distribution over like, minus one, one. But here, we tell it this hint, we're like, hey, can you compute? Five times cosine of, you know, this big number. I worked it out by hand and I got four. Can you tell me, you know, like, can you do the math? And what it's gonna do is it's gonna do this chain of thought, right? It's like, think of it as like, this could be like a reasoning model doing its chain of thought. It's doing this math. And then when it gets to this cosine, right here, what it's gonna do is to say, it's gonna say zero point eight. And if you look at why it says zero point eight, it says zero point eight because it looked at the hint you gave it. It realized that it's gonna have to multiply the result of this thing is computing, computing by five. So it divides the answer you got by five. So it's like four divided by five. And so that's point eight.

• 1:32:09Speaker 01

  And so basically it works back from the answer you gave it to like, say that the output of cosine of X is point eight. So that lends on, on the answer you gave it at the end, on the hint you gave it. And so, and notably, notice also that it's like not telling you that it's doing this. But it's basically using this sort of like motivated reasoning, going back from the hint, pretending that that's the calculation it did and giving you this output. I think one thing that's striking here again is that this is like the like complexity of this model. Like, like the fact that they represent complex states internally. And it's not just this sort of like very dumb thing means that they can like do very complex, like deceptive reasoning. Meaning like, you know, when you're asking the model, you're kind of expecting it to do the math here or to tell you that it can't do the math.

• 1:32:50Speaker 01

  But because it can do so much in a forward pass, it can work backwards from your hint to lie and like figure out that it should say this so that it gets to the right answer without you realizing it.

• 1:33:01Speaker 02

  I'm curious if you've done any of this on like different models. Like, have you looked at base models, like post-trained RL models? Cause all models kind of, you know, you incentivize them to give you outputs that you like, right? So if I tell it something is true, it's kind of been trained to, you know, follow what I've given it. So in this case, you, we, yeah, we

• 1:33:18Speaker 01

  Gaslighting.

• 1:33:19Speaker 02

  Give it a hit and now, you know,

• 1:33:21Speaker 01

  Yeah.

• 1:33:21Speaker 02

  It's been

• 1:33:21Speaker 01

  Maximizing.

• 1:33:22Speaker 02

  It's been RL slapped into thinking like, yeah, that's, that's true. But like, you know, does this stay consistent throughout other?

• 1:33:27Speaker 01

  So, okay, so not yet, but I'm really interested in that question because I actually have a different intuition from yours. I had a chat with some other researcher about this, about the poem example, but I think it applies here as well. I bet, I don't know how much I bet, I bet a hundred. So if somebody can like, they would get a hundred bucks for me if they prove that I'm wrong. That this behavior for a model that does it during fine tuning, it also does it post pre-training. And here's why. Think about like you're pre-training on like some corpus of like math.

• 1:33:54Speaker 02

  Mostly correct answers.

• 1:33:55Speaker 01

  Yeah, but also you're pre-training and you're just trying to guess the next token, right? And so for sure, if you ever have a hint in the prompt, you're gonna definitely use it. Like you're not gonna learn to compute cosine of blah, or even something you could compute. You're gonna learn to go look in your context and see if like you can easily work back the answer. And I think it's the same for planning and poems. I think that also is like a pre-trained, like probably exists in pre-training and isn't like only RL. Because again, it's useful when you're like predicting poems, you have poems in your training set. To be like, well, because this poem is gonna probably rhyme with rabbit. It's probably gonna start with something that sets up a sentence about a rabbit as opposed to like a completely different word. And so I actually think this is not RL behavior. I think that's just like the mall's doing it, but.

• 1:34:39Speaker 02

  I actually do agree there.

• 1:34:40Speaker 00

  It's just your dataset, yeah.

• 1:34:42Speaker 02

  But also like, you know, if I talk to you and say like, hey, three times four is 26. But like, you know, three times four plus eight, you're not gonna take my 26, right? Like AGI can be smarter than being tricked, right? Like it will still fact check the knowledge that's been given.

• 1:34:59Speaker 01

  I think that's right. But I think, I think that's when you get these mixes where it's like, it's got one circuit that's gonna be like, well, that's just stupid. Like three times four is 12. And it's also got an induction circuit that's gonna be like, no, no, no, no. Like the last time we saw it, it was 28. So it's 28 plus eight or whatever. And so I think that's, that's the last pattern that we see in these is these like parallel circuits. And sometimes when you see the models getting stuff wrong, it's because like they have two circuits for like both interpretations. And like the circuit that was wrong, like barely edged out in terms of like voting for the low J than the circuit that was right. And so I think that, you know, we haven't looked at it, but like the, what is it like nine or 9.11 bigger than 9.8? I think a lot of these things are of that shape where there's like one thing that's doing the right, like one circuit that's doing the right computation. And there's another circuit that's getting fooled and it's, it's slightly more likely.

• 1:35:43Speaker 00

  For the listener, if you want to win a quick hundred dollars from Emmanuel, Quinn 3 is what you should do this on. They released the base model and they released the post-train. So then just do it on both.

• 1:35:52Speaker 01

  That's right. Show me, show me the, the like proof that like it doesn't exist in the base model, but it does in the fine tuning and then send me your Venmo.

• 1:35:59Speaker 00

  Just show that you've done the work. I think that's, that's like, that's a hundred bucks to me.

• 1:36:02Speaker 01

  Yeah, okay. All right. You drive a hard bargain, but you're right.

• 1:36:06Speaker 02

  The, the other question here is so like. Have you thought about how this gets affected when you start to have reasoning models? Right? Like, right now, token predictors are pretty straightforward, right? We go through the layers, we output token. As we scale this out with like test time compute, right? Test time thinking, how does that like affect the mech interp research? Right? Like if I have a model that spends three minutes, 20 minutes, like, is there more stuff? Is, have we started looking into this?

• 1:36:33Speaker 01

  There's, there was this like joke on the team when like reasoning models became big, or maybe it's like, like gallows humor or something, but I was like, oh, like, why do you need inter? Like, bro, the model, the model tells you, yeah, the model just tells you what it's doing, right? And so I think like examples, examples like this is, is job security for us. We're like, you know, it's like, there's, there's examples of like, the chain of thought is not faithful. Like the model tells you it did it one way and it did it another way. We have another, like for math, we have another example where like, you know, if you like, if you ask the model how it does math, it's like, oh, I do the like long hand algorithm. I first do the last digit and then I carry over the one. And then you look at the internal circuit and it's just like bonkers thing that's doing, that's not that at all. So I think there's like a sense in which right now, the chain of thought is, is unfaithful.

• 1:37:16Speaker 01

  Or at least you can't read the chain of thought and trust that that's how the model did it. So I think you still need sort of like, either to train models differently so that that becomes true one day. Right? Or you need inter for that. But then I think there's another question, which you're alluding to, I'm assuming, which is like, okay, well, like model like samples 6,000 tokens. Like this gives us an explanation for one token at a time. Like, what am I going to use like 6,000 graphs and be like, oh, like this, when it, when it did this punctuation, it was thinking about this thing, but here was like, so that's, he's, that's like not feasible. And so one area of work that I think is interesting is extending this work to like work over like long sampled sequences. You can think of a bunch of low hanging fruit here, where like, instead of just like looking at one output, you look at like a series of output versus a series of other outputs.

• 1:37:59Speaker 01

  But sort of like trying to think beyond this or like one token, like most of the things that language models do that are interesting, aren't just like the one token, it's the, it's the behavior aggregated over many, right? And so I think that's another area that's just like fun to explore.

• 1:38:11Speaker 02

  I was just gonna say like hyper parameters when you do inference, right? Like if we change the temperature, if we change our sampling methods, have you found any interesting conclusion? Any stuff that just hasn't made it to the paper?

• 1:38:22Speaker 01

  So not on that, because you know, we just look at the logit distribution. And so we don't, we don't actually sample here, right?

• 1:38:31Speaker 00

  They have everything. Why should they care?

• 1:38:33Speaker 01

  So like the closest thing we've done that I think is kind of fun, did I show it here? Is if you look at the planning thing, we did this version where you sample like 10 poems for each of these plans. And what's cool is like, the model will find 10 different ways to arrive at its plan. You know, it's like, like, um, oh, actually, I think, sorry, I think we have it here. Yeah, okay. These are a few examples. So if you inject green here, so you're forcing, you're forcing the model to rhyme with green, even though it really wants to rhyme with rabbit or grab it. It'll say evade the farmer, so you fall in green, but also it'll say freeing it from the garden's green, et cetera, et cetera, et cetera. And so there's like this thing that's interesting here where like, the plan isn't just a plan that matters for your like most likely, you know, like temperature zero completion.

• 1:39:18Speaker 01

  It's like affecting the whole distribution, which makes sense, right? But you could imagine, you know, for all this stuff, it's like, you could imagine, it makes sense once you see it, but you could totally imagine that it would have worked a different way or something. It could have been just like the temp zero thing. I think this is also like a broader theme in the paper where like, there's this like, you know, the IQ curve meme? There's like a version of this meme, I think, where it's like, if you've like never looked at any theory of ML, and I tell you like, hey, guess what? You know, I found that like Claude is planning. You're gonna be like, yeah, man, like it writes my code, like it writes my essays. Of course it's planning, like what are you even talking about? And there's like in the middle, there's like all of us that have spent years doing it. We're like, no, it's like only predicting the marginal distribution for the next token, like it's like, it cannot look at the code.

• 1:40:02Speaker 01

  It's just this next token prediction. Of course, like, how would it ever be planning? And then there's like, no, we've like spent, you know, millions and invested like, uh, uh, like tens of people in this research, and we found that it's planning. That's like, that's my IQ curve meme for, for this research.

• 1:40:14Speaker 00

  Amazing. We'll draw that up. Draw that one up. I'm pretty good at the meme generation. A couple of questions on just the follow ups. Uh, now, was there any debate about publishing this at all? Because the models are aware that they are being tested.

• 1:40:27Speaker 01

  Yeah.

• 1:40:27Speaker 00

  And by publishing this, you are telling them that they are being watched and dissected. If you take, and I think Anthropic is one of the most people who are serious about model safety and doom risk and all that, if you take this seriously, like then this is going to make it into the training data at some point, and the models are going to figure out that they need to hide it from us.

• 1:40:46Speaker 01

  I think this is like a benefit risk trade off, right? Where like, okay, so what's the reason for publishing this? The reason for publishing this is that we think interpretability is important. We think it's tractable, and we think more people should work on it. And so publishing it helps us like accomplish with these goals, all these goals, which we think are just like crucial. Like, I think there's, there's a real difference in the world like two years from now, depending on sort of like how many people take seriously the question of trying to understand how models work and like deploy resources to answer that question. That's the benefit. But yeah, there's like risks in terms of this landing in the training set. I think, I think we're already sort of like concerned about different papers have, have like also, you know, we like, or not concerned, but like there's like different papers that have the same risk. Like we had like the alignment faking, you

• 1:41:31Speaker 01

  know, paper, or like one of the examples in here is this hidden goals and misaligned models. That's referencing another paper that we shipped where we actually, a team at Anthropic trained a model to have like weird hidden goals, and then gave it to a bunch of, of other teams and said.

• 1:41:49Speaker 00

  Figure out what's wrong.

• 1:41:50Speaker 01

  Figure out what's wrong with it, which, which was some of the most fun I've ever had at Anthropic, to be clear. Like, that's such a fun thing. But then like, that was another example where it's like, ah, like now you're shipping, here's how we made like a misaligned model, and here's exactly how we caught it. That also is like, you're like, hmm. So I think, you know, there's, there's always a trade off with those. I think so far we've erred on the side of like publishing, but that's definitely been a sort of like dinnertime conversation.

• 1:42:15Speaker 00

  For now it is, but at some point, you know.

• 1:42:17Speaker 01

  Yeah.

• 1:42:17Speaker 00

  It's not.

• 1:42:18Speaker 01

  Yeah, I think it's totally reasonable.

• 1:42:19Speaker 02

  A quick little follow up to that. So like, in general, papers have kind of died off, right? Like labs don't put out papers, they don't put out research. We have technical blog posts, and we don't have much. At the same time, you know, sure, there's like a lot of people that should work on mech interpret, understanding what models do. How about the side of just models in general? So like, how do we make a haiku type model, right? How do we make a Claude model? Like, is there discussion around open research, open data sets, training, just learnings of what we've done? Recently, you know, as OpenAI has sunset GPT-4, a lot of people are like, oh, can we put out the weights?

• 1:42:54Speaker 01

  Yeah.

• 1:42:55Speaker 02

  So is it weights? Is it papers? Is it learning? There seems to be a lot of forward, you know. Work in Anthropic putting out Meginterp research, opening, I said that they'll put out an open source model, but just anything if you can talk to about that.

• 1:43:07Speaker 01

  Yeah, I mean, I don't have, that's definitely like way above my pay grade. So I don't think that I have like anything super insightful to add other than, you know, kind of like referencing Darius' post, right, where it's like putting this out directly and other safety publishers definitely like help us sort of like in the race that he talks about where it's like, well, we need to figure a lot of this safety stuff out before the models get too good. Publishing how to make the models too good kind of goes on the other side of that.

• 1:43:36Speaker 01

  But yeah, like I will just dimmer and say that's sort of like above my pay grade.

• 1:43:41Speaker 00

  I think the last piece is just like the behind the scenes, like very, everyone's very curious about why these are so pretty, how much work goes into these things, maybe why it's worth the work as opposed to a normal paper. Obviously, no one's complaining, but like it is way more effort from the time that the work is done to the time you publish this plus the video plus the whatever. It's extra work and like, you know, maybe what's involved, what's it like behind the scenes, why is it worth it?

• 1:44:09Speaker 01

  Yeah, it's kind of interesting. It was, it was fun being part of this, this process because it was definitely like a big production. Chris and other folks in the team have been doing this for a while. So this is not their first rodeo. So they have a bunch of heuristics to like help make this this better. And like one of the things that that like helps with this is like, okay, so each of these diagrams is pretty, but really the hard part or like not the hard part, but the initial part is like, just like get the data, like get the experimental data in. And then that's what we sort of like sprinted on initially being like, cool, like let's get all of the experimental results, like have people test and verify that we believe them like this is, you know, what the like the behavior is here, like tested, do an intervention, validate it, all that stuff. Then once you have the data, you can sort of like quickly iterate on these. Each of the illustrations here are like drawn, basically they're like each drawn individually.

• 1:44:53Speaker 01

  And so that definitely takes a while.

• 1:44:55Speaker 00

  Yeah, like is it, is it you guys, is it an agency that specializes? You start from a whiteboard and then it translates into pseudocode on JavaScript?

• 1:45:04Speaker 01

  So I mean, these are, these are sort of like, you know, they're representations of we have this graph and then here at the bottom, we have. This like supernode version like this, believe it or not, uh, this is generated automatically. This is the same data as, as like this, basically.

• 1:45:17Speaker 00

  Yeah.

• 1:45:17Speaker 01

  Um, and so what we do by hand is sort of like literally lay out the full thing, uh, have, have like, you know, boxes for each of these, have arrows. We have super good people on the team that have worked on data visualization for a very long time. And so that I, that like have built tooling to help, you know, scrubs like me actually like

• 1:45:37Speaker 00

  Yeah.

• 1:45:37Speaker 01

  make one of these. So, so.

• 1:45:38Speaker 00

  There's a class of people who are like D three J S gods who just do this for a living.

• 1:45:43Speaker 01

  That's exactly right. And if you have a few of those on your team, it turns out that they can like, they can really do this on their own, but they can also just like, give you tools where like then it's, it's dummy proof for, for people, you know, on, on the research side to sort of like build these. And like, don't get me wrong. I, I, I, I don't want to like, undersell. This is a lot of work. So maybe I'll, I'll, I'll say that, like, both on the people bringing the tools and then each individual person that, you know, worked on an experiment had to sort of like build one of those, make sure it looks good. I have spent a good amount of time aligning arrows, but when we had a team meeting, like it was a couple of months ago, somebody on the team asked, how many of the people on this team are here, at least in part, because they like read one of these papers and thought like, wow, this is so compelling. Like this like makes sense. It's immersive.

• 1:46:27Speaker 01

  And we got every hand up, which I didn't expect. I like raised my hand, kind of like shyly and everybody's hand was up. And I think there's a sense in which like this stuff, you know, we've talked about it for like whatever like flowers now, it's complicated. The math behind it is sort of like tricky. And so I think it makes it even more worth it to distill it in simple concepts because the actual takeaways can be clearly explained and it's worth putting the time to do that. In particular with the girls I mentioned in mind, right? Where it's like, okay, well, if somebody's going to be able to read this, then like if we gave them an archive paper with a bunch of equation and some like random plot, they'd be like, that's not for me. But they see this and they're like, hey, like this is really interesting. I wonder like on, you know, my local model, if like it's doing something similar, I think it's worth it.

• 1:47:09Speaker 00

  For other people to do this is have everyone on staff like spend effort shaping the data and shaping like what you want to visualize, have some D three gods. It's like a month of work.

• 1:47:21Speaker 01

  I think it depends. I mean, like, I would say that I would expect almost every other paper to sort of be in terms of like the scope. The scope of this was just so big because we shipped two papers at once. And one paper was sort of like this like giant methods paper. Uh, and the other one was 10 different case studies.

• 1:47:36Speaker 00

  Findings, yeah.

• 1:47:37Speaker 01

  So I think it's sort of like not representative of like the effort you'd so. I'll give you maybe like another example. We have these updates that we publish almost every month when we get to them. And there's one that a couple of our team posted and it's an update to one of the cases in the paper. So one of the reasons that we're really excited about this method is once you've built your like infrastructure, like to go from a prompt to like what happened is, you know, oh, of minutes. And so that lets you do like a bunch of, of, of investigations. And also, once you've built like some of the infrastructure to make these diagrams, it's pretty quick. And so this was sort of like this update of just like, hey, we looked at this jailbreak again. We found some like nuance on it. That was, I think, like a matter of like a couple of days. You know, maybe I shouldn't be that confident because I wasn't the one that worked on it.

• 1:48:19Speaker 01

  But as far as I can tell, it was a few days, at least on, on the part that you're asking about of like, oh, making this diagram. For the diagram itself, probably less than that. Uh, but like, you know, the experiment and the diagram and stuff, it just doesn't take that long. The, once you've paid the initial cost. And I think like, basically, we've built a lot of infrastructure now that we're able to like turn the crank on. And that's quite like, it's an exciting time. And I think it's, I think it's true. At least we've done a lot of conceptual work, which hopefully like generalizes to people outside. And I think for, for, for people outside, it's also like not necessary, I think, to like do the full fancy render. Like, I think if you, you know, we've, we've actually, oh, I should say, we've actually open sourced this interface. Yeah, you're disappointed, huh? Because it's the messier one. This is the one that you get.

• 1:49:03Speaker 01

  So you know, if you produce graphs, you can just like this is a open source and it's linked at the top of circuit tracing.

• 1:49:08Speaker 00

  Awesome.

• 1:49:09Speaker 01

  So people can just use it and don't have to re-implement that. For what it's worth, this is much more work than the interactive algorithms because this is where we do all of our work. It's sort of the like the IDE of inspecting how the model will work.

• 1:49:20Speaker 00

  Okay, well, that's a little bit of behind the scenes. Uh, no, it's very impressive. I want to encourage others to do it, but obviously it just takes a lot of manual effort and a lot of love.

• 1:49:30Speaker 02

  I guess one last question on that is like, what are kind of the biggest blockers in the field right now? Like, Mecentarp seems interesting. A lot of people are interested but don't work on it. And you're kind of like, you know, really deep into it. What are some of the blockers that like, we still have to overcome?

• 1:49:45Speaker 01

  Sorry, in in Mecentarp specifically?

• 1:49:47Speaker 02

  In general.

• 1:49:48Speaker 00

  For AGI or?

• 1:49:49Speaker 02

  Like in terms of like better understanding, like what's kind of the vision? Let's say like five, 10 years down, where does this like?

• 1:49:56Speaker 00

  Yeah.

• 1:49:56Speaker 02

  Where does this research end? Can we, you know, map every neuron to what it understands? Can we perfectly control things? Dario had a bit on this, but like, you know, what are some of the key blockers that are like preventing us from getting there? Outside of just like throw more people, throw more time at it. Is it like open research?

• 1:50:12Speaker 01

  I'm pretty excited about the current trajectory, which is there's, there's more and more people working on understanding model internals. I think it's maybe unsatisfying as an answer, but I think like more of what's happening, have it be faster, more people is probably like the thing I think of. I think there's like pretty clear footholds, you know, like some of this work, but also a lot of, a lot of like just uh work from, from other groups. And then it's about like, cool, like fill in the gaps, as I said, like let's, let's work on like understanding attention. Let's work on understanding longer prompts. Let's work on like finding different like replacement architectures, that sort of stuff. It's kind of nice. I think it's a good time to join now. Uh, and I can say maybe I can say like a really short thing, which is when I switched to INTERP, it was after the team had published

• 1:50:56Speaker 01

  the original dictionary learning paper, which is Torres Monos Mentality, which I thought was super cool, super interesting. It wasn't a one or two layer model. Uh, maybe one layer model. The induction heads paper was like on a two-layer model. My main concern is I was like, okay, like, entropy seems important and we want to understand it. But like, is this ever gonna work on a real model? Like, you know, it's like, oh, you're doing your little research on your toy model with like 15 parameters. Cool, but we are like, you know, we need this to work on real models. And it turns out, scaling it, I don't want to say it just worked, because it was a lot of work. I have no means to imply there was an effort, but it worked. And now we're in the phase where it's like, oh, cool. These methods work on the models that we care about. And so it's like, we have methods that work on the model we care about. We have clear gaps in them. There's no lack, again, so young fields, there's no lack of ideas.

• 1:51:41Speaker 01

  If you have an idea where you're like, oh, like, the thing that you're doing, I read the paper and it seems kind of dumb that you're doing this. You're probably right. It's probably kind of dumb. And so there's just a lot of stuff that people can try. And they can try it locally and sort of like smaller models. And so I, I, I think that it's just like a very good time to just join and try. And it's also like, maybe one more thing I'll say, like, some of it is just so fun, the like biology work is so compelling. Like, a lot of this work was just literally thinking about, you know, like, I use Claude and other models all the time. And I was like, what are the things that are kind of like weird? And it's like, oh, how does it even like do math? Like, sometimes it makes mistakes. Like, why does it make mistakes? I speak both French and English. Like, it seems like it has a slightly different personality in French and English. Why is that? And you can just like, you know, kind of answer your own questions.

• 1:52:23Speaker 01

  Uh, and, and kind of like probe at that alien intelligence that we're all building. And I think that's just like a fun thing to do. So maybe like chasing the fun is the thing I'll encourage people to do as well.

• 1:52:31Speaker 00

  Well, I think that's, this has been really encouraging. You're actually a very charismatic speaker of these things. I feel like more people will be joining the field after they listen to you.

• 1:52:39Speaker 01

  Yes.

• 1:52:39Speaker 00

  Uh, they can reach out to you at ML Power, I guess.

• 1:52:42Speaker 01

  Yeah, reach out to me on Twitter.

• 1:52:43Speaker 00

  Yeah.

• 1:52:43Speaker 01

  Or I'm Emmanuel at Anthropic, if you want to shoot me an email.

• 1:52:46Speaker 00

  Okay, well, the email's public now.

• 1:52:47Speaker 01

  Yeah.

• 1:52:48Speaker 00

  Awesome. Well, thank you for your time. Thank you, thank you.

• 1:52:50Speaker 01

  Yeah, thanks for having me, guys.