E48: Mechanizing Mechanistic Interpretability with Arthur Conmy

Arthur Conmy sits down with Nathan Labenz for an accessible deep dive into the state of interpretability research online today. They discuss how pioneering researchers have painstakingly worked to isolate the sub-circuits within transformers that are responsible for different aspects of AI capabilities. Arthur also introduces us to a new ACDC approach that he and his co-authors have taken to automating some of the most time-consuming parts of this work. If you’re looking for an ERP platform, check out our sponsor, NetSuite: http://netsuite.com/cognitive We're hiring across the board at Turpentine and for Erik's personal team on other projects he's incubating. He's hiring a Chief of Staff, EA, Head of Special Projects, Investment Associate, and more. For a list of JDs, check out: eriktorenberg.com. RECOMMENDED PODCAST: The HR industry is at a crossroads. What will it take to construct the next generation of incredible businesses – and where can people leaders have the most business impact? Hosts Nolan Church and Kelli Dragovich have been through it all, the highs and the lows – IPOs, layoffs, executive turnover, board meetings, culture changes, and more. With a lineup of industry vets and experts, Nolan and Kelli break down the nitty-gritty details, trade offs, and dynamics of constructing high performing companies. Through unfiltered conversations that can only happen between seasoned practitioners, Kelli and Nolan dive deep into the kind of leadership-level strategy that often happens behind closed doors. Check out the first episode with the architect of Netflix’s culture deck Patty McCord. https://link.chtbl.com/hrheretics The Cognitive Revolution is a part of the Turpentine podcast network. Learn more: Turpentine.co TIMESTAMPS: (00:00) Episode Preview (04:40) What attracted Arthur to mechanistic interpretability? (07:49) LLM information processing: General Understanding vs Stochastic Parrot Paradigm (14:00) ACDC paper: https://arxiv.org/abs/2304.14997 (14:45) Sponsors: NetSuite | Omneky (24:30) Putting together data sets (32:39) How to intervene in LLMs network activity (36:00) Defining metrics to evaluate the production of correct completions (44:20) The future of the mechanistic interpretability research (50:00) Extracting upstream activations in the ACDC project and evaluating impact on downstream components. (56:00) Anthropic research findings (01:08:00) 3-Step process of the ACDC approach (01:22:00) Setting a threshold and validation (01:27:00) Goal of the approach (01:32:00) Compute requirements Correction: at (01:33:00), Arthur meant to say = "quadratic in nodes" (01:35:30) Scaling laws for mechanistic interpretability (01:40:00) Accessibility of this research for casual enthusiasts (01:46:00) Emergence discourse (01:56:00) Path to AI safety LINKS: https://arthurconmy.github.io/ https://arxiv.org/abs/2304.14997 X: @labenz (Nathan) @arthurconmy (Arthur) @cogrev_podcast SPONSORS: NetSuite | Omneky -NetSuite provides financial software for all your business needs. More than thirty-six thousand companies have already upgraded to NetSuite, gaining visibility and control over their financials, inventory, HR, eCommerce, and more. If you’re looking for an ERP platform: NetSuite (http://netsuite.com/cognitive) and defer payments of a FULL NetSuite implementation for six months. -Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that *actually work* customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off. This show is produced by Turpentine: a network of podcasts, newsletters, and more, covering technology, business, and culture — all from the perspective of industry insiders and experts. We’re launching new shows every week, and we’re looking for industry-leading sponsors — if you think that might be you and your company, email us at [email protected].

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  A really ambitious goal of interpretability, where like the whole architecture of the forward pass can be understood to a human. Or at least, yeah, like these high level concepts, like the whole routing to a particular expert has some meaning to humans. And I think it's possible that we can get to this stage with mechanistic interpretability. But I think it's worth noting that even if this fails pretty badly, it's still possible for the interpretability of narrow tasks, like an understanding of the dangerous capabilities. So we can at least remove those dangerous capabilities, even if we don't have an understanding of all capabilities of the model.

• 0:42Nathan Labenz?

  Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas and together, we'll build a picture of how AI technology will transform work, life and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Torenberg. Hello and welcome back to The Cognitive Revolution. Today, I'm excited to take a deep dive into mechanistic interpretability with Arthur Conme.

• 1:15Nathan Labenz?

  Looking back on the show over the last few months, I realized that I'd mentioned the topic of mechanistic interpretability many times, repeatedly highlighted it as one of the most promising paths to long-term safety and shared a few of the canonical results that most inform my AI worldview. But we've never really got into much detail about how mechanistic interpretability work is actually performed. Today, we're getting into those details. Now, this is an advanced topic. So while we definitely take our time to explain the key concepts in the simplest possible terms, we do assume throughout that you understand the differences between related concepts like parameters and activations or attention heads and MLP blocks. If those distinctions aren't already clear, I might suggest watching part one

• 2:00Nathan Labenz?

  of my AI scouting report first, as the fundamentals that I present there really are meant as a foundation for a conversation like this one.

• 2:09Nathan Labenz?

  Beyond that foundation, Arthur really is the perfect person to guide us, as he's just published a new paper and software library that aim to accelerate mechanistic interpretability work by automating the most cumbersome and tedious parts of the typical research workflow. Beginning with the core questions that mechanistic interpretability seeks to answer and describing the conceptual basis and experimental setups that are most commonly used, Arthur does a great job, both in the paper and in this conversation, of providing clear, even intuitive explanations for how researchers are starting to pry open the black box that our large language models. It's really fascinating to learn how subcircuits are discovered within transformers and to see the effective,

• 2:51Nathan Labenz?

  but often quite alien, problem-solving strategies that models learn. I'm also really excited by Arthur's vision for how mechanistic interpretability work could one day allow us to inspect powerful AI systems for the emergence of concerning capabilities, even during the training process. While we're a super-long way from being able to do that reliably today, such mastery of how AI systems work would be an outstanding development for safety, reliability, and performance.

• 3:21Nathan Labenz?

  This is an unusually accessible introduction to mechanistic interpretability from a practitioner who is among the best in the field. I learned a ton, and I think you will too. So I hope you enjoy my conversation with Arthur Conme. Arthur Conme, welcome to the Cognitive Revolution. Thanks, Nathan. Good to see you. Yeah, I'm super excited about this. So, boy, you know, folks who've listened to this show for a little bit have heard me certainly mention the concept of mechanistic interpretability a few times. And I have, you know, mentioned that I'm excited about it as a research direction. And it seems like one of the kind of most promising paths to long-term safety with AI systems is to understand them in a deeper way,

• 4:03Nathan Labenz?

  in a more penetrating way than we do right now. But we've not really got into the details too much other than looking at a few kind of headline results of Grokking and, you know, a few kind of seminal things that give people a sense that progress is happening here. This, for many, I think, will still be their first kind of real deep dive into mechanistic interpretability. So I'm excited to get into all that with you. Maybe for starters... Could you just tell us how you think about mechanistic interpretability? Like, what is it? You know, what are the goals? What attracts you to it perhaps as well?

• 4:36Speaker 00

  So mechanistic interpretability is the reverse engineering of the learned algorithms that neural networks implement into human understandable concepts. So the idea here is that neural networks, like machine learning models, are an algorithm which turns inputs into outputs. But it's very opaque how exactly that like model is turning inputs into outputs. And in mechanistic interpretability, we aim to explain how that happens in terms of the internal components of that model in a human understandable way. So not nearly that like this matrix is multiplied by this matrix and produces the outputs.

• 5:21Speaker 00

  But like, no, here are the like high level variables the model is using internally to produce outputs from inputs. And so I see like my sort of research goal as is to like improve the like world understanding of how neural networks operates by, for example, explaining more of the like behaviors of like neural network models in terms of their like internal components. And I think my background, which got me into like mechanistic interpretability, was like probably studying like mathematics in my undergraduate degree, where almost the whole subject is about like actually understanding how things happen. And like I was drawn to like this area in machine learning because machine learning is in general not a field where the understanding of how

• 6:06Speaker 00

  like algorithms are operating is the like way that further techniques are made. It's just like a blind optimization procedure. But it doesn't have to be that way. And like my, my like motivation is to like continue to like make like machine learning and the like the algorithms learned by neural networks more understandable in terms of like human concepts.

• 6:27Nathan Labenz?

  So there's a few kind of levels to this, I suppose, in terms of how much we could achieve, right? And I think we'll, different results kind of get to different stages of this kind of depth of understanding, let's say. The way I kind of think about it, and tell me if you would present this a little differently, is first, you might just ask, how is it that the models are doing what they do? Like, can we even just describe, you know, how information is being processed in any way that's more enlightening than just, you know, everything kind of connects to everything and we don't really know? If I then went up a level, I could say, okay, well, I could just, you know, describe then, and we're going to get into into the work that you've done to help kind of zero in on the part of the network that seems to be most important for a given task.

• 7:12Nathan Labenz?

  But then I could go beyond that, and I could say, well, okay, I can see that, like, these are the parts that are lighting up, but why does that work? You know, like, what is it actually doing? Is there a way for me to understand that in any sort of intuitive way? And then I guess maybe a third level would be like, to what degree can we understand or determine if the things that it has learned, the strategies that it has learned are in fact general and constitute some, like, meaningful understanding? Or are they sort of, you know, still in the stochastic parrot paradigm of, yeah, you might get decent results, you know, on things that look like the training data, but there's not really a deeper understanding here?

• 7:54Nathan Labenz?

  And I guess, you know, further too, you might say, well, is there anything we can do to encourage, you know, actual understanding? Or maybe discourage, I guess, depending on exactly, you know, what you're looking for from your systems? How do you think about those layers? You know, would you adjust that, um, that mental organization of this?

• 8:10Speaker 00

  Yeah, definitely. I agree with like the first two layers, Nathan, you proposed there. And like, think about those two a lot, that to me, mechanistic interpretability, like, has like, just like two steps essentially, that firstly, you find what the like important subcomponents of this huge neural network are, that sort of like matter. And then having established this like subset that is important, you can then ask, well, what is like the meaning of that subset? Which I think like, yeah, maps onto like your sort of two levels very well. And I think describes like a bunch of research that like has already been done. I'm sure we'll get into that further. And yeah, I think that then like the third level is like one of many like sort of

• 8:54Speaker 00

  uh potential use cases, essentially of mechanistic interpretability, where this could enable us to like uh answer the questions of like, are these models like doing reasoning or like uh modeling humans that they're like interacting with? Or is it just like a wad of heuristics and like statistics? That is not doing anything intelligent beneath just like a ton of like rote based rules? And like, we don't know the answer to that question at this point in time. Uh yeah, the sort of stochastic parrot's hypothesis is still a hypothesis that models are just like parroting nonsense. Uh but it like looks correct to us because we haven't probed deep enough. No one does probe deep enough with their evaluations. And if we can like actually understand the algorithm these models are implementing,

• 9:40Speaker 00

  we could have a yes or no answer to the question of whether they're just like in general, surface level heuristics, or like there are actual like algorithms which go beyond just like normal heuristics.

• 9:52Nathan Labenz?

  In my general sense, correct me if you would just agree with this, but my general sense is that it's kind of always both and we just don't really know which is the case for any given task and model that we might want to consider, right? Like it seems to me that maybe not always both, but certainly you get to a certain level of scale. There seems to be some generality that starts to appear, which, you know, even if we haven't proven it yet, we've seen enough examples of like, you know, meaningful grocking to believe that more is happening. But we just don't know for any particular task under consideration, you know, at the start, like, has this been understood or is it, is it still, you know, just statistical correlation that kind of appears to be making sense? Is that your understanding too?

• 10:37Speaker 00

  Yeah, I think that it's pretty clear that there are cases when like models are like reasoning or performing like the correct algorithm to like produce certain completions to problems. Like you can think of basic math problems or basic reasoning problems uh that have been like uh.

• 10:57Speaker 00

  Turned into benchmarks and then like destroyed by various like large language models as like affirmative tests. There are some cases of reasoning going on. But at the like the frontier of capabilities where like uh you suddenly have like the next like size of model that can do something that like previous models couldn't do such as G P T four surprising many people with its like coding abilities. It's unclear whether this is uh at that scale when like the ability first emerges. Like actually just pattern matching that's worked in enough cases to like convince humans. Or there's something deeper going on where at some level you reach some like understanding of code. And I think this like is uh quite an important distinction.

• 11:41Speaker 00

  Whether like frontier capabilities are incredibly surface level when they first emerge. Or whether they can be like learnt in generality straight away. Because like the emergence of capabilities and the unpredictability of new things that models can do. Is quite important for like the future risks of systems. Because if we can't like predict what's gonna come, then we would at least like to know that there's hopefully like a surface level heuristic maybe. Then like a completely general solution to something that we thought was very difficult. Because this could cause quite a lot of instability and unpredictability when we like deploy systems. So like quite a really getting quite into like the high level motivation for why like these issues are quite serious.

• 12:21Speaker 00

  But uh yes, I agree that there is a both uh surface level heuristics and general like general um reasoning ability in these models. And uh I think that the problem is like distinguishing the two, particularly at the frontier, if that makes sense.

• 12:35Nathan Labenz?

  Totally. I've said, you know, probably in a few episodes at this point that for me, the most important sentence in the G P T four technical report is the it's quite a short one. Certain capabilities remain hard to predict.

• 12:51Nathan Labenz?

  And there they show the reversal of what had been an inverse scaling law where I'm sure you're familiar with this, bigger models had been more susceptible to hindsight neglect or hindsight bias, sort of, you know, reasoning fallacies until GPT-4, when suddenly sure looks like, you know, that has been grocked. I don't think it's been, certainly hasn't been proven in public. I don't know what they know internally, but, you know, to go from everything's getting worse as the models get bigger to all of a sudden, you know, GPT-4 is perfect, definitely, you know, suggests that there are some kind of phase changes that happen on the frontier. And the fact that, you know, even OpenAI, for all the smooth curves that they can plot, you know, can't really predict on any given task whether

• 13:36Nathan Labenz?

  it's going to be understood or not, you know, that that definitely creates a lot of interpretability, as you said. So let's get back a little later to what we might do to encourage things to be more, you know, understandable from the start. And for now, just dive into what you have done to help us make sense of the models that we do have at present. So I love the way that you approached this paper and the, I also enjoy the clever name, the ACDC, which you give to the, the algorithm. I think first, it's really useful just to describe the general process that one takes to mechanistic interpretability. I think you did a beautiful job of that

• 14:20Nathan Labenz?

  in the paper. And this will be the first time that folks have heard, you know, this level of detail. So for starters, how about just kind of giving us this overview of the workflow that mechanistic interpretability researchers tend to pursue. We'll get into each, you know, of the three parts in more detail. And then of course, you know, get into especially the part that you've automated.

• 14:40Speaker 00

  Cool. Yeah, that sounds great, Nathan.

• 14:42Speaker 02

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 14:45Speaker 00

  So the way we like set up the like three steps of the mechanistic interpretability workflow, just to prepare in advance, are firstly, like choosing a like behavior or a task that like a neural network can perform.

• 15:02Speaker 00

  Such as, as we discussed, like the ability to like predict like modular addition. Uh like a language model trained to uh get the correct answer to modular addition sounds. Uh and then secondly, after picking this like behavior, we then define like the scope of the interpretation that we're aiming to explain. So this means that uh sometimes it's possible to explain the whole like computation that the model performs in terms of individual neurons, for example, which uh communicate between different layers of your model. But this is in general pretty difficult since there are a huge number of neurons and uh like computation is not often localized to individual neurons.

• 15:44Speaker 00

  So like other researchers and projects I've worked on have instead like looked at the like attention heads and MLPs of transformers. So uh an MLP consists of many neurons, uh but you can consider it as just like one component. And so a researcher who might maybe thinks it's a bit too ambitious to explain how a task is performed in terms of uh the like individual neurons may wish to explain like the task in terms of the MLPs, the whole MLPs that are important in the model. Maybe it's just a subset of them. So then that's the first two steps and then the third step is to perform a bunch of like intervention experiments

• 16:23Speaker 00

  that are find those exact like attention heads, MLPs or neurons that are important for that task at hand. And this is generally the most like laborious from the human side uh like part of the process. And hence why like we thought it was a good fit for trying to like automate in terms of the like circuit discovery process. So yeah, those are like the three steps, like a high level overview. Uh I'm happy to get into that if you'd like, Nathan, or if there are any of the questions uh that are seen to stick out to you, happy to discuss.

• 16:53Nathan Labenz?

  Yeah, I definitely want to get into each one in more detail. So but just to kind of echo it back to you. First is identifying a behavior of interest. I mean, let me let's just get into the details now. So the.

• 17:09Nathan Labenz?

  I'm struck by, you know, there's a lot of trade offs here, right? You've got, first of all, what models do you have access to? You know, what models can you kind of actually manage to scale your approach up to handling? You know, what are those models capable of? So those are kind of some very practical constraints. In general, people seem to be doing this work at this stage on relatively small models. And those, those models are, you know, only capable of relatively modest tasks, certainly compared to, you know, the likes of G P T four and Claude two. So maybe just, you know, give us a little bit more insight into how you choose what models you're working with and how you identify, you know, behaviors that you think are actually, you know, particularly worth this depth of investigation.

• 17:52Speaker 00

  Yeah, sure. That's a good question about like the, like the first step of like, what do you mean by like picking a task or like the criteria to like choose here? And I think that some of the criteria which feel important to me are like localization of a task, such as something which can be like, like thought about on its own as like distinct from the rest of the sort of natural language computation. So one of the tasks we considered was a task where a model is able to predict like a future year in a given century from like a previous year in that century.

• 18:35Speaker 00

  And in this case, this is like quite localized to individual token completions that are about particular years that have been like incremented from the like previous year. Whereas like tasks that are somewhat more vaguely defined, such as like the model produced like a non-toxic response or a non-harmful response are often really like open-ended and difficult to pin down. And so it's generally harder to like interpret tasks that have quite a vague, like a definition and can be like completed with a huge number of different tokens, for example, because this is just going to be distributed all through the network and be quite difficult to pin down.

• 19:21Speaker 00

  So that's like one consideration on like task choice. That's mostly a question of attractiveness, like how attractable is it to actually be able to like complete this interpretability project? And you could often like narrow down wide concepts into smaller ones. To like get around that issue. And then the like, oh, the second consideration would be something like, uh, is this like task relevant to something that like, uh, bigger models can do? And like, is confusing from like the relevance of like the actual capabilities of these models? So as examples of this, a lot of people are interested in models ability to recall facts.

• 20:02Speaker 00

  For example, like to produce completions, the sentences that require knowing something about like the objects in that sentence. And this is important because we'd like to know, like, if we train our language models on this much data, and this person is mentioned this many times, for example, well, will the model be able to like store that information? Will it know that information and stuff like that? And so this turns out to be like quite like an interesting problem because it certainly matters for like models we deploy like later. And like, there are lots of things that natural language models do because just the training data distribution is extremely diverse. And so selecting something which is useful as like models get bigger is another important consideration.

• 20:45Speaker 00

  So yeah, that's the two considerations for like choosing a task.

• 20:48Nathan Labenz?

  That's really interesting. So just on the first point about kind of locality, it strikes me that there's probably at least a rough mapping onto or maybe from tasks that we ourselves sort of have some ability to describe how we're doing and tasks that we might hope would be tractable. You know, for example, a toxic or non-toxic response, like if I look at my own behavior, I'm like, not always that super clear as to why I generated a toxic or non-toxic response. Whereas, you know, I think I have a clearer sense of, you know, how I'm thinking about something like, you know, for this example you gave about the years, you know, a prompt,

• 21:31Nathan Labenz?

  sample prompt from the paper, the war lasted from fifteen seventeen to fifteen, and then it's up to the language model to complete that. Introspectively, it feels like I have a better sense, you know, for what I'm doing. How much does that kind of introspective, you know, decomposition of tasks feed into your task selection? Not to say it should or shouldn't, but I'm curious.

• 21:57Speaker 00

  I think that we always would be choosing tasks that involve something that like humans know how to do. Because then we can like put metrics and measures of how much the model does them. Since we can like figure out how we do them. But I think the like interesting point here is that models often do things in very different ways to humans. And we like found like several examples of this in the circuits we discovered. Where when models are like choosing the correct name to put on a the end of a sentence. They often like aggregate all the names together and then remove the deleted names, which is not how like humans reason about names at all.

• 22:42Speaker 00

  So I think it's the case that we always choose tasks that like as humans, we can like understand like understand how we perform them. At least at this point in time. Uh, but we don't always observe that the language models, they can perform these tasks in the same way as humans at all.

• 23:00Nathan Labenz?

  Yeah, certainly. It's always important to keep in mind the, you know, just profound, profound differences between the way that we do things and the way that the models are often seemingly in fact, learning to do them. So, okay, well, there'll be more opportunity, I think, to unpack some of these examples and kind of, you know, see how this plays out. When you're putting together a data set, I also understand that it's important to have contrasting examples where you want to set up a situation where you can kind of look at the difference between a sample that's like doing or an example, you know, completion that's doing what you want it to do and one that's not doing it. Can you give us a little bit more intuition for that? Is it like for the year example?

• 23:42Nathan Labenz?

  Would the contrasting be like doing it right and doing it wrong, like getting the wrong answer, you know, from fifteen seventeen to fifteen fifteen, something nonsensical? Is that the kind of like super sharp contrast or is it just like other tasks, you know, that are kind of not this task?

• 23:56Speaker 00

  Cool. Yeah, that's a great question. So when we define these tasks.

• 24:02Speaker 00

  As we can like sort of explore in our paper, a crucial component of this is the selection of like two data sets. And the other first one is unsurprisingly, a bunch of like prompts or inputs to like a model which have the like behavior of identified. But you also do need like a contrasting set, as like Nathan you've just mentioned, that are like crucial to to find the components of the model that like actually do your like original task. And like, why do we do this? Why do we need like comparison of two different parts? This is to do with how in the language model computation, it's not possible to just

• 24:41Speaker 00

  find the like subset of the model which does your one task and just completely ignore the rest of the model. Because that is still going to need to do some computation in the forward pass when you're running the model to be able to produce the correct completion. Like you can't just like take your model in your like programming language and just say like, no, I don't, I don't want that component, like just like remove that. You have to be like somewhat more clever. And so the default path that's taken in most machine learning research is pruning. And what like pruning does is set certain weights in the network to zero to like remove like the effect of that that like component of

• 25:25Speaker 00

  the model. So if it's like a neuron, it will now now never fire if it's like weight is zero. But this is actually quite problematic, because the model in like training and normal like runs is not used to seeing just zeros from like a huge number of components in its like forward pass. It's used to seeing like different values which fall on some distribution. And then sometimes like something fires like slightly more than usual, let's say, and that will then cause the model to produce like the completion it does in our setting. And so this is a quite like long like point, but then the crucial like finishing

• 26:10Speaker 00

  touch here is that if you have a contrasting set of examples, you're able to just set the like models internal activations to the activations on the like corrupted like data set. And this doesn't have the problem that a lot of machine learning research has, that it just sets to zero and the model is now like essentially confused as to what it's like doing because it's never seen zeros before. Instead, the model is just like counterfactually seeing different outputs from like earlier components. So that's like the intuition for needing this like contrasting set of examples. And in the case of the like years, I think, where we're like predicting it, the war lasted

• 26:55Speaker 00

  from fifteen seventeen to fifteen. And this war example is from a great paper by Michael Hannah and like fellow authors, where they find how GPT two can do greater than, if you're interested. This then has a like a baseline example that those authors chose, where I believe it says just the war lasted from fifteen hundred to. And here, like literally any completion can work, like fifteen hundred to fifteen hundred, fifteen hundred to fifteen hundred and one, fifteen hundred to fifteen ninety nine. And so the model doesn't like need to do this like greater than operation to find out the future years.

• 27:39Speaker 00

  And so this serves as a perfect like baseline, because now when you compare your two like data sets, the model components that are important are on the like greater than fifteen seventeen to some future year data set, but aren't important on the baseline data set, fifteen hundred to literally any like year in the fifteen, fifteen hundreds.

• 28:05Speaker 00

  Then are like actually the model components doing that greater than computation. So I guess the key point here is that greater than is implicitly an operation, which is not just a super general algorithm of just a year. So our technique, like this formalism of the mechanistic interpretability workflow, is able to like specifically zoom into like the task at hand, which is about predicting a greater year rather than just like any year at all. So yeah, I hope that provides like a bit of a longer story for why like the baseline for like mechanistic interpretability is quite important. And it's not sufficient to just have like zeros as a baseline.

• 28:49Nathan Labenz?

  Yeah, cool. Okay. So then that's in anticipation, just to, you know, kind of send some of that back to you. Hopefully to make sure everybody listening along is with us. What I'm understanding is that in anticipation of the part three, where you're going to be, you know, systematically eliminating parts of the network to figure out which parts matter most.

• 29:17Nathan Labenz?

  It has been found, and you have an intuition for it, but I assume to start it as an empirical finding as most do, that just a hard elimination of different parts of the network, where we literally take them straight to zero, is in fact kind of too far outside of what the network has learned to expect and learned to process. And so it creates these other problems and then you actually can get better results by replacing, instead of actually deleting outright, you are replacing whatever values with something that's like representative and kind of, you know, normal in some sense for the model.

• 29:57Nathan Labenz?

  But to do that, that's where you you need these kind of baseline examples to to have a sense for what that normal would be in this case.

• 30:06Speaker 00

  Yeah, that's exactly right. So I think, yeah, a key like intuition pump that like I have for why this why we'd like to avoid setting things to zeros is that possibly the model's components have like an implicit bias term essentially in machine learning. That is not present in the literal bias parameters, but the weight matrix just on average, for example, is like outputting a value in some particular range. And it's unlikely that this range will be zeros overall. And so it's super useful to like use a baseline rather than use just the zeros.

• 30:42Speaker 00

  And this was like, yeah, it's not my primary research contribution, but when I was at a Redwood Research, who did like mechanistic interpretability research. Some of their research output provided evidence for this claim that it's not a great idea to set to like zero activations. But instead, like, yeah, corrupting your model with activations from a different data set example may be more representative of like the model's computation.

• 31:09Nathan Labenz?

  I'm as, I think I mentioned always kind of very careful about analogies, but again, just for intuition sake. If you were trying to do something like this on a human and you literally just removed part of their brain entirely. Then you might imagine that like other parts of the brain would be quite disturbed by that and be like, hey, wait a second, we're expecting signal from here and we're not getting it. And the whole system can kind of, you know, go haywire. So instead of literally, I mean, that's where you get into like literal lobotomies, I suppose. Instead of like actually, you know, totally disabling a part of a network, you kind of say, let me just return this to sort of baseline activity.

• 31:48Nathan Labenz?

  So that other parts of the network aren't like, you know, disturbed and they get something, you know, along the lines of what they're, you know, accustomed to seeing.

• 31:57Speaker 00

  Yeah, that sounds exactly right. And I definitely do think in terms of like similar analogies to like interventions you do on humans like sometimes because it is helpful to like choose between different like interventions you could do on models. And I agree that like the like zeroing intervention often is like equivalent to just like removing something in like a like a hu- human body or something if we were giving an analogy to like medical interventions. And in general, this like wouldn't be the way you would like treat someone. You'd rather go for like a placebo of like the hormone or something they're being treated for rather than just like removing their body's ability to produce that hormone. That would be the default strategy to like treating something.

• 32:39Nathan Labenz?

  Okay, cool. So we're in, you know, getting close to the end of part one and definitely anticipating part three. Again, the part one is identify behavior of interest, a data set that demonstrates it, and a metric to evaluate it. So let's then talk about the metric to evaluate. This was one of the areas of the paper where I was a little bit confused because my intuition out of the gate was just like, if I'm trying to figure out what parts of a network are important to doing a certain thing, then the it's, you know, the intuitive metric to me that I would want to look at first would be like, can I, you know, do some of this neutralization or, you know, corrupt

• 33:22Nathan Labenz?

  patching as it's called and basically get the same output. But you take take a few different approaches where it seems like sometimes you are just kind of trying to make sure that outputs are minimally changed, but other times you're looking at other kinds of metrics. So give us a little bit more about how you think about identifying metrics and why there are different metrics here in the first place.

• 33:44Speaker 00

  Sure. Yeah. I think that metrics are mostly like a question of the like practitioner's choice in selecting a task.

• 33:55Speaker 00

  So to give an example, in the case of the uh war sentences. So again, uh the language models complete sentences like the war lasted from fifteen seventeen to fifteen yada yada. Um the language model, we deem to be correct when it answers like fifteen eighteen or fifteen nineteen or some future year and like incorrect in when it like completes a sentence with lasted from fifteen seventeen to fifteen sixteen. And so there is some like human judgment call here that like some completions the model chooses like seventeen, eighteen, nineteen are like the correct ones and we want to measure that.

• 34:40Speaker 00

  And some completions the model probably places some probability on because these models like usually output like a distribution uh over each completion they could like create that we deem incorrect. So the like practitioners in this case decided to like measure how good the model was in this task by like summing the probabilities on the like correct completions, like the future years and subtracting the contributions from years that were less than that. And so it like wasn't sufficient for those researchers to just find like the uh subsets of models that were like similar to the original model.

• 35:25Speaker 00

  Because the original model was wrong in some ways, it would sometimes predict like the wrong earlier years. And so the like metric allows you to be a little bit more like fine grained and uh measure an exact like uh behavior that the model has rather than just uh hoping that the model's like distribution, which includes some incorrect parts, is correct. So yeah, I guess the like the high level thing here is that language models can be wrong and are often wrong. But you can like uh select for that if you know what like the right answer is and uh put this into a metric. Does that make sense?

• 36:06Nathan Labenz?

  Yeah, it's interesting. I mean, I guess we'll get into this a little bit in the upshot, you know, for kind of the findings portion as well. But it does still feel a little bit to me like. I don't know, confusing just in as much as, you know, if this model's been trained to do this thing, or maybe it's not been trained to do specifically this thing, but it's been trained and this is like one of the things that it seems to be able to do. If we then go in and start cutting out parts of the network or, you know, patching them with, you know, kind of neutral. It doesn't seem like it would, it should get better.

• 36:42Nathan Labenz?

  I guess in some cases it could, but that would, you know, be kind of a weird, you know, random result. Or seemingly random, if you have intuitions for that, I definitely want to hear them. But so then I kind of wonder like, why not just stick to the simplest thing of like, keeping the same original behavior.

• 37:01Nathan Labenz?

  As opposed to kind of looking at this, you know, this kind of task specific measure that's, you know, that's kind of cooked up seemingly kind of ad hoc.

• 37:10Speaker 00

  Yeah, I think I want to push back slightly on the like intuition there of like how language models produce their completions. Because often when we're like using like the GPT four and Claude two chatbots that are state of the art, it feels like they're like perfect in some completions, like it all looks correct and there's very few errors. But the like pre-training task for models that is like most of the computation procedure to produce that great model at the end involves predicting text from like across all the internet.

• 37:51Speaker 00

  So across all books and all like forums and all like just general sites on the internet. And the model like probably always has some like uncertainty over like what exactly the like locate, like the location on the internet essentially of the prompt is that it needs to use to produce better completions that are like more likely to be correct. And so in some sense, it's like juggling a lot of different heuristics. But some include just producing what to humans seems correct, like a year that's in future. But there's some balancing of like ironic completions or jokes or something.

• 38:33Speaker 00

  Or it has to like balance some probability that there's just like a dumb mistake here essentially. And then there's also some uncertainty that maybe this like webpage on the internet got like transcribed poorly. So that like the document will just cut off at this point. And when you think about just like how diverse the like internet of like different uh text pages is. It's then like less obvious that the model like should be completing what to us humans seems like the correct response. Just because there are so many contexts in which any particular sentence could arise.

• 39:12Speaker 00

  And this introduces just so much complexity that I guess my intuition for having like metrics for behaviors in mechanistic interpretability. Is you can control for this like what long tail essentially of possible reasons why completion would arise. By specifically choosing like completions that are correct in the sense of a year that's larger rather than smaller. So yeah, I I think that a language model training produces like a very wide diversity of like outputs. Some which are incorrect to humans and some that don't at all, and we can control for this.

• 39:49Nathan Labenz?

  To try to synthesize that for myself, I guess what you're saying is because the models in general have not been trained super specifically on this task. And they have not been trained on a super clean data set. You know, that's probably another reason that the frontier models are. Performing so much better is they probably have a lot cleaner data set than, you know, some of the earlier open source stuff that is accessible for, for this kind of work. But because they have such a mess, you know, kind of going into training, they've learned all these different things and they're all, you know, there's all these kind of sub circuits running in superposition on top of each other. And so actually there are, when you zero in on this task, it is actually reasonable to expect that

• 40:34Nathan Labenz?

  there is some sub circuit that in isolation could do a better job than the model itself is doing. And your goal with defining a task specific metric is to zero in on that perhaps even better than baseline model performance by stripping away this other stuff that in fact might be hurting overall performance on this task.

• 40:59Speaker 00

  Yeah, exactly. I agree entirely. Yeah. And just to put it in perspective, like the number of tokens these models are trained on is just mind blowing. And you imagine, or rather you look at the statistics of how long a human would need to spend reading like just text monotonously, it's like orders of magnitude longer than a human lifetime to consume the amount of text that like these models consume. And therefore, like as like individual humans, we like actually cannot model the full diversity of the distributions that like we're training these models on. So to me, it's like actually not surprising that like there's just substantially more complexity than there is to

• 41:44Speaker 00

  human text completion inside of like these like language models.

• 41:50Nathan Labenz?

  That's all my questions on part one. So we have identified a behavior of interest. We have put together a data set that demonstrates it and has some contrasting examples that are similar, but don't have exactly that same critical behavior. And we've got a metric to evaluate it, which could be, you know, by default, just minimal change to the model's output. But we do have some reason to expect that we might even be able to get better performance if we're savvy about defining a more intentional metric for evaluation. That's a big theme I kind of keep trying to reinforce for folks in general, just at the base level of, you know, model creation in the first place, that like clever formulation

• 42:31Nathan Labenz?

  of a loss function has been one of the big unlocks in, you know, the last few years, really, right? Just, just moving to this next, you know, word prediction or next token prediction in the first place was, you know, kind of a stroke of genius that allowed for all the data to be used and, but, you know, it's also, it's, that's just one loss function and there's certainly a lot more work to be done there to come up with better optimization targets. So that brings us to part two. And this part, again, I'm tempted, I don't, again, I don't want to be like overly reliant on analogies, but here it does seem like in a lot of different areas of science, one

• 43:14Nathan Labenz?

  of the kind of core decisions that you have to make is at what sort of level of zoom or what level of abstraction am I going to study this problem on? So in biology, you know, you've got ecology and you've got, you know, on the other end, like genetics and you've got a lot of layers in between. You know, you can study cells, you can study systems within the body, you can study, you know, an individual organism, you can study a species, you can study, you know, all these different kind of layers. So it seems like there is a reasonable, you know, analogy here that you, you kind of have to do the same thing, right?

• 43:55Nathan Labenz?

  You could look at every single activation, but for multiple reasons, that becomes either computationally intractable or just like too much of a mess. And so you, you have to kind of pick how zoomed in do we want to get for the purposes of this particular analysis? So. How can you, can you develop our intuitions a little bit more there for how you think about this decision?

• 44:22Speaker 00

  Yeah, the analogy to biology, while it's important to be like guarded around giving like analogies to like different fields. I think that like broadly speaking, I expect that like the developments of this like mechanistic interpretability field to progress more like biology than like different fields such as physics. Because there are like a number of parallels between the like development process of the complex systems inside neural networks that are neural networks essentially, and the like evolution process that also like trained on a very like stupid like goal function. But then also gave rise to like incredibly complex behaviors along the way.

• 45:06Speaker 00

  And so, beginning with an agreement with the analogy here, yeah, in terms of what is the like choice and how the choice is made in mechanistic interpretability research, I think it's mostly a question of like how ambitious the research is essentially. In terms of how they're like pushing this like frontier of the best explanations that we currently have of certain behaviors and certain like behaviors of different complexity with sufficient like depth and zoom it zoom into that like explanation of a behavior.

• 45:46Speaker 00

  And so it's a young field and there are really not many researchers doing research on mechanistic interpretability. But we already have like a neuron level explanations of how like toy transformers got complete the correct completion to like modular addition, which is work by like Neil Nanda and collaborators. And we have much worse understanding, but still some understanding of how GPT-2 small like completes its predictions. That solely in terms of like attention heads and MLPs.

• 46:24Speaker 00

  And then to give like a third example down the line, other researchers, the paper is called Rome, or like the Eiffel Tower is in Rome, have looked into like the factual recall models, and then edit that recall of models by looking at where facts are stored in terms of whole like attention layers, which include each a bunch of like attention heads like in parallel, and their work groups together whole layers of the like model to then say, well, the factual recall is isolated to these layers. And so really, it's a question of like, how ambitious like is your project?

• 47:04Speaker 00

  And the answer to how ambitious are you is, the correct answer rather, is not always like more, like if you try and like explain things at like a really low level, and this is just extremely difficult, then it's unlikely that like projects will be successful. And so just I think as a research community, people in mechanistic attemptability research are just trying to like improve this frontier of getting better and better at explaining harder behaviors, but in a more zoomed in way, and giving their contribution to the field that way. And so it mostly comes down to like a judgment call on the researchers like parts, but like where they're aiming at. And so, yeah, that's an example of three different levels.

• 47:49Speaker 00

  The neuron level in modular edition work, the attention head level in GPT two small work, covered in R work, but then the attention layer or like several layers level that's explored in different factual recall work. And I think all of these are like pretty good contributions at different parts of like the frontier.

• 48:08Nathan Labenz?

  And again, it just seems like there is intuition here around things like, you know, the best way to explain the flight of a fly ball is not to go to the quantum mechanical level. So you want to use a level of description that is actually.

• 48:27Nathan Labenz?

  Meaningful to the, you know, to the person who's absorbing the output. So in some sense, it's like you're optimizing for the human audience's ability to understand the results as much as, as anything else.

• 48:40Speaker 00

  Yeah. And like also as a quantum mechanic, to actually be able to finish your project to explain flight there. Like I, I wish you luck if you're a quantum mechanic trying to do that, but I also don't expect it to be successful. So that's like the, the trade off that is being made.

• 48:55Nathan Labenz?

  Yeah. So how in practice, you know, at a conversational level, this sounds sort of, I don't want to say easy, but it sounds like, you know, there's a few levels that you could zoom into that seem pretty natural. And, you know, if we're chatting about this over lunch, we can easily say, oh, why don't we try looking at the, you know, attention heads as kind of the, you know, the level of zoom for this particular project. But then obviously if you go back and actually code this and make it sort of, you know, something that can feed into the third step of actually automating the, the process of isolating these sub graphs. So how hard is that from a coding standpoint or a notation standpoint?

• 49:40Nathan Labenz?

  That's how, it sounds kind of hard to me. I'm, I'm not the world's greatest coder, but I would feel like I would have a hard time going from, okay, we've decided we want to zoom in on the attention head level or the, you know, an ML, individual MLP blocks will be our kind of unit of consideration to then actually figuring out, how do I express that in code as a, you know, causal acyclic graph? That's where it starts to sound a little more challenging. How hard is it in practice to do that?

• 50:05Speaker 00

  Yeah, I would say this was definitely one of the more fiddly parts of the like ACDC project to translate these like high level intuitions into something which is like,

• 50:17Speaker 00

  Able to be modified inside code. I do think that there are like pretty good libraries, this is mostly a coding question at this point, to be able to like extract and edit the internal states of machine learning models. So, uh, yeah, with any, like, when, once a researcher has some familiarity with how like a language model forward passes work, it's not so difficult to then add like attachments into your codebase to extract those like activations, because generally they are represented cleanly. And in our work, we like made a library for researchers to be able to more cleanly edit

• 51:02Speaker 00

  the impact of like one particular model component specifically on one like later downstream components, because that's the part which is somewhat harder from the like implementation perspective, editing the specific effect of one earlier model component has on one later model component. Because by default, just your like code just runs from end to end and one model component affects all downstream components. But to do interpretability, you'd like to be somewhat more fine grained and look at the impact that an earlier upstream component has on each individual downstream component.

• 51:47Speaker 00

  So that's the part that's difficult, though it is pretty easy to get at least started with like isolating individual attention heads. And there's now a lot of educational material, like trying to get more people to do mechanistic interpretability and to like have fun doing these sorts of experiments.

• 52:03Nathan Labenz?

  The difference here is that if you are trying to

• 52:09Nathan Labenz?

  Just execute procedurally the transformations of the transformer. You sort of take in data, you apply some mechanisms, you know, that ultimately cashes out to linear algebra. You take the results and you just kind of keep going, right? But once you're kind of past a certain layer, you can like leave that stuff aside. You don't have to, you know, by default, that inference time, right? You don't necessarily keep track of how layer one interacts with layer eight or whatever. And you don't even necessarily think of it that way in the implementation because you're just kind of doing one thing at a time. It feels like this happens, then this happens, and this happens, and it feels very linear.

• 52:51Nathan Labenz?

  But there is this kind of conceptual finding, empirical conceptual finding that because of the residual stream, if I understand correctly, actually you can have critical interactions that do not proceed layer to layer, but actually skip layers or, you know, you have these kind of if layer, you know, would, obviously, I'm just making up fake examples, but you know, if this happens in layer one of this particular network, then layer five also like lights up. And, you know, that's not the kind of thing that the naive kind of forward pass implementation really looks at. So the hard part is kind of, maybe not the hard part, but certainly a conceptual leap that one needs to make here is understanding

• 53:34Nathan Labenz?

  that like this causal graph is a bit more complicated than just the, the direct like procedural implementation.

• 53:45Speaker 00

  Yeah, that's exactly right. So this was a really beautiful like finding from the Anthropic Interpretability Team. So the AI lab that Nathan Claude, that as Nathan, I guess you mentioned, there's a residual stream, which in like normal machine learning is usually just referred to as just like the hidden state of the network, which is transformed from each layer incrementally. But the researchers at Anthropic uh realized that if you're uh implementing a model with residual connections, which just means that once you apply some transformation to this hidden state,

• 54:27Speaker 00

  you then add the original hidden state back to the transformation. So you incrementally update by adding a transformation of the current state at each step. And this gives you like a whole way to view like the forward pass of transformers and many other models as a continual stream of information which the model components read from and then write back to. And this is like a really useful finding for like all of the sort of mechanistic interpretability research field that has like this key consequence that we can model the impact

• 55:06Speaker 00

  that uh one very early layer component has on like a specific downstream component. Uh that is like a really non-trivial finding, but it's really like beautiful once it like makes sense. It's from the Anthropic paper, A Mathematical Framework for Transformer Circuits, that uh yeah, was a substantial like inspiration for our work that we like added to.

• 55:29Nathan Labenz?

  I do have, I think, the the graphical representation that they use in that paper, which presents the residual stream as kind of the central object. And then all of these other things as kind of side, you know, the attention heads and the MLPs as kind of side loops that are like making a contribution as opposed to centering those and having this kind of random line that kind of goes around them, which is how they're, you know, so many of the transformers are designed. That was definitely an eye-opening and kind of uh clarifying, reframing for me. And and that is in the uh early slides of the AI scouting report. So definitely, you know, another good reminder to check that out to get a little bit more grounding on on some of these earlier results.

• 56:15Nathan Labenz?

  So this ultimately does still sound kind of hard to me because you know you have kind of a clean forward pass implementation, but now you do have a many more kind of paths, causal graph that that ultimately turns into for the purposes of this type of sub-circuit isolation work. Come back to that in just a second. One thing that jumped out to me also is kind of a footnote, but and this is the kind of thing that may be really obvious to all the practitioners like you who do this every day, but is not necessarily so obvious to people who you know are kind of trying to catch up to you is the idea

• 56:55Nathan Labenz?

  that all of these circuits are acyclic, meaning there can never be a loop in the design of the circuit. It all has to be one way. It all has to be, you know, the kind of language of forward pass and back propagation sort of suggests that. But I'm interested in your thoughts a little bit on this kind of cyclicality. I guess for one thing, you know, do anything more to add? And then on the other hand, like, I wonder if that kind of is represents some sort of frontier in model development where if we could figure out how to have some sort of cyclical loop, you know, certainly that's something that we have, right? We have some sort of ongoing kind of feedback mechanism where our current state, you know, interacts with our future state and our past states in ways that are

• 57:39Nathan Labenz?

  not, you know, we're not just like, you know, waking up and executing a single forward pass in isolation each time. So yeah, I don't, do you have any thoughts on this kind of cyclicality as like, it's a little bit of a digression from the main topic, but does it feel like that is something that will never come online because our techniques just don't support it? Or do you feel like that is maybe, you know, one of a candidate for kind of another future capabilities unlock?

• 58:06Speaker 00

  Yeah, I really like this observation that I think is one that like researchers like me who spend like their days trying to like uh understand these models forgets is like a design choice which is informed all the things that I look at. That these models are just like ends to ends rather than like cyclic at all. And so it's I think it's a great observation because it's just one that I just wouldn't have noticed because I spend all my time working with the like the end to end models. And yeah, like the constraint of back propagation and requiring to have like gradients for each individual component as aggregated end to end.

• 58:44Speaker 00

  Does enforce that like the forward pass of the model then just computes like a purely linear process. Which goes through like different edges, but then is always like going forward in some sense. And uh yeah, like uh the like mechanistic interpretability research community. Has mostly focused or is mostly focused at the moment on these transformer language models, which all like operate under this like uh like a cyclic paradigm. And uh yeah, it's uh certainly just essentially a whole another like dimension which there's no current work on. To be able to like unlock some interpretation of which definitely like could be um.

• 59:26Speaker 00

  Something which would really like advance our understanding, uh particularly because a lot of the current use cases of like uh language models. Are maybe not quite like uh ace, like not quite cyclic, but certainly more recurrent. Which means like feeding back into themselves, uh which like in general mechanistic interpretability hasn't managed to like get a great handle on yet. So while we can talk a lot about the sort of like ability to understand how models produce. One token completion and how this is like a really exciting open research direction. We don't have much understanding of like how models produce helpful like rollouts of completions.

• 1:00:09Speaker 00

  So whole like poems or prompts that go on and do some like chain of thought reasoning, for example. We just don't really know the mechanics of um how the models like using uh its computed new like first token to then produce its computed next token. And we also do have a great understanding of how like these more common like agents deployed on like the internet such as like auto G P T models, how they or if they are meaningfully different from the models that just compute like one forward pass. But instead, like these models are continually produced, like more and more actions that the model then like tries to like uh do and then like observe some like consequence from that action and then acts on that further.

• 1:00:52Speaker 00

  So yeah, like uh uh all this sort of discussion is premised on like just really like individual single forward pass completions. And there are ways it could be extended, but we certainly haven't done them. And I'd like love for like future work and future mechanistic intelligibility research to hopefully grapple with like these like harder problems of like recurrence and then plausibly even like cyclic models if people find ways to like make this work with back propagation.

• 1:01:19Nathan Labenz?

  Yeah, I kind of see two, two more little follow ups in this digression and then we'll get to the, you know, the core of your contribution. What I was kind of, if I understand correctly, the, the core kind of constraint here and the reason for having this no, you know, cycle, no loops, uh acyclic constraint is basically that we just want to have easy computation for back propagation, right? Whereas if you, so you can kind of work your way back and you're only each, at each time, you can sort of say, well, we know how all the other stuff already plays out. So we're kind of, you know, using the, the chain rule and it's all at each step. It's like an easy calculation, whereas I guess if you had a loop, that would seem to

• 1:01:59Nathan Labenz?

  suggest something more like a differential equation type of math dynamic and then you would have just a lot harder math on your hands. Is that basically the, the issue there?

• 1:02:10Speaker 00

  Yeah, that, that seems correct that uh we like have found this easy way to train models under back propagation. And it's certainly not like the optimal way, but it's like a common lesson in like machine learning essentially, that a, a lot of progress is generally gained from pushing simple techniques super far compared to creating incredibly intricate and complex techniques over like long periods of time. And so, yeah, my, like a

• 1:02:45Speaker 00

  Article by Rich Sutton, like a famous, like a machine learning researcher, calls this the bitter lesson that generally are smart methods in machine learning. If they can't like absorb lots of amounts of computation power, lose to much simpler methods if the simpler method can scale a lot. So I think this like forward pass and non cyclic like paradigm of models is probably a consequence of like this simple backdrop setup for like language models. In fact, being very scalable to like large amounts of compute.

• 1:03:27Speaker 00

  Whereas cleverer architectures may like bake in better assumptions. And in theory, have like more useful properties, but aren't as easy to like just scale with a lot of like compute. And so I think that's the focus of like my research, I'd say overall, like focusing on the simple and principal like techniques that actually scale to like quite formidable consequences.

• 1:03:54Nathan Labenz?

  All makes sense. The bitter lesson, we learn it over and over again. What about a change to the loss function on the other hand? So I'm thinking of a recent paper, I'm hoping to interview the authors of this one as well. I believe it was out of Stanford where it kind of made the rounds for having a backspace token. That was kind of the headline, like we introduced a backspace token. Now the model can kind of course correct. I've only read the paper sort of superficially so far, but it also seemed to involve a a different loss function that they they kind of talked about re casting the process as an imitation learning challenge as opposed to just next token prediction.

• 1:04:37Nathan Labenz?

  And therefore the optimization seems to be over like a longer, you know, set of tokens. And then that can kind of feed into this ability to do the the backspace action when.

• 1:04:50Nathan Labenz?

  The results seem to be getting too far outside of, you know, normal distribution. What do you think about, does that, how does that kind of loss function switch potentially relate to this kind of mechanistic interpretability work?

• 1:05:02Speaker 00

  Hmm, yeah, I don't think as much as a researcher about the like sort of the loss function these like models are trained on, like what's the optimal choice of loss function there. But I certainly think it's like a really exciting direction for interpretability to try and like choose loss functions that are like more interpretable by default. And I think I was, I was excited to hear that you spoke with Ziming Liu, who's done some stuff on changing the loss function of models to make them more modular. And I think that this work is exciting, yeah, and it's not something which I personally worked on, but I always enjoy seeing changes to the default setup,

• 1:05:46Speaker 00

  which can hopefully incentivize like models to be more easily amenable to like our explanation techniques. And yeah, I think that one lesson here, which is quite useful, is that it's exciting to have interpretability and mechanistic interpretability techniques that can hopefully work no matter what the sort of like training setup is or how models change. So we'd like to have like approaches which will work even if the sort of like game changes slightly and people do things differently in future. And so this was like a substantial motivation for like the work on just pinning down models computational graph in full generality.

• 1:06:28Speaker 00

  Cause this wasn't tied to like having the particular transformer architecture that's basically ripped off of the like GPT two and GPT three papers, but could potentially be used on any sort of model. So yeah, I think that I'm, I think it's a good idea, particularly because machine learning moves so fast, to be open to approaches that will still work if the like board game changes as like machine learning progress continues.

• 1:06:56Nathan Labenz?

  So that brings us, you know, and I'm appreciate all your time and willing to go down some of these rabbit holes with me. But I think that brings us finally to your core contribution, which. Just zooming out for a second, this three-step process of identify a task, have a data set that can demonstrate the task, have an optimization goal to evaluate how a sub-network is doing against that task. That's all part one. Get set up. Two, figure out how to represent your network as this causal graph. And now, three, what you have created is a piece of software that can automate the otherwise extremely tedious process of just systematically working its way

• 1:07:39Nathan Labenz?

  through all the branches of this graph and figuring out which of these actually do anything and which can we cut as we look to zero in on what's actually doing the core part of this work. So tell us about how that works.

• 1:07:55Speaker 00

  Yeah, cool. That sounds great. We got so into just like explaining the sort of like side tangents of why mechanistic interpretability research has done all these things. But that was far longer than a necessary introduction to like what like the contribution of ACDC, so Automatic Circuit Discovery, is. Since ACDC is really just like a three-step algorithm that like imitates the human process for trying to interpret neural networks, but does this just via like a software rather than requiring a human in the loop. And so, given all the like extensive description of the setup that we went through,

• 1:08:38Speaker 00

  the three steps are firstly selecting this computational graph at some level of abstraction, and then add like a given node in that like out like that graph, looking at all the like input edges to the node in that graph. And one by one, like removing them by setting their like activation to the activation on the baseline data set.

• 1:09:12Speaker 00

  And then measuring whether setting the activation along this particular edge to the baseline dataset decreases the like model's performance on the downstream metric by a given like amount. And if this is a like a large decrease in model performance, then we keep this edge in the graph. But if it didn't seem to matter at all, we can remove this edge. And that's like the step two, which we then just recurs in the third step through all the nodes. So that's like the high level overview of all of ACDC. But

• 1:09:52Speaker 00

  it really is just like three steps to find a subgraph of the model's whole computational graph.

• 1:09:59Nathan Labenz?

  So you've got this process for kind of neutralizing components of the graph, asking how does performance compare on the output metric? And then if the performance is sufficiently degraded, then we decide, okay, that's one we need to keep. Whereas if it's not sufficiently degraded, then okay, we can throw that away. So I guess, two questions there. One is like just procedurally, you have all these different kind of length connections, right? Like if I'm starting at the end, you know, my last MLP block or whatever, it's influenced by all previous layers. But those are mediated through the residual stream. So what does

• 1:10:40Nathan Labenz?

  it actually really mean to say, if I'm looking at, okay, the connection between the eighth, you know, and final, let's say MLP block and the first attention layer, that's all kind of, you know, the direct connection there being mediated through the residual stream where all this other information is also flowing. What does it mean to knock that part out? Like you can't take the whole residual stream out, right? So what does that actually look like to cut that kind of connection?

• 1:11:08Speaker 00

  Sure. So in the example of a very early layer attention head, for example, that might be just one of the things that are the inputs to one late layer MLP.

• 1:11:23Speaker 00

  We would generally write the inputs to that MLP as the sum of like all the previous components, because as mentioned in the residual stream, the inputs to the MLP is just the sum of all the previous components that have added to the residual stream. So if we want to corrupt the effect that this singular attention head in the early pass of the model has on this like far downstream components like an MLP, what we can do is

• 1:12:03Speaker 00

  we can take the inputs into this attention, sorry, this MLP at the end, we can then subtract this clean contribution from the early attention head and then add in the like corrupted output of this attention head. And then this will preserve all the other clean activations, which are the inputs to this MLP, and will just corrupt the contribution which comes from that one early attention head. And so, yeah, that's the process that we use to like edit just this singular connection from, in this example, an attention head to an MLP.

• 1:12:47Nathan Labenz?

  Let's go back to the years example. So you're looking at the greater than task, you know, the war went from fifteen seventeen to fifteen blank, and you've got your contrasting example, which might be the war went from fifteen hundred to fifteen blank, such that, you know, you're not, you don't necessarily need to do the greater than because it's going to be some, you know, any two-digit number there would work, right? So you then have a situation where you're like, all right, I've got all my activations for the actual greater than task, I've got all my activations for the kind of very comparable but not requiring the actual greater than operation task. I hear you, I get, I get the idea that, okay,

• 1:13:28Nathan Labenz?

  I can express all the inputs to a late layer as the sum of all the outputs from the earlier layer. If I

• 1:13:38Nathan Labenz?

  Switch, you're only doing one adjustment to the sum at a time. So like, if you're just looking at this one layer, I'm a little confused still about if I were to change the outputs, those would also change how each middle layer of computation actually works. Right. But you're not doing that. So like, if I'm, if I'm looking at the connection between layer one and layer eight, I'm not necessarily changing the sum for the purposes of looking at layer seven. Is that right?

• 1:14:10Speaker 00

  Yes, you're totally correct that we do not edit the effects that in the example, the early attention head has on all the middle components. And to provide an implementation detail, which might help to understand the process here, we actually simply cache the corrupted value and the clean value of this early attention head. And the benefits of caching are that we can run the forward paths up to this final MLP without having done any changes to that early attention.

• 1:14:54Speaker 00

  But then once we're at this MLP, we have the cached clean and corrupted value. So now that we can like force this MLP to have like essentially seen the corrupted value, even when in the whole forward pass so far, we just had the like the early attention head, for example, on the like clean input. So it's a matter of caching those to like saving them as in your like piping code, just nothing more complicated than that. So that then, once we're at the downstream node, then we can do the editing. And this ensures separation of the effects of the early attention head on the middle components,

• 1:15:39Speaker 00

  separate from the effect of the early attention head on this particular MLP.

• 1:15:44Nathan Labenz?

  So again, what I, what I understand you to be doing for starters to be, make sure I have this straight, is defining how much degradation in performance on our optimization metric will we tolerate? And if it's, if the degradation is below that level of tolerance, then we'll cut that portion of the graph.

• 1:16:04Nathan Labenz?

  But I was kind of confused because I was kind of thinking, you know, different tasks, different graphs, everything might be so different. Does it make sense to make that kind of the free parameter? Or would it make more sense if it were tractable? I'm not, you know, this is where my abstract math isn't quite strong enough always to know what's going to work or not. But I kind of felt myself wanting to reframe the question a little bit and say, how sparse can I make this graph? How much can I cut it while still maintaining some overall level of performance? So why set that kind of individual kind of operation, you know, bit by bit threshold as opposed to some kind of global notion of how sparse can we go while still like succeeding

• 1:16:49Nathan Labenz?

  at the task?

• 1:16:50Speaker 00

  Yeah, this is a fantastic question because I would much rather have an algorithm which chooses the sparsity of the graph and then gives you back a subgraph that has that sort of sparsity. And the reason that instead we just have like a threshold which measures like the amount that a single edge matters is purely like a tractability question that we don't know like how to design an algorithm which like at the end, or at least I don't know at this point in time, that like at the end will give me like this like graph that's sufficiently

• 1:17:30Speaker 00

  sparse, let's say, but in the process like aims like robustly towards that conclusion. Because like in advance, you just don't know which of the huge number of subgraphs will be the one that like is the best at like explaining how a model does a task. And I think this is like a general machine learning iterative optimization problem that are really, we would like to specify what like we want in the end, but this doesn't really give us like a tractable way of getting there. Are you just asking for like a sub-

• 1:18:11Speaker 00

  X or Y sparse graph, this amount of sparsity in your end graph. But like if you just immediately select a graph with that sparsity, it just will be like in general, like absolutely hopeless at the task. And so we need like a iterative way to get there. And this could be gradient descent, or this could be the sort of ACDC algorithm, which goes node by node. But uh it just is like a pretty hard problem to like uh have uh that's like end goal be a useful target through the optimization process. So your intuition is completely correct that uh we would rather have like a this level of sparsity graph.

• 1:18:53Speaker 00

  But instead, what we've got is this like a proxy measure of the like local uh corruption amount that's allowed. But it certainly um can give you some indication of how sparse the end graph will be. Because the ACDC iterative process first will process the output node. So once you've run with a particular threshold, you'll be able to see like how many nodes is added to the output connection. And this will provide some guide to then how many nodes will be upstream. Because like you'll likely, if you have like one half of the nodes uh outputting to the end connection,

• 1:19:36Speaker 00

  then it seems like you're going to have a pretty dense graph overall. It's going to include most of those connections. Whereas if it's including just like two of the nodes and you like, for example, wanted a substantially more like that was not enough for you and you thought there were more components that mattered, you could increase it. So while sadly we don't have a met like a uh hyperparameter which can give us the exact sparsity of the end result. The early like performance of the algorithm does give you some indication of how sparse the end result will be. But yeah, it's a great concern. Uh I hope people in the future can make a version which gives graphs of like a specified uh sparsity. But for now, we like uh don't have a method of doing this.

• 1:20:21Nathan Labenz?

  So for now, you're kind of sweeping through the.

• 1:20:26Nathan Labenz?

  Parameter space for what that threshold should be. And then kind of eyeballing initial results and kind of saying, well, when I set the threshold such that it tolerates a lot of degradation, like everything got cut and it looks like this has kind of gone too far. Versus if I set it, you know, to only tolerate a tiny bit, then you might see something that looks still super dense and be like, doesn't feel like it's gone far enough. And you're kind of using a certain amount of taste to kind of figure out where in that trade off you want to be.

• 1:21:01Speaker 00

  Yeah, I think there's a good distinction here between two ways in which we like tested the ACDC algorithm. That involved like validation tests, which swept over a huge number of parameters to see what the like best performance was for like this algorithm. In different regimes, like in the very sparse regime to the like pretty dense regime was like one thing that happened. But then we also tested like at least a use case where this was used in practice by like researchers who were trying to find like a particular.

• 1:21:46Speaker 00

  Behavior of a model and where it was computed in the graph. And in this case, you're not so worried about sweep over all possible parameters. You're looking for like a revealing subgraph, which helps you like begin your like research into like how a model is doing a particular task. So I think there's like two modes there that like in the work, in machine learning work, you need to like validate that like actually your technique is helpful for recovering circuits. But then in practice, you can do like a early stopping essentially, once you found something which is revealing how models do certain tasks.

• 1:22:25Nathan Labenz?

  Yeah, I think the two ways that you validated the results are super interesting to me as well. The, I was a little bit surprised by the, the order of presentation, not that that's like the most important, but when I think about how would I, you know, I've come up with this technique, like how would I demonstrate that it actually works? To me, the obvious thing is like, show that it can continue to do the original thing, you know, as well, or not so much worse, or maybe even in certain circumstances, better than the fully dense graph. That part makes total intuitive sense to me. But is there a reason that you kind of prioritized the other one for discussion earlier, the other one being looking at what folks like Neil Nanda have actually found through their

• 1:23:06Nathan Labenz?

  own, you know, non-automated painstaking approaches and kind of comparing what the algorithm, the ACDC algorithm isolates to what they isolated by hand? Why was that like the first place to go for validation?

• 1:23:20Speaker 00

  Yeah, that's a good question. So just to clarify, the experiment that was performed, it was a experiment to see how well the ACDC technique, as well as differing techniques that we like repurposed from like the literature, were able to like find the circuits that previous work found. I don't think it actually included an example from like the Neil Landers line of work, but it did include, for example, the greater than year example that we discussed a lot. And I think we chose this measure, the measure of how much our technique reproduced the other

• 1:24:02Speaker 00

  like work that practitioners had done, was that our motivation was to like make something which is helpful for mechanistic interpretability research, that is like the first step in the process of like actually giving semantic meaning to the different components in models and like what these components are doing. And so I think that it is helpful to get some indication of how performance your like subgraph is, that you mentioned as the second like evaluation that we chose. But the purpose of the like ACDC algorithm was certainly something that we hoped practitioners would use to like explain models rather than like get models that are just like really

• 1:24:47Speaker 00

  good at predicting years, because this is not something which like is actually like useful to people, like how good is your model at predicting years? So our priority was like the First step on the path to understanding the semantic meaning of components. And high performances may be correlated with that. But it's not exactly as direct as just finding the important components that like researchers had like in practice found were like the semantically meaningful components. So I hope that makes sense as like a the two different evaluations and why we were excited about reproducing previous work. Though it is certainly flawed and we can get into that if need be.

• 1:25:31Nathan Labenz?

  I find it's think that it's kind of a an audience driven thing where you your goal is to create a tool that will be adopted by interpretability researchers and to convince them that this is actually meaningful. You wanted to show that you could recreate earlier results that they all know about and, you know, hold in high regard.

• 1:25:52Speaker 00

  Yeah, I'd say it's true that it's an audience problem. But I I want to clarify that I'm not like overselling this approach. In terms of an approach for finding like the best subcircuits that do different like behaviors, like correct years, this would probably not be a very competitive approach. Because we're doing these interventions on like the edges that involve like a substantial amount of caching and re-computation that would be inefficient compared to other ways you could like elicit a model capabilities. Because it's just like a substantially larger amount of compute.

• 1:26:36Speaker 00

  So that's just like not really the like uh the area that we're like competing for to make a good technique. We're competing for something different, which is like the uh simply the for the discovery of the semantically meaningful components. So I'm not overselling my work. I do think it would be a very good algorithm for getting very good subgraphs at particular tasks. But it wasn't the goal ever either.

• 1:27:00Nathan Labenz?

  So that would contrast to like the Ziming Lu from the Techmark Group work that we talked about earlier where they are taking a a different approach, modifying the loss function during the training process to kind of create sparse networks by design. Is that kind of what you're contrasting against? Like, that would be the approach to finding the, you know, the sparsest network that can do a task and and you are instead trying to create a tool that. Can kind of also do that, but is doing that already downstream of like this very messy training process? So it's not really optimized for the best possible circuit, but it's optimized for finding what circuits do in practice exist, given current techniques.

• 1:27:41Speaker 00

  Yeah, I think this is a useful distinction between post hoc interpretability, which our work is like an example of, and like training process interpretability, like selecting for interpretability, that I think that the Seemoo work is a great example of. So here, we're assuming we have some fixed model and it's a black box. We want to like open up the black box to understand what's happening within it. That's just like the premise of the work. But like complementary work, a different direction you could take is, yeah, like designing training processes that incentivize like interpretability through like modular structure

• 1:28:26Speaker 00

  in that example. And I just think these two approaches complement each other because on one hand, if the architectures that like we were studying were more comprehensible, then this would just make our job more modular, rather. This would make mechanistic interpretability much easier. But at the same time, work that's building into the loss function, some hopeful notion of interpretability does need to be validated to actually be interpretable down the line. Because models can learn like strange solutions which appear interpretable, but are not in reality as interpretable.

• 1:29:06Speaker 00

  And some work by Anthropic on a technique called Solu as an activation function instead of ReLU or GELU is actually an example where the researchers tried to like choose a training process, which led to a more interpretable model, but found that the model was like hiding its like superposition in this case via like confusing routes that they had created by like introducing their new technique. So I think that to reiterate, the two paradigms of like post hoc interpretability and designing training processes for interpretability are like complementary and different in terms

• 1:29:48Speaker 00

  of like approaches that you would take to like try and reach both of those goals.

• 1:29:53Nathan Labenz?

  It's just important to keep in mind that like sparsity. Does not necessarily mean it's super interpretable and certainly doesn't mean that it's like generalized in a way that, you know, we would consider to be like rocking or, you know, sort of representative of some like more fundamental, you know, non-stochastic parrot understanding. It could, you could have sparsity and still have all those other problems at the same time. Cool. Well, then let's talk a little bit. You kind of alluded to it for a second about like the compute that goes into this. How much compute did this take? Like, what kind of resources do you need? How accessible is this kind of stuff? You know, do the techniques

• 1:30:34Nathan Labenz?

  that you have today scale up to large scale models if you just have enough compute? Or is there not enough compute in the world to, you know, to apply something like this to a GPT-3? Tell me about just kind of, you know, all the compute considerations with this line of work.

• 1:30:49Speaker 00

  Sure. So this work was done with computes that was mostly from just like a far AI. So like a research group that one of the collaborators, Andrea Grigor Alonso, works with. And this was not like a super cluster of like one of the huge labs that we were working with. And we could see practitioners get results on the like the GPT-2 small, large language model in like half an hour or like an hour of like runs when they worked super well and were like pretty sparse. Though there are definitely cases where compute is somewhat

• 1:31:30Speaker 00

  of a bottleneck to ACDC and particular scaling hits. So the two cases that come to mind are that when you like don't select the threshold appropriately and you include lots of edges, then you tend to have to search through every single node of the computational graph. And since your like computation is like roughly linear in the number of like nodes that are present, this becomes extremely expensive as your technique includes and searches through each node. So the first case is when like, yeah, you don't choose like the correct threshold. And this sometimes can be frustrating and then leads to slow

• 1:32:14Speaker 00

  runs and slow feedback loops that we hope that like future work will be able to work on.

• 1:32:19Nathan Labenz?

  GPT two small is how many parameters, like ten million?

• 1:32:23Speaker 00

  Yeah, so that's the second sort of worry about the like computes that GPT two small is a hundred million parameter model. So this is like large, given like statistical models or things that people used ten years ago. But it's incredibly small compared to the like hundreds of billions of GPT three larger models. And because our technique is like an iterative over all the nodes, all the edges in fact, which scale almost like with the square of the number of nodes involved. Currently, this is not feasible at all for like the GPT three sized model. And isn't really even very efficient for models that are like at the billions of parameters.

• 1:33:06Speaker 00

  So we're not able really to scale up an order of magnitude beyond GPT two small at current. I at least am excited for further interpretability research to hopefully scale to those sizes. And already people have done some interpretability work on the like alpaca model, so like a seven billion parameter model. And I think that I know a number of like follow up works to this like work that could plausibly be able to scale up to that size. While also actually finding circuits. So an open problem, essentially, I would like to see more work on it.

• 1:33:42Nathan Labenz?

  Yeah, interesting. So even with GPT two small, though, I assume we're not using the individual neurons as the as the nodes, right? So it's, there's still some zoom out. So how, could you give an intuition for how that kind of number of nodes scales with model size? It seems like there's almost a different scaling law, you know, or different scaling intuition that one needs to develop here, right? Because.

• 1:34:14Nathan Labenz?

  It seems like more layers would definitely have a big difference. So even like with a certain number of parameters, you know, kind of depending on the the width and the number of layers, you could maybe set things up where the the number of edges to consider actually could vary potentially quite a bit.

• 1:34:31Speaker 00

  Yeah, that's a good point. I hadn't even considered that in the discussion. That, in fact, almost all of our research in the like the main text of our paper focused on the abstraction level of the important attention heads and the important MLPs in these like large language models, rather than being more specific or to the individual neurons of these large language models or the like attention layers that would be less specific. I was solely talking about the abstraction level of the individual attention heads and MLPs

• 1:35:11Speaker 00

  and connections between them, including, in fact, the individual like query and key and value parts that are like the inputs to attention heads, for example. We were able to isolate those. And I think that this is roughly mirroring the pace of progress of people's interpretability projects. Because after all, it is just like last year that I was like fortunate to work with collaborators on the I O Y or interpretability in the wild paper, which was the first work that was able to reverse engineer a circuit inside G P T two small.

• 1:35:54Speaker 00

  And then this year, we now have the greater than circuits in G P T two small, for example. So, yeah, and these works are both at this attention head and MLP level, which is like the only like the point at which we can do experiments on G P T two small with A C D C. I think it would still be too slow because there would be too many connections if you were looking at individual neurons. But the existing interpretability research hasn't really been able to understand these G P T two models on neuron level. So, yeah, it's a sort of a bad news and good news there, I suppose, for like understanding G P T two neurons with A C D C.

• 1:36:35Nathan Labenz?

  Even keeping that level of abstraction.

• 1:36:39Nathan Labenz?

  You know, where the focus is on the attention heads and the and the MLP blocks. If you were to do, try to take the leap, you know, a thousand X parameters, right? From whatever order of magnitude, a hundred million on GPT two small to order of magnitude, a hundred billion on WAM two or GPT three or what have you. How does the compute requirement of this process scale? Does it go as like the square of the increase? Like, does a thousandfold increase in parameters end up being like a millionfold increase in compute? Or?

• 1:37:15Speaker 00

  Yeah, I think it's on average slightly worse than like a scaling with the square of the number of like nodes or parameters you're introducing. Obviously, nodes and parameters not the same, but this is because as you increase like the number of nodes by a factor of two, say, you're roughly increasing the number of edges in the graph by a factor of four, because these networks are highly connected. Like, it turns out that your layer zero heads have an impact on almost all downstream layer heads. And so as you increase the like, yeah, the number of nodes by a factor of two, you roughly increase the number of edges by a factor of four.

• 1:38:00Speaker 00

  And because this algorithm is iterative over each of the edges, this leads to like, yeah, like the quadratic increase. And then like your whole forward pass is now more expensive as well, because you're dealing with a bigger model, which accounts for something on top of the quadratic cost of more iterations. But I don't think would give you, I mean, like, yeah, it turns out that the process of just iterating over each edge is the slow part of this network, like the bottleneck, rather than forward pass cost. So that turns out to be the biggest bottleneck.

• 1:38:34Nathan Labenz?

  In practice, how easy have you guys made this today? You know, for somebody who obviously has a an interest in the subject, but, you know, I have, I definitely feel like I have some weaknesses when it comes to. Notation and you know, I'm not the greatest at managing like a ton of indexes. You know, there's a lot of indexes to manage when you're doing this kind of work. Is the work that you've put out like developed enough where somebody like me can actually get in there and do it? Or like how, how much of a burden still remains for, you know, the kind of casual investigator to get in and start to figure some stuff out?

• 1:39:12Speaker 00

  Yeah, we tried to make our library like well-written for like all practitioners. By like building it on top of a really great mechanistic interpretability library called a Transformer Lens. So a Transformer Lens, which you can like find on GitHub, is a library which makes mechanistic interpretability of generative language models far easier than the default implementations in like a hugging face, for example, or in online tutorials. And so it was originally developed by Neil Nanda, who again has like helped a lot with making this mechanistic interpretability field

• 1:39:57Speaker 00

  easy to skill up in and have a bunch of like tractable research directions. So it's thanks to him for this sort of resource. But now the ACDC library can load any of the models that are in the Transformer Lens library as a computational graph with all the connections between the like nodes as different edges. So this includes like the whole GPT-2 line of models as well as a bunch of the like smaller toy language models that have been found by like Eluder AI and the Pythia models, for example,

• 1:40:35Speaker 00

  and some of a bunch of toy models that have different activation functions, such as the like the GELU and SOLO activation function. So out of the box, you can use this thing with a ton of language models that are available for mechanistic interpretability researchers in the like Transformer Lens library. And this includes like models like the LAMM models, but we think that probably ACDC will be a little bit too slow on these larger models for now. And so like we're excited to see future work that can scale it up.

• 1:41:06Nathan Labenz?

  If all goes according to plan, we're going to presumably see people starting to isolate a lot of subgraphs. What would you say then is the state of our ability to actually make sense of these subgraphs? Is that, my understanding is that's remains like a very artisanal sort of process and kind of has its own workflow of like trying to figure out what algorithm this, you know, really instantiates and, you know, is it something that constitutes understanding? All that is kind of in the eye of the beholder. There's these various techniques around like editing and looking for behavioral change.

• 1:41:48Nathan Labenz?

  There's like probes, you know, to kind of try to figure out what internal states, you know, actually map on to, to real world states that maybe the model didn't even necessarily see in its training data. But how, how in like in general are these things like, do we get to a satisfying conclusion for most of these subgraphs that we identify or not so much?

• 1:42:09Speaker 00

  Yeah, I think this is mostly like an open question and I'm excited to see work that provides evidence like either way for hopefully the interpretability of just these like just raw subgraphs of huge numbers of attention heads and MLPs or that like they would be somehow misleading or confusing and not useful. This would be a useful piece of evidence that like mechanistic interpretability is hard. And like so far, the use case that we found with like a practitioner who used this to find out whether like the GPT two small model was able to produce completions that were

• 1:42:48Speaker 00

  like the correct gender of different like names in the sentence or expected gender. So it would turn like ordinary names of women into like the she pronoun. So this was just like the bias essentially of the model to expect that as a completion and ordinary male pronouns, male names rather into he. So again, like a bias of the model to produce that completion then like reveal this structure. The model was like aggregating information about this like name on like this surprising next token to the name. So the model would just like take the name information and then move that to like the next token in the residual stream, so a different residual stream.

• 1:43:33Speaker 00

  And then that would be what we like, would be like funneled downstream into like the like. Uh, normal, like a, a gender completion, uh, like the he or she completion that was expected based on the, like, the biases in the training data. And so this was a case where, uh, the, like, the H D C algorithm could have gave, like, a confusing mess as to, like, how the model did this particular pronoun completion, but actually was, like, fairly interpretable. Oh, it was aggregating information on this, like, position that was the, like, uh, the, the token after the name, and the researcher could clearly see that through, like, a bunch of the MLPs, and then could, like, make that conclusion, which was certainly,

• 1:44:14Speaker 00

  like, a, a non-trivial conclusion, and would have taken a long time to find by, like, hands, since it's like a computation which occurs, like, in the internals of the model, like, it's not a function of the inputs, and it's not like something which is a function of the outputs. It's just this internal position which matters a ton. And so, in the limited examples we have so far, it turned out to be a pretty easy process. But I expect there are definitely cases where it's much harder, and I'd like to see, like, further evidence, whether it's in general a lot easier or in general still quite hard.

• 1:44:47Nathan Labenz?

  Fascinating. I'm trying to kind of envision that, and I certainly appreciate the importance of these internal states, which, you know, some might be bold enough to call emergent, uh, properties or emergent behaviors. Do you have a take on the, I guess I'll just call it the emergence discourse?

• 1:45:06Speaker 00

  Yeah, on emergence, it's definitely a concept which is, um, attractive to talk about because of its, like, connection to unpredictability and the, uh, like, longer term worries that different AI, like, or new AI systems will be qualitatively, like, different from current AI systems. And I do think that it is, however, often a question of which metric you choose to measure your property under. So, abilities of large language models often seem emergent when we look at token completions that our

• 1:45:50Speaker 00

  Like billion parameter model, for example, can suddenly do three digit addition. Like we give it three digit addition sums and it can now like suddenly be able to like generally produce the correct answer. Whereas like the hundred million parameter models can produce just like rubbish on the same outputs. And this feels to us like something that's emergent because like suddenly the model's great at this and previously it was like absolutely hopeless. But often when you like hear these statistics or read these papers about the emergent capabilities of models. They're solely looking at a particular metric, like in this case, the probability that the model gives the correct like addition completion.

• 1:46:33Speaker 00

  And actually, language models and models generally are trained on the logarithm of the probability that the model gives to certain completions. And so often follow up research has found, like it's been called like a mirage in one paper. But once you're looking at like the logarithm of the probability that the model gets the correct three digit addition sum correct, for example. Then progress looks really smooth and it just increases gradually in like the log of the number of parameters of the model. But it just happens to be that like exponential growth is extremely fast.

• 1:47:14Speaker 00

  And so at one moment, you're at like one percent likelihood of producing the correct addition sum. And then suddenly you're like timesing by like fifty or whatever and we're at fifty percent. And this looked like it was a qualitative change and came out of nowhere. But really you were just staring at like the wrong metric. And so my broad take is that for now, we are not very good at like finding the right metrics to measure models under. And so we resort to just looking at their outputs and sampling what happens. Even the like best evaluations that exist, the like team from the Alignment Research Centre who found a bunch of like somewhat dangerous capabilities of G T four still.

• 1:48:00Speaker 00

  In general, we use the technique of just looking at what the model's outputs were. And we should expect that these like probabilities on completions are growing like exponentially in the sort of like scaling curve because we train on like logarithm of the probability of the completions. And so I think that currently we're likely to see more emergence, but it's mostly because we're looking at the wrong metrics. And I'm certainly excited about digging deeper into the internals of models through interpretability or other methods. Because of the fact that by default, I expect we'll see emergence, but we could do so much better.

• 1:48:38Nathan Labenz?

  Yeah, that's a fascinating, I've been kicking this question of emergence around from a bunch of different angles. As well and trying to just figure out first of all, like what matters and I guess. One way to maybe think about what matters, tell me if you see this differently is. Just asking like, how practical is it to? Zoom in on these things in in the process of training. Cause I guess my intuition is that. You know, what really matters for like users, society, companies, you know, is. At the end of a training process, what can a model do or not do and how general is that capability and you know?

• 1:49:21Nathan Labenz?

  Is it grogged in some way that reflects a meaningful understanding or is it still stochastic fair? That seems like the key thing that like matters most. It does seem to be true that, you know, per that Mirage paper. That if you find one of these kind of surprises. And then rewind and say, well, okay, let me actually measure that performance at every. Every, you know, increment of the training process. Then you can plot like a smoother curve and it seems like there's this kind of. Phase change that is often happening between like a correlation paradigm and a more algorithmic

• 1:50:02Nathan Labenz?

  paradigm and those are kind of, you know, one is is dropping while the other is rising and that does from what I've seen often take like, like an order of magnitude more training to to make that phase change or sometimes even more perhaps. But it seems like it's still going to be really hard. You know, if you're if you're training a system like GPT four. First of all, you don't know what list of as you're training it, you don't know what things will like emerge that you could then later come back and plot a smooth curve on. So you're going to have a hard time knowing what things to even look at, you know, incrementally along the way. And then just like the compute tax of that, you know, if you were to say every batch, I want to like run a million diagnostics and, you know,

• 1:50:42Nathan Labenz?

  kind of benchmark a thousand, you know, a million things, whatever, at every step, like that becomes a massive overhead. So I kind of look at that Mirage paper and I'm like, you definitely found something that is quite helpful to understanding what's going on in the training process. But from the standpoint of like society or even model developers, it doesn't feel like that allows us to get around this problem of we don't know what's going to come out of the model at the end of a big training run, or at least not without like a significant overhead imposed on the training process. Would you challenge anything there or correct me on anything?

• 1:51:24Speaker 00

  No, I definitely agree that it seems an extremely difficult problem to predict what are essentially like unknown, unknown capabilities that we don't, like, we don't just don't know how far training on predicting next words and then maybe being RLHF fine tuned on top of that gets us in the limit. How many capabilities does this actually get us? Will this solve like? Will this be like competitive with the best humans at maths, for example, or will it never reach anywhere close to like a like a graduate maths student?

• 1:52:01Speaker 00

  I don't know what the answer is here. And so therefore I agree that and there are just like so many other tasks in this like a ballpark like that that plausibly could emerge or plausibly can't. We don't, we just don't know where they'll come from. But I think that I I'm more concerned about mostly known unknowns in the evaluation space of different evaluations of things. Models can do, as like an example, a lot of AI safety research has established like that there are often like convergent instrumental goals that models will have.

• 1:52:40Speaker 00

  So if the end's training target involves one of a huge number of objectives, it is useful for the models to gain like power or resources to achieve those goals. Because power and resources are just like very helpful for a huge number of things that the model could want to do, such as convincing people to send certain messages on the internet or acquiring certain like like objects on the internet or something. You would like to have more money and more influence to get those things.

• 1:53:17Speaker 00

  And so my take here is that we don't just, we don't need to look for the unknown unknown capabilities to have helpful evaluations and predictions of different model capabilities. We can sort of think about these known unknowns of like concepts which theoretically are likely to emerge through sufficient training because of the like instrumental convergence arguments. But then current models don't do very much, like seek power essentially. So once we restrict to a certain number of capabilities that seem could be like quite dangerous if we have future powerful AIs, then I hope that we can develop better evaluations

• 1:54:02Speaker 00

  to figure out how close or how far our current models are from doing these like certain dangerous like or gaining these certain dangerous capabilities that are like known purely from the theoretical angle for now.

• 1:54:16Nathan Labenz?

  This this debate kind of always comes up around interpretability, or maybe not always, but I think it's personally just fascinating work that, you know, I'm very curious about. Kind of independent of its consequences, you know, for me, it just passes the, you know, it's interesting on its own merits test. But, you know, as I mentioned at the top, to me, it feels like it's a pretty promising path to safety. It seems like you're kind of sketching out like a vision of sort of the holy grail of mechanistic interpretability, at least for safety purposes, would be to figure out how models might implement some of these most concerning behaviors and then be able

• 1:54:59Nathan Labenz?

  to detect that, the formation of those subgraphs in the training process. That would be like, you know, the, the dream scenario, right? Any, anything to add to that?

• 1:55:09Speaker 00

  Yeah, this sounds like exactly what I think of as like a speculative, but like incredibly beneficial application of like the mechanistic interpretability techniques that like I and a number of other researchers and my collaborators have worked on. So I, I agree with this characterization and I will just like point out that I'm, I'm well aware that this like problem's been sketched out where there are no theoretical dangerous capabilities that powerful AI systems could have, can definitely be approached with like other approaches to safety. We don't need to have a mechanistic understanding of AIs to be able to like hopefully steer them like away from dangerous capabilities or at least know when the dangerous capabilities are present.

• 1:55:54Speaker 00

  But it's certainly the case that mechanistic interpretability has like a uniquely, like a uniquely specific approach to like isolating and understanding those capabilities because it would hopefully be able to explain those capabilities in terms of the exact like location in models and the exact reasons why this capability emerged, rather than just like a litmus test that goes like positive or negative for whether this capability is there. So that's like the, the wider dream of interpretability with regard to like applications and safety.

• 1:56:28Nathan Labenz?

  What do you make of the argument that I, that I do sometimes hear that is like, yeah, but everything's kind of dual use and yeah, we can understand this stuff better, but also that's just going to feed into accelerating the increasing power of systems in general. And so, you know, maybe it's not so good. I don't find that super compelling. I don't really have a great knockdown reason for it, other than just. I don't know what else to do but try because it certainly seems like everything is progressing regardless, right? So I wouldn't pin, you know, the sort of potential for a runaway kind of loss of control scenario on mechanistic interpretability by any means.

• 1:57:07Speaker 00

  Yeah, I want to be careful because I guess we both have a similar opinion here. But I also like don't find the arguments or the danger of mechanistic interpretability research extremely compelling. And since we both sort of perhaps have this opinion, I don't want to misrepresent the opposite view. But to me, it seems like the vast majority of capability gains in machine learning that have been relevant to the development of the most powerful systems have not come from advances in transparency or insights about how models work.

• 1:57:47Speaker 00

  There's a great discussion of this exact question in like a alignment forum research post on pragmatic AI safety, which discusses that, like, you can just survey where capability gains to vision models in machine learning and to language models in machine learning have come from. And the vast majority have come from basically engineering hacking to find something which works slightly better than the alternatives, while no one really understands why this works slightly better than the alternatives,

• 1:58:27Speaker 00

  such as picking a loss function that's just predicting the next token that turns out to work really well at absorbing capabilities. Or in the like RLHF process to pick a reward which just chooses between zero and one. It's just like a preference between one and the other thing. To me, these things didn't like arise from a deep understanding of how to model language or how to model human preference. But as far as I understand it, arose from trying a number of alternatives. And then eventually, selection pressure leading to these being the best of the bunch.

• 1:59:09Speaker 00

  And so, under this worldview of progress in machine learning, I think that currently mechanistic interpretability is very unlikely to contribute to like the bulk of further performance improvements in like uh machine learning models. I guess that's my first um disagreements with the perspective that uh mechanistic interpretability could be dangerous for its dual use to making ever more powerful like AI systems. And then my second like uh disagreements with the perspective that mechanistic interpretability uh could be like harmful overall is that I think that mechanistic interpretability, if it works,

• 1:59:53Speaker 00

  like this is all premised under it like being useful because currently we haven't found like a stellar application to the models that matter, but we hope we can get there. This second reason that uh I think it has like a greater positive side to a negative side is that it plausibly gives us a way of like designing and understanding AI systems in a different way to the current understanding of systems, such that we could develop maybe more powerful AI systems. Like this is the the worry, but they would actually be understandable to us. We would understand how these AIs are like computing the outputs that they're like processing from inputs. And to me, this may involve more powerful AIs,

• 2:00:38Speaker 00

  but would substantially reduce the like risks of deploying these systems because a lot of the risks from like the alignments of AI systems come from being able to specify your objective and trying to get something from an AI system, but not understanding the process through which the AI system achieves that end goal. And this essentially is like the alignment problem that uh. Specifying the end state isn't enough because it either is really hard to specify that end state like as an outer alignment problem in the jargon, or the AI system learns

• 2:01:20Speaker 00

  a solution which was just totally unintended and maybe internally optimizes uh that is like this inner alignment problem, even if you chose the right goal. But to me, interpretability and mechanistic interpretability could be like a way, if it can work, to develop AI systems where we understand that middle like process between our like specification of the go- like the goal of the system and the AI system being able to like actually execute and like achieve that goal. So that's my like two reasons for being optimistic about uh the the impact of like mechanistic interpretability research.

• 2:01:58Nathan Labenz?

  What I try to envision what that might look like, you know, a a a future that sort of combines better understanding, hopefully better control, but also, you know, increasing power and, you know, maybe power per unit compute or whatever. First thing that comes to mind is kind of a mixture of experts, mixture of sparse experts sort of architecture. I'm kind of imagining something where, you know, take like the the Ziming Lu paper we've talked about a couple of times that are, you know, creating these very small but like very sparse and kind of crystalline almost looking subgraphs. And, you know, scaling that up where there's some sort of mechanism where you've got a

• 2:02:41Nathan Labenz?

  lot of those and you only use, you know, a certain number at a time. And so you can kind of see like what was, what modules were loaded in to to handle this particular case. What do those things do? It seems like that is like potentially pretty promising to me. Is that, how does that relate to your kind of obviously still somewhat vague uh vision for what might eventually come online?

• 2:03:04Speaker 00

  Yeah, I think this is an example, a nicely concrete example of like a really ambitious goal of interpretability, where like the whole architecture of like the forward pass can be understood to a human, or at least, yeah, like these high level concepts, like the whole routing to a particular expert has some like meaning to humans. And I think it's possible that we can get to this stage with mechanistic interpretability. But I think it's worth noting that even if this like fails pretty badly, it's still possible for the interpretability of narrow tasks, like

• 2:03:44Speaker 00

  the mentioned power seeking in certain scenarios, can be achieved like through like mechanistic interpretability. Like there could plausibly be a circuit which does this particular power seeking task. And having like just understood that circuit in a network, we can like, uh, understand why the training process got to this solution, or we can just remove that circuit entirely before we deploy a system. And I don't think this is like ideally what like my future looks like as in, oh, man, we have like this like misaligned system, but we'll just like remove that part and deploy it anyway. But I think that there are ways which this uh is like

• 2:04:25Speaker 00

  a graceful degradation of the like ambitious goals, having a whole architecture which makes sense, like an understanding of the dangerous capabilities. So we can at least remove those dangerous capabilities, even if we don't have an understanding of all capabilities of the model.

• 2:04:42Nathan Labenz?

  Well, you've been extremely gracious with your time. I have maybe one more question. I'll give you a chance to if you want to touch on anything else we haven't. But what are you looking at? You know, I kind of try to keep my eye on the horizon in my, you know, self proclaimed role as AI scout, you know, I'm kind of always looking for what is happening that maybe isn't being talked about a ton yet, but seems like it has kind of transformative potential. Are there things that you see right now, or, you know, are, or maybe haven't seen, but are keeping an eye out that you think could kind of? Change the game, so to speak, either to make things a lot easier, perhaps, you know, so that, you know, maybe something like the Ziming strategy becomes mainstream and things become much more easy to interpret.

• 2:05:25Nathan Labenz?

  Or on the flip side, you mentioned like recurrence, you know, makes things harder. There was just this paper in the last few days, I believe out of Microsoft Research, around retention. They propose a somewhat different mechanism, which I don't really understand yet, but they are bold enough to call it a possible successor to the transformer. Seems like, you know, there's potential here for paths of, and this may be an indication too, that the history in its particulars could end up really mattering. If you can imagine like, there may be, and it seems to me very likely that there is, or there are, multiple very viable architectures to be found, just starting with the fact that we have the human brain that works pretty well, we have the transformer that works

• 2:06:10Nathan Labenz?

  pretty well, probably other things that have been work pretty well. It seems like some of those things may be much more or much less amenable to being understood. So I wonder what you're kind of, you know, keeping your eye out for in terms of things that could, you know, kind of shake the snow globe or, you know, kind of rearrange the game board in a substantial way.

• 2:06:30Speaker 00

  Yeah, that's a really good question about like, looking forward, what are like the things that, like, I'm thinking about looking for, and I think that it's worth others having on their radar. And in terms of like, yeah, mechanistic interpretability and interpretability research, I think a common theme, which I would expect to be part of a lot of the, like, the next generation of contributions to the field, will involve more higher level motifs in language models.

• 2:07:08Speaker 00

  So we spoke a lot today about the circuit framework. We break up like a large subgraph into these like individual components like attention heads and MLPs that are given to you by the architecture. You read what a transformer is and then you learn, okay, it has attention heads and MLPs. But both myself and current work, and I've heard of a number of other groups, are working on trying to go beyond these abstractions of just the heads in the model and the MLPs in the model to look for higher level motifs in models.

• 2:07:46Speaker 00

  So to be concrete, like you can find, as like I think in current work I've been working on, that like certain behaviors such as suppressing these like negativity heads in GPT-2 small generalize to the whole distribution of training text. And you can find this motif that we call copy suppression in upcoming work that works across all the like training distribution rather than just on these narrow tasks. And it's also distributed across like several different heads rather than in like singular heads.

• 2:08:27Speaker 00

  And I know of a number of other groups who are also going beyond the sort of circuit paradigm where you explain like different model behaviors in terms of these given components like attention heads and MLPs, but aggregates different components and weights them in clever ways to build like higher level motifs that like at this point in time, there are like very few examples of in the literature. And I expect that this is like the next big phase of mechanistic interpretability research going beyond like narrow circuits and low level details to high level motifs about like how these large language models are doing computation.

• 2:09:10Speaker 00

  So yeah, I'd stay on the lookout for higher level motifs that occur across different model components or like between different model components. That's my personal current direction and what I'm excited to see other groups do work on.

• 2:09:25Nathan Labenz?

  Cool. I love it. We will certainly keep an eye out for that. You've got the motifs notion, that's kind of a different sort of thing that you're looking for. Is there also a frontier on? The automation of that or or just better automation that you think will be a a driver of a lot of value?

• 2:09:42Speaker 00

  Yeah, I think that the uh automation to find motifs or explain motifs is probably like quite a lot harder than the default path of uh just narrow circuits. But I think that uh there are a number of efforts which could be scaled up to either work with ACDC or would like go off on their own if they worked out particularly well to be able to like explain motifs. So the example would be the like OpenAI research where GPT-4 can be used to explain the neurons in GPT-2.

• 2:10:22Speaker 00

  And here, this is a useful complementary technique to like this automatic circuit discovery where we're like just finding structure because it's like by default assigning semantic meaning to different components. And I think that what using these like uh language models in particular to try and explain what different components or even different subsets of different models are doing is like an approach which would be super exciting for uh understanding how these uh like motifs are present in different models that forcibly is somewhat more difficult uh with just the pure circuit discovery approach.

• 2:11:03Nathan Labenz?

  I love it. There's so much for us to continue to explore and learn about and um you've given us a great tour of uh one corner of the world, but uh we've got a lot more work to do. So Arthur Conme, thank you for being part of the cognitive revolution.

• 2:11:18Speaker 00

  Thanks so much, Nathan. Thank you for having me on. It's been a pleasure.