Untangling Neural Network Mechanisms: Goodfire's Lee Sharkey on Parameter-based Interpretability

Today Lee Sharkey of Goodfire joins The Cognitive Revolution to discuss his research on parameter decomposition methods that break down neural networks into interpretable computational components, exploring how his team's "stochastic parameter decomposition" approach addresses the limitations of sparse autoencoders and offers new pathways for understanding, monitoring, and potentially steering AI systems at the mechanistic level. Check out our sponsors: Oracle Cloud Infrastructure, Shopify. Shownotes below brought to you by Notion AI Meeting Notes - try one month for free at ⁠⁠⁠⁠https://⁠⁠notion.com/lp/nathan Parameter vs. Activation Decomposition: Traditional interpretability methods like Sparse Autoencoders (SAEs) focus on analyzing activations, while parameter decomposition focuses on understanding the parameters themselves - the actual "algorithm" of the neural network. No "True" Decomposition: None of the decompositions (whether sparse dictionary learning or parameter decomposition) are objectively "right" because they're all attempting to discretize a fundamentally continuous object, inevitably introducing approximations. Tradeoff in Interpretability: There's a balance between reconstruction loss and causal importance - as you decompose networks more, reconstruction loss may worsen, but interpretability might improve up to a certain point. Potential Unlearning Applications: Parameter decomposition may make unlearning more straightforward than with SAEs because researchers are already working in parameter space and can directly modify vectors that perform specific functions. Function Detection vs. Input Direction: A function like "deception" might manifest in many different input directions that SAEs struggle to identify as a single concept, while parameter decomposition might better isolate such functionality. Knowledge Extraction Goal: A key aim is to extract knowledge from models by understanding how they "think," especially for tasks where models demonstrate superhuman capabilities. Sponsors: Oracle Cloud Infrastructure: Oracle Cloud Infrastructure (OCI) is the next-generation cloud that delivers better performance, faster speeds, and significantly lower costs, including up to 50% less for compute, 70% for storage, and 80% for networking. Run any workload, from infrastructure to AI, in a high-availability environment and try OCI for free with zero commitment at https://oracle.com/cognitive Shopify: Shopify powers millions of businesses worldwide, handling 10% of U. S. e-commerce. With hundreds of templates, AI tools for product descriptions, and seamless marketing campaign creation, it's like having a design studio and marketing team in one. Start your $1/month trial today at https://shopify.com/cognitive

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome back to The Cognitive Revolution. Today, I'm speaking with Lee Sharkey, principal investigator at mechanistic interpretability startup Goodfire, about fascinating recent work that he and co-authors have done to start moving beyond analysis of the concepts that neural networks represent between layers and actually begin decoding how they compute within and across layers. We begin by discussing why an understanding of concepts isn't enough. On one level, this needs no justification. An approach that explains only the meaning of the intermediate results between layers, while it does take a serious bite out of the black box problem, leaves the layers themselves as smaller but still unexplained black boxes.

• 0:42Nathan Labenz?

  But more concretely, research has also highlighted important weaknesses of the feature-centric approach. Conceptually, a sparse representation of features inherently loses a lot of potentially important information, which is encoded in the structure of how features are clustered together and otherwise meaningfully arranged in space. For example, in some networks, the days of the week are represented not by seven random directions in activation space, but by a set of directions that lie together in a plane, such that a simple rotation operation can act as a sort of next-day function that converts a given day of the week into the next. If history is any guide, there is presumably a lot more such critical complexity to be discovered.

• 1:25Nathan Labenz?

  And so with this motivation in mind, and taking inspiration from sparse autoencoder and similar techniques that effectively separate clean concepts out of their usual state of superposition, Lee and team have similarly set out to decompose a neural network's parameters, which of course are used to process layer inputs into outputs, into simpler subcomponents that they hope will correspond to interpretable mechanisms that the network has learned.

• 1:51Nathan Labenz?

  Their first approach, published earlier this year, was called attribution-based parameter decomposition. Lee describes this as a sort of model unmerging. If you imagine, counterfactually, having started with a giant mixture of experts' model, and then merging all the experts into a single instance of the architecture, parameter decomposition would be the process of unmerging to recover those hypothetical original experts. Personally, I visualize it as expanding a neural network, which of course has width and depth defined by its architecture, into a third vertical dimension, where each vertical slice consists of a simpler sub-network that's presumably needed only a small fraction of the time, such that when all the vertical dimensions are again collapsed down to a single model, they all add up to the original network weights.

• 2:37Nathan Labenz?

  Whatever mental model you prefer, amazingly, by constructing a loss function that incentivizes three things: first, faithfulness, or the accuracy of the sum of the sparse networks back to the original full network; two, minimality, or the idea that as few network slices as possible should be active for any given input; and three, simplicity of the discrete sub-networks derived from the process. You can actually get this to work. I really do think it's worth taking a moment to appreciate how incredible this is. As Lee says, channeling Ilya, sometimes the models really do seem to want to learn.

• 3:13Nathan Labenz?

  That said, given the complexity of the target, you shouldn't be surprised to learn that this first method was still far from ideal. In addition to being computationally expensive and requiring massive memory for all those parameter copies, training success was extremely sensitive to hyperparameter choices, and there were conceptual issues with the gradient-based way that performance was attributed to parameters. Specifically, in some cases, critical components that had approached a local performance maximum would have a near-zero gradient and would be misclassified as unimportant.

• 3:45Nathan Labenz?

  These weaknesses naturally inspired the second method that we discussed: stochastic parameter decomposition. For efficiency, instead of describing network subcomponents as sparse instantiations of the original network architecture, the new approach breaks each weight matrix into rank-one components, which are matrix operations that read from one direction in activation space and write to another, and which can later be regrouped into larger units. Also, to better identify which subcomponents really matter, they replaced the gradient-based approach with a novel approach that uses stochastic masking to help the network learn to predict each component's causal importance, effectively identifying which elements the overall network can't do without.

• 4:28Nathan Labenz?

  Overall, it's a more scalable, stable, and accurate approach, and on toy problems with known ground-truth answers, it is able to successfully recover the expected mechanisms. Of course, as with any branch of interpretability, plenty of work remains. One big question is how can we algorithmically group the rank-one subcomponents into semantically meaningful mechanisms? Much as sparse auto-encoders require a feature labeling process, we still need to figure out how the rank-one components ladder up to conceptually intuitive transformations. And of course, there will be plenty more challenges with scale. But as Lee explains, the promise of this work is tremendous. This kind of understanding could enable everything from surgical unlearning of capabilities

• 5:12Nathan Labenz?

  to the identification and interpretation of novel scientific insights that models may have learned, including and perhaps especially from non-language data. All things considered, this conversation is a great chance to visualize what's going on inside neural networks, to ponder how much we've learned and how little that still seems to explain, to marvel at the fact that gradient descent and backpropagation can effectively optimize such complex architectures even under such complicated multipart constraints, and to appreciate the combination of brilliance and plain hard work that goes into figuring all this out.

• 5:47Nathan Labenz?

  Right up there with understanding our own biology and arguably no less important or challenging, understanding how and why neural networks do what they do is one of the grand scientific challenges of our times. So I really hope you enjoy this conversation about decomposing neural networks in parameter space with Lee Sharkey of Goodfire. Lee Sharkey, Principal Investigator at mechanistic interpretability startup Goodfire. Welcome to the cognitive revolution.

• 6:14Speaker 02

  It's great to be here, Nathan. Thanks so much. A huge fan of the show.

• 6:17Nathan Labenz?

  Thank you. That's an honor. I'm excited for this conversation. I always love to learn and I expect to learn a lot over the next hour and a half or so. Real briefly, before we get into the technical work that's going to be the primary focus, you've moved recently. We've had two guests from your previous organization, Apollo Research, on the podcast over time, focusing on risks of deception and identification of deceptive behaviors in frontier models. And we've also had two different episodes with folks from Goodfire. Dan and Tom have done a great job orienting us to everything that's going on in mechanistic interpretability. You were doing MechInterp at Apollo and recently

• 7:00Nathan Labenz?

  moved to Goodfire. So maybe give us a little bit of an update on how that came about and then we'll get into the work itself.

• 7:07Speaker 02

  Yeah, absolutely. So yeah, Apollo Research and I, you know, I started Apollo Research with a bunch of folks, including Marius. Marius Hoban, who mentioned you had on. And yeah, you know, we started and we were focused on detecting and ideally mitigating deceptive behaviors in frontier AI systems. One of the goals there for my team was to focus on the mechanistic interpretability side of that, where if we can, you know, read the thoughts, so to speak, of these frontier AI systems, and maybe that will give us a leg up on detecting deceptive behaviors without necessarily needing to rely

• 7:47Speaker 02

  on their outputs. And yeah, I think we did, you know, like, I was really happy with the work we got done there and like really, you know, love the organization we built and like really, you know,

• 8:01Speaker 02

  Continue to support the the work that Apollo does. I continue to be somewhat involved there. But yeah, we figured that it was probably better at this time to for Apollo to really double down on the evals side of things, which left, you know, my team, which primarily focused on mechanistic interpretability rather than evals, you know, a bit in the lurch. And so we, but you know, we all agreed that this was the right step. And meanwhile, you know,

• 8:32Speaker 02

  say, maybe six months before we actually decided to make this decision. I had, it seems like engine history now, but I ended up connecting Tom Gra and Eric Ho, who, you know, went on to to co-found Goodfire together. And so I had a, you know, a B character role to play in that, in the founding story of of Goodfire. And so, you know, I was obviously very aware of Goodfire and, you know, massively supported the work they're doing. I think, you know, the team is absolutely amazing. And it just felt like a really natural fit to to move over from Apollo to to Goodfire. And it made sense for, yes, some of the team to come along as well.

• 9:15Speaker 02

  So we can basically continue the stuff that we were working on there.

• 9:19Nathan Labenz?

  Was it three of you that made the move?

• 9:21Speaker 02

  Ah, yeah. So me, Dan Braun and Lucius Bushnak.

• 9:25Nathan Labenz?

  Gotcha. Cool. Well, that's great. I, and I assume the fifty million dollars Goodfire raised to

• 9:35Nathan Labenz?

  support both team and compute certainly didn't hurt the value proposition either, right? And we'll get into a bit more, you know, where the compute costs are for some of this work, but everything seems to be, you know, dependent on a healthy dose of compute. So the ability to draw those resources in from the private sector, you know, definitely as opposed to, you know, Apollo being philanthropically funded. Also, it seems like it could be a big differentiator long term for the ability to scale what you're doing.

• 10:07Speaker 02

  Yep, 100%.

• 10:09Nathan Labenz?

  Cool. Well, we're going to, I think, primarily focus on two papers today, one of which came out still under the Apollo banner and the more recent one under the Goodfire header. First one is called Interpretability in Parameter Space: Minimizing Mechanistic Description Length with Attribution-Based Parameter Decomposition. I think there's a, you know, try not to retread well-covered ground too much, so I definitely would refer listeners who want to go deeper on that particular paper and have a really thorough exploration of it to the AXRP podcast that you did not too long ago. I thought Daniel did a great job there and I listened to the full thing to help inform myself coming into this. I

• 10:53Nathan Labenz?

  guess it seems like, maybe I'll just ask you the first question. What is attribution-based parameter decomposition, or maybe even more narrowly, what is parameter decomposition? And, you know, how does it differ from the schools of interpretability that people are probably familiar with? And you could probably assume people, if they've listened to our feed much, at least are aware of the basics of SAEs.

• 11:16Speaker 02

  Yeah, so what is parameter decomposition then? Well, let's contrast it then with activation-based decomposition. So in interpretability, typically in the past, what we've done is we've, you know, wanted to understand what is going on inside a neural network. And so what we've done is we've collected, you know, lots of activations of how the data, how the the the network has processed its input data. And, you know, you've got these intermediate activations and then it spits out an output. And what we've typically done is we started with the idea that, well, these activations should, in some sense, be involved

• 11:56Speaker 02

  in different things that the network is doing in order to compute its, you know, intelligent output. And there is some sort of like learned algorithm that the network has has has learned, basically, to, you know, exhibit its intelligent behavior. And activation decomposition, basically, is the idea that we can look at these intermediate activations, piece them apart, and say, this part does this and this part does that. By contrast, parameter decomposition, it is philosophically very similarly motivated. But instead of picking apart the activations, the idea here is to pick apart the parameters.

• 12:39Speaker 02

  Now, why might you want to look at the parameters? Well, in some sense, the, you know, the thing that we're really interested in is, you know, the neural network. And the neural network is not its, its input data, although it interacts in very special ways with it. But the neural network is in some sense, you know, the parameters, the architecture that pieces them together, the nonlinearities that connect them. And it uses all these components to transform the input data into activations and transform those activations until it gets to the output. And in some sense, then, like, the parameters and the architecture and the nonlinearities are the thing that we really want to understand.

• 13:22Speaker 02

  They are the, like, they are the algorithm that, they are implementing the algorithm that the network has learned. And the, you know, activations in some sense are, in some sense, along for the ride. But they do interact in very particular ways where it's not quite as separable as that. But the basic idea here then is that the network is using its parameters in different ways for different inputs. And for, you know, an input where there is a cat, it will take these inputs and spit out a, you know, a cat label, say.

• 14:05Speaker 02

  And for another input, maybe, you know, a picture of the Eiffel Tower. It may not use very many of the same parts of the network as it did when looking at a cat. And so in some sense, there's this idea of modularity. There's like a, parts of the network are doing one thing, other parts are doing some other thing. There's like a specialization in what the parameters are doing. And we don't really, you know, the aim for a parameter decomposition is to find these, like, these modules that are doing specific, specific jobs, specific computations. And the term that we're, that we, you know, say we're looking for here is we're looking for the mechanisms, like the mechanisms that the network is using in order to compute its behavior.

• 14:50Nathan Labenz?

  Sort of just flesh out that cat example a little bit more.

• 14:56Nathan Labenz?

  And to say some really basic stuff, all these neural networks are composed of layers. Often the layers are the same, you know, exact structure repeated over and over again, although not always. We've observed by doing interpretability on the activations that we can see things like fur has been detected and eyes have been detected and a tail has been detected. And we sort of see this like gradual move from low level features to higher order features as we go through the layers. And so there's this story of kind of understanding, it kind of mirrors, especially in these visual

• 15:37Nathan Labenz?

  cases, kind of mirrors what I understand to also be happening in the human visual system. And so that's a remarkable coincidence. But when we are purely looking at the results of the intermediate calculations, we can sort of say what concepts are active at any particular time with lots of caveats referred to earlier episode with Dan and Tom about the sort of philosophical gaps between like the labels of these concepts and the underlying sort of, you know, what exactly is is being activated there and how does it relate to the labels is a bit fraught. But nevertheless, bracketing that for the moment, we have these sort of labeled concepts that we can say, okay, this seems to, this feature always lights up for all these inputs,

• 16:22Nathan Labenz?

  all these inputs, you know, seem to be fur. And so that seems to be the fur concept that's activated. When that and the and the tail and the, you know, whiskers and the eyes are all in the pointy ears or whatever are all activated, then in, you know, future layers, we get the cat activated. But we're not saying anything there about how the transformations are happening from layer to layer, right? That's the big gap that has been sort of left by the sparse autoencoder work for future work. And and this parameter decomposition is basically is that future work that is now saying, okay, how do we actually move from these concepts through the layers, one to another?

• 17:05Speaker 02

  I agree with that. I'll say also that it's not the only approach that you might consider using in order to piece together how these representations at one part of the network may become other representations at other parts. There are other approaches that remain in the activation space that you might consider also using. But yeah, they both try and achieve this idea of characterizing the computation between representations rather than just identifying the representations themselves. Maybe one way of thinking about this is identifying the variables used in computations rather than identifying the

• 17:49Speaker 02

  computations. And parameter decomposition and other approaches aim to find the computations rather than the variables.

• 17:57Nathan Labenz?

  So why do we need that? Why isn't it enough to say, well, we've got concepts A, B, and C in layer one, and then in layer two, we see concept D? And isn't that like, doesn't that tell us what the neural network is thinking, so to speak? What is not answered by that level of analysis that we still need to get clarity on?

• 18:19Speaker 02

  It gets somewhat philosophical. However, I'll try to convey how I think about this. I would say that we don't... What does it mean for the network to have, to use particular variables? We can go in and we can say, look, there's this set of activations, and this set of activations seems to correspond to data points on which there are cats. But maybe we can also find groups of inputs where there's a cat, but it's a cat standing in a particular position. And there's, you know, one group of your inputs is a cat standing

• 19:04Speaker 02

  in one position and another in another position. Now...

• 19:08Speaker 02

  Does the network use the fact that there are two different positions of these cats? Or does it just have a cat variable? Like, what would it mean to be able to distinguish between a network that just has a cat variable and a network that has multiple different cat variables? My proposition here is that what it means for a network to use particular variables is that these variables are like the variables that the network does its computations over. Now, it may be the case that this network that had multiple cat in different positions variables actually might not use those things.

• 19:53Speaker 02

  They all just, to the network, they're all just cats. But you may nevertheless be able to look really in detail at the data set and find lots of little distinctions within the activations. But the network might not itself actually use these. In some sense, we want some way to identify the set of things that the network is using. And to do this, then, in some sense, you might say, well, this is putting computations first and representations second, rather than putting representations first and then computations second. This is, it's like, yeah, it's like saying the fundamental, so to speak, variables that

• 20:36Speaker 02

  the network is using are those over which it does computation. And we can maybe find things that, I don't know, the network seems to represent. But there's not a particular reason to say that this is a feature that the network uses just because we can find it in there. Does that make some sense?

• 20:56Nathan Labenz?

  Yes, I think so, though, as you said, it does get a little philosophical. I was also thinking of the finding of multidimensional features as sort of another window into why the assumptions that underlie the SAE paradigm may need to be elaborated and may need a little richer treatment. You want to tackle that as well?

• 21:25Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 21:29Speaker 02

  Sure, yeah, so like, I guess,

• 21:33Speaker 02

  since, you know, the, some of the earlier, like, SAE work in the latest, you know, bout of sparse coding work in, in the deep learning space, which kind of maybe started late 2022 and kind of picked up in 2023. You know, some of the observations that people had since that time were things like, well, it looks like the representations that we're finding seem to be spread across multiple different layers. And it's kind of confusing to think about what that might mean. It's, there are also observations that these things that we were finding, well, they, they were, they're directions in activation space.

• 22:18Speaker 02

  They're, they're, you know, particular patterns of neural activation. But that didn't seem to be the whole story. There seemed to be some sort of like multidimensionality to them. It wasn't just one direction. It was, you know, a, perhaps a plane of directions or even, you know, even higher dimensional spaces potentially.

• 22:40Speaker 02

  And it was kind of unclear how to think about,

• 22:44Speaker 02

  you know, these kind of higher dimensional features, so to speak, as,

• 22:50Speaker 02

  like, it just was philosophically somewhat confusing to, to think about this.

• 22:56Speaker 02

  And one of the reasons that, it was indeed, these, these were several of the reasons that we, that motivated some of our, you know, some of the, the thinking that led to parameter, our primary, sorry, attribution based parameter decomposition, which was that, well, what if, say, a multidimensional feature just is an input variable to a multidimensional computation? And that's indeed why the network has bothered to go the length to structure its representations in this, in this way, like. Without that, without there being some sort of multidimensional computation applied to it, you might ask, well, wow, like, why did the why did the network bother

• 23:39Speaker 02

  to to structure its activations in this like, really, you know, ordered ordered way? There must be like some something for which it is using that order. And this was why we were thinking, well, maybe it's the computations that are giving this that are kind of defining the structure of these of these features. And maybe then the same kind of thing might apply in for the multi-layer case. You might think, well, it doesn't for certain for certain networks like transformers, which use a residual stream, it's very easy for computations to be spread over multiple layers

• 24:19Speaker 02

  because it's, yeah, it just works out kind of straightforwardly for them to do this. And it was kind of confusing to to think about this in terms of SAEs as well. And it kind of just made some natural sense to think about, well, maybe it's the same feature if it interacts with like a a part of the parameter vector that happens to span multiple layers. And for this kind of edge case as well, this multi-layer edge case, it just made a bit more sense to think about the computations as well, rather than representations as as, you know, primal.

• 25:01Nathan Labenz?

  And so an example of that is like, the days of the week, or the, I think, the original cracking result with modular addition would probably fall under the setting where there's a, it's not like there is a, I guess, you know, it could vary, obviously, across different networks, but at least in some settings, it has been found that it's not like there are seven days of the week features that light up independently and are all like, you know, if this is a really strong Monday signal or a really strong Tuesday signal, but rather there seems to be a plane in space through which the day of the week kind of

• 25:44Nathan Labenz?

  rotates and remarkably like seems to sort of crystallize into this like, you know, the graphics on this are like pretty amazing sometimes where it's like you pretty much divided the circle into seven equal shares and

• 25:59Nathan Labenz?

  Each day of the week has a different direction in this plane as opposed to, you know, each of them having their own independent linear directions. And basically, I think the same mechanism seems to have, or a similar, similar structure, if not the same mechanism, has been found in the Grokking result where to do this modular addition, things are first kind of translated to a angle, then the angles are added trigonometrically, and then the result of that trigonometric calculation gets mapped back on to the final number. And that is seemingly like hard to, maybe you can give a little more technical intuition

• 26:44Nathan Labenz?

  for like, why that would be really hard to figure out just with features, but it seems like it's, on some level, it may just be that, I guess I don't know exactly what you would find if you applied the standard SAE methodology to one of those things, because if I'm not oversimplifying the standard SAE methodology, you would have, you would be expecting to see these like seven distinct features, or, you know, in the modular addition case, you'd expect to see like, I'm not even sure exactly what you'd expect to see, but I don't know that you would expect to see a sort of a cyclic trigonometric structure if you were looking for all independent features.

• 27:23Speaker 02

  Yeah, I think it's probably easiest to think about the days of the week case. It, so, to be clear, it can, it can be both, that there is both a, like a multi-dimensional aspect to these like representations, as well as individual, individual days aspects to these representations. And the reason is that

• 27:52Speaker 02

  Sometimes you will, you might want to, um, in some tasks, you might want to say, well, the day after Tuesday is, and you, you might just apply the, uh, the computation that like, you know, rotates the day of the week, uh, feature, because now you've just got one mechanism that, you know, you can apply to every single day of the week. But, um, you might just, in, in some, uh, settings, um, you might really just want one day, uh, you might just say, I was on my way to the shop on, and there's really just, you know, a one direction that you wanna, wanna say here, say it's Monday.

• 28:33Speaker 02

  Um, and so in some settings, you may just want, uh, computations that apply to these individual, um, days, and some settings where you might want this, like, uh, higher dimensional computation, this kind of two dimensional one that might, uh, rotate it around. Um, and this is, I think, you know, an important, um, aspect of, of parameter decomposition. There's not like one basis, uh, that you might use to interpret, you know, the, the whole model. And it's, it's more that there are steps in an algorithm. And, you know, sometimes, uh, these steps will, um, yeah, I guess just sometimes these, some steps will be useful and others will not.

• 29:17Speaker 02

  And, yeah, I guess, let's see. So the SAE and the SAE kind of approach might find, uh, might, might not find this like shared, um, you know, uh, shared computation that just rotates, um, from, you know, one day to the next. It might find, um, you know, conditioned on prompts that, um, say the day after, or, you know, the fol- the day following, or the day, so on, um, following Tuesday is, uh, you, you might find such, you, you might find like one for Tuesday, one for Wednesday, one for Thursday, just to, to rotate around every single time. Um,

• 30:00Speaker 02

  This is just because, you know, it's kind of trying to find sparsely activating parts of the data set that have a particular shared property.

• 30:13Speaker 02

  And in particular, parts of the data set that have a particular shared like direction in activation space, which is not necessarily, you know, the all the all the ways in which inputs might share properties. One of the ways in which they might share properties is they use similar mechanisms. And these mechanisms may do the same operation, but to different inputs, depending on what the what the prompt is. I'm not sure if that was was clear, but yeah, this is one of the differences you might expect from parameter decomposition and an essay like approach. Now, this is not, there are other approaches

• 30:55Speaker 02

  that might get closer to what you might expect or want to find with a an approach that identifies the computations. For instance, the Anthropic recently released a an update on MOLT. I will need to remind myself what that stands for, but it's, you know, there's a similar principle here where it's instead of looking at individual directions, it's looking at more like multi dimensional computations.

• 31:25Nathan Labenz?

  Yeah, so I guess maybe another way, a very simple way to say it is we want to also understand the functions. Like if the whole, the whole project here is to understand why the AIs do what they do and possibly be able to intervene in in certain ways. We're just going to have to dig in and figure out the nature of the transformations as well as the intermediate results. And the intermediate results can shed some light on that. But at times, there may be assumptions that in that approach that leave us blind to.

• 32:05Nathan Labenz?

  I thought a key point you said was functions that are reused on different inputs and do the same semantically meaningful transformation on a given input. That is something that the SAE approach could be entirely blind to, but which obviously like is a is a pretty important aspect and potentially happens a whole lot. Like, I guess we don't really know yet how much that may happen, but certainly it seems like it probably should happen a lot, right? I mean, the higher, especially as we get to more and more powerful networks, it would seem like we are moving away from, or graduating perhaps in a sense, from sort of simple

• 32:50Nathan Labenz?

  rule-based like combinations of distinct features and probably toward higher and higher level abstractions, which would presumably then correspond to more functions that can operate over over ranges of inputs and do, you know, do the same useful transformation to some, you know, some space of inputs.

• 33:15Speaker 02

  Yep.

• 33:16Nathan Labenz?

  In a way, that's like, that's chunking reality, right? And we, that's like an important aspect of of chunking reality that it seems like the SAEs are sort of hinting at, but not really directly characterizing.

• 33:27Speaker 02

  This feels reasonable, yeah. I think one of the one of the cases that I like to think about is in an SAE or say a a transcoder, which is an SAE that takes as input, you know, some activation and spits out a prediction of what the the activation will be at the next layer. Both of these will have a, you know, a problem where suppose the network is just doing a simple transformation, suppose it's a rotation, or suppose indeed that it's actually just an identity transformation.

• 34:01Speaker 02

  You might get, you're going to have to spend a lot of, a lot of representational capacity to represent the computations that the network is doing at this particular transformation. Why? Because, well, you've got, say, you know, a million different features and there's a million different features in the output, especially, you know, in the case of the identity transformation, there's going to be a million there. And you would need one, one, you know, input feature and one output feature for each, each such, you know, input-output pair. Whereas, in essence, what the network is doing is actually very simple. It's just a transformation of this, of this type, whether it's an identity or a rotation,

• 34:45Speaker 02

  and you've just got one object doing that, that transformation. That's the kind of thing that we want parameter decomposition to be able to find if it is, if it is indeed the case that networks are doing that kind of thing.

• 34:58Nathan Labenz?

  Yeah. Okay. So we've got some examples of that coming up later.

• 35:02Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 35:06Nathan Labenz?

  So, let's break that for a second and maybe just talk about sort of how it works and the sort of intuition for how, how the whole setup works. There, I think there are a lot of similarities to SAEs. I mean, the, I'll do the quick recap of the SAE. So we, to do an SAE, you insert this very wide layer between two layers of a network, the thinking being that we know that the, the normal width of a network is way smaller than the number of concepts that it can handle. So clearly, there's some, you know, superposition of concepts, the concepts, it's not like one neuron corresponds to one concept.

• 35:48Nathan Labenz?

  It's sort of any direction in this four or eight or 16 or however many thousand dimensional space, any direction in that space can represent a concept.

• 36:00Nathan Labenz?

  And so the goal is to untangle those into a sparse thing. And for that, you need this really, really wide layer. So you train through this reconstruction loss where the goal is to project out into this really wide space, but have a sparsity term in the loss function so that, you know, some small, reasonable number of truly relevant features are activating and then project back into the dense space so that you are recovering the behavior of the model, but then can look at, you know, which things are activated. And then you go through this labeling process at the end of that to say, okay, here are all the inputs that led to this particular thing lighting up, like what does that in fact seem to be? And with that, we now have this thing that we can use, you know, hopefully for monitoring,

• 36:43Nathan Labenz?

  detection of, you know, potentially bad concepts being activated, potentially steering, that's where Golden Gate Claude comes from, et cetera, et cetera. It seems like there's a pretty similar motivation here saying like, okay, if concepts are densely packed in this superposition way at the activation, you know, as a result of the intermediate calculations, which we call activations, then perhaps the same is true about the computations themselves.

• 37:16Nathan Labenz?

  The network itself represents or combines a huge number of probably much simpler, more, you know, semantically sort of intuitive calculations, computations, transformations, functions, whatever the exactly right word is there. And because they're all so densely packed and overlapping and the sparsity of the data set also is a key thing here, right? Because like the reason that this can work is because certain concepts that may be, you know, pointing in very similar directions almost never occur together.

• 37:59Nathan Labenz?

  And so even though they may look very similar, since they don't occur together almost ever, like that's fine, you know, the network can sort of get away with that. And there's probably that sort of happening in computation space as well. And so the same question kind of applies. Can we break this up into a really wide version of it that separates the?

• 38:24Nathan Labenz?

  flattened layer or the flattened network that we originally trained into all these more atomic units of computation. And if we can do that, then does it appear that they actually have intuitive meaning so we can start to go through them, you know, painstakingly and understand what each one is doing. How am I doing there in terms of just creating the the motivation, anything I'm missing or anything you would correct or complicate?

• 38:58Speaker 02

  Um, there's plenty how it complicate, but I think it's a really good, you know, intro like, yeah, I think,

• 39:05Speaker 02

  yeah, I think that's just the right way of thinking about it, you know, we think that there are many different computations going on inside this network, more indeed than it has neurons. And the way it might,

• 39:20Speaker 02

  the way we think networks can do this is that they, you know, spread their computations out over their like computational units, where these computational units are the neurons. And so this by spreading it out, you kind of get some nice properties such as, you know, avoid, even though you might overlap with other computations and because you've spread this out, you can kind of suppress this noise that comes from this overlap. And you don't necessarily need to,

• 39:56Speaker 02

  you can basically fit more in because you're you're kind of like silencing the stuff that you don't really want to be there, the kind of computations you don't really want to be active right now. It's it's very similar principles to the idea of like representation and supervision. But yeah, just brought into computational space.

• 40:16Speaker 02

  And yeah, I think the other thing that you mentioned, yeah, it's kind of like, there's this. Extra layer to what the the network is is doing. We've kind of split it out into this.

• 40:30Speaker 02

  It's not just, you know, we don't just have a a layers and width dimension anymore. We kind of have this this extra dimension where, you know, you're asking like what computation is the or what computations is the network using at this particular layer, at this particular, you know, in this particular width dimension as well. And like, this does have precedent in other areas of deep learning. A mixture of experts might is a kind of example of this where, you know, you have layers, you have individual networks, but you're there's there's kind of, you can do different computations at the same layer, at the same, you know, neuron dimension, so to speak,

• 41:12Speaker 02

  depending on which expert you use. And so there are senses in which this is kind of like, well, suppose every single network is just a big mixture of experts that's been smushed together. Can we separate out the the experts? One way I've heard it described is it's like model unmerging, where model merging is just like you've combined these two networks that do different things into one network that can do both.

• 41:42Speaker 02

  And we kind of want to to do the opposite. We want to, you know, split this network out, this network that can do many different things in out into networks that can only do one thing. Or as a smaller number of things as possible.

• 41:57Nathan Labenz?

  Yeah. So the setup is to to actually do the training of of this, to to start to actually do this splitting. I I think your work does a great job of really emphasizing the loss functions. And so the constraints that you put on this are, first of all, that the and I think there's a couple of different versions of it. We'll maybe start with the first one and you can, we can explain how the second one is new and improved and better in a couple of different key ways. But if I understand the first one right, it's basically saying, okay, let's make like a bunch of copies of the network with this essentially the same footprint to start.

• 42:43Nathan Labenz?

  And we'll have the constraint that all these different copies must sum to the original. So if you had

• 42:53Nathan Labenz?

  Whatever, a billion parameters in your original model and you decided you were going to split this into a million, you know, different subcomponents. You would then have a quadrillion parameters, which we can talk maybe about like the, you know, computational challenges and the memory requirements for things like this. But, okay, now we've got the constraint, though, that those million copies, they must add up to the original. We then want to say, okay, akin to the SAE sparsity requirement, to make this, you know,

• 43:33Nathan Labenz?

  to make what comes out of it hopefully intuitive and tractable and semantically like natural, we want to have as few of those be active as possible for any given input, which you call minimality. And then the final one is for each of those things, we also want to make them as simple as possible so that they individually are interpretable. So, again, the, we can split this into, and I sort of think of this as giving the network height. Obviously, we have, you know, width of the network is sort of how many neurons it has at each layer and depth is like how many layers it has. I think of this as sort of splitting it into a vertical height dimension where instead

• 44:16Nathan Labenz?

  of having all this stuff, you know, happening in the same, you know, track of computations in a way that we can't untangle, we're now untangling it into all these different, hopefully, atomic computational subcomponent units, only a few of which will be in use at any given time. And ideally, those can be kind of distilled to their simplest form while still, of course, reproducing the original network behavior. A visual I have of this comes from the Tegmark Group. I think it was Xingming Liu was the lead author on this paper, Seeing is Believing, where they just trained toy

• 45:02Nathan Labenz?

  Very small models on like, simple problems, but with a strong sparsity feature. And you could watch, they had these great graphics where you could watch an initially randomized network, learn a particular function, but also you could see that most of the weights would drop out to zero as the thing sort of crystallized into its simplest form that could still do the task that it needed to do. Those are relatively simple toy problems. So, again there, what would you add to my setup of the, you know, it's those, and so

• 45:43Nathan Labenz?

  now we're gonna do this giant training, optimizing all of that together, which is kind of an amazing thing to me. I felt this way about SAEs, and I think I feel it even more strongly here, that you can put all those constraints into a single optimization problem, and it works. Like, if you told me that in advance, you know, if you said like, here's my idea, I'm gonna put all these different terms into one single joint loss function, try to optimize it all together at the same time, I would say, good luck. But that seems like, you know, it seems hard to find something that's actually gonna work in that space. So, yeah, tell me if I'm missing anything, or again, anything you think is important to add for

• 46:27Nathan Labenz?

  understanding there. And then I'm really interested in like, how do you account for the fact that this actually can work?

• 46:34Speaker 02

  Yep, yep. So, yeah, I think you did a great job on introducing the various losses. So just to recap, we have the faithfulness one that makes them sum up to the parameters of the, first of all, what are these things that we're summing up? These are, we're calling them parameter components, and these are the things we want to, you know, approximate one individual job, or one individual mechanism that the network has learned. So we have the faithfulness loss, which makes all these parameter components sum up to the parameters of the target model. We have the minimality one that basically makes it do its job, do the same job as the original network, but using as few of these parameter components as possible. And then the simplicity one, which is like, we want these things to use as little computational machinery as possible. And that one is pretty important because, well, they're all important, but this one's

• 47:18Speaker 02

  important because, well, a great way to satisfy the first two losses is just to use the original network itself. You're only using one thing, it sums up to the parameters of the original model, but you haven't done any, you know, work to decompose this. So you need them all to be simple. And the way we operationalize simple is that this should be, you know, low rank and involve as few layers as possible. And we can get into the technicalities of that in a bit. But yeah, I think like the idea, I think, you know, I hadn't actually made the connection between, say, the, the work that the, Shimin Lu, did you say? I think it's, yeah, I think so. My apologies for messing up the name.

• 47:59Speaker 02

  But yeah, so, you know, I hadn't really made the connection between this kind of work and, and parameter decomposition. But the, the idea here is that suppose you aren't optimizing a network with the kind of sparsity constraints, and I believe some sort of locality constraint that they might have used to, so that, so there was kind of like neighbors, neighboring neurons did similar things, if I recall correctly. But, you know, suppose you weren't optimizing it with these constraints. Well, the network is in some sense like still, it's still able to learn. I would guess that the network is able to learn a very similar algorithm, no, no matter

• 48:40Speaker 02

  whether or not it's like trained to be sparse in a particular basis and constrained so that, you know, these neurons are close together. I would guess that it's like, we just need to like figure out the way that the network is doing the same algorithm, but in some sort of, you know, basis that we don't have direct access to. The computations, if the computations are the same, the computations both have to be sparse, and we just need to find the basis in which they are sparse. And so, yeah, it's just, we are assuming that the network is. Doing sparse computation, even if we haven't optimized for it. And then we're just trying to find the ways in which, you know, the basis in which it is indeed sparse.

• 49:25Speaker 02

  With regard to, yeah, whether or not, sorry, how, you know, how do networks manage to do these things? Well, the networks, you know, I think it was Elias Discover, as he said, you know, the networks, they just want to learn. And like, I think there's, there's something really deep to this in that like, there's just many different ways for the net, like the larger your network, the more ways it has to get things right.

• 49:53Speaker 02

  And this is a pretty powerful principle, I think, just because it means like,

• 49:59Speaker 02

  it's actually just much, much easier to find the right way to do something if you've just, if you're simultaneously looking in many different directions, many different directions in parameter space.

• 50:11Speaker 02

  And this, so it's kind of, in some sense, somewhat unsurprising that you can add, you know, we could add many more different constraints and it'll find some, you know, some like reasonably satisfying way to satisfy all of them. But, you know, it is kind of counterintuitive to our low dimensional brains to, you know, like think in these terms. And it's still kind of amazing, but yeah, I should also qualify this and say, well, the algorithm that we're talking about attribution based parameter decomposition, which where your parameter components are very large and it's very computationally expensive. And where the algorithm is basically, has got a bunch of problems.

• 50:51Speaker 02

  It is like still somewhat hard to get this particular algorithm working, which is why we ended up doing future work. But, and so even though it is still kind of amazing that they can find some solution that satisfies all these things. It's caveated with the idea that, well, at least for this algorithm, it was at least somewhat hard for that algorithm, but less so for others.

• 51:11Nathan Labenz?

  Yeah.

• 51:13Nathan Labenz?

  One practical question on the faithfulness concept, which is the idea that the, all the different

• 51:23Nathan Labenz?

  vertical dimensions, let's say of the.

• 51:28Nathan Labenz?

  The identified subcomponents that they all have to sum up to the original. Can you help me just, I guess, develop my intuition for exactly what's going on there? Like, if I, and I should say, you know, most of the problems that you've worked on so far, I'd say we're kind of at roughly, you could, you know, again, complicate this, but it seems like we're roughly at the toy models of superposition phase of this work, where the models that we're studying are rather small. And that, you know, a big question, of course, is going to be like, what does it look like to scale this up? But when, so it's not like a million layers yet is the key point there. But when one parameter in one particular layer gets turned up,

• 52:13Nathan Labenz?

  that means that that same parameter position in all the other layers has to be turned down by the corresponding amount, right?

• 52:25Speaker 02

  In these different parameter components, yes, as long as they, they all sum to the, to the, you know, the parameter.

• 52:32Nathan Labenz?

  Yeah, and I said layer there, but I meant vertical layer, which is a concept, maybe we should retire. But is, I am thinking about it in this like vertical visualization, I guess, for myself. One thing I wasn't clear on there, and I was, I was thinking about this concept of attention sinks that I've seen in the past, where we're calling this on the fly, but like, it's been found in certain cases at least, that having a few sort of junk token positions at the beginning of a transformer can be really useful because in the absence of that, AKA in the normal approach,

• 53:15Nathan Labenz?

  there's, you got to look back to something in attention, and so. Whatever tokens happen to be in those first, you know, five to 10 token positions become really important because everything kind of has to look back to that. And so the introduction of this like buffer, which they called an attention sync, was a way to say, let's not overweight the initial tokens of this sequence. In some cases, you know, you might just want to look back to sort of nothing and kind of recognize that like these, these initial tokens aren't actually like super critical to predicting what comes next. So let's not, you know, force all attention to kind of land on, on those initial tokens.

• 53:56Speaker 02

  Mm hmm.

• 53:57Nathan Labenz?

  And that attention sync concept, I was kind of wondering, could there be sort of a junk sync concept in this setup where, like, how do I not end up in a situation where my, the new subcomponents that I've created are learning whatever they're learning, which is recreating their initial model behavior? But maybe some of the other layers, which, you know, would never need to get used, are in fact just sort of taking on the opposite of whatever the ones that are actually doing the computations are learning? And could that create a sort of conceptual disconnect where my new vertical layers are learning computations?

• 54:42Nathan Labenz?

  Yes, but how do I know that those are faithful to? I know that they still sum, you know, in that sense of faithfulness, they still sum. But maybe like the computations that have been learned in these layers could perhaps be like quite distinct from what is happening in the original network? And, and maybe that's being hidden by the fact that some of, some of the other new vertical layers are just like absorbing all these like gradient changes in ways that like don't actually mean anything because those, you know, never get used at all. And you've maybe just kind of trained like some new stuff that didn't actually exist in perhaps, or, you know, doesn't necessarily correspond to a mechanism in the original.

• 55:22Speaker 02

  Let me see if I have understood what you're saying. So we have these parameter components and we, you know, they satisfy the constraint that they all sum to the parameters of the original model. And you're worried about the possibility that even though they all sum, there's a lot of, you know, different ways that in which you can sum. And in one of these, we're worried about the case where.

• 55:47Speaker 02

  There are some of the computations that, say, one of these parameter components is doing is kind of cancelled out by another one. Is it the case that both of these would be active?

• 56:03Nathan Labenz?

  Well, I'm assuming that if there is sort of a junk sync component, that it would presumably very rarely be active.

• 56:13Speaker 02

  Hmm hmm. So we do, in many like networks, expect expect there to be a junk component and a junk component that does actually do nothing. And this is just because networks, neural networks are, you know, degenerate. There are many different ways to implement not just something that works well, but implement the same algorithm. One degeneracy you might think of is if you scale up the weight before a ReLU and scale down the weight after a ReLU, well, this is very much the same algorithm. It's the same kind of amount of activation gets put back into the residual stream. But there are, you know, there's a two dimensional space here in which we can, well, in fact, it's a one dimensional space here,

• 56:57Speaker 02

  but in which we can move parameters such that exactly the same thing is done. And there are many more such degeneracies in neural networks. And so there are, you know, we basically expect one of the degeneracies that networks might use is that, well, they say, well, none of the, some of the, sorry, all of the activations that the network ever sees are, say, orthogonal to this direction in parameter space. So this direction in parameter space is in some sense never used. And you could indeed oblate this direction entirely in parameter space. And it would just not affect the outputs of the original model.

• 57:40Speaker 02

  Now, this would need to be a over parameterized model. You might not expect this kind of thing in the under parameterized case, such as language models, but in, say, an MNIST model, you might expect, you know, a junk component, an MNIST model of a sufficient size, you might expect a junk component that is just not really used for the algorithm. Now, would you be able to?

• 58:06Speaker 02

  Yeah, I guess, but I'm not sure this is exactly the same kind of junk component that you're talking about. Maybe there's, maybe it's something else in mind.

• 58:13Nathan Labenz?

  Well, I guess maybe another way to frame the question is, how do you know where the real learning or algorithm, where is the real learning happened, where is the real algorithm implemented? Another kind of inspiration for this question is going back to like some of the Neil Nanda work on like the Othello board state type thing. I recall that there was, there is at least the possibility of confusion where you want to know like, what is being represented inside the model? But if you train, you know, a big detector model on the internal states and try to predict board state from the internal state of the model.

• 58:56Nathan Labenz?

  It's, you know, at some point you're like, well, maybe my new network has learned to do that. But that doesn't really mean that my original network had the, you know, had a sort of semantic understanding of this concept. Maybe that kind of only came online in the, you know, interpreter model that I trained.

• 59:17Speaker 02

  Yep.

• 59:18Nathan Labenz?

  And so I guess I'm worried about a disconnect between the original model. You know, we've split it into these, you know, many copies.

• 59:29Nathan Labenz?

  Those individual layers, those vertical layers, those subcomponents are evolving through this training process as they learn to be simple and reproduce and be, you know, only a few of them activated at a time.

• 59:40Speaker 02

  Yep.

• 59:41Nathan Labenz?

  But I guess I'm just wondering like, what is tethering those resulting subcomponents to the original? Cause I could imagine that I can change, change, change, change, change in a functional way that gets me the desired behavior.

• 59:55Speaker 02

  Yep.

• 59:56Nathan Labenz?

  And, you know, I can offset that in some other one of these subcomponents, which maybe never gets used. And then I could potentially, if I do that long enough, hard enough, I could arrive at a spot where

• 1:00:09Nathan Labenz?

  potentially the algorithm of the new thing implements is like quite distinct from the original. And I'm not sure how I would know that.

• 1:00:17Speaker 02

  I think that's a reasonable concern.

• 1:00:18Nathan Labenz?

  Or if there's something that I'm not catching that is preventing that from happening conceptually.

• 1:00:22Speaker 02

  So for attribution based parameter decomposition, that is, I think, a fairly reasonable concern. I don't know if it happens in practice. I don't know if like the other losses in some sense implicitly penalize it. It will, if, whenever we move on to the stochastic parameter decomposition, I think it will become a lot more obvious why this is, this can't be the case. We can talk about that now, or I'm happy to park it until, until we get there.

• 1:00:46Nathan Labenz?

  Yeah, well, let's do it. I think we're pretty much there. I mean, the, so we can add this to the list of possible problems, whether or not it's an actual problem with the original method. But let's just run through maybe like what made this, you know, not the last word, or you know, why the, why you needed an upgrade. You mentioned that it's not super easy to get working. Like, there's a lot of hyperparameter dependency. It's not super easy or, you know, stable to train. So that's one. There's computational expense, which maybe you can unpack a little bit more. There's this like conceptual drift possibility that I'm flagging.

• 1:01:22Speaker 02

  Yep.

• 1:01:23Nathan Labenz?

  And then I also wanted to hear a little bit about like reliance on attribution methods, and why that's a problem as well. So maybe you can just run through those and then we'll move on to the, the new hotness.

• 1:01:36Speaker 02

  Yeah, yeah. So, yeah, this algorithm was extremely janky. It was just very difficult to get it to do what we felt was a sensible thing to do. My later algorithm has much less of this issue. But for instance, we, you mentioned, you know, there was a bunch of different hyperparameters that were necessary to tune in order to get a decomposition that made sense to us. And, you know, there's just, one of the reasons here was that, well, we're using like a top K parameter where you're only allowed to activate, you know,

• 1:02:16Speaker 02

  the top K most attributed parameter components for any, for any given input. And you want to, you know, train these attributed ones to be able to do better at the task that they were, on which they were attributed. And so you're basically training them to reconstruct their output. And so.

• 1:02:40Speaker 02

  Yeah, this this top K parameter is one of the one of the very fiddly things to get right here. It's very

• 1:02:49Speaker 02

  it's very what's the discontinuous, basically. You know, you can

• 1:02:56Speaker 02

  because you can very quickly go from one component being active to, you know, you slightly update the parameters ever so slightly such that another one comes online instead of that one. Well, you know, now you for a very small nudge in your parameter space, you've got a very different, you know, output. Now you're implementing a very different function for a very small change parameter space. That's just inherently hard to optimize with gradient descent. You know, another issue is just like the sheer number of different hyperparameters. But yeah, I think the other main one is these you're you're really you're really relying on your attributions to be right. And if they're, you know, in some sense, wrong or in some sense systematically biased,

• 1:03:39Speaker 02

  you might end up, there may not even be a kind of stable optimum for in your training landscape. Because if your biases, you know, suppose you were at a a good point in your training landscape, but your biases were slightly off, they might nudge you, you know, somewhere else. And then, you know, you might move over there and your biases are systematically, you know, biased in some other. Sorry, your attributions are systematically biased in some other way. And so it might nudge you somewhere else. Bias, sorry, the like the attributions that we were using were gradient based, which, you know, there's some like some intuitive sense in which this is kind of trying, kind of capturing what we wanted. What we wanted was like

• 1:04:25Speaker 02

  some number that told us this was highly important. This parameter component was on this interview on this input. And gradients are a kind of proxy for this because, well, if your output changes a lot for a small change in this parameter component, that is to say, if the gradient of the output with respect to this parameter component is large, well, you can say that it's, you know, a an important parameter component. But there's some cases where this is just like straightforwardly not the case. One example might be attention, where if you've got a really suppose you have a transformer and it's, you know, attending very, very, very strongly to one particular sequence index.

• 1:05:07Speaker 02

  And attention there is almost one and everywhere else zero. Well, if you nudge the parameter component that is responsible for this very strong attention to this sequence index, you nudge it by some small amount. Well, the attention is basically saturated here. And so the gradient is actually very small. It's actually not going to move your attention by very much, despite this being a very mechanistically important parameter component, because it's implementing such strong attention. And so this is just like one of the ways in which gradients may not be a good attribution method.

• 1:05:47Speaker 02

  We actually just want some other method that tells us more accurately how important was this was this parameter component. And this was one of the developments that we introduced in the follow-up paper on stochastic parameter decomposition.

• 1:06:03Nathan Labenz?

  I think that's a perfect transition. So that brings us to stochastic parameter decomposition. My general read of the paper is it's like very much a natural follow-on, right? In fact, I actually read, I went through the XRP podcast and the first paper and then wrote the first half of this outline. And then I read the introduction to the new paper and I was like, basically the first half of this outline sort of mirrors the introduction to the new papers. I felt good about that. So it's basically same goal, same concepts, same, not exactly same loss functions, but

• 1:06:43Nathan Labenz?

  sort of same conceptual constraints, just operationalized differently this time and addressing the weaknesses that you just ran through in the first, in the first case with better results. So maybe take us through it. I do have some questions on kind of the intuitions for some of the decisions. But take us through the, you know, the set, what's new and improved first, and then we can kind of unpack exactly why you did it.

• 1:07:11Speaker 02

  Yep, yep. And so the, one of the new things is in the headline, stochastic parameter decomposition.

• 1:07:20Speaker 02

  And this was, this, the stochasticness relates to the replacement for the attributions in the previous, in the previous paper. Now, it's still reasonable to think about, you know, we, we still need in some sense these attributions.

• 1:07:38Speaker 02

  But we will, you know, just to avoid confusion with, with the previous method, we can, we'll, we'll call them something, something else. We'll call them causal importances. If something is causally important for the network's algorithm, it should be attributed. And we just want some, something that will let us approximate how causally important this, this subcomponent is. And as a, before we get into the details on the, on the causal importance calculations, well, it's also important to note that one of the other major differences with the previous method was that we are now not using this frankly ridiculous

• 1:08:18Speaker 02

  size for each of the parameter components. They are not just a sing, you know, something, they're not a randomly initialized copy of the original model. They are in fact just a randomly initialized rank one matrix for each of these layers. And we're basically like, the way we think about this is that if in the previous method, we had these parameter components and that spanned all layers and spanned all potential ranks in these layers.

• 1:08:49Speaker 02

  Well, we Are basically just pre-splitting it up. We are using, we're saying, well, pretend everything was just localized in rank one matrices in each, in one layer. And later on, the idea will be that we group these things together back into, you know, full parameter components such that if we did indeed have like rank two parameter components or parameter components that span multiple layers, we could indeed find them. But no post hoc, after we group these things together. So those were two of the major differences between the APD, attrition based parameter decomposition and SPD, stochastic parameter decomposition.

• 1:09:32Speaker 02

  And so it's probably, you know, now reasonable to get into the causal importance calculations, which is where the stochasticness comes in.

• 1:09:41Nathan Labenz?

  Can we take just one more second on the, on how you're breaking the original network down into these rank one components?

• 1:09:50Speaker 02

  Mm hmm.

• 1:09:51Nathan Labenz?

  So again, in the original, you've got a network that has a certain architecture, it has a certain parameter structure and you make any of those with, is it really like random initialization on all those copies?

• 1:10:07Speaker 02

  Uh, yeah, yeah, yeah.

• 1:10:08Nathan Labenz?

  Yeah, interesting.

• 1:10:09Speaker 02

  And it's similar in the, in the, the sub-component that we're now using in stochastic parameter decomposition. These are called sub-components rather than components because we will, you know, group them together later on. But yeah, these sub-components and parameter components are, they're all, they're all randomly initialized and it's this faithfulness loss that like makes them all summed to the parameters of the original model. And that faithfulness loss is pretty strict, so they don't really get to DP it very much from, from the parameters of the original model as you sum them.

• 1:10:42Nathan Labenz?

  But they, they do at first, they just converge quickly to

• 1:10:46Speaker 02

  Yeah.

• 1:10:46Nathan Labenz?

  To like sum back to the, yeah, interesting.

• 1:10:49Speaker 02

  It's because there's

• 1:10:50Nathan Labenz?

  I guess I would have naively thought in the first case, I would have thought more along the lines of like, and I don't know why this wouldn't work, you could probably tell me. Why not, you know, if you're going to divide the thing into a thousand vertical layers, just take all the parameter values and divide them by a thousand and then have sort of a thousand weak copies of the thing, some of which can then be turned up, down, whatever. Why randomly initialize instead of doing something more? You know, principled or sort of, you know, a simple transformation of the train network itself.

• 1:11:24Speaker 02

  Yeah, I think we explored a bunch of different initializations. And I actually don't recall the exact one that we used in the APD paper. This was the example that you gave of dividing it down and making it just exact copies is, I think, something that we explored and we may even have used a randomly randomized version of that. But you can't really do this in the stochastic parameter decomposition case. Why? Because, well, you have, say, a, I don't know, a, I guess, like, one of the ways you

• 1:12:02Speaker 02

  might consider doing this is having, say, a thousand by a thousand matrix and, you know, dividing it up into, say, 10,000 subcomponents. Now, I don't know if there's a guarantee that if you sum up all but one of these parameter components, that the remainder is also rank one. Does that make any sense? So there's like, you assume that your 10,000 subcomponents sum to the parameters of the original model.

• 1:12:43Speaker 02

  And the idea would be, well, we might as well just constrain this sum to be exactly the parameters of the original model. And so what we'll do is we'll just take the sum of the first 9,999. And then for the remainder, we will just let that be whatever makes exact this sum to be the parameter of the original model. But I think then this final one may be of arbitrary rank. I

• 1:13:18Speaker 02

  That feels intuitive to me, but I'm not sure if it's 100% true. But yeah, it feels like that one may be a arbitrary rank. So if we want them all to be rank one, then we can kind of just randomly initialize. There is a slightly more, you know, involved randomization. Oh, sorry, slightly more involved initialization such that they do point in similar directions to the to the original model, such that their cosine similarity with the original model is not, you know, is at least not negative at initialization despite being rank one. But yeah, it would, the the details of that are probably not, you know, super important. And I think it should just work with random initialization, random initialization.

• 1:14:01Nathan Labenz?

  So how how should we think about these? It is, the original setup is like pretty intuitive to me to sort of envision, okay, I've got this thing, I split it into all these vertical layers. Each one, you know, gradually over time becomes simple. Not that many are active. And so I sort of can visualize like instead of this sort of dense and unintelligible computation, I now have like N of, you know, 100N or N of 1,000N that are actually active. And those, you know, are each relatively simple and they create the same output. I can sort of visualize that. And I'm like, yeah, okay, so there's sort of these, and they feel like circuits. Each of those things feels like a circuit.

• 1:14:43Nathan Labenz?

  And it's kind of, okay, if I activate these eight circuits for this input, then it works. And these other, you know, this one takes 12 for a different input and it works. But each one, you know, sort of intuitively feels like it's doing some sort of, you know, information processing through the original architecture. And so that's kind of intuitive. Here, I'm a little less able to tell that story because I'm like, at each layer, we are breaking up whatever matrix exists at that layer into a bunch of rank one parts. How should I think about those rank one parts? I mean, I had to go like re-familiarize myself a little bit with what exactly is rank? And what does it mean? And

• 1:15:26Nathan Labenz?

  In the simplest terms, it's like just a much simpler, smaller part of the overall transformation. Specifically referring to dimensionality, you have some n-dimensional space, something that is rank one, basically exists as a line in that space, right? And something that is rank two exists as a plane in that space and so on.

• 1:15:47Speaker 02

  Yep.

• 1:15:47Nathan Labenz?

  So we've got now a bunch of like, instead of a transformation that can operate in n dimensions all at once, we've got a bunch of transformations that all operate on one linear direction in that broader space. And now we're composing those

• 1:16:07Nathan Labenz?

  sequentially, I guess, in a way that

• 1:16:11Nathan Labenz?

  recreates the original. But how do, how should I be thinking about those rank one things like, why do that? I guess.

• 1:16:18Speaker 02

  Yeah, I, I guess like, I don't really view them as very fundamentally different. So suppose, you know, consider the case in the old algorithm where we have this parameter component and this parameter component can span all layers and all potential ranks. But the, one of the penalties that we were emphasizing was this like simplicity penalty, where we don't want it to involve too much computational machinery. We want to be able to study it. We want to be able to study as simple objects as possible.

• 1:16:49Speaker 02

  And the way we operationalize this was that we penalized indeed the, the rank of the matrices in this parameter component. So, and also penalized it for existing in multiple layers. And so what this, you know, the idea here being that if it, if this parameter component really was just localized in a few layers and in a few ranks, the rest should become, the, the rest of its parameters should become zero in, in all these different layers. And there should just be a

• 1:17:22Speaker 02

  low rank set of matrices in the, in the layers that, in which it is involved. And so we are hoping that, like, just to maybe go into a little bit more detail about why that is like simpler. If it's rank one, as you mentioned, you know, this is just a rank one matrix basically looks for and it reads and writes in a particular, in one single direction in activation space. And the only, the only direction in which it like reads is the, the, the direction defined on its, the right hand singular vector. And the direction to which it writes is the, the left singular vector.

• 1:18:06Speaker 02

  Either it, this space, this matrix will not be used if the activations don't, in some sense, overlap with these, with that, that read direction. And if the right direction just has no relevance downstream, it has no causal importance downstream, it doesn't do anything, then this, you know, this rank one matrix won't do anything. So, given that we're already looking for these like rank one, these low rank, like matrices in these, these full parameter components, the idea of stochastic parameter decomposition is that we can, well, one of the ideas we introduced in stochastic parameters decomposition, it's not a fundamental, you know, part of the algorithm.

• 1:18:51Speaker 02

  But the idea that we introduced here was that we'll just start with these low rank versions of the, of the, the parameter, sorry, of the subcomponents. And then later on, we can like aggregate them such that if we want to combine two, in order to create a rank two, you know, parameter component, then we can do that. And if we wanted to, you know, have, exist in multiple layers, we can aggregate them together as well. And so they're really just chunked versions of the same fundamental object, this like parameter component, this, this vector in parameter space. But one is just, you know, something that we'll group together later on.

• 1:19:31Nathan Labenz?

  Okay, well, let's come back to the how the grouping is done, because I think that's another.

• 1:19:38Nathan Labenz?

  Quite vexing, potentially quite deep question. But for the moment, okay, so we've broken the matrix into, if it's a rank N or an N dimensional space, we've now got not just N, but actually could be more than N rank 1. So how about a little help on that intuition as well?

• 1:20:01Speaker 02

  Yeah, so yeah, it's one of the, one of the core ideas in the representations and superposition line of work is that there are these features in activation space that, and there's more, that each individually represent a single feature, a single thing in the world. But they, there are more of them than there are neurons that represent them. And the same idea holds true here. In parameter space, we can represent more directions in activation space

• 1:20:44Speaker 02

  by using just more rank 1 matrices. And this is, this gets back to the read and write direction. So if we want to read from, you know, more directions than we have neurons, well, we just have more ranks than we have, you know, dimensions in our, in our activation space. More ranks than we have columns in the matrix. And so we're like, it's possible for this to, you know, read, read directions, read, you know, read these representations in superposition if we have more of these rank 1 matrices in our individual matrix.

• 1:21:27Speaker 02

  The idea then is that this, these rank 1 matrices are what, you know, will implement individual or parts of individual computations that, that the network is, is doing to compute its, compute its behavior. But only, you know, whenever, uh, the, the, the challenge is basically to, um, identify which ones are, you know, important for a given input. Uh, yeah, an input data point, um, like, and when they, when, when it's not important. And that's when, where the, the stochastic, you know, calculation of the, of the causal importance comes in, which I can get into.

• 1:22:06Nathan Labenz?

  Yeah, maybe one more question before we go to the causal importance. We've broken these things into all these some, and there's like an interesting, uh, question too of like, what is the ratio of how many of these rank one? Uh, subcomponents do you need for different size and complexity of original network? In the relatively simple problems studied so far, like that ratio isn't super high. I'm kind of wondering, is that ratio going to become like extremely high if we get into the language modeling space? It seems like probably yes.

• 1:22:39Speaker 02

  Uh, potentially like we, you know, it, it, we will probably have more computations than we have neurons. Um, that's just the, the name of the game in taking things out of superposition. But, um, one of the questions that I, um, ask myself often is, well, maybe like, is it going to be more or less efficient than say the, the dominant, um, approaches? You know, at the, at, at the moment, um, and like the, these dominant approaches, maybe say sparse auto encoders, transcoders, uh, cross layer transcoders and so on. And like the basic question here is, well, is it going to be more or less efficient?

• 1:23:22Speaker 02

  And like, it's unclear, but I think there's reason to believe that, uh, if indeed this is a reasonable way to decompose, decompose networks, there's probably going to be like fewer of these, uh, subcomponents than there are, say, uh, latents in a sparse auto encoder or a transcoder or so on. The reason being that, uh, we're, there's no upper bound to, um, how many latents you can have in your sparse auto encoder or your transcoder. And

• 1:23:49Speaker 02

  Whereas there is a kind of an upper bound on the the number of parameter components that you might have, why? Because, well, they have to all sum to the parameters of the original model. And this prevents you from just like adding in an extra, you know, an extra component or an extra latent and just like cranking up your, your, you know, reconstruction loss. You know, basically, there is kind of like a, it should level off a bit faster, I would say. But this is an empirical question that, you know, remains to be, remains to be answered for the kind of models that we care about, you know, like large language models or other similar, similarly interesting models.

• 1:24:33Nathan Labenz?

  Is that sort of like saying that while reality is complicated, the rules of physics are are simple, like the sort of, you know, the number of fundamental transformations, the number of sort of fundamental functions that are used to carve up reality should in in theory just be a much reduced space that can then operate on like many more inputs?

• 1:25:02Speaker 02

  Maybe.

• 1:25:03Nathan Labenz?

  Sort of thinking like platonic representation hypothesis type notions?

• 1:25:09Speaker 02

  It feels like it's getting at something true, although, yeah, I, it's not obvious to me right now how to connect it to, yeah, the way in which I think of things. But yeah, it feels like it's getting at something true.

• 1:25:22Nathan Labenz?

  Yeah, well, more work to be done for sure. Is the, and how expressive can these rank one things be? Like, they can represent, for example, a rotation. So is that right?

• 1:25:35Speaker 02

  Uh, I think they can represent a rotation of one direction to another, but.

• 1:25:44Speaker 02

  I'm just, I'm hesitating because I don't, I'm trying to recall the like technical definition of a rotation, but like the, you know, I think it is true, yes, that it basically can do a rotation and also a scaling. And the rotation is a very simple kind of rotation. It is just like, to the extent that an activation or an input activation projects into one direction, it will now project into this other direction. You can get arbitrarily high dimensional rotations up to the dimensionality of the, you know, the input.

• 1:26:18Speaker 02

  But yeah, so I think it is true to say that it can do a rotation, but it's a very simple, low dimensional kind of rotation.

• 1:26:24Nathan Labenz?

  But that would be enough, for example, to handle the transformation of like Tuesday into Wednesday with a sort of next day operator?

• 1:26:34Speaker 02

  I think you would basically need two for this, for this particular transformation, because these variables live on a plane. And so you basically would need to project, you know, to

• 1:26:46Speaker 02

  two directions in a particular way. Whereas like, if it were, yeah, you're basically leveraging two such one dimensional rotations in order to do this two dimensional.

• 1:27:01Nathan Labenz?

  Okay.

• 1:27:02Speaker 02

  I think.

• 1:27:03Nathan Labenz?

  So this gets us back also to the grouping concept being quite important. So let's come back to that again in a second. I think I've belabored the rank one thing enough. We've broken this thing up into some ratio greater

• 1:27:22Nathan Labenz?

  rank one subcomponents from the original N dimensional matrix. Now tell us about this swap out of the attribution method to the causal importance learned, the learned estimation of the causal importance.

• 1:27:41Speaker 02

  Yep, yep. So. One of the things, one of the ways you might think about what the attribution method was doing was that we wanted something that said, well, if we turn this parameter component off, it shouldn't really affect the output. And so the way we did that was just, we literally did turn them off and only kept on this top K parameter components. And what this is like really saying is that it shouldn't really matter if this, if this parameter component is on or off. It could be on, you know, to its full extent, or it could be off to its full extent, or anywhere in between, and it really should not affect

• 1:28:20Speaker 02

  the, the output. The top K like approach was just not really great at, you know, optimizing for this such that if you did turn it on like a little amount, it probably still would do, do stuff. Whereas what we're basically doing in stochastic parameter decomposition is we are in fact, um, learning a function that predicts for a given input. If, like, how, how ablateable is this, how turn offable is this subcomponent? And if it is turn offable, then again, it shouldn't matter how much we turn it off by. It is just like, it just

• 1:29:02Speaker 02

  causally irrelevant. And so we have this causal importance function that predicts this one number, how causally, you know, we call it the causal importance value for this subcomponent on this, in this, on this data point. And it is just a number between, supposed to be between zero and one. And to the extent that it is zero, we can basically randomly mask this subcomponent on this input, anywhere between zero and one. That is, say, we can turn it on randomly anywhere between zero and one. And if it's, if this causal importance value is one, well, then we don't really get to randomly modify, randomly mask it at all. I mean, it just has to be on because it's, you know, really, really causally important.

• 1:29:44Speaker 02

  And so the way, yeah, we literally just have a neural network that spits out this, this one number for each subcomponent. And

• 1:29:55Speaker 02

  If the subcomponent is very causally important and it gets the right causal importance value of 1, well then great. But if it is causally important and accidentally the network spits out, well, 0.5, well, now this parameter subcomponent can be masked anywhere between 1 and 0.5. And if it's masked close to anywhere that's not 1, so anywhere along this random distribution, well, it's going to damage the loss. And so what the this means that gradients will be able to flow into this causal importance function that produced this number using, yeah, basically the reparameterizing trick to let us basically

• 1:30:40Speaker 02

  pass gradients through this random sample. And basically, I put this this random distribution such that it actually does learn to approximate how ablateable this this parameter component is for a given input. And I think, like, this was not my idea, but I think it's a I think it's a really great idea that I haven't seen elsewhere in the attribution literature. And I think it has applications far beyond just the the approach where, you know, the interpretability approach we were having here. It doesn't have to be applied to parameter components. It could indeed be applied to, you know, parts of an input image and in

• 1:31:25Speaker 02

  the kind of way that attribution methods have been done in the past. We haven't done that ourselves. We'd be keen to see someone do it. But yeah, it's just a an attribution method that actually gets to the, you know, really captures just how causally important this this component was for for this this input.

• 1:31:47Nathan Labenz?

  And so is that, that's a pretty, first of all, that's a pretty big. Matrix, right, that's been added for that or a network that's been added for that learning because it's got to go from the input space to the output space of all the subcomponents, which are all the rank one breakouts at all the layers.

• 1:32:10Speaker 02

  And this causal importance function can basically be of arbitrary architecture. It can indeed, as you suggested, like be a map from the input to this, you know, like the number of components per layer times the number of layers, like that would be a very large network. And this is not, in fact, how we implemented it in the paper. In the paper, we just had a very, very small, you know, 16 parameter, I think it was 16 or it varied somewhere between, say, 16 and 128 parameter, like thresholding network that took as input the, I mean, it gets a little bit, you know, mathematical here,

• 1:32:55Speaker 02

  but it's not super complicated. It's just like the, you multiply the activations by the right singular vectors of the of the parameter components, which is basically, you're asking how much are these parameter components reading from the activations? Is this one reading a lot? Is this one reading not very much? Just measuring, you're measuring the overlap between the activations and the right singular vectors of these components. And to the extent that the parameter subcomponent has a lot to read there, to the extent that it's active, this is like a number that you can basically, you know, do a fancy thresholding with a small neural network. And that's just, you know, 16 times however many, you know,

• 1:33:38Speaker 02

  parameter components you have, which, you know, in the scale of the overall network is not very large. And it doesn't really even need to be a, you know, a very fancy neural network at all. It can just be some sort of simple learn threshold we found in practice. It was, you know, it it helped it learn nice kind of thresholding functions that were not completely,

• 1:34:05Speaker 02

  you know, not completely straightforward, but and they had some kind of structure to them. But overall, it was just like a slightly fancy threshold.

• 1:34:14Nathan Labenz?

  That's interesting. And and then when the whole thing is run, when it's trained and when it's run, it's that's now part of the reconstruction, right? It is, because again, all this is jointly optimized. This whole prediction of which component is or which subcomponents are going to be active and the structure of the subcomponents themselves, that's all being jointly optimized in in one big compound loss function. And then when you're actually running the replacement network for study on different inputs, that is still part of it, right?

• 1:34:54Speaker 02

  It can and usually is part of it, but you can throw away this causal importance network and just sum up the parameters of the sum up all these subcomponents together and just run it like the original, the original model. And if it if training has gone well, well, this should look very much like just running the original model. Now, it's it's like running everything with a mask of one instead of zero. It, you know, it should do exactly the same thing. And the causal importance function is is not necessary. What the causal importance function is is letting you do is basically telling you which parts are actually being used on this on this input.

• 1:35:37Speaker 02

  And, you know, you can imagine a world where you can throw away, suppose there was a subtask of particular interest, you might be able to throw away most of the network and only have and only keep the parameter or subcomponents that are used on this, you know, distribution of interest. And that may indeed, depending on how large the the model is, may indeed be a, you know, a much smaller set of parameters than the than the whole model.

• 1:36:09Speaker 02

  But yeah, that's future work.

• 1:36:12Nathan Labenz?

  Yeah, interesting. Okay.

• 1:36:17Nathan Labenz?

  So why does, why does this work better? I guess, how does it work better? Tell us about the results of, you know, maybe go problem by problem that we had in the last version and talk about the updates. You can get into the maybe toy problem, you know, compare and contrast. And then again, I'm really interested in sort of, do you have any intuition for why this works? It still seems very like magical that all this just shakes out.

• 1:36:44Speaker 02

  Yep, yep. So I think with regard to attributions, this is just like much closer to what we actually wanted from gradient attributions. It doesn't have the kind of biases that gradients would have. It should, for instance, be able to tell us, you know, it should be able to cope with the saturated attention case that I mentioned, you know, earlier when we were talking about gradient attributions. I think it might also be time to, you know, revisit the garbage component that you mentioned, where because basically, you know, I think we were, we were talking about, you know, a case where maybe the network has learned the subcomponents. And in fact, you know, maybe the subcomponents are actually doing slightly different algorithm from the original network.

• 1:37:27Speaker 02

  But because, you know, in APD, you could switch them off and never activate them. Well, it, you can kind of compensate for one component that is active doing a different thing while still summing up to the parameters of the original network. In stochastic parameter decomposition, you can't really do that anymore because you're just randomly activating every part of the network, basically, all the time. And if it is, you know, doing something that it shouldn't be doing, that was going to, you know,

• 1:38:03Speaker 02

  Like modify the activations in a way that the original network just didn't modify the activations. And so it's going to, you know, be bad for the loss. And it's like, it's important to realize that like, there's just, we're not actually able to turn off subcomponents in SPD. There's no like dead components or anything. What it means for it to be a dead component is that you, not that you never activate, but that you activate with always randomly. And I think this is just fundamentally different from just being completely silenced all the time and gets rid of, you know, a lot of pathologies like the one that you mentioned. And it also gets, you know, around this idea of the discontinuity that I mentioned in the top K,

• 1:38:50Speaker 02

  top K optimization, where it's no longer the case that you're only activating,

• 1:38:58Speaker 02

  you know, the top K and the rest are silent. Everything again is active all the time, which means that if you're, if you're in APD, where you're using top K, well, if you're silenced, well, gradients aren't really flowing through you. You can't really learn to be, you know, your parameters of your parameter components can't really learn to be better. Whereas if you're always on randomly all the time, well, gradients are always flowing through every subcomponent. And so it feels just like there's a lot more, a lot more continuity in the learning process. And this, you know, pans out in terms of like,

• 1:39:38Speaker 02

  it's not so much, you know, the, the, the algorithm is still somewhat, you still need to get hyperparameters like somewhat right. However, it's just not like in attribution based parameter decomposition where not only was it sensitive, but it was just not well behaved. You change it one way and it changed in a kind of unpredictable, unpredictable fashion. Where at least in stochastic parameter decomposition, fine, there's a range in which you need your hyperparameters to, to exist. But because things are well behaved, you can actually just find them without having to, it's, it's just.

• 1:40:13Speaker 02

  Easier to understand which thing you need to change in order to to get to the right right solution. So it's it's just much nicer to work with than than previous than the previous algorithm. Now, that said, the method that we introduced in the paper is just not, you know, it's not perfect. There's plenty of things that even since publishing that paper, we've, you know, focused on changing and found better versions for, you know, sampling this, you know, randomness and the the gates were used in causal importance functions and so on, just to make it a better behaved algorithm overall. And there's still optimization

• 1:40:54Speaker 02

  on algorithms to do, but it is just significantly better to work with than the previous method, which makes us excited for scaling this up to larger models, which we've begun to do for EG, like small transformers, including language models.

• 1:41:10Nathan Labenz?

  Okay, let's talk about then feature splitting. You had said that the some of the results here show that the new method avoids feature splitting, I guess. And in coming from an SAE perspective, like, I sort of think feature splitting is is good. I think of it as being just like, as I, you know, make my SAE bigger and bigger, then I get like finer grained resolution on the features, and I expect to see these sort of, you know, vanilla get resolved into like French vanilla and, you know, normal vanilla or whatever. And that seems good. But in it, tell me like, in what sense is it bad? And like, in what sense, you know, this is, why are we glad? And how has this helped us avoid that problematic version of feature splitting?

• 1:41:55Speaker 02

  Yep, yep. So we've talked already about one of the examples with the cat and the cat in a number of different positions. And it's not clear whether or not, you know, the network really is actually like using in some sense, these like more fine grained features that you can identify. And so this is just, you know, from a practical point of view, we don't want to, we don't want our explanations to be To explain more than they need to. Now, we can explain, you know, potentially how the network, you know, takes one of these cats in one position and, you know, applies circuits to it. And then we can do it for the next, you know, cat in another position and so on. But like, we're wasting our time.

• 1:42:35Speaker 02

  If the network is just not really, you know, using these variables in its, in its algorithms. And so we basically want to avoid doing unnecessary explanation. We want our, you know, our descriptions of what the network is doing to be minimal in some sense. So another particularly like pathological case of feature splitting is, say, feature absorption, where you might get, say, you have a, in one SAE, a smaller SAE, you have a feature for words that begin with the letter E. And, but then, you know, in a larger SAE, one that has more fine-grained features, we

• 1:43:19Speaker 02

  have a feature for elephant, and then a feature for everything else that begins with the letter E. Or, you know, it splits into other words that begin with, with the letter E. And in some sense, the like, this, this other decomposition, the one with more features, is just like less interpretable because, well, there was no need for us to split out elephant, maybe. It was just, you know, maybe we want, we just, again, just want the, want the variables on which the, the network is, is working. And this is just, there's no obvious way to avoid this using, say, SAEs. Now, it is, it may be the case that, you know, there's a perfect stopping point for

• 1:44:04Speaker 02

  SAEs, such that, you know, beyond this point, you're using variables that the network is using, and before that, you're, you're not. Sorry. Yeah, there may be a point at which you go into a regime where you've got SAE features that are the corresponding variables that the network is not using. And before that, you've got only variables that the network is using. But that's not obvious either, because, well, fine, you could find these sparsely activating directions in activation space. But there's no real reason to expect, well, no guaranteed reason to expect that these are indeed the way in which the network is chunking up its computations.

• 1:44:46Speaker 02

  Maybe it is in fact a group of these SAE features over here that the network is actually using. And maybe you actually did need to split further for this group over here. There's no guaranteed reason that SAE features will find variables for computations. Now, this is not necessarily a criticism that applies to other variants of the sparse dictionary learning paradigm. And I do want to caveat everything we've been talking about here in that there are

• 1:45:26Speaker 02

  people whose views I respect a lot that see issues with our approach. We see issues with theirs. But overall, I don't want to give the impression that the algorithm that we're putting forward here is the final word on anything in interpretability, or that it's definitely finding the right thing. I think there's a lot more research to be done on this. And yeah, I just want to make clear that there's still more to be done.

• 1:46:02Nathan Labenz?

  So then how about the actual grouping, labeling, and semantic understanding of

• 1:46:11Nathan Labenz?

  All these rank one units. There was one bit, and this is one of the toy problems. We don't have time to do the full setup. But there was this sentence that SPD splits its corresponding W out into 50 subcomponents. In other words, this like one matrix that's been, you know, broken down into into rank one. It has 50 rank one things that all remain active. And then the paper says that these 50 subcomponents appear to be effectively part of a single rank 50 component, comprising the entire W out matrix. And there I'm like, how do I know? What does that mean? And how do I know it's right?

• 1:46:54Nathan Labenz?

  Like I'm, and doesn't it leave us kind of with a similar problem to where we started? Which is like, what's going on there? You know, if, if we've got 50 things and we group them back into one thing, like, how did you know to do that versus determining that there, you know, that there are 50 different things happening? That part seemed to go by very quickly and I wasn't quite clear on it, let alone how to extrapolate it into, you know, larger and larger models in the future.

• 1:47:25Speaker 02

  Yep, yep. That's a really great question. So I think what's important to appreciate is that we're kind of,

• 1:47:32Speaker 02

  whenever we're optimizing these things and whenever we're saying that, you know, this is like the right decomposition. There's like an asterisk here. We're saying this is for a given, for a given amount of decomposition. This is the decomposition that breaks it up into as few components as is necessary to explain a given level of, you know, functionality of the network.

• 1:48:03Speaker 02

  We, and so, you know, we can basically scale up and down the amount of decomposition that we want to do.

• 1:48:10Speaker 02

  And, you know, break up the network into larger or smaller parts.

• 1:48:17Speaker 02

  There is like some reason to prefer some of these larger, there are some reasons to prefer, you know, one, some regions of this spectrum over others. One of these reasons might be, for instance, that the reconstruction loss, you know, is actually, as you scale things and decompose them more and more and more, there comes a point where you actually just, your reconstruction loss kind of gets a lot worse. And correspondingly, there may be a point where your causal importances, these values that we, you know, are supposed to tell us like how important each subcomponent are, they kind of behave a bit pathologically.

• 1:49:02Speaker 02

  For instance, they might all typically take, you know, fractional values rather than values that should be like one or zero. Because ultimately, you know, if something's causally important, fine, it can kind of be, you know, partly causally important, which will let us, which will shed light on like how much loss we might lose by ablating it. But by and large, we kind of prefer regimes where it's a bit more binary. And there's like points along this spectrum as we like modify these, say, these hyperparameters that control how, you know, much decomposition or activation sparsity is going on, that

• 1:49:46Speaker 02

  give us clues as to like what, you know, what decompositions make sense. And no, in some sense, none of these decompositions are like right. This holds true for sparse dictionary learning, for, for, for parameter decomposition. There's not, none of these decompositions are right in some sense because, well, we're just dealing with. One like fundamental truth that is the original network and like we're just really trying to get a lens on to like get a lens on to what it is doing under the hood, but we're always going to, you know, none of our lenses are going to be like super perfect and they're always going to introduce like

• 1:50:27Speaker 02

  approximations of some sorts. This is just a necessity. We're all trying to discretize a fundamentally continuous object in various different ways. And whenever you do that, you're just simply going to throw out some non-zero, you know, amount of approximation quality.

• 1:50:46Speaker 02

  But we are willing to pay these, you know, costs to get tractable explanations, even though, you know, there is not one like actual ground truth, true decomposition.

• 1:50:57Nathan Labenz?

  Yeah. Okay.

• 1:51:01Nathan Labenz?

  Maybe last question, I'll maybe bundle a couple of last questions into one and you can take as long as you have. Where do we go from here? Like, how expensive is this going to be to scale up to like the, you know, let's say, small but like meaningful language model regime, your sort of three Bs or your seven Bs? Is that like the next leap that you're going to try to make? And what sort of practical applications do you find to be most exciting? Obviously, again, SAEs can do the sort of monitoring and steering. Are there like fundamentally new use cases that you see opening up here?

• 1:51:43Nathan Labenz?

  And maybe if you have a, if you have time for it, is there an intersection between this and the science models? I know, you know, Goodfire has been doing some interp work on science models and that's one of the things that excites me most in addition to just keeping the, you know, the general purpose models on the rails, the idea that we could like make scientific discoveries by understanding what the models have learned is, is pretty exciting. So, yeah, what's the next, what's the next chapter? What's it cost scale up wise? What are the, you know, the new capabilities that that unlocks and, and any comments on science? And then we'll let you go.

• 1:52:21Speaker 02

  Cool. And so, yeah, like the next steps are definitely to scale these up to more respectively sized models, eventually getting, I think we've, you know, we've tried out.

• 1:52:34Speaker 02

  In the region of low-digit billions, we've scaled this down to even smaller language models in the millions, just to get some early traction before scaling back up to those sizes. There's a lot of interesting empirical questions that we want to answer there, like, how much do we actually get to decompose these language models into low or high? Maybe the things that they decompose into are too high-ranked for us to understand, which would be, we would basically need to do extra work after this first step of decomposition,

• 1:53:13Speaker 02

  but we'd at least get a decomposition into a bunch of components that are themselves easier to isolate and understand.

• 1:53:26Speaker 02

  Then, yeah, I think some of the applications that I'm, that I think this opens up, or could open up, are, I think it should make, say, unlearning more straightforward than, say, in SAEs, because you're already working in parameter space. You already know this is the direction that does X. We can just modify this vector in parameter space.

• 1:53:54Speaker 02

  Like, we have a more straightforward lens onto functions as they relate to parameter space. Whether or not this beats some gradient descent-guided approach is a separate question. But I think monitoring is also something that I think will be useful for, to the same or a greater extent than, say, sparse autoencoders or other approaches, would be, why? Because, well, sparse autoencoders might be detecting a particular region of the training dataset. It's a too-narrow region, perhaps, and instead, what we really wanted was like a, whenever

• 1:54:38Speaker 02

  the network was using function X. That's when we want to, you know, like, pay a bit more attention and go into detail on what it's doing rather than was there a, you know, an activation in this particular direction? There's going to be some correlation between these two, but we, you know, there will be just some things we want to monitor for that might be better thought of as in terms of functions to get less abstract, like maybe there are, maybe there's a function for deception and there's many different situations in which a a model might be doing deception. The input directions might be different and so the intermediate you know, computations might point in different directions in a way that like

• 1:55:18Speaker 02

  SAEs might struggle to identify as this is the one direction that does deception. It may instead have a multiple, like many different directions that do this, but in, you know, sparse parameter, sorry, stochastic parameter decomposition. And other, you know, we hope later versions of this approach might be able to identify the the individual components that are, you know, doing, doing this functionality. And then lastly, yeah, we really want to extract, you know, knowledge from these models, some of which are superhuman at you know, some tasks that we've trained them to do. And we want to be able to, you know, actually think in the terms that the network itself is thinking in.

• 1:56:04Speaker 02

  Because that's the, you know, that is the knowledge in some sense that the, that the network is, is using. And so we hope that this may be a way to, to get a lens onto that. In a way that, you know, may or may not generalize better than, than, than other approaches, but yeah, that remains to be seen.

• 1:56:22Nathan Labenz?

  Cool. Well, we'll definitely be following your work here. It's fascinating stuff and I'm glad to have you on, you know, while we're still in the relatively early phases of it. I think the

• 1:56:36Nathan Labenz?

  You know, this mega project of figuring out why these things do what they do. And, uh, you know, if there's any way to really make ourselves confident that we're going to be able to predict what they're going to do in, in novel circumstances, uh, is about as high a priority as any in, in my mind. So I appreciate that you're working hard and have taken, uh, at least one good bite out of that, uh, problem. And I'm really looking forward to hopefully seeing this line of work mature into, you know, something that can tackle the, um, the biggest models and, and the hardest problems. It'd be great to ablate that deception circuit if we can find it. So, uh, any other thoughts you want to leave people with before we break?

• 1:57:15Speaker 02

  I think that's honestly it. Nathan, it's been a really great conversation. Yeah, we had a lot of fun. Um, and yeah, just super pleased to be here.

• 1:57:23Nathan Labenz?

  Lee Sharkey, Principal Investigator at Mechanistic Interpretability Startup Good Fire. Thank you for being part of the Cognitive Revolution.

• 1:57:32Nathan Labenz?

  If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts where experts talk technology, business, economics, geopolitics, culture, and more, which is now a part of A16Z. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at

• 1:58:15Nathan Labenz?

  ai podcast dot I N G. And finally, I encourage you to take a moment to check out our new and improved show notes, which were created automatically by Notion's AI Meeting Notes. AI Meeting Notes captures every detail and breaks down complex concepts, so no idea gets lost. And because AI Meeting Notes lives right in Notion, everything you capture, whether that's meetings, podcasts, interviews, or conversations, lives exactly where you plan, build, and get things done. No switching, no slowdown. Check out Notion's AI Meeting Notes if you want perfect notes that write themselves. And head to the link in our show notes to try Notion's AI Meeting Notes free for 30 days.