The Neuron Basis: Goodfire's Curt Tigges on Agentic Interp, Monosemanticity & Reverse Engineering

Goodfire technical staff member Curt Tigges discusses research revealing that individual MLP neurons in modern language models are far more interpretable and functionally bounded than previously thought. Using an automated agent to test hypotheses across architectures like Qwen3.6-35B-A3B, Tigges explains how modern gated MLPs act as legible conjunctions and control signals capable of extensive dynamic sparsity without lasting benchmark loss. By showing that many internal representations can be directly enumerated, this work offers an alternative to complex external featurizers and provides practical avenues for behavior monitoring and persona steering. For full show notes, links, and references, read the episode page:https://www.cognitiverevolution.ai/the-neuron-basis-goodfire-s-curt-tigges-on-agentic-interp-monosemanticity-reverse-engineering/ Sponsors: Claude: Claude is the AI collaborator built for problem solvers, extending your thinking across coding, modeling, and strategic workflows. Get started and discover Claude Pro at https://claude.ai/tcr OutSystems: OutSystems is the leading agentic systems platform that enables enterprises to engineer, orchestrate, and govern AI applications on a single unified platform. Learn more and see how it works at https://outsystems.com/tcr Tasklet: Tasklet empowers your business with AI agents that connect to your tools and automate recurring workflows with no code required. Visit https://tasklet.ai and use code cog rev for $50 in free credits CHAPTERS: (00:00) About the Episode (05:55) Rethinking individual neuron interpretability (12:25) Understanding gated MLPs (22:57) Sparsity and ablation experiments (Part 1) (23:03) Sponsors: Claude | OutSystems (26:20) Sparsity and ablation experiments (Part 2) (Part 1) (39:07) Sponsor: Tasklet (40:44) Sparsity and ablation experiments (Part 2) (Part 2) (41:30) The neuron scientist agent (55:10) Monosemanticity versus circuit mechanics (01:09:37) Reverse engineering for alignment (01:20:24) Factual recall and experts (01:28:41) Designing interpretable architectures (01:36:16) Restoring reasoning performance (01:45:43) Future directions and wrap-up (01:51:34) Episode Outro (01:54:15) Outro PRODUCED BY: https://aipodcast.ing SOCIAL LINKS: Website: https://www.cognitiverevolution.ai Twitter (Podcast): https://x.com/cogrev_podcast Twitter (Nathan): https://x.com/labenz LinkedIn: https://linkedin.com/in/nathanlabenz/ Youtube: https://youtube.com/@CognitiveRevolutionPodcast Apple: https://podcasts.apple.com/de/podcast/the-cognitive-revolution-ai-builders-researchers-and/id1669813431 Spotify: https://open.spotify.com/show/6yHyok3M3BjqzR0VB5MSyk

• 0:00Speaker 00

  Hello and welcome back to The Cognitive Revolution. Today, my guest is Kurt Tigges, member of technical staff at Goodfire and with Nick Camerata, co-author of the new paper "Neurons as an Interpretable Basis: The More Academic and More LLM-Focused Companion to Nick's Recent Viral Essay, Neural Networks Are Already Clean." This work directly challenges what had been the received wisdom, that neural network internals are such a tangled mess, and that individual neurons are so frequently reused across disparate concepts, that we can't realistically hope to decode them directly and instead need helper models, such as sparse auto-encoders, to make sense of their internal states.

• 0:43Speaker 00

  Kurt's position, as you'll hear, is that individual neurons are in fact much more directly interpretable than previously believed, that most neurons in fact have one main function, that these functions are often better thought of as verbs and control-flow gates than as entity recognizers, and critically, that LLM agents are now capable of grinding through a network and describing each neuron's role with remarkable accuracy. We begin with a detailed overview of the LLM they study, Qwen-3.6 35b-a3b, a 35 billion parameter model containing 5 million MLP neurons, which are spread across 40 layers,

• 1:24Speaker 00

  each of which has 256 experts containing just 512 neurons each. At every layer, eight experts are active in any single forward pass. We linger on the highly expressive architecture of modern Swiglu MLPs, which provide two distinct sets of signals to each neuron, one linear and one gated by the Sylou activation function. This specific MLP architecture was proposed as a transformer improvement by Noam Chazir in 2020, in a five-page paper where he famously wrote that, quote, "We offer no explanation as to why these architectures seem to work. We attribute their success, as all else, to divine benevolence."

• 2:07Speaker 00

  Six years later, I'm struck by how similar this structure is to some of the gated architectures we've seen in Mamba and other highly expressive states-based models. And my own intuition is that these parallel paths allow the model to simultaneously gather semantically relevant information and effectively filter it for relevance. Interestingly, because the activation function does allow for negative values to be transmitted, each neuron can have different meanings when activated in its positive and negative directions. From there, we move on to the techniques that Kurt used to support the idea that a model's large activations are all you need for solid performance. The key concept is sparsity.

• 2:48Speaker 00

  In short, Kurt shows that you can ablate roughly 90% of the individual neurons, keeping only those with the strongest activations, with surprisingly minimal performance degradation. In some cases, such as on the ArcAGI puzzles, such intensive ablation actually improved performance. And while it did cause the model to struggle on reasoning tasks, the authors do show that a very small amount of additional post-training on the sparsely activating model is enough to recover a large majority of the performance loss. Having established that the model's weaker activations matter less than most had believed, we move on to discuss the neuron scientist, the LLM agent that they use to scale the work of characterizing individual neurons.

• 3:31Speaker 00

  Given a corpus of text and relevant interpretability tools, a frontier model generates and then iteratively refines paragraph-length hypotheses that attempt to describe what each neuron does. These hypotheses in turn are validated by asking another model to indicate, based on the description, where in a given bit of text that specific neuron should be expected to fire, and then showing that the original model's behavior is substantially restored relative to ablating the neuron entirely when the other LLM's guesses are patched in to the model's activations. Along the way, we also touch on monitoring as the first and most obvious application

• 4:11Speaker 00

  of this work: the small circuits called neuron communities they identify and begin to explore; the fact that individual experts within the mixture-of-experts architecture are also more interpretable than previously discussed; how this work relates to the current consensus understanding of how models store and recall facts, as well as the roles of the residual stream and attention mechanisms; the possibility that small activations may matter much more in long-context scenarios; the path from here to ambitious reverse engineering; how we should be thinking about scaling laws for interpretability; and what, if anything, this work suggests we might ought to do differently in the training phase in order to produce more readily interpretable models in the future.

• 4:56Speaker 00

  The bottom line for me is that this work does constitute a significant step forward in interpretability. Where much of the field previously assumed that helper models were required to disentangle model internals, there is now good reason to believe that we can study neurons directly in a highly scalable way. At the same time, whether such low-level understanding can be effectively laddered up to higher-order understanding of circuits and behaviors remains to be seen. There is much more work to be done, and while I totally understand how hungry people are for hope after a summer of so much bad news about our prospects for alignment through behavioral training alone, we should be very careful, as Kurt is throughout this conversation, not to jump to unwarranted conclusions.

• 5:40Speaker 00

  Interpretability remains unsolved. With that, I hope you enjoy this deep dive into the surprising legibility of individual neurons with Kurt Tigges, member of technical staff at Goodfire. Kurt Tigges, member of technical staff at Goodfire, welcome to The Cognitive Revolution.

• 6:00Speaker 01

  Thanks, Nathan. It's great to be on the podcast.

• 6:02Speaker 00

  I'm excited for this. This is the latest in a long line of Goodfire mechanistic interpretability episodes, and in some ways might be the biggest one yet. Occasion here is that you have worked closely with Nick Camerata, who recently made some big waves in the AI space with his essay "Neural Networks Are Already Clean," and now we've got the follow-up paper, which goes deeper, especially into the language model side of this, and quantifies it in a variety of different ways and brings some new techniques to bear on the problem as well. And so I'm really excited to go into it and really get into the nitty-gritty details.

• 6:44Speaker 00

  But maybe before we do that, just to kind of get us started, I would be really interested to hear, because I think one thing that's been interesting about this particular essay is that, more so than most papers, there's like a sense that you want to change people's minds, that you're kind of trying to induce a paradigm shift with this work. So maybe just lay out, like, what is the core argument that you're making, or the core takeaway that you want people to take on board and use to inform their work, and then we'll proceed to unpack.

• 7:17Speaker 01

  So the core takeaway of what our experiments and what our writing tries to convey is that basically the individual neurons in MLPs and neural networks in general are more interpretable than the field has thought to this point. They are very difficult to interpret, they're complicated objects, they were very confusing in the initial, like, days of interp, and because of that, I think it became not very common to study them in detail. There always has been a group of people, like, trying to do some interp on individual neurons.

• 7:56Speaker 01

  But yeah, I think it just was seen as a very difficult thing, and that representations would be distributed against, across many neurons rather than localized in individual ones. So our message is that we can understand them, that you often don't need very many active at once, and that even when a neuron has more than one meaning, which can be very confusing, it's usually not like an unbounded number of hundreds and hundreds of meanings that are just like very difficult to understand, rather it's like a compact set of meanings.

• 8:41Speaker 01

  And yeah, that's the main message.

• 8:45Speaker 00

  One detail from the essay that I would like to understand better was the idea, basically, I would summarize it as they actually used to be like a few years ago, when a lot of this work was originally done and hypotheses were formed, that maybe the neurons at that time were in fact less interpretable because of somewhat less good training techniques. And so there's been kind of a improvement in the underlying substrate, like the problem isn't maybe gotten easier in some ways. And we weren't like looking because we'd kind of thought that this was hard and given up on this line of inquiry. What has changed?

• 9:28Speaker 00

  How would you describe the, it's, you know, I feel like most things have drifted toward being like less interpretable, you know, we're worried about chain of thought, not being faithful, or not being monitorable. Why has this gotten easier?

• 9:41Speaker 01

  Well, there's a confluence of two different things that has made this easier, I think. One of them is that we have better tools available. So we have agents that can like comb through all of the situations where a neuron activates and just like look for patterns and kind of extract these and give us like a compact set of descriptions of what the neuron is doing. So we have like one such agent that we've scaffolded, the neuron scientist, who will go through these and provide an analysis. We also have better ways of like looking at the individual circuits that go into and out of a neuron to trace and like quantify these behaviors. But another thing I think is that you're right, that

• 10:24Speaker 01

  like a lot of the earlier networks were kind of difficult to interpret. They were messier, certainly smaller. And this might be going a bit into the weeds there, but also I think that there is an effect with the activation functions. So ReLU is the most well-known one, but in the networks that we've looked at so far, the activation functions, ReLU and GLU are harder to interpret. And then Swigglu becomes easier to interpret, probably because it allows for like a sort of like, it's like a little circuit that's able to

• 11:04Speaker 01

  do some logic. And so that opens possibilities for greater interpretability.

• 11:12Speaker 00

  I'm certainly no stranger to trying to get agents to do all my work. So I empathize very much with that approach. Let's take one minute also and just orient ourselves on the model that is kind of the core object of study in the paper. In the essay, there was a lot more stuff on vision models of various kinds. Here, we're talking more language model. So the model that you studied the most is Quen 3.6 35B A3B, which folks will know is a mixture of experts with 3 billion active parameters at any given time. Can we just kind of describe that thing in a little bit more and potentially tedious detail?

• 11:54Speaker 00

  Like, how many layers does it have? I read that there's 5 million neurons, but I'd just kind of like to get a little bit of a sense of the layout of spread across how many layers, how many experts in a layer, within an expert, you know, what is the sort of width and depth of an individual expert? And then I do want to spend a little bit of time too on the two signals that feed into each neuron and how we should understand those. I honestly think that is something that I have not understood as deeply as I probably should have before this conversation.

• 12:26Speaker 01

  The structure of a gated MLP is probably not something that a whole lot of people are familiar with. So Quen, as you said, it's a mixture of experts model, and it has 40 layers. Each layer has about 130,000 total neurons, if I'm remembering my numbers correctly. And those are divided into 256 experts for each layer. So at each layer, the router will decide to send the input to eight out of those 256 experts. And so it'll just like go through the experts one at a time as a little sub-network of the entire model. So

• 13:11Speaker 01

  this means that, you know, only a few of the model's neurons are active at any one time, just because, you know, only eight out of 256. So each expert has 512 neurons, and those are always on to some degree, usually just a tiny bit, but some of them will be, will have strong activations. And those are the ones that we look at in particular. So some other interesting things about this model, it's worth knowing that even though we have 5 million neurons, there are effectively 10 million things that the neurons can express

• 13:52Speaker 01

  because each neuron has a positive and a negative activation, and those can have different meanings. And they're read geometrically in a different way by the model. So sometimes if you have one neuron that's about a topic, or one pole will be about a topic, and the negative pole will be about a different topic. Sometimes they're related, but there's a lot of variety there. So other reasons why we chose this model, it performs quite well on benchmarks, and it also has a form of attention called a gated delta net, which

• 14:33Speaker 01

  has regular attention, but then that's only one-fourth of the attention layers. The three-fourths are these gated delta nets. And this model just has a lot of features that are also present in like the biggest open source models, the frontier open source models. So that made it a good subject for our investigation. Now, when it comes to the Swigglu activations, or the individual neurons, and what you were alluding to before with the two paths into the neuron, basically, you can think of each neuron as having an up projection and then

• 15:17Speaker 01

  a gate projection. So instead of, input will come into the neuron layer, and then it will go through two, those two sets of parameters instead of just one. And those are two like parameter matrices that are kept completely separate, so they can recognize different things. So what happens in practice is that the gate matrix will detect some aspect of the input, and then the up matrix, the up projection, will detect different thing about the input. And then, once they do their computations, the result is multiplied together. So that allows for you to have like a conjunction, where basically, if you see, you know, maybe

• 16:02Speaker 01

  the gate covers like context, and then the up projection covers a specific topic about the context. Like maybe the context is cooking, and then the up projection recognizes an ingredient like garlic. And then when they're multiplied by each other, it activates extra strongly.

• 16:20Speaker 00

  So maybe, I don't know if you know like the intellectual history of this very well, but like the sort of traditional MLP that I think most people imagine when they think of an MLP is basically just rows of neurons where each one has a parameter that says how strong its activation will be multiplied by as it connects to each of the different positions in the next layer, right? That is vanilla. That's kind of what I've become accustomed to thinking about. When did this more complicated mechanism from layer to layer start to become the norm? Do we know why it became the norm? And is there anything

• 17:05Speaker 00

  more that I should be just kind of, like sometimes with, with the attention mechanism, you know, we've got like our keys, queries, value matrices, and we have kind of a story as to like what each one of those is doing. You know, this is saying this token has this information, it's looking for this information. Now, having found that, it's going to load this other kind of information. Do we have a story around that here that is like principled that led to its design? Or are we kind of post-hacking that story based on the fact that this just kind of seems to work?

• 17:37Speaker 01

  To be honest, I'm not sure of the exact history of the development of this. But the larger family, the gated MLP family, that includes Swigglu, but also includes ReLU, well, it's a little bit different, but it uses ReLU instead of Sylu in the middle. These have been around at least since the Llama 3.1 family. So that's been a few years now.

• 18:02Speaker 00

  More than two years.

• 18:04Speaker 01

  Yeah, yeah. And probably predating that. But right now, the vast majority of modern language models use gated MLPs, and among those, most of them are Swigglu. They just perform better in practice. And I think the people who developed it weren't thinking about interpretability at all. Just it worked better, probably because it's more expressive. You know, you get twice as many input parameters to the MLP. We do have a post hoc story about how it allows the network to function. And we go into this in the paper a little bit

• 18:45Speaker 01

  as well, which is that if you have this combination of gate and up, it means that you can switch contexts more easily. So for example, this is, if you have a neuron that has two meanings, and then you'll probably want it to be read by two sets of downstream readers. And so that each group of downstream readers will interpret the neuron differently. That way you won't get your signals mixed up. For example, there's a neuron where the one meaning is oral health, and the other meaning is marine wildlife. And these are very different

• 19:28Speaker 01

  concepts, right? In practice, when the neuron has one meaning, the set of readers is completely different. When it has another meaning, the set of readers is, again, separate.

• 19:41Speaker 01

  And what we see is that the context of the text that's sent to the model will activate these downstream readers in advance so that they'll be prepared to like, receive the target neuron signal. So it's like you have a switching function, basically.

• 19:59Speaker 01

  So this just allows more combinations of things to happen in the network.

• 20:04Speaker 00

  If we zoom in on one particular neuron, it's getting now basically two signals from every neuron position in the layer before it. One is the sort of simple signal of just like the strength of that firing times a parameter that says, you know, in a linear way, here's how strong you should fire based on my contribution. And then there's this additional parameter, which, correct me if I get anything wrong, I probably will. Again, comes from every neuron in the layer before, but now it's the strength of the activation is put, is multiplied by a parameter and put through this

• 20:48Speaker 00

  gate function, which is the Swigloo or the Silu or could be a variety of different ones. But you said Swigloo, I see Silu in the paper, but you said Swigloo is like the most commonly used. So maybe you can de-confuse me a little bit on that, but it's not necessarily the case where we shouldn't think of it necessarily, I guess, as the, in fact, I'll state it the other way. We probably should think of it as the gates coming from different places than the signals because if the gates and the signals came from the same place, they'd be, it'd be inseparable. And the whole point of this is that we have a neuron being told when to

• 21:30Speaker 00

  be active and then also kind of how strong to be active, but coming from different places so that this can, these two signals can reach it independently. And that independence is where we get this greater expressivity.

• 21:44Speaker 00

  How am I doing?

• 21:45Speaker 01

  Yeah. So that's basically right. It is, it does get a little messy because the strength of the activation can come from both the up projection and from the gate.

• 21:57Speaker 01

  So to, also to answer one of your earlier questions, so the the gated MLP as a whole is called Swigloo. This particular design, that includes the up projection, the gate projection and the decoder direction. Then inside of that, the gate is multiplied by the activation function, which is Silu. If you had Regloo instead of Swigloo, then the gate would be multiplied by a ReLU activation. And what's the Silu does is allow the gate to be like a little bit negative sometimes, but mostly positive. Whereas ReLU in that place would only be like only positive.

• 22:42Speaker 01

  But yes, we do indeed see that they often have somewhat disjointed sets of things that activate them. Sometimes there's some overlap, but usually it's quite separate, which allows for that express activity.

• 22:57Speaker 00

  Gotcha. Okay. Hey, we'll continue our interview in a moment after a word from our sponsors.

• 23:04Speaker 00

  Today's episode is brought to you by Anthropic, makers of Claude. To show off Claude's latest capabilities, I asked Claude to take the lead in making this ad.

• 23:15Speaker 00

  Given a short creative brief, Claude, using Opus 5.5, studied my past Claude ads, wrote this script, directed a clone of my voice, and designed and coded the animation you're watching.

• 23:28Speaker 00

  My job, give feedback and decide.

• 23:32Speaker 00

  Claude is the AI for problem solvers. It's the collaborator that understands your entire workflow and thinks with you, not for you. Whether you're debugging code at midnight, building a financial model, or strategizing your next business move, Claude extends your thinking to tackle the problems that matter. And with Claude code routines, it keeps working in the cloud even after you close your laptop.

• 23:58Speaker 00

  For problems worth solving, get started with Claude at claude.ai/tcr.

• 24:04Speaker 00

  That's claude.ai/tcr. And check out Claude Pro, which includes access to all of the features mentioned in today's episode. claude.ai/tcr.

• 24:21Speaker 00

  This episode of The Cognitive Revolution is brought to you by Outsystems, the leading agentic systems platform. We've all seen the headlines. Companies are pouring money into AI. But the big question on every executive's mind right now isn't just how fast can we adopt this? It's where is the ROI? We're seeing a real trend toward AI chaos. You've got teams deploying standalone coding tools, random agent builders and experimental scripts. It sounds innovative, but in reality, it's creating a massive headache. Fragmented tools, ungoverned data, serious security blind spots and costs that are spiraling out of control. If you don't bring those agentic applications under control now, while they're still embedding into your core processes,

• 25:06Speaker 00

  you are looking at broken systems and damaged customer trust down the road. That's where Outsystems comes in. Outsystems is the leading agentic systems platform for the enterprise. Instead of managing a patchwork of disconnected tools, Outsystems lets your team engineer, orchestrate and govern your entire agentic ecosystem on one open unified platform. It's built for the speed of AI, but with the reliability and security that enterprises actually require. We're talking about real results like KeyBank, who used Outsystems to deploy a customer facing app that delivered 75% faster onboarding times.

• 25:47Speaker 00

  Or the global logistics leaders who built agentic systems to completely eliminate their engineering bottlenecks. You don't have to choose between speed and control. Whether you're a small team or a massive enterprise, Outsystems helps you engineer, orchestrate and deploy agentic systems that actually scale. Stop chasing the hype and start owning your agentic future. You can see how it works and learn more at outsystems.com/tcr. That's outsystems.com/tcr. So one big maneuver that plays out a few different ways in this paper that's really core to all the findings is

• 26:29Speaker 00

  sparsity. In a few different ways, you sort of look at how sparse can we make this thing and still get, you know, pretty similar results. One thing that I would love to understand is kind of when we move things to sparse, I'm not entirely clear on what that always means. Like the simple way to think about it would be like turn them all to zero. But I, I take away that it's actually a little more complicated than that. Like there's some sort of baseline average activation that's non-zero that is considered to be like the, the sort of default or neutral state when a neuron is ablated or you know, essentially turned off, but turned off doesn't mean going all the way to zero. Help me understand that a little bit better.

• 27:14Speaker 01

  Yeah. So, um, it's a bit tricky in practice because when an expert is activated, all of its neurons are on a little bit. You know, there is no true zero for Swigloo or Silu.

• 27:28Speaker 01

  When we do the sparse experiments, there's a variety of things we do for

• 27:35Speaker 01

  where we, in some cases, we ablate everything except for the extreme activations that we've measured precisely for a given neuron. And then when we do that, we set the other values of all the other neurons to their average. So it's a mean ablation. Now this mean ablation in practice is actually quite close to zero because the neuron is bipolar. It's both positive and negative. The average just ends up being like very, very close to zero or small. Another kind of ablation we do where we just select the top

• 28:09Speaker 01

  K neurons from each expert, for example, top 64 neurons from an expert. In this case, those are the only signals we allow to pass through. The other items we do not, they don't have any signal. So they're effectively zero.

• 28:24Speaker 00

  So is, is much at stake there? Like in the scenario where you use the average? If you just literally set them all to zero, meaning like there was no contribution from that neuron to the next layer, would that be, would that like yield dramatically different results or is it not a big difference in practice?

• 28:44Speaker 01

  So in practice, when we do this, the model performs quite well still on a number of benchmarks. Just basically equivalent to the original model.

• 28:54Speaker 01

  So when we initially do this,

• 28:57Speaker 01

  Then reasoning does suffer. So the models will start going into loops. Sometimes it'll like be unconfident about its answer and just like self-check too many times. But with a little bit of fine tuning, we can correct that and the model like can perform as well or even better in the benchmarks that we tested in this ablated state.

• 29:19Speaker 00

  Yeah. So that's really weird. I definitely wanted to try to get some intuition around that. So just to one more fine point of clarification. So there's these different ablation techniques. Top 64 is, I think, where the like 88% number comes from, basically, right? And then you also have, and then you look at top 32, and it seems like there's significantly more degradation of performance when you go from top 64 to top 32. But then there's also this like percentile thing where you ablate like, because it's bipolar, I understand you're taking like the middle, right? So you're saying the strongest positive,

• 30:00Speaker 00

  strongest negative, we'll keep, but we'll ablate like the 10 to 90 percentile or the 5 to 95. Why is it, why is it meaningfully different to have top 64 versus just keeping the bottom and top 5 percentile? Is that like, it seems like it's almost two ways of saying the same thing, but there must be something more there that's causing those to be like substantially different to the point where you're even bothering to report them separately in the paper?

• 30:27Speaker 01

  Each form of ablation has kind of a different meaning to the model. One thing we noticed is that if you rank a single neuron's activations or the places where it activates by these percentiles, like P75, P90, P95, and so forth, different neurons start to descend into noise at different levels. So one neuron might be meaningful all the way down to P75, and another neuron might be meaningful just down to P99. And this can also differ by layer. So sometimes just a strict ablation based on percentiles doesn't

• 31:10Speaker 01

  always make sense or might not be principled. But another way that you can do it is if you do the top, top N, top K, top 64, for example, then what this says is, given that you've activated this expert, what are the most important things in the expert? And this kind of lets the model decide in a way what's important, rather than just like the pure percentiles. And the percentiles can be kind of dependent on the data set as well. But by letting the model do that, we get a little better performance. Practice,

• 31:50Speaker 01

  what this turns into is the vast majority of things that activate are indeed already above P95 on the positive side or below P5 on the negative side. But it'll include like a handful of neurons that are like, you know, P90 or something like that. They're a little bit weaker.

• 32:11Speaker 00

  So is the core difference that the percentile based thing is happening model wide versus the top K is happening at the per, on a per expert basis? Or is the percentile also per expert?

• 32:26Speaker 01

  The percentile is actually per neuron. So it's like, given this neuron's historical activation range, you ablate based on that individual neuron's value for P90 or whatever the case may be.

• 32:40Speaker 00

  But that's like global in the, in the context of all 5 million neurons?

• 32:45Speaker 01

  Yeah, all at once.

• 32:46Speaker 00

  Gotcha. So the, the top K is sort of like a stratified approach where you're going to ensure that on an expert by expert level, you keep like a similar amount of activity present?

• 33:01Speaker 01

  Yeah, yeah, exactly. Yeah. Because maybe you have an expert where most of the neurons have, still have some kind of meaning below, you know, P95. Maybe they still, maybe all of the neurons in that expert have a meaning at like P90, and it's like very clear and still useful. And then you have another expert where nothing is usable below P99. And so in such cases, you'd want to let each expert decide what the threshold should be.

• 33:37Speaker 00

  Can you maybe give a little more of the intuition behind the language you're using there around like, meaning down to, you know, could be P99, could be down to P90? Because I was thinking of it maybe wrongly, but kind of in a reverse way where I was thinking like, the most important neurons would be the ones that would make the more stringent filters, right? Like the 25 to 75, you're cutting half of the neurons out. The 10 to 90, you're cutting 80% of the neurons out. As you push this middle band wider, you're left with presumably like the most important neurons. But I think you're saying something different, which is like, something could be in a high percentile and not that important, whereas something else could be in a lower percentile and still important.

• 34:20Speaker 00

  And that's why it's worthwhile to go down to the per expert level. Do I have that right?

• 34:28Speaker 01

  Yeah, essentially. So to be clear, the variance is quite small. It's almost always like the extreme activations that are the most meaningful and most, the most causally relevant to the model's computation. So to give some intuition about how we see this, we collect a whole bunch of examples where the neuron activates at every percentile, like the top 1,000, and then P99.99, and so on, just like down these ranks all across the activation range. And then we study where, like where the coherence stops. So we just

• 35:09Speaker 01

  like read the examples and study them and see like sometimes it dissolves into noise at a certain point. But it's always in the 90s, pretty much. It's just like, you know, sometimes it's going to be P95, sometimes it's going to be P99 or something. But it's also worth noting that the quantiles themselves do depend on the data set. So we use a pretty large mixed data set, but it's not as large as the model's training distribution, which was, I believe, something like several trillion tokens. We use 10 billion tokens to extract these examples.

• 35:54Speaker 00

  So as I'm looking at the performance table, what you're saying makes sense as I look at P25 to 90, or P25 to 75. So the middle half of neurons that are least extreme in their activations being ablated, you scroll down that column and look at all the different performance results on all these different benchmarks, and there's like, not that much change. So from that, you could say at a very high level, like you can cut out the half of neurons that are least extremely activated, and basically not much goes wrong. And even interestingly, on the reasoning

• 36:34Speaker 00

  stuff, like you don't see much damage there either. Then you go more extreme, cut the middle 80%, and then cut the middle 90%, and the impact to performance becomes more significant. So we're kind of pushing like, where does this curve get steep in activation extremity space? And trying to find like how far we can go and still have things like mostly work. I'm really interested to get your intuition for, how is it that some of these things do better? It's notably across the top, you've got four different ARC AGI

• 37:16Speaker 00

  sections, and the middle, 50% doesn't move it much, but then moving, removing 80, removing 90% of neurons, you actually get a little bit of a bump in performance. Do you have any account of why that is happening?

• 37:30Speaker 01

  So some of these benchmarks will care about how quickly the model reaches an answer. So like if it reasons for too long before it puts out an answer, then, then it counts as like a failure. Basically, if you cut out a lot of the stuff inside, the model becomes more confident in those benchmarks where it performs better. It reaches the answer more quickly. And I suppose some of the benchmarks that you mentioned don't involve reasoning, but the ones that don't involve reasoning, it's just because the model's more confident. It doesn't have as much internally to distract itself from the right answer. And the most confident thing rises to the top. In the case of reasoning,

• 38:15Speaker 01

  then with our fine tuning, we see the same thing, or it just reaches the answer that the original model knew, just like faster.

• 38:22Speaker 00

  Hmm. Interesting. Less to distract itself. That's fascinating. So so far, just to kind of put a signpost in the ground here, we have talked through in somewhat excruciating detail, the structure of the network, what it means to ablate neurons, and we've established that you can ablate a significant majority of neurons, up to 90% or 88% if you take the top K 64, and still get surprisingly non-degraded performance, at least on these like non-reasoning tasks. Hey, we'll continue our interview in a moment after a word from our sponsors.

• 39:07Speaker 00

  The Cognitive Revolution is brought to you and powered by Tasklet. I've been running Tasklet agents for almost a year, and they've become a major part of how the show gets made. Before every recording, one agent researches the guest and sends me a suggested outline of questions, complete with links to all sources. Meanwhile, another agent scores the overwhelming number of pitches that I'm getting from PR firms. And that's really the core idea behind Tasklet. Run your business on agents by giving them real jobs to do on an ongoing basis. Setup is incredibly easy. Just describe the work in plain English. Research and New sales lead, update the CRM and draft a personalized reply.

• 39:50Speaker 00

  Tasklet connects to your business tools and can run around the clock, on a schedule or when something happens, like a new email arrives. Each agent works with your tools and business context and remembers useful information across threads. You can also share agents, connections and reusable skills with your team. So the workflows that one person figures out becomes something the whole company can use. No developer needed. You decide what agents can do on their own and what needs your approval. And there are no per seat fees. You pay for usage, not for adding teammates. Start with one recurring job, like researching the people you're meeting. And then just keep building from there.

• 40:34Speaker 00

  Visit tasklet.ai and use code COGREV for $50 in free credits. That's code COGREV at tasklet.ai. Okay. Now, the next question, though, of course, would be like, well, it's one thing to say, only the extreme ones matter. It's another one to say, can we interpret what's going on? So maybe walk me through the process that the neuron scientist goes through.

• 41:05Speaker 00

  You already kind of described, like, you have a big corpus. You collect a lot of activations. You can rank them. You can, you know, put different thresholds in place. Now, how do we turn that massive blob of activation data into five million stories about what each individual neuron in a whole big 35B, A3B network is actually doing?

• 41:31Speaker 01

  Great question. So I like talking about the neuron scientist because I think it's a pretty cool

• 41:38Speaker 01

  agent, basically. And we put a lot of work into like trying to improve it. But so as you said, we basically collect these thousands of examples of prompts where this neuron becomes active. And then we group them, we cluster them by similarity. We have a few ways of doing this. One of them is you can look at the input connections and group them by or cluster them by that. You can also look at the just the text itself and what other neurons activate and cluster them by what other things activate it. And in practice, that gives you like different contexts where the neuron becomes active. But the central thing is that we just get an initial grouping.

• 42:22Speaker 01

  And then we hand these examples and the groups to the agent, the neuron scientist.

• 42:30Speaker 01

  So the neuron scientist's job is to come up with a hypothesis that unifies these different examples where the neuron activates and then predicts where it'll activate in the future. So sometimes that will be one meaning, like that's more common. Sometimes it'll be multiple meanings.

• 42:50Speaker 01

  And there's different patterns, like different words that neurons prefer to activate on, different contexts, different concepts.

• 43:01Speaker 01

  We give the neuron scientists a lot of latitude in which kind of hypothesis it comes up with. But we give it a bunch of tools as well. So it can create example sentences or modify example sentences and send them to the model. And then get the result of how much the neuron activated on them. It can run other experiments like

• 43:24Speaker 01

  two very similar sentences with like one minor difference and then see like what the difference is in activation there. It can do steering. It can do basically all the same things that an interpretability researcher can do.

• 43:43Speaker 01

  So it'll come up with an initial hypothesis and then propose this. And then we will test it objectively by with another system that is separate from the agent. So that the other system will just basically see, okay, based on this description of the neuron,

• 44:00Speaker 01

  can we predict where in this example text, the neuron is going to fire and by how much? And this will happen for like a few texts. And then if it fails, if the description just isn't good, then it'll come back to the neuron scientist with feedback. It will see how it needs to adjust its hypothesis, improve it a little bit, send it back and so forth.

• 44:30Speaker 01

  And then at the end, a helper agent will create a page that basically shows us the results of what the hypothesis is.

• 44:42Speaker 00

  So the beginning of that sounds pretty similar to auto labeling of SAE features. I think people have seen that in the past, right? Where there's like, okay, here's a bunch of passages that caused this particular SAE feature to light up, like, hey, model, what do you think that appears to be? And then where we've taken this farther now is we're not just asking that question, taking the answer and calling it a day, but rather saying, burn some tokens to really like dial in your explanation, test it and refine it. And, you know, let's really try to make this good. Because obviously a lot of SAE features are deemed uninterpretable. And certainly, like, honestly, I have looked at like lists of passages that make

• 45:26Speaker 00

  SAE features fired, some of which were considered uninterpretable, others which were considered interpretable. And at times I've been like, I don't know, dude, like you're saying that's what this means. And it's not like really jumping out at me that that's like really what this means. So like, what do you see the agent doing as it goes down this path? Could you tell a story of one where it was like not obvious at first, but through use of tools, it was able to kind of get to insight as to what this particular neuron meant?

• 45:58Speaker 01

  Yeah, for sure.

• 46:01Speaker 01

  So yeah, a lot of the auto labeling stuff, the auto interp stuff that's used for SAEs and that we first used for neurons. When the function of the thing is obvious, it will tell us what's going on. But when it's not obvious, it'll usually be some vacuous label like this fires on words in the context of medical discussions. And that's just not very useful. That doesn't teach us much at all.

• 46:28Speaker 01

  So one thing that we have noticed is that as we apply increasing levels of intelligence, then more and more of what the neuron is doing becomes legible and predictable. We have a graph of that in our paper where like the auto interp stuff is the lowest. And then as we get better agents and they're allowed to do more work, then we get better predictions. And the neuron scientists usually will use a frontier model.

• 46:56Speaker 01

  Like whichever the latest one that seems to perform well is.

• 47:02Speaker 01

  So one example

• 47:06Speaker 01

  of a neuron that seemed uninterpretable at first, but then as we applied more intelligence, became more interpretable. There was a neuron where if you just look at the examples and just like the 50 token window where it fires, it seemed like completely uninterpretable. It was always, I mean, it would just be some phrase that could be across the field of cooking or instructions or

• 47:36Speaker 01

  like legal documents or anything, really. And you couldn't really tell

• 47:44Speaker 01

  what it was. There was just nothing unifying the patterns.

• 47:49Speaker 01

  But with a lot of investigation using agents, we were able to identify that this actually fires, the neuron fires when it's the second item in a list where contents of the list had been introduced in a different format a long way before in the text, maybe a thousand tokens before in the text. But this was being recapitulated here in this place in a different format. So like maybe before it was a commented or comma based list and now it's like 1234 or something. So this is like a structural feature that you wouldn't necessarily think to look for, but was present.

• 48:33Speaker 01

  And there's a lot of features like that where the idea isn't obvious at first.

• 48:42Speaker 01

  But that's just, just one example.

• 48:46Speaker 00

  So the, I thought this was pretty creative and I, correct me if I'm wrong, but I think all this technique would be pretty equally applicable to SAEs. Like, there's nothing that would prevent you from giving an agent a similar set of tools around SAEs and allowing it to tinker and, you know, change text and rerun and all that, that same kind of thing, right? So that's an interesting innovation regardless of, you know, which lens we're using to look at things. So let me see if I can get this right. So the, the neuron scientist kind of iterates for a while. It does, it does its experiments. It does counterfactuals. It maybe, you know, expands scope to look at longer bits of text to try to see these like bigger picture patterns that weren't apparent in the very, you know, excerpted text that

• 49:30Speaker 00

  we're accustomed to seeing. And it eventually settles on basically what sounds like a paragraph length description of what the neuron is. Okay. Now, how do we validate that that in fact is what it is? The answer is give it to another model from, I guess, like a different model maker or, you know, different family of models and say, okay, your task is to take this description of a neuron, look at this text, go token by token, and tell us for each token, how strongly that neuron will fire. And then, based on the strength

• 50:12Speaker 00

  of those neurons, you actually like patch its answers into the model. Uh, now we're back to the original model, the model being interpreted, and say, how well does the model perform if instead of its actual natural activations, we force the predicted activations from this other model that was based on the combination of the description and the text at hand? And the answer there is, we recover 74% of the KL divergence loss, which I'd like to understand a little bit better. Like, I'm not, I don't have great intuition.

• 50:53Speaker 00

  I don't see in like KL divergence color, I don't think. Maybe you do. Um, so tell me, did I get anything wrong about the process? Is there anything else we should understand about the process? And then, how should we understand that 74% recovery bottom line out of that?

• 51:09Speaker 01

  So that's essentially correct. Yeah. So the, uh, the neuron scientist comes up with a paragraph length description, um, of the neuron, highlights, you know, what sorts of tokens it fires on, as well as the context and the concept overall. So this is one of our evaluation methods. It's currently our primary one. And, um, as you said, we first ablaze the neuron completely from the passage. And then, this other model will, um, say, okay, based on the description, out of a score, out of, uh, zero to nine, how strongly will this neuron fire? Um, and then it labels each token in the sequence with that. Most

• 51:53Speaker 01

  of them are zeros, but a lot of them are going to be high. And then, uh, as you say, we patch those into the model. And then we see how much of the damage from deleting the neuron entirely was recovered by patching it in, uh, to these places, at these values.

• 52:09Speaker 01

  So this works pretty well. And the, a model that we, um, use right now is Jev. So it's like very different from the model that bring rate of the prediction.

• 52:21Speaker 01

  And this gives us an idea of how much of the damage is recovered. And we use, um, as you mentioned, KL divergence. So this basically says, at each step, for each token in the sequence, the model is predicting, it has a prediction for every token in its vocabulary, each of the logits in its vocabulary. And so that's a distribution of predictions over all of those tokens. And, um, the KL divergence is best understood as like, the difference between the distribution when it's damaged versus the distribution when it's patched in or repaired. And so, uh, that distribution of predictions over tokens basically shows how much damage we've caused to the model, the model's ability to predict the next

• 53:05Speaker 01

  token. Um, and we measure that for each of, like, we combine it over the entire sequence.

• 53:13Speaker 01

  So at each token position, we have that value. And then we just like, uh, measure it over the whole sequence that the model's predicting. Now, something that's worth noting is that we do weight it by damage. So when we say 74% damage recovery, that means that cases where the neuron basically didn't matter, or it mattered very, very, very little, get a lower weight than cases where the neuron mattered a whole lot to the model's prediction. So that's the, the basic setting. And this is, um, our primary evaluation, but we have a couple of others that we use too, that are worth mentioning. So a very intuitive

• 53:55Speaker 01

  one is, we say, we give another model the description, and we say, based on this description of the neuron, make a text, write a text that's going to activate the neuron more than anything in the data set. Like, we've given, we've run a 10 billion token data set through the model and collected these examples. We say to this other model, exceed that. We don't give it any examples. So it has nothing to work from except for the description. And then it just like iterates until it's produced a sentence that will activate the neuron more than anything in the data set. Um, and then other metrics that we might

• 54:35Speaker 01

  use are, we say, given this sequence, can you tell me whether this neuron should activate in this sequence at all? Um, is it relevant at all? And, um, at this one, you know, we often see rates in the, the 90s, um, because it's an easier task than token by token prediction.

• 54:58Speaker 00

  So this all adds up to, we should have some, you know, reasonably high level of confidence that like, the process by which the paragraph, um, length description of the neuron is created is like working. Because if it wasn't working, we wouldn't expect to be able to recover this damage. We wouldn't expect to be able to get these predictions right. How do I, how now do we, you know, now we, we start to get into the interpretation phase of this, right? So now I'm looking at a paragraph length description. It's still a little bit in the eye of the beholder. Like, a paragraph can obviously contain two different concepts or related concepts. We have the question of like, how do we decide to draw boundaries between concepts? So like, how do you think about modus manticity coming

• 55:43Speaker 00

  out of this? Like, what counts as one concept? Um, I definitely want to hear your thoughts around the nouns versus verbs thing. You know, maybe give me a little more of like a qualitative, uh, intuition for like, what are these, what are these paragraph length descriptions? Somebody, I'm sure you've read, you know, thousands of them. What, what is the nature of these paragraph length descriptions?

• 56:07Speaker 01

  It can vary widely between neurons. Um, we see neurons doing so many different things. As you said, sometimes they'll contain multiple ideas, like this neuron will activate, you know, on, on oral health and marine environments, usually on the point at which someone decides to get oral surgery, or at the point at which, um, a specific animal in the marine environment was described. The description will read something like this. Um, most neurons do have something of a central concept that unifies them. But, um, as I've looked at, you know, thousands of neuron descriptions and what they actually

• 56:50Speaker 01

  tend to represent, I think I've, uh, like, come to a somewhat different philosophical perception of things than simple monosemiticity or polysemiticity. It's like monosemiticity or just like semanticity in general is this idea that a, a neuron or something else like a feature represents a single thing, a single concept. It's just detecting whether that's there. Uh, but in practice, I think it ends up much more complex. So each neuron can connect to thousands of other neurons upstream. And they, uh,

• 57:34Speaker 01

  That provides a lot of information to this neuron.

• 57:40Speaker 01

  One thing that you might see is a neuron that responds to like discussions of insects, for example. It'll fire specifically on three types of things. The food that insects prefer to eat, the habitats they prefer to live in, and the places where they prefer to lay their eggs.

• 58:01Speaker 01

  Now, is this three concepts or is it one concept? Kind of hard to say because it's not firing on everything about insects. It's not firing on every preference of insects. It's just firing on like those three subtopics within the insect sphere. Now, a neuron that fires on, you know, musical instruments like banjos or legal documents about alimony, that's probably going to be, that's very easy to see, like, okay, this is two different things, right? Another problem that we see also with the concept of human recognition of monosemitism or polysemitism is that we don't always know

• 58:45Speaker 01

  why the model's putting things together. There's an example, like, let's say you have a neuron that fires on clothing, it fires on exercise equipment, and it fires on dishes and on tablecloths. Now, seems very polysemitic, right? Like several different meanings. But what if the concept that the model has is things you might find at a garage sale? Then suddenly those things are unified, but you might never see it from the outside. So for me, the important question about mono and polysemitism is less about is a thing strictly monosemitic?

• 59:30Speaker 01

  Is it strictly polysemitic? It's more, can you elucidate all of the things that the neuron represents? And sometimes you might want to zoom in and find, like, it looks like a single concept. You zoom in, you find a few concepts inside, but then you just list them, right? You just, you know what they are. You elaborate over that set of things. And when those things are disjoint, when they're like totally different from each other, then you just list that as well. And the interesting thing about this also, from a mechanistic perspective, is that we can see the signatures of these concepts in the weights themselves,

• 1:00:11Speaker 01

  which means like, you can basically find, among the thousands of neurons that are connected to a neuron, a subset that goes to the neuron when it's discussing one topic and a subset for another topic. And you can just kind of analyze thousands of examples and pull out these subcircuits in this way. And then that gives you, you know, this perspective of like how

• 1:00:40Speaker 01

  the model actually has these primitives inside of its own mind.

• 1:00:44Speaker 00

  So as you've looked at these circuits, do you have kind of a simple story around like, I mean, this sounds silly to say, but like, kind of how it works? Like, I mean, what, what sort of motifs or, you know, moves stand out as sort of the, you know, notable primitives that could give us some intuition? I mean, I guess this gets to, you know, one of the big questions, of course, with all of this is like, at what point do we get to the ambitious reverse engineering phase of all this? And I think that's been one of the things that has been like most discussed about this essay and now the paper, or I'm sure will generate a lot of similar discussion.

• 1:01:29Speaker 00

  To what degree do we now have a better story of, it's one thing to say, okay, we can kind of say, actually, look, these neurons do have a pretty interpretable thing that they do individually, but how does that fit into context? How does that start to turn into behavior? Like, how do we aggregate that into, you know, real understanding? How far are we on that? Do you have any kind of story that you can share that gives us a little bit better intuition around, again, like how it works as opposed to just like what an individual neuron means?

• 1:02:07Speaker 01

  Well, some great questions in there.

• 1:02:10Speaker 00

  Only the big ones for you.

• 1:02:12Speaker 01

  Yeah.

• 1:02:15Speaker 01

  So first of all, I think the shift from noun to verb can be seen in a couple of different ways. But one way that I like to think of it is, this is less of a detection of a single feature or object and more like the neuron, the neuron is doing something, right? It's not just detecting a thing. It's like sending a signal that is going to induce the model to do something else. So in that sense, it is a verb.

• 1:02:46Speaker 01

  But also it detects basically shifts of what's going on in the text or what things are happening and shifting towards a different kind of generation of text. I guess that's how I think of the noun versus verb distinction, although

• 1:03:09Speaker 01

  Nick might think of it slightly differently, but I think it's quite similar between our thoughts.

• 1:03:14Speaker 00

  So let me just, let me echo that back to you real quick. So the verb is like,

• 1:03:23Speaker 00

  with respect to the model, the verb is like, this neuron sends a signal downstream that activates a certain set of circuits. And presumably some are still very like nounish, but these sort of, these are like control flow nodes that we can describe as like, if this, then that kind of operations.

• 1:03:48Speaker 00

  But, but we should understand their effect as not like representing something, but sort of directing traffic or routing energy or activation through the model to the right places downstream to be used.

• 1:04:02Speaker 01

  Exactly, yeah. They're not just like static, like each layer just detects a certain set of concepts and now these concepts are there in the model. They are, like you said, control flow. They direct the model towards specific things or away from specific things. And we can see this with some interesting neurons that respond to reasoning in particular. So there's this whole class of neurons that they fire on and direct the model to be more or less competent in its reasoning, or to do things like wait or to like recheck, or to like continue along a specific direction or pivot to a different approach.

• 1:04:46Speaker 01

  There's a lot of reasoning specific ones that are just really interesting in this way. And these are very much like verb and steerable type neurons. There's others that are like, oh, this number is wrong, or this one's like off by one, or like, yeah, just like these sorts of operations.

• 1:05:08Speaker 01

  Relating to the topic of ambitious reverse engineering, I think, so certainly there's more than just neurons to reverse engineering. We need to understand attention and other things in the model. The residual stream in particular is like a really complicated thing to unpack. But I think understanding neurons and the specific circuits and subcircuits, they instantiate is pretty critical to that.

• 1:05:37Speaker 01

  Now, you can understand every single neuron in the model and just have a big list of what they all represent and what they all do and not have an ambitiously reverse engineered model. You need to see how they operate together. And turning individual neurons into understanding requires, like, basically understanding circuits. And you can get circuits through a variety of ways, like you can run millions of prompts through the model and gather attribution graphs for them using one of the existing methods and then see like what the average,

• 1:06:19Speaker 01

  just like pick out these communities of neurons from there. That's one thing that we mentioned in our paper. You can also see what the circuit is for a specific prompt. To turn this into understanding, then you need to find the right ones and then see how they impact behavior.

• 1:06:40Speaker 01

  So from this lowest layer, we need to go like slightly up this tree of abstractions and find like a subcircuit for like all these little pieces of knowledge that the model has and for all of its like little decisions, perclivities. Maybe you find like how it decides to use a particular Python package or how it decides to decide which painters are impressionist versus post-impressionist, or figure out like which capital belongs to which state. So we actually have done some work in there too, and we see how this sort of knowledge is represented.

• 1:07:24Speaker 01

  The, like, for example, for the impressionist example, there's a community of, I think, just a few dozen neurons that represent this idea of which painters are impressionist, neo-impressionist, or post-impressionist. And then the neurons representing individual artists are linked to the tokens for those movements. And it's all like, just a simple little mechanism that will decide this topic.

• 1:07:57Speaker 01

  So I think we have to work our way up the tree of abstraction and just kind of compile this huge list of things at the neuron level, at the sub-neuron level, where we see the individual circuits for the different meanings that a neuron could have, and then up towards neuron communities and super circuits and so on, that just do these given tasks. Now, I think there's two things you have to do. One of them is you want to do this for the model in general, at least a few times, right? You want to understand the full model, all of the little circuits, all of the pieces of understanding that you can.

• 1:08:38Speaker 01

  If you understand the neurons, then you have the right primitives to like put into these slots. You can say, okay, now that we have a neuron community and we know what the neurons are, we can say what it's doing. So going really wide and getting this huge collection of things is important, but you also need to do things that are like really specific to your alignment and safety objectives, too. So that means doing a lot of experiments with data sets that elicit specific safety relevant properties, like reward hacking behavior, or scheming, planning, or whatever the case may be.

• 1:09:22Speaker 01

  You need to see how the communities affect this behavior and like how you can steer it in one way or another. And you need to see, yeah, basically what fires when those things are happening.

• 1:09:37Speaker 00

  I have a bunch more specific questions, too, but like, where do you think we are on this story? I mean, when I hear a little circuit that, you know, determines what wave of impressionism, you know, pre or post or whatever a particular artist is in, I'm like, okay, that's something. You know, it's a result, there would be, you know, a zillion of those, right? So in a sense, we could, you know, map out a bazillion of those things and then kind of still be sort of in the same place that we're at with respect to the neurons themselves, where it's like, okay, well, now we've got, you know, an equally long list, or maybe even a longer list of like micro circuits that do things.

• 1:10:22Speaker 00

  How does this ladder up into like a holistic understanding? Do you feel like you see that path from here? Because I think this is one of the things that people sort of initially were like, oh my God, you know, they've reduced interpretability to a normal science now, we can just grind the agents away forever and we'll like have a pretty good understanding of how the models work. And then other people were kind of like, I don't know, you know, we can like have, as you said, like a long list of neuron descriptions, or maybe, you know, an even longer list of these micro circuits that doesn't really tell us like macroscopically, like, what's it going to do next? Or, you know, what's it going to do if I put it in this situation it's never seen before, right? Like, yeah, I mean, I guess out of, out of distribution, generalization is always like

• 1:11:07Speaker 00

  a critical question. Do we feel like there's any traction on all this labeling, all this stuff? Like, can we now make any better predictions about what will happen when you put a model into a novel situation?

• 1:11:23Speaker 01

  Well, a lot of the neuron stuff is new enough that we have just begun to do this. There's a lot of interesting work online, inspired by our first essay on this, where one user found a single neuron that fires on reward hacking text. So, you know, that neuron could be used for monitoring potentially, at least for that subtype of reward hacking, maybe not all of reward hacking. There's also a lot of interesting work on persona creation and steering with neurons. And that's surprisingly practical.

• 1:12:04Speaker 01

  It seems like a lot of neurons just affect how the model responds and its personality. But getting back to your question about having this huge collection of micro circuits, huge collection of neurons, where does this get us? Like, how does it like actually turn into larger level primitives? I think to some extent, we haven't yet figured out what those higher level primitives are. Very much like how, you know, we have physics, we have chemistry, and we have biology. And theoretically, you could describe everything in physics or in biology in terms of physics,

• 1:12:45Speaker 01

  but you would never want to, because it's too, too granular, too detailed. And you don't need to, because there are higher level laws that have been discovered at the chemical level and then at the biological level. So I think for neural networks, we still need to find these laws and the science of the higher levels of abstraction. But I think by understanding the lower level stuff better, we can more quickly come to this, these next levels of abstraction and also validate them and like see cases where they might not apply.

• 1:13:28Speaker 01

  So you could have a persona steering vector, right? Which I think, you know, we've seen work in this before from Anthropic and others, where it's just a vector that can steer the model's persona in different directions. We can understand that in a neuron basis by just seeing like what neuron circuits lead to this persona, or which of them are important to it, which of them, if you ablate them, will cause problems with it. Are there any neurons that would like sabotage it if it got a particular piece of input where it like break the persona and cause it to do something different?

• 1:14:13Speaker 01

  Like how so many, so many models will like sometimes just snap and do something really crazy, like one out of 10,000 times or something like that. If you want to debug why the model does things like this, then I think neurons are a good place to look. Not necessarily guaranteed to have the answer there, but it'll be useful. But, okay, so I see our place being somewhere like we're discovering these primitives, like molecules, that are important to the model's behavior.

• 1:14:54Speaker 01

  But we need to understand alloys, we need to understand fluid dynamics, we need to understand these higher levels of science. But if you have the neurons, you can model all those things better. I see it also as a process that's happening from two ends. One of them is the lower level, get all the little pieces, get all the little components, see how they work. From the top down, we also see just like finding these vectors, like difference of means vectors or steering vectors that do these like big things. We don't necessarily understand how they work mechanistically.

• 1:15:37Speaker 01

  Or you do some training, and that also is very top down, but you start with the lower level, you start with the higher level, and you see, you keep working until they converge. So build higher level abstractions, more and more of these, until you reach the level where you see what's actually happening. When you train a model to do X, like what changes at these levels that makes it want to do reward hacking, that makes it eval aware, that makes it, you know, better at solving problems, that makes it more cooperative, and so on. Yeah, so basically, we have a lot of work left to do, but I think neurons are a big

• 1:16:22Speaker 01

  piece of that, and they will enable all the work that we want to do.

• 1:16:27Speaker 00

  So maybe I'd summarize that as saying first, you mentioned monitorability, that certainly stood out to me as like the obvious immediate application of this kind of work.

• 1:16:41Speaker 00

  You also mentioned, and it's right to call out, need to be careful that you don't just take the first neuron you find for reward hacking and monitor that and end up blind to other places where reward hacking may be invoked in the model. But nevertheless, it's like, you know, it seems like a, there's a great angle on monitorability there for sure. And then the other angle, I guess, when you talk about debugging, the way I'd tell the story is kind of like, when we thought that we had to use these like SAE-type constructs because everything was so messy inside, and now we're like, creating this thing, we've

• 1:17:21Speaker 00

  sort of created this additional somewhat opaque layer that when we have questions, we have a harder time getting to ground truth because we've now got to go through this. And so, to the degree that we can just do away with that additional rigmarole and get closer to ground truth, we're now like, at least debugging the thing itself as opposed to some reconstruction of the thing itself, which just means like, we're probably going to make a lot less mistakes in the debugging process and hopefully have a lot greater clarity that we're, you know, we're looking at what we think we're looking at. Would you, would you modify that? Or is that a decent account?

• 1:18:04Speaker 01

  Um, certainly, that's the hope. I think with, so SAEs can be used for a variety of parts of the model. I think for MLPs specifically, for neurons specifically, if you can get a better understanding in the neuron basis, it makes sense to use the neurons instead. And we have some things mentioned in our paper, and also there's a great paper by Stepan and Matush, Stepan Shabalin, and I think this just came out a couple of weeks ago. It's about how transcoders, which are like the, you know, the neuron basis version of MLPs, work slightly differently, but similar idea, dictionary learning. Basically, these

• 1:18:49Speaker 01

  just recapitulate neurons. They, the features are neurons, and when they're not neurons, they represent sparse combinations of neurons that often fire together and compose into a specific meaning. So certainly, if you use transcoders instead of looking at the neurons, you know, it might help to like, see which combinations of neurons often fire together and whatnot. But you have more loss if you substitute a transcoder for the MLP than if you just use the top K neurons instead, which really suggests that you should be looking more at the neurons. Now, SAEs still have some use for like, the residual stream,

• 1:19:34Speaker 01

  which we don't understand yet in terms of neurons, and is a more complicated object. Certainly, there is, you know, superposition and complicated geometry in the residual stream. But even for that, SAEs may not be the best choice. There's like, block sparse featureizers, which we have released a paper on. And basically, I think you need to understand the geometry of the residual stream better. That's one of the bigger things I think we need to do to do this reverse, ambitious, I can term ambitious reverse engineering,

• 1:20:14Speaker 01

  is understand the residual stream better. But yeah, when it comes to neurons, I think it makes more sense from my perspective to try to understand them really well.

• 1:20:24Speaker 00

  Let's maybe do a little lightning round, if such a thing is possible in this context, on just like, how this work interacts with, or maybe should cause me to update my understanding of some other parts of, you know, what I think I understand about what's going on in these, in these networks. Factual memorization and just factual recall, like, clearly, there's like, an unbelievable number of facts memorized in models. My sense had been that the facts

• 1:21:02Speaker 00

  live in the MLPs. I kind of always knew that was somewhat of a general, you know, an oversimplification. How do you, how would you update my thinking about facts? Like, do they, do we still think they live in the MLPs? Do they, like, this sort of one core role per neuron does leave me like, okay, then where, where are the facts in this thing?

• 1:21:30Speaker 01

  Yeah, that's a frequent question, because, you know, a model might have 5 million neurons, but no, way more than 5 million facts, right? But that's because you don't have a fact in a neuron. A neuron is not a fact, but the fact is in the connections between neurons. And you can have exponentially more connections between neurons than you have individual neurons. So, you know, a set of 10 neurons networked together can represent way more than 10 facts, just because the number of possible paths between them is just so high. So yeah, from what we've seen, certainly, we haven't found every single

• 1:22:10Speaker 01

  fact that's in the model and can say 100% they're only in the MLPs. But we find so many facts in the MLPs just implemented by these connections that it seems like the natural place for them to be. Like, the post-impressionism thing, you know, that's a whole bunch of artists and a whole bunch of concepts. And in a smaller set of neurons than there are, things that could be elaborated from the community.

• 1:22:38Speaker 00

  Do we see a different profile of factual recall degradation with sparsity as compared to like, like ARC-style tests? They can even do better with sparsity reasoning, they do worse. My gut would say factual would also be like, significantly affected by sparsity, but what did you actually observe?

• 1:23:01Speaker 01

  Um, so we did test some MMLU data sets, and were able to recover performance on those pretty well. And that does test factual recall. We do have some tests that we're still working on for like, medicine knowledge or chemistry knowledge. But for the most part, we really haven't seen much degradation. Let's see. Yeah, for MedQA, actually, a little bit of fine tuning restored that so that it's within a point of the original model.

• 1:23:40Speaker 01

  GPQA Diamond, also the same. We were able to restore super GPQA as well. So it's possible that if we did just like, huge sets of millions and millions of facts, and threw that at that, we might see some drop. But from the benchmarks that we've seen, there's, it performs quite well on factual data sets. And this is because, you know, we're not obliterating every neuron across every step, but every single token, you know, it's only a small set of neurons that need to operate. So every neuron still gets its chance to express

• 1:24:23Speaker 01

  itself. And every connection between neurons still gets its chance to express itself, but only in cases where it seems important and relevant.

• 1:24:31Speaker 00

  Yeah, interesting. I guess that that really just speaks to like, localization, which then brings to mind the nature of experts. The old story about experts was the experts themselves are also not interpretable bad news. You know, it's, we can't really identify what role the experts are playing. It seems like if the, based on everything you've said, like, that almost can't be true, right? Like, these circuits, like, they're localized, the facts are there, the router has to get that expert active. So there's got to be like, some meaningful expertise localization, or otherwise, the router would just have an impossible time picking out which of the eight out of 256 at each layer would actually

• 1:25:13Speaker 00

  be active, right? So like, should I now update my thinking that the experts actually like, are much more coherent in terms of the role that they play as well?

• 1:25:24Speaker 01

  Yes, actually, you should. The experts. So there is some earlier research on this. I think, basically, I forget the names of the papers, but they did run some data sets through experts and found that some experts tend to activate more on specific topics than others. But now we've found that at the neuron and representation level, where indeed, the experts that we examine are very, like, concentrated. They're like, mostly about one to three topics, and that's it. They're not about like, 100 different things

• 1:25:59Speaker 01

  mixed all together. Yeah, some really interesting ones that we found are like, let's see, there's a lot of mathematical and notational ones. Another one is like, legal slash constitutional jurisprudential text specialist. That's a one type. One about algebra. There's so many about different things with math. There's also like, timekeeping, units of time, and basically like, identifying how much time has passed. Geometry expert. Certainly, a

• 1:26:40Speaker 01

  lot of medical ones that are just like, about specific subtopics of medicals, medical things. But yeah, the experts are pretty focused on what they represent.

• 1:26:53Speaker 00

  Do you think that this

• 1:27:00Speaker 00

  Suggests like future directions in architecture. I mean, if we zoom out to kind of the super big picture, right? We're in this moment where we've got agent swarms on the loose. We've got frontier model companies saying, hey, we need some like antitrust waiver or something so we can coordinate on pacing the frontier. Jensen Huang says like, you don't need that. Just don't release unsafe products and like get ready to enter the big leagues, which I've been in for a long time, which is like, it's only 10 to 20% of my resources that go into making the thing. It's the other 80 to 90% that go into making it safe and reliable. So like buy more GPUs and burn more flops on monitoring and interpretability and all these things.

• 1:27:44Speaker 00

  And so I kind of imagine a world in which we maybe start to make that shift and we're like, okay, another way to frame this question might be in terms of like scaling laws for interpretability. But it kind of does make me think that maybe we want to go, or we should at least be open minded to designing architectures that are maybe somewhat less quick learners or take longer to train, maybe are somewhat even less efficient at inference because hey, we're going to be putting a huge amount of flops into interpretability and monitoring. But now we take into account this like 80% potentially that Jensen's talking about that's going into understanding and control.

• 1:28:29Speaker 00

  And so we can afford to pay those costs more because now it'll make like interpretability easier. Does this lead to any upshots in terms of architectures that you think would be more interpretable?

• 1:28:42Speaker 01

  It's a great question.

• 1:28:45Speaker 01

  Certainly the idea of an alignment tax where you just like pay some cost in the model just to make it more interpretable and more monitorable. Certainly there's some trade off where it's worth it.

• 1:28:59Speaker 01

  Although, you know, many companies will not want to pay that alignment tax, as it were. But I think one of the cool things about what we see here is that the models are already fairly interpretable and more than we thought. Like there's maybe, it's plausible, but not guaranteed, that maybe we need less of that, create an interpretability specific architecture that is expensive to run than we thought, because maybe we can just interpret the existing ones. But with that said, I do think that there are probably some tweaks that would make models easier to monitor and maybe easier to shape through training so that we can like have intentional design.

• 1:29:44Speaker 01

  And one of those things is probably if you have a more sparse inference regime, there's just less things that you have to monitor, right? Instead of tracking 5 million neurons at every step, you can track a much smaller number. And then you can also probably find cleaner circuits. So there are probably ways to incentivize models to be more sparse, either during training or after training. So like what we did with the sparsified Quinn model, you know, it performs as well or better on most benchmarks while still being quite sparse.

• 1:30:25Speaker 01

  There are some papers that take that further.

• 1:30:30Speaker 01

  I think maybe not purely in the MLP basis, but in terms of parameters. But I think

• 1:30:40Speaker 01

  basically things that push the model towards more sparsity will be useful. I also think localization is important too.

• 1:30:51Speaker 01

  There's one paper, I think it's from AE Studios that talks about like localizing, for example, potentially harmful biological knowledge or W and D knowledge in like parts of the model so that you can just like not have those working or just not even have those be part of the public release of the model.

• 1:31:14Speaker 01

  So if with our knowledge of neurons and expert specialization, we can sort of incentivize the model to put buckets of knowledge in a particular place, then we can just kind of ablate that when necessary.

• 1:31:32Speaker 01

  But yeah, I think, I think this points towards the potential for models that could be pretty interpretable without paying that like a huge alignment tax or a huge safety tax.

• 1:31:46Speaker 00

  Thank you for raising the topic of Gram gradient routing, something, something. I'm a Gram maximalist and I bring it up probably every other episode. So yeah, that's to the degree that anybody's playing a drinking game as they listen to the cognitive revolution, it's time for your Gram shot on this episode. The bingo card, it should be like all Gram squares. But I had this question around like the activation function. And there's this weird kind of double negative phenomenon that happens with some cases. You can maybe help me understand a little bit better, like what's even going on there?

• 1:32:24Speaker 00

  But the key thing is that because the activation function has this like ability to be positive and negative, unlike the original like Relu one, which was just like zero if negative or linear if positive, right? Now you have the potential for there to be double negatives. And it seems like this is like a very confusing thing that we don't really know what to make of it yet. When I saw that, I thought, well, geez, maybe this could be the sort of thing that could be the alignment tax that we'd pay. Like maybe we go back to a simpler, you know, monotonic, if not Relu specifically, you know, some monotonic activation function that would potentially eliminate this sort

• 1:33:09Speaker 00

  of bipolarity. And then, you know, maybe we don't have to worry about double negatives and maybe that learns a little less fast and maybe it requires a little bit bigger network, you know, more parameters for the same value or whatever. I'm sure that that won't pan out exactly as I just described it, but like, should we be thinking about like maybe somewhat simpler mechanisms in some cases?

• 1:33:32Speaker 01

  I think not necessarily. So the thing with Swigglu is that I think it allows for more

• 1:33:41Speaker 01

  more monosemiticity or more simplicity, because even the polysemiticity, when you have it, can just like be switched between a few different things through that mechanism. But I agree that the Q3, the double negative activations where you have the gate projection is negative and the up projection is negative. And then these sort of cancel each other out and turn into positive thing. That's kind of a weird case. And like you said, those are less interpretable examples that have, they're definitely more divergent from what the model is doing. Sometimes they're useful for what the model is doing, but sometimes not. There are some modern architectures that don't allow this to happen.

• 1:34:22Speaker 01

  For example, if you have Red Glue, you still have the gate and up projection, but the, instead of Silu in the middle, that gets multiplied by the gate, you have Relu. So I believe the Gemma four models use this. But this one by design is not going to have that double negative activation. Those perform quite well. We haven't done detailed analysis of the neurons there to say how interpretable they are, but it is certainly an option to have. Now, something that I believe I've seen when looking at Kimi K3 neurons is that

• 1:35:09Speaker 01

  they tend to almost never use this double negative activation. So I'm not sure in this case whether it's architectural or just that's the way the model's wired, but it may be more of a confusing thing that's particular to the Quinn 3.6 family or to Swigglu in general in these specific models.

• 1:35:35Speaker 01

  With that said, they're, they're very weird activations because you can ablate them sometimes, just like get rid of all of them. And other times it screws with reasoning or memorization, but you can just like put them back in swapped in the direction, like the token order is reversed and then that restores reasoning or you can like swap in all of these double negative activations from a completely different sequence and that will fix reasoning too.

• 1:36:11Speaker 01

  We're not sure exactly what they're doing. That's going to be the subject for future research.

• 1:36:16Speaker 00

  One thing I definitely also wanted to make sure we touch on is long context, basically. And this, you know, connects to reasoning to some degree as well. Maybe we should just establish how easy it was and give me any caveats on, you know, exactly how well it worked, but it was strikingly few tokens that needed to be brought to bear on the sparsified model to restore the reasoning performance.

• 1:36:44Speaker 00

  How do you account for the fact that, I think it was like, it feels like 10 million or something, like a really a very, very minimal number, basically brought reasoning back? Why in retrospect, like, how do we, how do we explain that to ourselves?

• 1:37:00Speaker 01

  Well, it was 100 billion, but yeah, pretty low.

• 1:37:02Speaker 00

  Still a small number in this context.

• 1:37:05Speaker 01

  Very, very small. Yeah, yeah, that's definitely way smaller than your average post-training run and way, way, way smaller than the pre-training would have been. Yeah, basically we were able to do it in like a single day on a few GPUs. So

• 1:37:25Speaker 01

  the primary problem with the long context degradation in the initially sparsified model was that the, I think the model just got confused. It would get, go into loops. It would lose track of where it was. It would self-check over and over again. Something about confidence was, was off. And I think that what the fine tuning experiment shows us is that the model still had the knowledge, it still had the basic reasoning mechanisms there, but there was some kind of calibration that was off that was the issue. And we don't know exactly what that was, but the fact that it took so little to restore it indicates to me at least that the model was really close.

• 1:38:10Speaker 01

  It was probably just like confused at having 90% of its brain mass effectively gone in this process of reasoning.

• 1:38:21Speaker 01

  We also saw, you know, as we did some experiments before fine tuning, where we had the model try a problem multiple times, but pass a four score or pass an eight score. And in these cases, its performance was equivalent to the original. So it had the knowledge, it just wasn't sure how to get there until we fine tuned it.

• 1:38:46Speaker 00

  Like, how, how much of a recovery did that achieve? I mean, I, I, I don't know quite how this plays out on like all chips, but there are at least some chip designs that will like skip multiplication by zero in a way that, you know, you could imagine, geez, if we could,

• 1:39:12Speaker 00

  on a layer by layer runtime basis, if we could just like round a bunch of things to zero and then not have to do a bunch of matrix multiplication because we've got a bunch of zeros there and we just know that, then you could imagine like potentially a negative alignment tax for some of these things, but it has to work, right? So like, when we talk about recovering performance, would you like, want to use the, you know, sparseified and like lightly post-trained model? Is it, is it like indistinguishable from a user experience standpoint? Or is it, you know, more of a like 90% sounds good, but there's still like a, you'll, you'll feel the gap kind of a story?

• 1:39:55Speaker 01

  I said, I think part of that you could only see if you use it a lot for a long time, but I have talked with the ablated model a decent amount. And from the queries that I gave it and from my experience with it, it was just as useful as the original model. It would still reason through things and give me the answer to the questions that I wanted. And, you know, they'd be pretty much the same answers as the original model.

• 1:40:27Speaker 01

  The recovery in terms of like, how much?

• 1:40:32Speaker 01

  So for several benchmarks, we saw not just recovery, but even improvement of, you know, five to 15 points of performance, which was like really, really surprising. We didn't continue post-training too long because we just stopped at the point where we reached parity for the most part. But it would be really interesting if this model could actually be trained to be, if you just gave it more than a hundred steps of RL, maybe it would turn out to be better in some ways. So we tried a wide range of benchmarks for mathematics

• 1:41:14Speaker 01

  and for code, as well as MedQA and basic knowledge, performed, you know, at parity or better on all of those. We're still running on some benchmarks to see if we can get, you know, better performance even on some coding tasks. But yeah, it's a great model still.

• 1:41:36Speaker 00

  Yeah, that's really super interesting and definitely kind of suggestive. I'll just point out that this is the sort of thing that a pause would allow us to maybe understand a lot better than we currently do.

• 1:41:49Speaker 00

  One other thing I wanted to understand a little bit better, again, this is on the long context thing. So some of the experiments were done with like attention frozen. And then you ask like, okay, now we've had, you know, whatever context up until this point, and the, you know, model has kind of figured out how to pass information from token to token. Now, how sparse can we go to just like predict the next token or, you know, to like run reasoning for a little while? And it seemed like the story there is like the, the single token prediction, you can go like extremely, extremely sparse. But if

• 1:42:29Speaker 00

  you want to allow reasoning to run for longer, then you can't go quite as sparse. And so ChatGPT actually gave me what I thought was a pretty interesting description of this, which is like similar to language in the paper, but a little different. And ChatGPT says, this suggests something like each new token's computation uses a small circuit acting on representations that were gradually built up by previous computations. And this sort of had me thinking like, okay, maybe that is sort of what some of these weak activations are doing, especially as we think like really long contexts. I mean, benchmarks in general don't have like super long contexts. And I don't know how many, you know, turns deep

• 1:43:13Speaker 00

  you're getting with the, you know, the ablated model when you're doing these tests. But, you know, in practice, in like real use, right? We're going like many turns and we're filling up the million token context window and then we're like, you know, compressing and won't be too long before, if not already, we like have sort of, kind of neural ease, you know, carry over from one context window to the next. And so this could be a way where maybe because it's just like relatively convenient to look at like single turn or like relatively short things, I could at least see a story based on everything I understand so far where weak activations don't matter on short time horizons much, but maybe they

• 1:43:58Speaker 00

  do matter over like a million tokens. And it's like a big part of what allows the model to sort of pick up on like the rich texture of the whole conversation we've had. What do you think? Do we have a, do we have a read on that? Or is that still something we, we will need to answer with, with further research?

• 1:44:19Speaker 01

  When it comes to million token windows, I think we'll definitely have to see how that pans out in future research. We've tested it on tasks that go up to 30, 32K tokens, and it still seems to perform well at those. And that's for like coding specifically.

• 1:44:41Speaker 01

  So one of our experiments suggests that for different topics, you need different amounts of basically neuron fires retained. So for reasoning and coding, you need more. For telling a story, you need much less, like by an order of magnitude. Certainly, there is something to the need to have information accumulate through the context and be stored, you know, wherever it may be, residual stream, KV cache, et cetera.

• 1:45:16Speaker 01

  This does give the model something useful to use, but up to 32K tokens, it seems okay.

• 1:45:24Speaker 01

  I suspect we can find, you know, thresholds that will like give us exactly how much, you know, we can ablate. But

• 1:45:35Speaker 01

  my suspicion is that what works for 32K will probably work further out as well.

• 1:45:43Speaker 00

  So where do we go next? I mean, I was kind of trying to think, what are like canonical results in interpretability that I would bump this up against? Or, you know, do kind of imagine like, what is the sort of neuron basis version of tracing the thoughts of a large language model came to mind? You know, unbelievably complicated setup there with like transcoders everywhere. I guess an optimistic take would be like, we should see something like tracing the thoughts of a large language model in the neuron basis as kind of your next act? Is that the sort of thing that you think will be achievable in the not too distant future?

• 1:46:25Speaker 01

  Yeah, I do, actually. Probably in multiple ways. You know, one thing we can do now, in fact, with the neuron basis is you can find the circuit for a particular passage, like you give a model a paragraph, then you just look at the circuits, the neurons that are firing there, and you ask another model, hey, here's the attribution graph and the neurons that are firing. Can you tell what, you know, what, what the paragraph is saying? And their answer is that usually they can. They can identify the entities, their relationships to each other and so on. Now, that doesn't tell us any more than what the text would have told us, but sometimes we want to be able to trace, you know, is there any hidden

• 1:47:09Speaker 01

  thought going on in the chain of thought? Or for these models that are trained to like communicate in neural ease or like direct transfer of KV caches, then we'll just like want to be able to read what's going on in there. So I'm definitely interested in exploring this more and finding out a structured representation of the model's thought processes. And that's definitely one of the big hopes for being able to understand neurons, that we can do this better.

• 1:47:38Speaker 00

  How accessible is this on a compute basis, by the way? Like, when you think about the unit of work, is it sicking the neuron scientist on one neuron or some, you know, higher order thing? And like, how much does that cost, so to speak? How do you map that on to like, you know, whatever, whatever the unit is, if we're going to roll it out across the whole model, what kind of compute budget did you spend on all this so far?

• 1:48:06Speaker 01

  Um, there's a lot of variables there, so it's hard to like give a single number. Certainly, I've also experimented with, you know, more and less complex neuron scientists. The most expensive one took forty-five dollars of inference for a single neuron, which is pretty crazy, but they're in general, yes. In general, they're much cheaper. Um, and we haven't tried to optimize for cost too much, but if you're running a local model that is in a scaffold to do the neuron analysis itself, um, I would say that's probably the best and cheapest way to go about it, like running a local GLM 5.3 or

• 1:48:50Speaker 01

  something like that, which performs quite well.

• 1:48:54Speaker 01

  Yeah, so my suspicion is that if you wanted to interpret an entire model, for the vast majority of neurons, you probably don't need to do the full neuron scientist on every single neuron, but you need a way to find out which ones are more difficult and then selectively apply intelligence and compute to the harder cases. So like you have a course pass over the whole thing and then just like some areas you get really intense with it.

• 1:49:22Speaker 01

  But yeah, I think it's definitely not a crazy price tag if you do that sort of process.

• 1:49:31Speaker 00

  And Lord knows, uh, there's a lot of compute out there these days. So, um, what, what cost could be too high to, uh, understand the alien minds that we're rapidly, uh, bringing online?

• 1:49:46Speaker 00

  Fascinating stuff. Anything that I didn't ask you about or that you would wanna, you know, clarify or just touch on or leave people with?

• 1:49:54Speaker 01

  I guess one of the things that I would want to leave people with is, um, the idea that you can try this stuff yourself, like, um, you can run a local model, extract the neuron activations and play with like neuron steering and neuron identification.

• 1:50:12Speaker 01

  We plan to release some labels for neurons that can be played with, but, um, for a smaller model, it's very feasible to just like get a bunch of labels and do attribution graphs and see how it looks. Another thing I want to say too is that, um, you know, from this work, we're definitely building on, um, what other people have done. Like we're not the only people who have explored neuron interpret. We've definitely gotten some heavy inspiration from other people, uh, including Transluse, who have done great work on neurons.

• 1:50:51Speaker 01

  There's always been like some group of people who's like still try to interpret things in a neuron basis. And so we reference those in our, um, report, but yeah, neurons are great. You should play with them.

• 1:51:03Speaker 00

  Well, it'll be a great thing indeed, if we can simplify the way in which we can analyze and make sense of neural networks. And this does seem like, uh, a notable step in that direction at an absolute minimum. So really appreciate, uh, the hard work that went into it and appreciate your time today, helping me make sure I understand what you have come to understand. Kurt Tigges, thank you for being part of the cognitive revolution.

• 1:51:29Speaker 01

  Absolutely. Thank you so much, Nathan. I've, uh, really enjoyed it.

• 1:51:41Speaker 02

  Walked by a yard this morning, had to stop and stare. A wool coat on the hedge and a kitchen chair. Dumbbells in a laundry basket, plates in the grass. I thought, bless their heart, and I started to pass. Coat and dishes and a rowing machine. Could it really be? One little sign and I see what you mean. Mono semanticity. I've known a few folks like that yard, hard to tell. Tuesday in a quiet loft, Friday raising hell.

• 1:52:23Speaker 02

  I used to call them scattered, I used to call them wild, till I found the one thing they wanted, and I smiled. Coat and dishes and a rowing machine. Could it really be? One little sign and I see what you mean. Mono semanticity.

• 1:54:14Speaker 00

  If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at ai podcast dot I N G.

• 1:54:58Speaker 00

  And thank you to everyone who listens for being part of the cognitive revolution.