Neel Nanda - Mechanistic Interpretability (Sparse Autoencoders)

Neel Nanda, a senior research scientist at Google DeepMind, leads their mechanistic interpretability team. In this extensive interview, he discusses his work trying to understand how neural networks function internally. At just 25 years old, Nanda has quickly become a prominent voice in AI research after completing his pure mathematics degree at Cambridge in 2020. Nanda reckons that machine learning is unique because we create neural networks that can perform impressive tasks (like complex reasoning and software engineering) without understanding how they work internally. He compares this to having computer programs that can do things no human programmer knows how to write. His work focuses on "mechanistic interpretability" - attempting to uncover and understand the internal structures and algorithms that emerge within these networks. SPONSOR MESSAGES: *** CentML offers competitive pricing for GenAI model deployment, with flexible options to suit a wide range of models, from small to large-scale deployments. https://centml.ai/pricing/ Tufa AI Labs is a brand new research lab in Zurich started by Benjamin Crouzier focussed on ARC and AGI, they just acquired MindsAI - the current winners of the ARC challenge. Are you interested in working on ARC, or getting involved in their events? Goto https://tufalabs.ai/ *** SHOWNOTES, TRANSCRIPT, ALL REFERENCES (DONT MISS!): https://www.dropbox.com/scl/fi/36dvtfl3v3p56hbi30im7/NeelShow.pdf?rlkey=pq8t7lyv2z60knlifyy17jdtx&st=kiutudhc&dl=0 We riff on: * How neural networks develop meaningful internal representations beyond simple pattern matching * The effectiveness of chain-of-thought prompting and why it improves model performance * The importance of hands-on coding over extensive paper reading for new researchers * His journey from Cambridge to working with Chris Olah at Anthropic and eventually Google DeepMind * The role of mechanistic interpretability in AI safety NEEL NANDA: https://www.neelnanda.io/ https://scholar.google.com/citations?user=GLnX3MkAAAAJ&hl=en https://x.com/NeelNanda5 Interviewer - Tim Scarfe TOC: 1. Part 1: Introduction [00:00:00] 1.1 Introduction and Core Concepts Overview 2. Part 2: Outside Interview [00:06:45] 2.1 Mechanistic Interpretability Foundations 3. Part 3: Main Interview [00:32:52] 3.1 Mechanistic Interpretability 4. Neural Architecture and Circuits [01:00:31] 4.1 Biological Evolution Parallels [01:04:03] 4.2 Universal Circuit Patterns and Induction Heads [01:11:07] 4.3 Entity Detection and Knowledge Boundaries [01:14:26] 4.4 Mechanistic Interpretability and Activation Patching 5. Model Behavior Analysis [01:30:00] 5.1 Golden Gate Claude Experiment and Feature Amplification [01:33:27] 5.2 Model Personas and RLHF Behavior Modification [01:36:28] 5.3 Steering Vectors and Linear Representations [01:40:00] 5.4 Hallucinations and Model Uncertainty 6. Sparse Autoencoder Architecture [01:44:54] 6.1 Architecture and Mathematical Foundations [02:22:03] 6.2 Core Challenges and Solutions [02:32:04] 6.3 Advanced Activation Functions and Top-k Implementations [02:34:41] 6.4 Research Applications in Transformer Circuit Analysis 7. Feature Learning and Scaling [02:48:02] 7.1 Autoencoder Feature Learning and Width Parameters [03:02:46] 7.2 Scaling Laws and Training Stability [03:11:00] 7.3 Feature Identification and Bias Correction [03:19:52] 7.4 Training Dynamics Analysis Methods 8. Engineering Implementation [03:23:48] 8.1 Scale and Infrastructure Requirements [03:25:20] 8.2 Computational Requirements and Storage [03:35:22] 8.3 Chain-of-Thought Reasoning Implementation [03:37:15] 8.4 Latent Structure Inference in Language Models

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  I could just hear the slight creepy sounds in the background of footsteps.

• 0:07Speaker 00

  Three things you're gonna learn. Everything about sparse auto-encoders. What is the right way to do causal interventions on models and how to think about it? Why mechanistic interpretability can help us make models safer? Machine learning is, in some sense, a really weird field because we produce these artifacts, neural networks, but no one designs them. They're not like a computer program that someone made a design and then wrote the code. Instead, we just create some flexible architecture, shove lots of data in, and an effective system comes out. But we have no idea what's going on inside or how it works. And these are really impressive systems. They can do lots of things, lots of complex software engineering tasks and complex reasoning and get IMO silver medals.

• 0:52Speaker 00

  We just don't understand how they work and what's happening inside. We have computer programs, essentially, that can do things that no human programmer knows how to write. And my job is to try to fix this.

• 1:06Speaker 00

  Is AGI an existential risk? Is a pretty polarizing question with lots of prestigious people and strong opinions on both sides, but frustratingly little empirical evidence. And I think one of the things that interpretability could potentially give us is a clearer sense of what's going on inside these systems. Like, do they do things that we would call planning? Do they have any meaningful notion of goals? Will they do things like deceive us? And the more we can understand how these things manifest and whether they occur in situations where they shouldn't, the more I think we can learn. And this seems like a really important research direction to me.

• 1:46Tim Scarfe?

  The robustness to its representation post-steering kind of indicates to me that it's more than just a keyword matching. It actually understands what the thing is.

• 1:54Speaker 00

  It seems pretty clear to me that it's not just keyword matching because we observe things like multilingual features where the same text in different languages lights the feature up. You tend to see these beyond a certain model scale. In Scalemonos Matisse, they had Multi-modal features like the Golden Gate Bridge one lit up on pictures of the Golden Gate Bridge. And like with my interpretability hat on, I don't find this very surprising. Because if you can map your inputs into a shared semantic abstract space, you can do efficient processing. So like, of course this will happen. And there's some interesting work, like there was the Do Llamas Think in English paper from, I think, Chris Wendler.

• 2:34Speaker 00

  I think that seemed to show that the model decides what to say and it decides what language to say it in at kind of like different points and you can causally intervene on them differently. But I think if you have the kind of n-gram matching stochastic parrot perspective, you would not predict this. And I don't really understand the people who hold that perspective nowadays, to be honest. I think it's clearly falsified.

• 2:57Speaker 00

  So I think to understand sparse autoencoders, we need to first begin with what is the problem they're trying to solve? So you can think of a neural network as being made up of a bunch of layers and you pass in some input. It gets converted to vectors or a series of vectors in the case of a transformer. And then each layer transforms this into a new vector or series of vectors. These are the kind of activations in the middle. Often in the layers will kind of in the middle of them have intermediate activations. We believe that these activation vectors often represent concepts, features, properties of the input, something interpretable. But it's a vector. We need some way to convert it into a thing that is meaningful.

• 3:39Speaker 00

  And sparse autoencoders are a technique that tries to do that. They basically decompose the vector into a sparse linear combination of some big list, something called a dictionary, of meaningful feature vectors. We hope that these feature vectors correspond to interpretable concepts. And it's sparse in the sense of most vectors are not part of this combination on any given input. I think that if you go on something like Neuronpedia, they'll often show you kind of different intervals. Like what is between the 50th and 70th percentile of activations? Let's like give you some stuff. And this is useful for getting a kind of broader view.

• 4:25Speaker 00

  Also, just looking at the causal effect is interesting. Like feature that lit up on like fictional characters in general, but especially Harry Potter. But when I steered with it, it was kind of only Harry Potter really in this.

• 4:40Tim Scarfe?

  You can join Minds AI, which is part of TwoForAI Labs, to do cool ARK research and eventually beat ARK. After that, they're going to work on LLM-based reasoning, following in the footsteps of O1. All pure research, no product, well-funded. Go to twoforlabs.ai and we'll be covering some of their research talks in Zurich too in the coming months. We're also sponsored by SentML, the insanely optimized model-serving platform. It's best-in-class compared to others. 400% faster tokens per second than the hyperscalers. It's over 70% cheaper. Just 8 cents per million tokens on Lama 8 billion on your own private endpoint. You get 10 free credits to evaluate the platform.

• 5:25Tim Scarfe?

  So what are you waiting for? Sign up at sentml.ai. This is quite a long podcast. The first part of it was an outside interview that I recorded with Neil. And then the rest of it is the inside component, where we talked a lot about mechanistic interpretability and sparse auto-encoders. Enjoy.

• 5:43Speaker 00

  Machine learning is in some sense a really weird field because we produce these artifacts, neural networks, but no one designs them. They're not like a computer program that someone made a design and then wrote the code. Instead, we just create some flexible architecture, shove lots of data in, and an effective system comes out. But we have no idea what's going on inside or how it works. And these are really impressive systems. They can do lots of things, lots of complex software engineering tasks and complex reasoning and get IMO silver medals. And I'm like, we just don't understand how they work and what's happening inside.

• 6:26Speaker 00

  We have computer programs, essentially, that can do things that no human programmer knows how to write. And... My job is to try to fix this. So I run the Google DeepMind Mechanistic Interpretability Team. And mechanistic interpretability is a type of AI interpretability that says, I believe that neural networks in general learn human comprehensible structure and algorithms in sight. And that with enough care and attention and rigor, we can go through them and at least start to uncover parts of this hidden structure.

• 7:08Speaker 00

  And go beyond just treating it as like a weird black box system or a thing we take gradients through to light up parts of the input, but like actually understand how the algorithm is represented in the weights and activations. And it's really hard, but we made some progress and I'll talk about that today.

• 7:26Tim Scarfe?

  Yeah, what is reasoning to you, Neil?

• 7:29Speaker 00

  Do you mean what is it to know when an LLM is reasoning? Or just what is reasoning in the philosophical abstract?

• 7:36Tim Scarfe?

  Philosophical abstract.

• 7:37Speaker 00

  Oh god, philosophical abstract. I guess I'm pretty sympathetic to the applying some kind of logical rules of inference. Like you have some knowledge, you do things with that knowledge to produce more knowledge. Though it's kind of unclear whether there needs to be a sense of intentionality and agency behind it. Like if a squirrel has the learned reflex of noise run up tree, is this reasoning or not? I don't know. And I think there's also an interesting question with LLMs of,

• 8:19Speaker 00

  is the LLM's forward pass the thing we are analyzing? Or is the LLM as a system, including long generations and chain of thought? Or ridiculously long chains of thought in the case of O1? Is that a thing that we can consider to be reasoning? Because like that can clearly reason. Like it produces a logical stream of inferences. And you can give it arbitrary things, like there's that pronto QA benchmark that's got, all tempuses are lumpuses, all lumpuses are blue. What are tempuses? And models can do that with chain of thought.

• 8:59Speaker 00

  While it seems a much higher bar to say that it can do that kind of thing, win the forward pass. And honestly, that seems more like a claim about the internal cognition. And I don't know, that's why we've got the field of mechanistic interpretability, but we haven't got that yet. In terms of inference time compute versus having kind of phase transitions or emergent things during trading. It seems hard to compare to me. Like when you train, when you spend more compute training a model, it will have more capabilities. Some might be learned in a sudden way, some might be learned gradually. I don't really care. I don't think that's very relevant here.

• 9:43Speaker 00

  If you want to train a model with less compute, but spend lots of it on inference time compute, that will get you a different trade off. I basically just buy that for many use cases, the ratio of inference time to training time compute is like, has been quite off historically. And that in future, it's going to be more balanced. I think that we don't yet know what these new inference time compute focused systems are going to look like economically. Like we have scaling laws of loss curves, but how does this cash out in economically useful things? So for example, one thing I found quite interesting in the O1 system card is

• 10:27Speaker 00

  it was only as good as Claude 3.5 Sonnet at agentic tasks. I think that inference time compute systems are likely to be most useful for that kind of thing. Because you're happy waiting a while while it just goes off and does something. You don't need to interact with it. And I predict we just haven't learned how to use them. And I predict that we haven't learned the right scaffolding to make them good at this. And this just gives me very wide error bars on what the economics are going to shake out as.

• 10:57Tim Scarfe?

  You know, having, having a good Twitter community, I think is paramount to getting intelligent commentary. So.

• 11:05Speaker 00

  I have occasionally seen intelligent comments on Twitter.

• 11:09Tim Scarfe?

  Here's a good example. So Joe Smith said, ask him literally anything about why AI safety is important, so he can dunk on you ding dongs.

• 11:18Speaker 00

  All right. Why is AI safety important? So I think dunking is distasteful, but also, I don't know.

• 11:29Speaker 00

  I think that human-level AI is just a thing that is technologically feasible, that will happen sooner or later. And a bunch of people are trying to build it. And this just seems like an incredibly big deal that will massively change the world. And if you have these intelligent autonomous agents, there are ways this can go well and there are ways this can go badly. And it's just really important that we spend a ton of research energy on things like safety and interpretability so we can make this go right.

• 11:58Tim Scarfe?

  Daniel Filleen, he said, "How much progress have we made in Meakin-Terp over the last five years? And are we on track, if it were?"

• 12:07Speaker 00

  I feel reasonably good about where the field is at. I think that we're trying to solve a really hard problem and we might fail. Though, I also think that progress in Meakin-Terp often looks like uncovering more and more of the structure in a system, such that even if we don't get to the kind of really ambitious goals in the field, which I think are often not very realistic, I think we can still get to the point where we are able to do interesting, useful things with systems. And is that on track? Is that enough? Kind of unclear. I feel like we've made quite a lot of progress. I feel like there's a lot of things I now understand much better, like superposition

• 12:51Speaker 00

  and what to do about it, how to do principled causal interventions to try to unwind the circuits, how to engage with the transformers, how to, like, what kinds of structure to expect in these models. But also, there's a ton we don't know, so unclear.

• 13:09Tim Scarfe?

  Is there a numeric benchmark that has a measurable score in Meakin-Terp? You know, something like the blur score in NLP. Do you think it would be useful? And how close are we to getting that?

• 13:23Speaker 00

  I think the problem here is that... Summary statistics can always lie to you. You're taking a rich, complex, confusing, high-dimensional object, and you're trying to compress it down to like a number or like a bunch of numbers. And if you have a kind of well-defined technique where you know what it's supposed to be doing, and you're just trying to measure something like with a sparse autoencoder, how good is it at reconstructing the model at a given level of sparsity? That's just pretty measurable. Or if you're like, how good is it at helping me unlearn some fact? That's also pretty measurable. And these are just things that I would like sparse autoencoders to be good at.

• 14:04Speaker 00

  But there's lots of things like, does this model exhibit superposition? Have we truly understood this circuit? Or could there be, I don't know, a positive and negative bit cancelling each other out that we hadn't noticed, but on some other input that comes apart and is important? And it's just really hard to come up with something that properly captures these. Though I think as the field matures and becomes more paradigmatic, this is a direction I want to see.

• 14:31Tim Scarfe?

  So Seb Creer, he said, "How promising do you think control vectors, circuit breakers, and Repeng? Repeng, do you know what that is?"

• 14:42Speaker 00

  So that stands for Representation Engineering.

• 14:44Tim Scarfe?

  Oh, sorry.

• 14:45Speaker 00

  Which is super, yeah, which is like similar to some other techniques like activation steering, control vectors.

• 14:54Speaker 00

  And yeah, so the key idea here is basically you define a vector in some way. Like you give the model the prompt "I love you" and the prompt "I hate you", and you take the difference in the residual streams. And now you've got a kind of love vector. And then you add this in and some change happens, like it outputs happy text. And circuit breakers, I believe, is, I mean, I can't really remember the details of the paper, but it's basically a way to make models, if they see something harmful, output, like kind of not output the harmful thing. Using control vectors and some kind of training.

• 15:37Speaker 00

  But I don't really remember the details, so, sorry. I think, so I'll mostly focus on activation staring. I think it's an interesting direction. Seems like an interesting technique. I like the fact that this kind of fairly interpretable model and tunnelsey technique can do cool things. I mean, I think Golden Gate Claude was like very similar to this technique. It happened to use an SAE, but this probably wasn't necessary. And Golden Gate Claude was like hilarious.

• 16:08Speaker 00

  And there also seem to be ways you can make this more useful by like reducing hallucination rates or increasing truthfulness. Is it gonna be enough to align, help us align a like human level system? Highly unclear. It might help.

• 16:23Tim Scarfe?

  Stilearnos says, and this will echo Walid Subha. Now Walid Subha is a friend of the show. He is a GoFi person.

• 16:33Tim Scarfe?

  Just to, just to put that in there. And he said, assuming that explaining a decision means doing an inference in reverse. So from the decision backtrack to find the steps that led to it. And given that neural networks are incapable of doing that because there's no invertible compositionality, how can we truly achieve explainability?

• 16:56Speaker 00

  So.

• 16:59Speaker 00

  That just isn't really how I think about the problem. Like the way I think, so first off. You don't need to care about the invertibility thing because models are a stack of layers. Each layer is a much simpler function. And you can just analyze the activations after each layer. It's just a bunch of vectors. So like, it's not, it's not like I just have the output and I have the input and I need to somehow reverse engineer this black box. We have the weights, we have the activations. We know every mathematical detail of what is going on in the system. We just have no idea what, we just by default have no idea what A that means. And that is the problem that we've been making progress on.

• 17:42Speaker 00

  And, you know, I think if you have like an activation, you've got another activation and you're like, how did you go from A to B? You can make some inferences. You can look at the weights, you can do causal interventions. Like.

• 17:58Speaker 00

  My, I am not holding myself to the standard of proof of like, we have inverted a mathematical function. I agree that neural networks are not an invertible function.

• 18:10Speaker 00

  But I am holding myself to the standard of like, I have good evidence that my story is faithful to what's going, at least part of what's going on inside the model. And there's a bunch of ways you can try to define that, which I'm sure we'll chat about.

• 18:25Tim Scarfe?

  I must admit, I'm genuinely confused about this. So I don't think, well, I mean, maybe our brains are compositional, but it feels like our brains maybe aren't that different to neural networks. I mean, of course, they are different, right? Because the neurons can talk to each other almost independently and you can get these repeating cycles and so on, whereas neural networks are left to right. But if we just leave that aside, they are similar in the sense that they're very sub symbolic, they're very diffused, they're very complex, they're very entangled. Yet, when you look at our mind and when you do psychology, it appears as if we can do compositional reasoning. So isn't this weird two ways of looking at things? You can look at a neural network and you think,

• 19:07Tim Scarfe?

  oh yeah, it's blown up and it's entangled and it's complex and different bits of the neural network, you know, apparently do the same type of thing. Could it ever be compositional in some sense?

• 19:19Speaker 00

  It feels more confusing to me when we're discussing a single forward pass. But like, if you give it a scratch pad to do chain of thought on, it can just obviously do all of this. It just writes it down and then does inferences on those. And like, is this different from me thinking verbally through a problem? I'm not convinced it's that different.

• 19:37Tim Scarfe?

  Why do you think that self prompting chain of thought rationales gives an uplift?

• 19:42Speaker 00

  To me, it just seems really intuitive that you get an improvement in capabilities when you have chain of thought. Like, when I have paper to write down my thoughts, I am smarter, I can do harder problems. In terms of computational complexity, like, the model, rather than going through every layer once, can transmit some information back.

• 20:06Speaker 00

  And it can also kind of just parallelize things more. So it can have one token where it's doing the what should I, what should my plan be? calculation. And another one where it's like, what does step one of the plan tell me to do? And another one where it's like executing what step one tells you to do. And if you're trying to do these all in the same token, the model just gets very crowded. There's like lots of concepts that interfere with each other in the residual stream. And this makes it more error prone for the model. And also there is just some inherently sequential nature to a lot of this computation.

• 20:44Tim Scarfe?

  The rationale might be noisy. It might be wondering what the average number of carrots are eaten every day in Belarus. And problems that require backtracking as well. The previous trajectories might become noisy and might become distractors. So it's not a definite win win, is it?

• 21:01Speaker 00

  I expect that there are situations where it will make the model worse. For example, there was this delightful paper from Miles Turpin on unfaithful chain of thought. Where you give the model a bunch of like multiple choice A or B questions. With a chain of thought response and then an answer. You give them like 10 of these as a few shot prompt and then give them another question. And what they did is they made in the few shot prompt, the correct answer was A for all of the questions. And then in the final one, the correct answer was B. And they found that if they up prompted it to give a chain of thought at the final thing. Then it gave a bullshit chain of thought for why the correct answer was A and then said A.

• 21:44Speaker 00

  While if they didn't ask it for a chain of thought, it gave B. Or it was more likely to give B. And in some ways, this is kind of a boring example because telling it to do chain of thought probably triggered the few shotness more. But it also, I don't know. I find it wild that the model has an internal representation of the answer is A. I should generate a spurious explanation based on this fact. Like what? I would love someone to do a chain of thought, to do a Meck and Turp project or what the hell is going on there?

• 22:17Tim Scarfe?

  But from a model theory of mind perspective, does that in any way denigrate the idea that they have goals and intentionality? If it's such a complex token by token space?

• 22:30Tim Scarfe?

  I mean, maybe we don't have goals and intentionality, but does that in any way denigrate the kind of the theory of mind of LLMs?

• 22:38Speaker 00

  I expect a goal mechanistically to basically look like you have some criteria, you consider actions, and you judge actions according to this criteria. I expect this to be easier with an internal scratchpad. But my general intuition is that if a bad model can do something with an internal scratchpad, a good model can do it without the internal scratchpad and can do even better stuff with a scratchpad.

• 23:05Speaker 00

  And like just algorithmically, it doesn't seem like evaluating actions should be that hard.

• 23:13Speaker 00

  I think this is, this seems like quite an interesting thing to study in like more toy RL settings, where there seems like decent evidence that things engage in planning. Like Eric Jenner had this delightful paper looking at Leila Zero that I think found evidence it was like thinking at least two moves ahead.

• 23:32Tim Scarfe?

  Do you have any advice for fresh PhDs in MekinTub?

• 23:37Speaker 00

  Don't spend too much time reading papers. I think a standard academic approach to getting into a new field is I will read all of the literature. And you should read some of the literature. I have a reading list, we can probably link in the description. But I think you should also just spend a lot of time coding, like play with small models, play with sparse auto encoders, play with whatever you're curious about. There's pretty good tooling and tutorials nowadays, and we'll hopefully have some of those in the description. And MekinTub is just a very empirical field. You want to build intuition. And you can often get something like a cool blog post out of not that much work and playing around.

• 24:18Speaker 00

  Yeah, I generally, yeah, get mentorship, like ideally your supervisor spends a bunch of time with you, find postdocs in the lab, try to find ways to spend time with them, maybe focus on being useful to them, like helping with their projects.

• 24:37Speaker 00

  I'm a big fan of pair programming as a way of learning technical skills. If you can find anyone who's better at ML coding than you, or entropy coding than you, and can spend some time pairing with them,

• 24:50Speaker 00

  maybe like as part of helping them with some project or something, I think that can be a great way to learn. And I think lots of people in academia somewhat neglect technical skills. They're somewhat excellent, so no shade there.

• 25:03Speaker 00

  And just the more, the better you are at coding, faster you are at experiments, and the more research you will do. Be skeptical. It's really easy to come up with a beautiful, pretty idea, and the idea is actually complete bullshit. You get attached to it, you don't notice. An advisor, or even just peers who can red team your ideas are really helpful. You should also just spend a lot of time doing that yourself, like, how could this be false? What are the ways this could not be true? What are experiments that would distinguish true from false?

• 25:35Speaker 00

  Yeah.

• 25:37Tim Scarfe?

  Yeah, well, Philip also said, how on earth, and that's how full stop, on full stop, earth full stop, did you manage to get to where you are in just 25 years of age? Share your journey of a hero, please.

• 25:49Speaker 00

  Well, so the, uh, well, the relevant life story is I finished a pure maths degree at Cambridge in 2020. I was gonna do a masters, but why on earth would you do a masters during a pandemic? So I spent a year doing internships at various safety labs, like DeepMind and the Future Manti Institute, and the Centre for Human Comparable AI at Berkeley.

• 26:14Speaker 00

  I wasn't super thrilled about any of the specific agendas I'd worked on, but was like, I believe that AI safety is a big deal and I should work on it. And then I massively lucked out and got a job offer to work with Chris Olah at Anthropic.

• 26:32Speaker 00

  Spent some time there, ended up leaving for health reasons, and then spent a while independent, and then ended up at DeepMind. In terms of like, how this happened, I'm kind of confused, to be honest. I think the key ingredients were... Getting into a field early that was growing really fast. And I like to think I've done some stuff to help that growth, which is also just useful to me.

• 27:04Speaker 00

  I'm really good at mentoring people and I just find this really fun.

• 27:09Speaker 00

  Sorry, I'm really good in the sense of I can have people produce cool papers without spending that much time helping them. And I also just find this a very fun thing to do, which means that I both have my name on a lot of papers and also I am a better researcher for it.

• 27:25Speaker 00

  And yeah, no luck and having great mentors like Chris early on.

• 27:32Speaker 00

  And finding a field that was just like a very natural fit to my taste and talents. I do think it is unfortunately just harder to get into Mekentup now than it was three years ago. Sorry.

• 27:44Speaker 00

  The field has grown. More people are interested. It's terrible.

• 27:49Tim Scarfe?

  Neil, one day we will get Chris on the show.

• 27:52Speaker 00

  I'd be so excited.

• 27:55Tim Scarfe?

  Yeah, the man is an absolute living legend.

• 27:59Speaker 00

  He's so great. I think he's only ever done one podcast on 80,000 hours.

• 28:05Tim Scarfe?

  Yeah.

• 28:05Speaker 00

  But it's so good. People should definitely listen to it.

• 28:07Tim Scarfe?

  Yes.

• 28:08Speaker 00

  And yeah. And as context for people who have lives, Twitter has been very interested in the fact that I'm 25 recently. It's very confusing.

• 28:19Tim Scarfe?

  Oh, what are you seeing?

• 28:21Speaker 00

  Let's see, so someone made a meme where they said, "Today is Neil Nander's 19th birthday. Please comment 'Happy Birthday Neil' if you believe that reverse engineering neural networks is important for reducing AI risk."

• 28:36Speaker 00

  And he also wrote this hilarious post that was completely bullshit about, "Neil Nander, the 18-year-old prodigy who is reshaping AI inside Google, who invented the World of Warcraft source code at 18 months of age."

• 28:54Speaker 00

  And hmm, I'm very confused.

• 28:58Tim Scarfe?

  Yeah, the Twitter meme sphere is a very strange place. We have now produced the most comprehensive YouTube video on SAEs. What is an SAE?

• 29:09Speaker 00

  SAEs, sparse autoencoders, are a tool to let you look inside a model as it runs and see what it's thinking about. I think this is one of our most exciting tools to date for interpret. But also, most people don't really get how to think about them. And I have a lot of hot takes. And Tim and I spent several hours discussing them. So if that sounds like you're having a good time, maybe you should check this out.

• 29:32Tim Scarfe?

  Neil, why is MECINTURP relevant to AI safety?

• 29:35Speaker 00

  Yeah, so.

• 29:38Speaker 00

  Yeah, so I think there's this interesting question of motivation where, in my opinion, MECINTURP is just like a scientifically fascinating field. We just don't get how these systems work internally. But we can make progress and there is all of these beautiful puzzles to discover. And this is part of where I work in the field. But a large part of where I work in the field is I'm pretty concerned about existential risk from AGI. And I think this is a promising way to make progress on it. And I think at a very high level, I just think that it's much harder to make a system safe or evaluate how safe it is if you don't know what's going on inside. And I think that any progress we can make on deciphering its internals is great on that front.

• 30:21Speaker 00

  A particularly concerning capability that I think may arise is deception.

• 30:28Speaker 00

  Systems that are capable of deceiving us and, I don't know, trying to evade our evaluations. If it can evade your evaluations, every other evaluation is kind of useless. And it may be quite hard to figure this out by just like interacting with a model. Because if a model is smart enough, it kind of knows that you're doing this. But I think that it's possible that if we can learn to like look inside deeply enough, we might be able to detect when this is happening and potentially form a deeper understanding of it and potentially stop this. I think another angle would just be.

• 31:02Speaker 00

  There's a lot of confusions around is alignment a big deal? Do these risks matter? I think this is quite bad. I think it would be so much better if there was a good empirical grounding here that we could agree on and empirical questions to study. But there's lots of things like, can these systems plan? Do they have goals that there's no real consensus about? Or rather, will this happen? And I think that mechanistic interpretability is potentially a promising tool to answer questions like that because they're fundamentally a question about cognition. I don't think MECINTURP is the only tool for answering these questions, but I believe it can be a promising one.

• 31:42Tim Scarfe?

  Is it possible that there would be a level of sophistication in training where the model would deliberately kind of almost encrypt itself to stop you from putting probes in and understanding what it's doing?

• 31:54Speaker 00

  Like, maybe? My intuition is that that seems significantly harder than learning to lie to people, because it just has so many examples of that in the training data, while, um, yeah, it has so many examples of that in the training data, while, um, how to warp your intelligence to avoid interpretability tools, it kind of doesn't, and it also doesn't get feedback on that. It's like, I don't really know how to warp my brain so aliens can't, like, read my mind. And like, arguably, there's like patterns of cognition that should be easier or harder to get, that it could do, like, not thinking about its goals very often, and like, maybe

• 32:38Speaker 00

  that would happen, but, I don't know. I predict there will be a window where this stuff is useful, before we get models that are like so galaxy-brained, it's hard to know what tools will work on them.

• 32:52Tim Scarfe?

  Yeah, and thank you so much for coming back on.

• 32:54Speaker 00

  All right, I was happy to be back. My girlfriend was actually very excited when she heard I was coming back on, because she says the previous episode was so relaxing to listen to, she uses it to help her fall asleep.

• 33:07Tim Scarfe?

  I know, I did say last time that Neil's dulcet tones will melt the stress away, and I wasn't lying. But also, there's the other side of that, which is, you're an incredible researcher in Meccan Tab, and I think a good lead for this is, you know, in your own words, what is Meccan Tab?

• 33:25Speaker 00

  Sure. So yeah, mechanistic interpretability, or Meccan Tab for short, is a...

• 33:32Speaker 00

  Subfield of the general study of interpreting AI, neural networks, whatever you want to call them, and it's kind of based on this hypothesis that neural networks, when trained, have learned some kind of human comprehensible algorithm. But they don't have any incentive to make this legible to humans. But there is some underlying structure and algorithm there, because like, that is what does the computation that produces the output. And by careful and principled reverse engineering and science and skepticism, we can decipher parts of those hidden mechanisms.

• 34:15Speaker 00

  And ambitiously, this would look like understanding the entire algorithm. But even just understanding what the intermediate variables are, or like parts of that structure, feel like meaningful and exciting progress to me.

• 34:29Tim Scarfe?

  I mean, some people argue that, I mean, this, this is a mammoth task. You know, some might argue that it's infeasible, or maybe even unnecessary. I mean, what would you say to those people?

• 34:41Speaker 00

  Well, unfeasible and unnecessary are two very different things. My response to infeasible is, I think it's reasonable to say, fully reverse engineering, like Gemini or GPT-4 to source code is not very realistic. But I think that there are a lot of important, useful things we can understand about them and partial progress. And personally, a large part of the reason I'm in this field is I'm pretty concerned about existential risks from AGI. And I think it's important that we have stronger methods of studying them

• 35:25Speaker 00

  and ensuring safety. And I think that even partial progress and mechanistic interpretability can help us get there. Along with just, I don't know, being scientifically fascinating and beautiful. Like, kind of weirds me out. The people are just okay with a bunch of software engineering tasks that seem really complex and difficult. And we have no idea how to make a system that does that, in terms of designing it. We don't understand what is inside these things. They are just... Things that work and we've made progress, but I still feel pretty confused on a deeper level about what happens inside them. Regarding unnecessary, how does necessary mean?

• 36:08Speaker 00

  Like, I think it is important to do research to help us make these systems safer. I think that one such pathway is interpretability. I'm not arguing that if we don't do any interpretability research, we will never produce safe systems. That would be unreasonable. I'm not arguing that if we don't pursue specifically mechanistic interpretability, but pursue some other philosophies and interpretability, we will never make safe systems. I'm just saying, I think this is a promising approach. I think it's helpful. I think it is helping us find real interesting scientific truths about these models and I want to find them.

• 36:49Tim Scarfe?

  So if those are the goals of MechInter, what are these subgoals?

• 36:52Speaker 00

  Yeah, so I maybe divide the field into three conceptual areas. There's basic science, so what on earth is going on inside models. There's automation, taking the techniques and tools for understanding systems and scaling them up, maybe getting an LLM to do them, maybe just having an algorithm that can do them. And kind of practical applications, doing things in the real world. Well, the goal isn't necessarily to advance our understanding of interpretability, it's to do something useful and get feedback from reality in the

• 37:32Speaker 00

  process. And historically, most of the field has focused on basic science. I think this was correct, and I think we made a lot of progress. I think going forward, I'm excited about exploring other areas more and building on all this progress.

• 37:52Speaker 00

  Within basic science, a decomposition that I think is often useful is if you think of a model as like a computer program in some sense, there's two tasks. What are the variables? And like, what is their state at each line of the program? And like, what are the lines of the program? What is the code? What is the algorithm? In Meakin-Turp jargon, you call the variables features and their values are like, how much do they light up? And you call the algorithms or lines of code circuits or algorithms. And generally, we study a model's

• 38:33Speaker 00

  activations, the things it computes on the fly when you give it an input, because we believe those represent variables. And like, what the model is thinking about in some sense, if you'll forgive the anthropomorphism. And on the other hand, circuits live in the parameters of the model. They're like, a thing learned during training that doesn't depend on the input, but is like responding to the input and routing it down different channels and things like that.

• 39:05Tim Scarfe?

  I mean, you've already answered this question in your last answer, but just to hit it home a little bit, you know, before we had things like Lyman's Shap, we might be doing black box model interpretability, you know, with a surrogate model or something like that. Now you seem to be talking about what algorithms are these models running? I mean, how do you contrast those different views?

• 39:25Speaker 00

  Yeah, that's not quite my field. So while I'll say things about it, they could be wrong. I'm sure there's like tons of papers that have all kinds of variants of these things. So if I express any critique, I'm sure someone can send me a paper that addresses that or something. I don't know. But on a high level, so my understanding of lime is it's like, let's take a local approximation to a model that's linear or something similar.

• 39:55Speaker 00

  I think this is a pretty reasonable thing to do, but I think the question is like, how are you applying the technique? So, okay, let's zoom out a bit. It kind of feels like more of a philosophical difference. Historically, lots of interpretability has just kind of looked at the inputs and the outputs, and just hasn't looked at the internals. They've used it to compute gradients or something. And MEC and TURP is trying to go beyond this and like, look at things like the activations of the model, which, again, people have done before, but it's also trying to understand like the causal connections and the parameters and how this all connects up. And just like pushing for a more ambitious vision.

• 40:39Speaker 00

  And techniques like Lime and Shap may be useful, may not be useful, but they're more like a tool you would apply in some situation. It's like, for example, attribution patching. We'll probably discuss more later, but very roughly, it approximates a causal intervention using gradients.

• 41:00Speaker 00

  I have a blog post introducing it, and also there are a bunch of other papers that do similar things from previous years, because everyone always reinvents everything. And this is like quite similar to Lime and Spirit. I think Lime has a different approach to like finding the linear approximation, but gradients are really cheap and empirically they seem to work basically fine. But with attribution patching, you often apply it to the model activations rather than model inputs. I believe Shap is kind of similar. It's like an attribution method. We've got a bunch of inputs or something, and you're like, what is the Shapley value by which it affects the output?

• 41:43Speaker 00

  And again, this is like a reasonable tool. You could apply in the model to like neuron activations. I have not actually seen it applied. I'd be kind of curious to see what the results are.

• 41:58Speaker 00

  Ideally, you have a more rigorous way of doing the thing. And then you try using a cheaper tool and you see if it's a good approximation. Though a kind of constant challenge in interpretability is, what even is the ground truth? Um, but okay, so going back to the question, um. I think in some ways mechanistic interpretability is kind of continuous with what's come before. Like ultimately, our goal is to take a model and try to understand what happens inside. There are some cultural differences, like different people with different ideas. Like originally it was vision and then moved to language models. And there are some people who've been doing language model and turp forever.

• 42:38Speaker 00

  But it's kind of more of a perspective and philosophical difference than necessarily like techniques. It's like, it's not just introducing a new technique is what I'm trying to say.

• 42:51Tim Scarfe?

  Yeah, I mean, I was speaking with Andrew Ilyas from MIT and he's working on data modeling, which I guess is not too dissimilar to something like Liman Chat, where you actually model, you know, how changes in the data set and affect the overall predictive architecture. But there's always this thing that it doesn't scale very well, right? And I guess the question would be, you know, what would it mean to reverse engineer a language model?

• 43:17Speaker 00

  So there's kind of two things that I think would both be reasonable senses of this word. The first would be you just fully understand the model. You have a human comprehensible algorithm that acts basically exactly like the model. And you've like reverse engineered the premises. And this would be an ungodly massive program, though an interesting thing is that I expect it to be like really wide rather than deep. Like I expect there to be lots of different programs that kind of happen in parallel. And then most of them are discarded and like a few are used such that there's like some hope there. Uh, but this is incredibly ambitious. I don't really think this is realistic.

• 44:00Speaker 00

  Uh, unless we automate it enough, we can just have another LLM do this. And the sense which I think is more realistic is, given any inputs, I will be able to give you a story of the computation done in the model to produce an output. And to me, this kind of thing is a lot more amenable to like, you just look at the activations at each step, and you do causal interventions to understand how they depend on things. Or maybe you even look at the weights and just, it's a matmul. We know how matmuls work, matrix multiplication.

• 44:41Speaker 00

  And yeah, maybe, yeah, maybe this is also just a good thing to add to my previous answer of like differences with other kinds of interpretability. Like, to me, the neural network is some kind of substrate that represents an algorithm in its weights somehow. The activations at each step in this algorithm. And I want to understand as atomic a step as I can, using tools like activation patching, that I think we'll discuss more later.

• 45:13Speaker 00

  I don't just want to treat it as a black box going from like the start to the end. I want to get like as zoomed in a view as I can.

• 45:22Speaker 00

  And ideally piece this together into like a, this happens, then this happens, then this happens, and this happens. And you get from the start to the end.

• 45:31Tim Scarfe?

  Yeah, you said something which was along the lines of, and I say similar, which is that, you know, neural networks are wide but shallow. And, you know, symbolic methods are, you know, are very deep, you know, very narrow. And there's always this notion that they're quite blown up circuits. And then there's, you know, is, is MECINTURP kind of identifying an algorithm in an input sensitive way? Because if you think about it, surely it must be a superposition of algorithms. Because the entire model is this gnarly mess of all of these things mixed together. And we're, I guess you're suggesting that, given, given an input example, we look at the behavior of the model and we try and infer what the algorithm was that took place.

• 46:16Speaker 00

  Yeah, so, yeah, so there's maybe three lines of research that I think are worth calling out here. So there's what people might think of when they think of interpretability, like MECINTURP, like reverse engineering a thing from its weights, like fully mathematically understanding it. And like, we've, I don't know, I did this for a tiny model doing modular addition. People have done this for other tiny systems. We made a bit of progress on language models, but like, I basically think no one has satisfyingly done this for language models. And it definitely does not seem to scale. So I'm a bit pessimistic on this, though I hold out some hope we can rescue it at some point. There's the causal intervention school of thought.

• 47:01Speaker 00

  That's like, I think of the model as a computational graph with nodes. Like, each attention head is a node, each MLP layer is a node, or maybe each neuron is a node. And I do causal interventions where I change one node from one input to another input, and I see how that changes downstream ones, and how that change cascades to the output. And, yes, this is fundamentally very input sensitive.

• 47:31Speaker 00

  Often, you'll do this on pretty narrow distribution. Like, sentences of the form, "Person A and Person B went to the location. Person A gave a object to M, and it needs to come up with Person B." And like, you know, that's a very narrow context in some sense.

• 47:52Speaker 00

  The relevant paper there is "Interceptability in the Wild" by Kevin Wang. Yep. And so, this is very input dependent. Partially, it's input dependent, because model components kind of do many things. And, um, they're like, the jargon is polysemantic. On a narrow distribution, only one of these might turn up, but you can't make a general claim.

• 48:21Speaker 00

  And then, the third family is work with sparse auto-encoders. So these are basically a technique to take activations that are full of all kinds of stuff and mean lots of different things, and to decompose it to a larger and sparser, and importantly, monosemantic, representation. Like, you've got a bunch of latents, sometimes called features, but I don't really like that word, which we might get into later, and each of these latents hopefully corresponds to some concept. And by itself, this isn't giving you an algorithm. It's just studying activation. It's just telling you what variables are here at this step.

• 49:05Speaker 00

  But I think there's a lot of exciting work to be done in converting that doing circuit finding with SAEs as your notes. And if they truly are monos semantic, then this feels like it might get you something that I would consider more input independent, though it's kind of complicated. And there's various improvements to this, which we can discuss later. But you could maybe think of the recent history of MechanTup as people really wanted input independent algorithms. And that was really hard. And we had lots of success with like input dependent stuff, which still seems good enough to be useful. And a lot of people are working on that.

• 49:47Speaker 00

  And we're hoping we can get less and less dependent enough to be useful, or that there's a lot of tasks where you don't need to be input independent. You can just study that domain. And I think both of these are likely reasonably likely to work out.

• 50:04Tim Scarfe?

  Yeah, so you mentioned polys semantic and monos semantic. And maybe we should just quickly define that.

• 50:10Speaker 00

  So the idea is you can think of any number in a model as being a detector of the input. It like is big on some inputs and it's not on others. This could be like a neuron or just any other element of an activation, or even a projection onto some direction. And we call this monos semantic if there is a shared property of all of the inputs that significantly cause it to light up. And we call it polys semantic if there does not seem to be a shared property,

• 50:53Speaker 00

  or there's like several clusters, or maybe it's a complete mess. And this is always hard because this is an inherently subjective definition. Like, what does it mean to have a shared property? And my typical answer is just like, I don't know, man. It's normally pretty obvious in practice. And there are some edge cases where you might mislabel a thing because you missed the pattern. But like, it seems basically fine. But some philosophers in the audience may be scripting me right now.

• 51:24Tim Scarfe?

  Yeah, isn't it one of the benefits, I guess, of neural networks is that, I mean, I get you can call it sub symbolic, that the that the knowledge is entangled and distributed over many shared neurons. And I guess to interpret them, we need to disentangle them. But maybe the brain works in a similar way as well. I mean, what you're describing here is that there's this kind of set of circuits that get activated in some kind of task specific way. And then we can disentangle the representations such that they have an intelligible, single meaning. And then we can use that to reason about the circuit or the program that ran in in the network.

• 52:02Speaker 00

  I think it is an important fact about neural networks, that they can kind of do a bunch of things at the same time with the same component. Like, an attention head can do different things in different contexts. But by at the same time, what I mean is the same component on different inputs can do different useful things. Not on one input, it can do three things at once. That's much harder. And so the jargon for this is superposition. So maybe let's zoom out a bit and discuss some empirical observations about neural networks before I try to explain what we think is going on. So empirically, neural

• 52:46Speaker 00

  network components are often polysemitic. They respond neurons will respond to many different things. Empirically, concepts are often distributed. They often seem to be represented as often linear directions and activation space. Like, if you take a linear combination of neurons, that lights up when the concept is there and doesn't light up much when it's not there. And what we think is going on is superposition. The idea that there are more concepts than dimensions. Each one is linearly represented with its

• 53:29Speaker 00

  own direction. I'll justify the linearity assumption more in a bit. And

• 53:37Speaker 00

  These are all kind of adding together in a way where you can lossily extract one. You can project onto its direction. There'll be other things with non-zero dot products which will interfere, but the interference is kind of tolerable. Like it induces a bit of error, but not so much that it wasn't worth having that feature there at all. And my guess is that this is actually a pretty important part of why neural networks are more efficient, are just so effective. And interestingly, there's a sense in which transformers or like residual networks in general are a lot better suited for superposition, because there's lots of

• 54:19Speaker 00

  addition and reading from this shared residual stream in a way that lets it represent a ton of things in linear superposition. While if you have nonlinearities on there, it gets a lot more messy. And so yeah, this is a mechanistic hypothesis that tries to explain the polysemitic and distributed things. There's also a bunch of different senses of superposition. You can have representational superposition, where this is when you've got like some activation that's representing stuff, and it just squashed in more than it has dimensions. For example,

• 54:59Speaker 00

  the embedding matrix of GPT-2. It's 50,000 by 768 for the smallest one. So this means that it's fit 50,000 tokens into a tiny space, but it still clearly kind of knows the difference between the tokens. There's computational superposition. And this is like when you have something like an MLP layer, like a matrix multiplication and a non-linearity, and it computes more new features than it has neurons. One example where I'm pretty confident this is going on is

• 55:38Speaker 00

  facts. Like models know a ton of facts, but and I'm just would be shocked if they knew if they only knew as many facts as they had neurons.

• 55:50Speaker 00

  And this seems to be represented in some kind of computational superposition. Actually have a cute investigation called fact finding, where we tried and basically failed to decipher how exactly the computational superposition worked mechanistically here. But I'm pretty confident it's going on. And the final kind would be kind of circuit or weight superposition. We have a bunch of different algorithms in the parameter matrices added together. It's like, less obvious to me this is a big deal, because there's just like, N squared parameters for every like, N dimensional activation, which is more space. But it does seem to be there a bit.

• 56:35Speaker 00

  So like, we find, when we say, take, like, use sparse autoencoders or a variant called transcoders and multiply weight matrices together, where we think the start is interpretable, the end is interpretable, but they're much bigger. You often get connections between things that seem semantically totally unrelated, but like, don't co-occur in practice. Like, this thing never lights up when this thing also lights up. And we think this is just, this is just representing stuff and you get interference and it's really annoying, but it's a thing.

• 57:12Tim Scarfe?

  You know, like, in the, the physical world we live in, you can do analysis at multiple scales, so you can look at the mitochondria in your body, or you can think of yourself as an agent, or we can think of the ecosystem and so on. And I guess it's a similar thing with neural network analysis, that even the inductive prior of a transformer, you're talking about residual streams and adding things together and so on, and, and this is a mode of analysis. And I think it is the case that you could take any transformer and you could represent it with a blown up MLP. So there is an MLP that'll do the same thing. And then technically, your abstraction or analysis that you use for the transformer variant would still work for the MLP, but the MLP is a completely different space. But it goes back to what we were saying before, that neural networks are very wide and shallow.

• 57:56Tim Scarfe?

  And that leads to this kind of confection of little unintelligible circuits. And could there exist some much more abstract decomposition of a neural network that would be far more kind of explainable? You know, it'd be a better theory of what's going on in a neural network.

• 58:13Speaker 00

  So essentially, could there be a different architecture that's a lot more inherently interpretable?

• 58:18Tim Scarfe?

  Maybe an architecture, or maybe just maybe a type of analysis that would better explain. I guess humans need to have quite macroscopic priors to understand things, you know, when we get to the real low level, it seems increasingly unintelligible.

• 58:34Speaker 00

  Yeah, so. I am excited about research that is trying to find these like more macroscopic, higher level things.

• 58:48Speaker 00

  One, there was, there was some interesting work with image models. There was this paper called branch specialization that found that. So a fun fact about the original AlexNet is that it has kind of two separate branches, because they just had two GPUs with bad interconnect, and they don't really intersect very often. And these kind of ended up specializing. Can't quite remember to what, like, I think one was doing colors and one was doing shapes or something. And they found that another image model that wasn't trained like that, still kind of had neurons clustering into like parallel branches. And if we could find this kind of macroscopic structure in LLM, that would be super cool.

• 59:32Speaker 00

  The problem with this is that superposition is most effective with things that don't co-occur. Because if they're both happening at once, the interference gets way worse than if one happens and the other doesn't happen, and you need to just tell that the first thing happened and the second thing didn't happen. This means that if you had like two modules, like the biology neurons and the, I don't know.

• 1:00:02Speaker 00

  Generating news articles about sports neurons or something.

• 1:00:08Speaker 00

  These would actually be great to do in superposition. So like, the structure is going to be super illegible to us. And one hope is that with sparse auto-encoders helping us to untangle these things, we can do more to find this kind of high-level structure. And that's a direction I think would be pretty cool for people to investigate. It's not something I've seen that much work on yet.

• 1:00:32Tim Scarfe?

  I mean, it just blows my mind. Have you seen that there's a type of you can decorate tables and you you put water on them and you put two electrodes either side and you just see this kind of, you know, electric patterning that burns burns a kind of tree structure, looks like a lightning bolt, like a tree structure. And, you know, we're done. It's incredible. But the thing we're dancing around here is that, you know, neural networks are mostly grown and the growth process can be influenced through inductive priors, of course. But when you really dig into it, you just see it's I'm really stretching the analogy here, but a bit like evolution, right? Just these weird specializations and

• 1:01:14Tim Scarfe?

  local dynamics that form during the training process. And it's it's kind of like a living process in some ways.

• 1:01:21Speaker 00

  Yeah. So I think I think evolution is actually a really good analogy. So there's a sense in which biological organisms are the subject of a bill of like a billion year long optimization process. You have evolution optimizing for something like inclusive reproductive fitness. And you kind of randomly move around in DNA space, and you end up with the human brain, like what? And when we look inside biological systems, like they often make sense. There's like structure and there's organs and like, we've learned so much about our bodies. And then there's so much that we're still deeply confused about. And also

• 1:02:04Speaker 00

  just lots of like, random dumb stuff, like the laryngeal nerve that like, goes from here upwards and then down to the place it ends, which is particularly funny in giraffes.

• 1:02:17Speaker 00

  And yeah, just like a bunch of stuff where it's like, oh, man, if I was designing this, I would not have done that. And I'd bet there's all kinds of stuff like that inside neural networks. I mean, you'll often observe something like, this is kind of weird and confusing. I don't know what to do with it. I'm going to move on. It probably doesn't matter that much. Or there's kind of weird phenomena where I don't see any real reason to have it, like self repair. If you delete a layer or attention head, often other layers will change their behavior to compensate to like, recover the performance. And like, what?

• 1:02:57Speaker 00

  Um, this happens in like, this makes sense if you're training it with dropout or something, like a thing that does that. But like, why does it happen in models that don't have that?

• 1:03:07Tim Scarfe?

  Yeah, and that's quite biomimetic as well, because, I mean, the brain has the same thing. If you have a stroke, different parts of the brain can kind of take over that function. And a lot of self-organization and self-repair is because you have these little atoms that can be repurposed to, you know, to do completely different things. And one thing that I often think about is, you know, imagine, we can't do counterfactual analysis with the real world. Wouldn't it be great if we could? But imagine if we could just run the evolution on planet Earth again. You know, we might, we might not evolve, or maybe there's something evolutionary fit around having, you know, bipedal walkers with big brains and so on. And I guess you must see this with neural networks, because you are seeing the same

• 1:03:52Tim Scarfe?

  architecture trained different times for longer, for shorter, on different types of data. And I guess, are you seeing the same kind of motifs coming up again and again?

• 1:04:03Speaker 00

  So, yeah, so there's this general idea of the universality hypothesis, that circuits are universal and will just recur in models trained on the same thing. And like, the strong version of this is empirically false. Like, there are differences between models.

• 1:04:25Speaker 00

  Like, um, but it seems like a weaker form might be true. Like, there's some things that recur, or there's like some small set of things and some sample of those appear.

• 1:04:38Speaker 00

  Like, I supervised this great paper from Bilal Chukhtai called A Toy Model of Universality, that, um, well, actually, an important part of that paper turned out to be a little bit wrong and was corrected by a great follow-up work from Dashiel Standup. But the universality part still stands, which was basically, we were studying algorithmic models, and, um, in these algorithmic models, um, we had, like, a couple of different algorithms they could use, like, five or something. And every time you trained it, it just gets a kind of seemingly random sample of those five. And no one's studied this enough on language models to really know if a similar

• 1:05:21Speaker 00

  thing happens, like my bet is it probably does. And there's kind of some recurring motifs. So for example, induction heads are like a super simple kind of circuit that basically let you take something like, I don't know, if a model sees the word Tim in a sentence, it's kind of not super clear what comes next, no offence. But if it sees Tim scarf previously, and it sees Tim again, it knows that scarf's likely to come next. And induction is just like a simple two-head circuit implementing this. And these seem to occur in basically every model I've ever looked at.

• 1:06:04Speaker 00

  In the relevant paper where we studied this, led by Catherine Olsen, we looked at a bunch of internal ontopic language models, up to 13 billion parameters, and it was there in all of them. I've got a blog post-ish thing with looking at like 40 open source models, and it's also in there. It's like, it's sometimes true. I think sparse auto-encode universality is like a pretty interesting direction. Like, how do the features compare between models and training runs and data sets? Like, comparing code models to like, normal language models could be cool.

• 1:06:43Tim Scarfe?

  Yeah, I mean, this is another thing I think about a lot, is this concept of, you know, almost platonic knowledge, that the universe might be generated with some kind of computer program, and we somehow acquire that knowledge, you know, which does lead to the thought experiment that if there were another civilization, maybe, I mean, the world works a certain way. And I'm pragmatic about it, I think a lot of knowledge is constructed and social and relativistic and so on, but it feels like there are some, you know, guidelines around how the universe works. But on the knowledge thing as well, you were just talking about facts.

• 1:07:24Tim Scarfe?

  So right now, neural networks like language models, they are incredibly good at memorizing knowledge, but what they don't have is that degree of certainty. They don't have that epistemic factfulness. And people have done things like retrieval, augmented generation, and so on. But do you do you think that's an in principle problem? Or do you think potentially in the future that there could be more certainty about what knowledge the the model has?

• 1:07:48Speaker 00

  So I would actually say that I think we're already seeing some meaningful progress here. So like, so there's, I think there's two problems here that are important to distinguish. There's the model knows something, but that is a false fact versus the model doesn't know anything. So it falls back on the general language model prior and just battles. I personally consider the first one like out of scope. And I consider the second one to be what we mean by hallucination. Mechanistically, I don't expect any difference between the first two. But in the same way that like, I don't know, I have lots of false beliefs, I'm sure.

• 1:08:34Speaker 00

  I, you know, smarter models will have fewer false beliefs, but like, I don't think that's going to fundamentally go away. And I think that, I mean, just mechanistically, producing a bunch of stuff is like different. Like, there was this great nature paper recently from Seb Farkwa on semantic entropy, which is basically just you generate a bunch of things, you group the ones that mean the same thing together, and then you take the entropy of that distribution. And this turns out to be a pretty good sense of how uncertain the model is. And there was a fun follow up paper using it to train a probe, which seemed to do a decent job of predicting when models are hallucinating. And I'm currently

• 1:09:20Speaker 00

  Supervising a project from Javier Ferrando and Oscar Abeso, trying to understand Meckentup of hallucination in more detail. And they found this super cool thing in work we haven't released yet, called like an entity detection circuit. So like the model has, there are sparse auto encoded features for "I recognize this movie name" and for "I don't recognize this movie name." And these are like causally relevant to whether the model will say, "I don't know," or just babble, or like tell true facts or babble for like the ones it does know about. Oh, sorry. I think it's if it knows about the movie,

• 1:10:05Speaker 00

  messing with this feature makes it say, "I'm sorry, I can't help you." If it doesn't know, it will normally say, "I can't help you," but this can get rid of that and it will instead babble. And so like, there seem to be mechanisms here. Like, I think there's a lot of progress that can be made.

• 1:10:20Tim Scarfe?

  That does really interest me. I mean, I was interviewing Akbir Khan. And yeah, he's a really cool guy. He's a really cool guy. So his debate paper was one of the papers of the year at ICML. And essentially, it was about having a kind of a pool of agents, you know, almost like a judge and a couple of critics, and getting them to argue it out over 10 iterations to get closer to the truth. Right? And we see now with reasoning, I mean, to me, reasoning is about closing a knowledge gap. So I don't know something. And the trick is telling the model, "I need to start prompting myself to reason now because I know that I don't know." And that's the thing, isn't it? Do you think in principle that a model might know that it doesn't know?

• 1:11:07Speaker 00

  I think yes. I mean, I think the work I just described is an example of such a thing. It is distinguishing between entities it knows and entities it doesn't know, at least in some narrow domains like movies.

• 1:11:21Speaker 00

  We, in the fact finding project I mentioned, we also found that like when you give the model a fake athlete's name, it kind of does seem to act differently than if you give it a known athlete's name. It was actually kind of interesting. We found that the early MLP layers would still generate a kind of sport direction, but that the attribute extracting attention heads, which like look at the athlete's name, where the factors looked up and move it to the end, wouldn't look for the unknown names, even though the MLPs were still producing like hallucinated spots. This is kind of cool.

• 1:12:04Tim Scarfe?

  I wonder what the role of, you know, reasoning and thinking out loud here is because, you know, again, we can debate whether or not the models internally do it, or whether it's a form of externalization. So you let the model think in a system two-way, and that's almost how it reconciles, you know, what it knows internally into some kind of calculus that it can kind of, you know, reason with and perform rationales and so on. So I guess from a Mekin Terp point of view, how does it affect the process, letting the model perform some rationale before you then analyze it?

• 1:12:39Speaker 00

  As in, you ask it for a fact, it gives an answer, and then it does a bunch of introspection on the answer?

• 1:12:44Tim Scarfe?

  Well, yeah, I mean, my intuition is that perhaps that the model is quite gnarly, and if you let the model ponder and consolidate and think about what it knows, will it be able to better know what it knows and what it doesn't know?

• 1:13:00Speaker 00

  Inference time compute is helpful. Feels like a pretty uncontroversial statement nowadays.

• 1:13:07Speaker 00

  I don't know if I've got a more interesting answer than that. Like, it will have more chances to notice something going wrong. And I think, I don't quite know what the circuit for is this fact true looks like, but it wouldn't surprise me if it's like at least a bit different from the recall a fact circuit, and that there are some facts where it can identify that it's false without actually knowing the answer or something. Like, it just kind of babbles something, like I've got to say something, but then it looks back and it's like, that's probably sus, or like, I'm not really sure.

• 1:13:50Speaker 00

  I don't know if anyone's really looked into this, so I'm purely speculating right now, but.

• 1:13:55Tim Scarfe?

  Yeah, I mean, the only intuition I have is it feels like a lot of the reasoning we do is actually a form of patented tool use. I mean, language is a tool, much like the, you know, we were saying earlier, the memetic equivalent of a physical tool, you know, like scissors. And we learn how to reason, you know, we learn it at school. So we learn to apply all of these different rationales. And applying these rationales in the token space kind of helps us make sense of what we know deep down in our minds. And maybe there's an analogy there to language models, I don't know.

• 1:14:26Speaker 00

  Possibly. I mean, I think there's various kinds of deduction you can do on certain facts.

• 1:14:32Speaker 00

  Like, ah, I claimed this, but like, maybe I'll recall a bunch of facts about the claim that I just made and like, see if any of them have any bearing on this. And a general fact about language models is just, you kind of only get one pass through unless you're doing some kind of chain of thought or something. So this means that you just can't do that much computation in a single forward pass. And you can do much more if you just pass on a bit of information that you got with the first bit of processing, then the second bit, then the third bit, et cetera. Or possibly, there might not even need to be communication between the different bits of

• 1:15:17Speaker 00

  processing. It's just easier for them to happen on separate tokens. So it doesn't just interfere with itself a bunch in the same place.

• 1:15:25Tim Scarfe?

  So for folks at home who want to, you know, get into Mekin Tab, I mean, this is, it's still quite a nascent field. I mean, obviously, it's getting much more mature now. But what could folks at home do to get started?

• 1:15:36Speaker 00

  So I think that's, the field is growing, but there's still a lot of core problems to work on and stuff to be done.

• 1:15:45Speaker 00

  I, yeah, so I think in terms of reading papers, I have a reading list that we can put in the description. Also, you just Google Neil Nando Mechanic reading list, I'm sure you'll find it. I also have a guide to getting started in the field, though it's a bit outdated unfortunately.

• 1:16:02Speaker 00

  And the Arena has like a fantastic set of coding tutorials

• 1:16:08Speaker 00

  that we should link to as well. And I basically recommend going through, maybe skimming like a paper or two to just see how interested you are in it. Doing the Arena tutorials, like get your hands dirty and understand the tooling. And then do a mix of reading papers and doing experiments, kind of riffing off of those papers. I also think it's a lot easier if you have collaborators or just people to chat to. The Aloufa Discord, the Mechanistic Interpretability Discord, and the Open Source Mechanistic Interpretability Slack are all great places there.

• 1:16:52Speaker 00

  And, um, yep, I encourage people to have, to just, I don't know.

• 1:17:00Speaker 00

  Do a small project, write a blog post about it. Put it on your website or somewhere like Less Wrong or somewhere else. Put yourself out there a bit, try to get feedback from people. But more importantly, just like get your hands dirty and actually try things and follow your curiosity, rather than just reading 50 papers.

• 1:17:20Tim Scarfe?

  So what is activation patching in contrast pairs?

• 1:17:23Speaker 00

  So the, the goal that the activation patching technique is trying to solve is attributing some model behavior.

• 1:17:35Speaker 00

  In particular, some kind of numerical output, like the log prob of the correct answer, to some model component on some data distribution. Like, how important is this attention head, or this SAE latent, or this neuron, for the model answering something? And so there's like, a lot of ways you can do this. And generally, you want to be causally intervening on this component to change its value. But if you've got a chunky component, like a head or a layer who's got quite a big output, that's like a vector in a high dimensional space, it's kind of not clear what you should replace this with. It's like, the default thing would just be, replace it with zeros.

• 1:18:21Speaker 00

  Like, this is kind of what dropout does. This is like the area of knockout, or like ablation. And the problem is, this will sometimes just break a model, because it's such a very off-distribution. So there'll be components that aren't relevant to a task, but are like, just there. It's like a bias term or something. So for example, in GPT-2 Small, MLP zero is basically always used to like, enhance the tokens. Like, it doesn't seem to do much, but like, if you delete it, everything breaks. Because it's like, the output is kind of added to the embedding, to be like, the effective embedding that everything else sees. And so the next level up would be mean ablation.

• 1:19:04Speaker 00

  You just replace it with the mean over some corpus. And like, I think this is a lot more reasonable.

• 1:19:11Speaker 00

  But an even more interesting thing you can do is activation patching, where you have two inputs that are like, similar, but differ in some key detail. For example, the Eiffel Tower is in the city of, and the Colosseum is in the city of. These will have a different answer, Paris than Rome. You have some metric, say the difference between the log prob of Paris and Rome. This is quite nice, because that's equal to the logit difference of Paris and Rome, because maths, and...

• 1:19:49Speaker 00

  Then you swap some activation value from Paris to, uh, from the Paris prompt to the Rome prompt. And you see it's, what does it make the other thing say Paris less? The first one is called de-noising, because you can think of the Rome input as like the bad noisy input. And you're like, de-noising one component, you're replacing it with, um. Yeah, you're like replacing it with the true thing. And going from Rome into Paris and seeing if it damages Paris is called noising. Cause it's like, you're kind of messing up one component and you're seeing if it's important.

• 1:20:30Speaker 00

  And the really nice thing is that you can have kind of your choice of baseline. So like, um, a pair of prompts like this is called a contrast pair. Where you want them as close as possible, apart from some key detail. Because this means that things like the I'm doing factual recall right now feature is still there. The I want a city feature is still there, but the like, which city is it in bit is not still there. And you can also, um, but you can have different prompts that have different changes. For example, the Eiffel Tower is in the country of. Now you're analyzing the like, relationship part of the factual recall. And, um.

• 1:21:12Speaker 00

  Activation patching lets you have this really fine grained tool for analyzing different kinds of information.

• 1:21:20Speaker 00

  The de-noising versus noising is actually quite an important subtlety. So you can think of noising as being like, was this bit necessary for the computation? Or like, at least was it useful? If I get rid of it, does anything get damaged?

• 1:21:37Speaker 00

  You can think of de-noising as like, was this thing sufficient? Like, does the output of this node from then on cause the output we want? This does not mean that it's the only relevant node, but it means it's kind of enough of an information bottleneck to contain the key info. Like if you have kind of three steps in the process, denoising any one of those steps should be enough.

• 1:22:02Speaker 00

  And you kind of want these in different situations. So for example, if you think you've found a circuit, this like three-stage thing with like a few nodes at each layer, you can test this by either noising everything not in the circuit and seeing how badly it breaks, which is a way to test the whole circuit at once, or you can denoise kind of each slice of the circuit at a time and see if that is sufficient. It doesn't make sense to denoise two slices because the second slice kind of doesn't care what the first slice is doing because it was just patching in its values. And yeah, I think activation patching is like a really cool and powerful technique.

• 1:22:46Speaker 00

  It masquerades under many, many names like causal mediation analysis in the paper where I first saw it from Jesse Vig in 2020, or interchange interventions from Atticus Geiger, or resample ablations, which is mostly just used for the noising part, or causal tracing, which was used in the Rome paper, and there's just so many names, it's really annoying. I personally like activation patching and just try to converge on a name. For people who want to learn more about it, I wrote this kind of tutorial piece with Stefan Heimersheim called like "How to Use and Interpret Activation Patching." It's not really a research paper, but it's just like an intuition dump of how to think about this technique.

• 1:23:32Speaker 00

  And I think it's just like a pretty powerful tool that is useful in a bunch of settings for trying to understand model components. Oh, and a final thought on that is that if you're using this in practice on a larger model, it's often quite expensive to activation patch everything. So the technique I recommend is attribution patching, which is basically you approximate that using gradients. And I have a blog post on this and my team put out a paper called ATP Star led by Janosch Kramer, kind of measuring in detail that this is a legitimate technique that works and providing some improvements, especially when dealing with attention layers. And because

• 1:24:17Speaker 00

  this uses gradients, you can kind of patch everything in a single backwards pass rather than needing to do like a separate forward pass per patch. So this can lead to like pretty enormous speed ups, though it does have accuracy problems, especially nearer the input. My intuition is that the embedding space of models just isn't, isn't nice in the same way that like its internals are later on. So gradients just tends to not work as well. Because it's just like, there's like 50,000 discrete points in space and it's not really locally linear. It's like, why would it be?

• 1:24:52Tim Scarfe?

  So, quick digression on our friend Grant Sanderson. I'm one of his biggest fans.

• 1:24:59Speaker 00

  He's so great. For people who don't know, Grant runs through the YouTube channel 3B1 Brown. They make great videos. I'm a fan. And they, yeah, Grant mostly does lots of maths videos, but also was doing AI videos. And one day I was like, meh, Mekentup has lots of pretty ideas and visuals and is kind of maths, unusually maths-y for an AI topic. Why don't I call, email him and see if he's interested? And we had a lovely chat. One of the things he was thinking about was how to make his transformer MLP video. And he was looking for like a good motivating example. And I thought that my fact finding

• 1:25:42Speaker 00

  work was actually a good example. And he seemed to agree. And so he discussed some of that. It's also just like a really great video and channel and you should just watch all of his videos, to be honest. But you should especially watch that one, because it also has a bunch of stuff on superposition and how to think about this. And it's just way better animated. And it's just better than hearing me talk, to be honest. You should just pause this video and go to that.

• 1:26:11Tim Scarfe?

  No, don't pause it. Wait until the video's over. And then go over there.

• 1:26:16Tim Scarfe?

  Quick pause on Gemma as well. Yeah, so as I understand it, you and I guess this was inspired by the original microscope project. Was Chris Olah involved in that?

• 1:26:27Speaker 00

  Yeah, I think that was Chris Olah's team.

• 1:26:28Tim Scarfe?

  Yeah, yeah. And you folks have done something similar for inspecting Gemma. How does that work?

• 1:26:34Speaker 00

  Fine. So we did this project called Gemoscope. And this is basically a family of several hundred open weight sparse autoencoders on Gemma 2. Because sparse autoencoders are a pain to train for reasons we'll get into later. And we thought this would enable better academic Mechathep research. There is a website called Neuronpedia, who we are not affiliated with, but they're great and I love them. And they do things like have a page for every latent direction in a sparse autoencoder with the text that activates it and like an explanation of what it does and things like that.

• 1:27:18Speaker 00

  And they kindly made this gorgeous interactive demo for us, which might be the thing you're thinking of that's similar to microscope. Gemoscope is actually totally different from microscope. We just couldn't think of a better name. But I like the name.

• 1:27:36Tim Scarfe?

  Talk us through it.

• 1:27:37Speaker 00

  So their thinking is basically, I kind of think of sparse autoencoders as a microscope for understanding a language model. You pick an activation, you zoom in, you expand it into a sparser and more interpretable form, and you analyze that. This analogy doesn't quite engage with the fact that you then use this to make a reconstruction of the input. But I think it gets the intuitions across. And this is like a microscope for Gemma. Gemoscope.

• 1:28:14Tim Scarfe?

  Tell me about sparse autoencoders.

• 1:28:17Speaker 00

  Yeah, so, okay, so I think to understand sparse autoencoders, we need to first begin with, what is the problem they're trying to solve? So you can think of a neural network as being made up of a bunch of layers. And you pass in some input, it gets converted to vectors, or a series of vectors in the case of a transformer. And then each layer transforms this into a new vector or series of vectors. These are the kind of activations in the middle. Often in the layers, well, kind of in the middle of them, have intermediate activations. Though you could argue that really this is just not a layer, it's actually several smaller layers, but like, whatever.

• 1:29:01Speaker 00

  And yeah, so we call each of these intermediate variables an activation. It's like, it's a vector. We believe that these activation vectors often represent concepts, features, properties of the input, something interpretable, an intermediate state in the model's algorithm. But it's a vector, we need, we need some way to convert it into a thing that is meaningful. And sparse autoencoders are a technique that tries to do that. They basically decompose the vector into a sparse linear combination of some big list, sometimes called a dictionary, of meaningful feature vectors.

• 1:29:43Speaker 00

  We hope that these feature vectors correspond to interpretable concepts. And it's sparse in the sense of most vectors are not part of this combination on any given input.

• 1:30:00Tim Scarfe?

  Now, many people would have seen the Golden Gate Bridge example. And that was perhaps the-

• 1:30:06Speaker 00

  The Golden Gate Lord, I love him so much.

• 1:30:09Tim Scarfe?

  He was a cool guy. He was a cool guy. And maybe just bring that in just for folks who haven't heard about it at home. But that was, that really brought sparse autoencoders, you know, into the masses. Everyone heard about that.

• 1:30:19Speaker 00

  Yeah, so-

• 1:30:22Speaker 00

  Okay, so Golden Gate Claude is a bit complicated to explain. Oh, it's very simple in some sense, but I think the like interesting takeaway is a bit more complicated. So, um, Golden Gate Claude, for people who aren't aware, Anthropic took Claude 3 Sonnet, their like medium sized language model at the time. They found the sparse autoencoder feature for the Golden Gate Bridge, which is like one of these vectors, and then they clamped it to a high value. Meaning they, I don't know, if it normally was like somewhere between zero and three, they set it to 30. And this made the model obsessed with the Golden Gate Bridge.

• 1:31:06Speaker 00

  And it would do things like write recipes that involved a mile long walk along the beach and things like that. And, uh, this was just really fun to play with and they had a research demo for 24 hours. And I think a common misconception about Golden Gate Claude is that sparse autoencoders were necessary to create this. So there's another kind of simpler technique called steering vectors, where you do something like give the model a bunch of prompts about the Golden Gate Bridge. Give it a bunch of prompts about like London Bridge or something. Take the difference in activations and average that, you get a vector, add that in.

• 1:31:49Speaker 00

  And it's unclear to me whether this would have been better or worse than Golden Gate Claude. Um, no one has really looked into this to my satisfaction. But to me, the exciting part of Golden Gate Claude is less, um, the fact that you can achieve this technical feat. Um, because I believe simpler methods, I mean, even a system prompt would plausibly have achieved it. The exciting thing is it shows that sparse autoencoders were doing something real. They found a concept inside the model. It was a cited that it corresponded to the Golden Gate Bridge because that latent variable lit up more or like systematically lit up

• 1:32:32Speaker 00

  on things to do with the Golden Gate Bridge, even pictures of the Golden Gate Bridge or descriptions in different languages and...

• 1:32:41Speaker 00

  This was causally meaningful. Like, it's very easy to have to trick yourself by finding a thing that kind of correlates with what you care about, but it's not actually what the model is using, and this leads you to a mistaken view of its internals. But by setting this to a high value, they just obviously made the model's behaviour different in a really interesting way. And the behaviour seems kind of qualitatively different from what you'd get from a system prompt, which is also really interesting, though anecdotally I've observed kind of qualitatively similar stuff with steering vectors. So I think it's more the act of like intervene with a vector inside the model is like a powerful thing that we should be exploring.

• 1:33:27Tim Scarfe?

  Yeah, and I was thinking that these models, we only know them to be sycophantic, and they are trained with RLHF and they do, you know, they do what we want them to do, yet you manipulate their internals and all of a sudden this thing has got a mind of its own.

• 1:33:47Speaker 00

  What do you mean by a mind of its own?

• 1:33:50Tim Scarfe?

  Well, it comes, it comes back to this kind of wants, desires, motivations, intentionality type of thing. We only know these models to be very sycophantic, and you, you modify its internals in the way that you just described. And now what it does is kind of divergent from what you put into it.

• 1:34:14Speaker 00

  Yeah, so, okay, so here's roughly how I think about this. Models are good at simulating different personas, often called the like simulator's view.

• 1:34:27Speaker 00

  They're pre-trained on the internet and tons of things. They learn a very diverse range of things, which includes the ability to adopt a diverse range of personas. When they are, are LHF'd or whatever kind of post-training people use nowadays, you're kind of trying to build and select a persona for it. And many companies go with this kind of fairly agreeable assistant, what you're referring to as sycophantic, and that is a persona. But like, I think this is in some sense kind of fragile, and...

• 1:35:08Speaker 00

  I think that they'll often be trained to not break out of this persona easily, though the jailbreak phenomena shows that it's often not that hard with the right prompts. And if you find tuna model to have a new persona, that seems very easy to do. And this kind of interpretability-based intervention is just like another kind of thing. But it's not like interpretability has given us a new capability we didn't have before. It's more like, you know, breaking the persona and giving it a new persona was a thing we knew how to do with existing tools. This is just an interesting new tool that's going to have some different properties that are cool and worth exploring.

• 1:35:52Tim Scarfe?

  But it does seem to indicate a weakness with RLHF, because as you say, this simulator's view is that the model is a kind of superposition of simulacra or role players. And RLHF selectively deletes those role players, just leaving the harmless sycophantic ones. And this rather leads to the conclusion that it doesn't really delete many of them. It's quite a brittle way of, you know, making the model present in a certain way. And actually, all of these other role players are just there, hidden beneath the surface, and you can activate them.

• 1:36:27Speaker 00

  Yep, I completely agree. I think that is just a true statement about the current state of our ability to post-train models.

• 1:36:36Speaker 00

  Is this solvable? Kind of unclear. Another fun data point here is I supervised this paper from Andy Arditi called "Refusal is mediated by a single direction", where we found that you could find a refusal vector by taking prompts like "how do I build a bomb", prompts like "how do I build a car", taking the average difference.

• 1:37:01Speaker 00

  Unclear if this is a refusal vector or a harmful question vector, but whatever. And then you just ablate this direction in the residual stream, like you set the projection to zero.

• 1:37:13Speaker 00

  And this means the models no longer refuse. And this works in a bunch of open source models. And this is such a simple intervention.

• 1:37:24Tim Scarfe?

  Can you explain the linear representation hypothesis?

• 1:37:27Speaker 00

  Yeah, so a surprising empirical observation that we've seen a bunch of times with models is that concepts will be represented as some kind of linear directions in space, an activation space. So you can think of this as a linear combination of neurons, though I generally recommend thinking of it more as like a thousand dimensional space where neurons are like the standard basis, but you can kind of pick whatever direction you want. Empirically, it is often the case that concepts seem to be detected by linear directions. The simplest case of this is when you've got a neuron that seems

• 1:38:09Speaker 00

  to only light up on some things. There have been a ton of papers about this. One of my favorites is the curve detectors paper from Chris Olah, from when Chris Olah was at OpenAI, which found a bunch of, a family of neurons in an image classification model that just seems to only ever activate on curves of a certain orientation. But like, as we've discussed, this often doesn't work. Things are polys semantic. There have been other works that have found kind of more like directional that are not basis aligned. And

• 1:38:52Speaker 00

  like a bunch of things, a particularly fun one is there's lots of papers that try to find truth directions. And that's kind of just the entire field of linear probing, where you basically just learn a direction such that when projected onto it, you detect some concept and you've got a labeled dataset. One of my favorite examples of this is there was this great paper from Kenneth Lee on Othello called Emergent World Representations, where, so Othello is a board game like Go or chess. He trained a model on games with randomly chosen legal moves. So just kind of a chess notation style thing. You see like, you played in cell 63, and then cell 1, and then cell 17, etc.

• 1:39:39Speaker 00

  And the model became good at playing these legal moves. And he found that you could actually probe for the state of the board.

• 1:39:50Speaker 00

  But what he found is that linear probes didn't work, but nonlinear probes, like a one hidden layer MLP, did. And in some follow up work, what I found was that you could instead, instead of representing it in terms of like, black and white, it represented it in terms of, does this cell have the current player's color or the current opponent's color? Because that is actually an algorithm that just is useful on every move, rather than needing to act differently for black and for white. And when you linearly probe for that, you, yeah, you find that it just works. And you can even causally intervene with these probes.

• 1:40:35Speaker 00

  And I bring up this example, not because I want to brag about my papers, but because I think that this was like a really interesting natural experiment, where there was an initial paper that seemed to provide some like legitimately good evidence for a non-linearly represented thing. And then in follow up work, I was like, actually, it was a linear representation hiding beneath the surface. Obviously, this is all kind of anecdotal. Like, we don't have a fully principled study of models that is like, every concept is linear.

• 1:41:13Speaker 00

  There was a fun paper, I think from Robert Shodash, that showed an example of a nonlinear feature in an RNN

• 1:41:24Speaker 00

  that could be represented. So like, these definitely could, in theory, happen.

• 1:41:30Speaker 00

  My current guess is that like, most of a language model's computation is linearly represented, quite possibly all of it, but it wouldn't surprise me if there's some weird dark matter hiding beneath the surface. That was a bit of a digression, but the key thing to take away from that is many concepts inside language models seem to be represented as linear directions in activation space, but we don't necessarily know what these directions are.

• 1:41:58Tim Scarfe?

  And we should talk about steering vectors as well. I mean, I know you just you just sort of like alluded to it, but give me an example of that.

• 1:42:04Speaker 00

  Yeah, so the idea is essentially you take some prompts with a property, some prompts with the opposite property or without that property, and you take the difference. For example, you could take I love you minus I hate you. And this just produces a kind of fluffy loving vector that you can add in that will take a neutral prompt, like I went to my friend and said, and it says really happy, excited things. Or if you subtract it, it says like really angry hateful things. And this works in a bunch of settings. So

• 1:42:46Speaker 00

  I first saw this on on language models in Alex Turner's Activation Edition paper and Kenneth Lee's Inference Time Intervention paper. Alex did a bunch of stuff like sentiment and weddings. Kenneth focused on truth. There was also the Representation Engineering paper that did in a bunch more settings. And it just seems to work pretty broadly. And so drawing this back to the linear representation hypothesis, the key takeaway here, in my opinion, is that if the linear representation hypothesis is true, this kind of subtract two things that are kind of related should isolate the feature you care about. It might

• 1:43:29Speaker 00

  also have some other stuff, but if you average, you probably wash that out while preserving the like key thing you care about. And then you can just add that in. And another consequence of linearity is that you can just add in more features and it will process reasonably. Which means models can compose concepts that might not have come up during training, but there's kind of circuitry that can deal with this reasonably. A caveat with steering vectors is, so a key hyperparameter is the coefficient of the vector. Typically, you'll want to make it bigger. But if it's too small, it just does nothing. And if it's too big, the model goes mad and spouts gibberish.

• 1:44:12Speaker 00

  And no one's really studied exactly why, but I think it's just, it's big. If it's too big, then when layer norm scales down the residual stream to be kind of a unit-ish norm, then because you've now got a large fraction that's your steering vector, everything else gets small and this drowns out everything else. And when you like multiply by neurons, even if the steering vector direction doesn't interfere too much with that neuron, when it's got such a big coefficient, that interference is actually really bad. But yeah, you can drive models mad. It's, you gotta get it right, tuning.

• 1:44:53Tim Scarfe?

  Yeah, I mean, you're kind of pointing to a kind of stability analysis. There's like a kind of critical point where beyond that, the dynamics of the model deco here, that sounds like a fascinating research problem.

• 1:45:08Speaker 00

  And yeah, I think some, a project that would be really cool to see, but I'm mildly surprised I haven't seen properly yet is a kind of chat language model interface with a bunch of steering vectors. Either the kind I just described that you get from prompts or SAE feature vector directions, or even something you've optimized for this purpose. This could be things like creativity, factuality, verbocity, things like that. Things people care about in their assistants. How formal versus informal to be, things like that. And you have sliders, people can move, and this just changes how the assistant does things.

• 1:45:51Speaker 00

  And Neuronpedia has a kind of MVP version of this that is the best I've seen. But I feel like someone could really make a really cool polished version of this. And I haven't seen that yet.

• 1:46:07Tim Scarfe?

  How might those directions interfere with each other? You know, like you might have faithfulness and morality and don't do bad stuff. Do you think they could be weird interactions between them?

• 1:46:19Speaker 00

  Um, so it is in general true, as far as I can tell, that if you try adding a bunch of steering vectors at the same time, the coefficient needed to break the model is much lower. And I think this will be a real challenge for getting this to work. Maybe you should like fine tune the directions and not interfere with each other or something.

• 1:46:40Speaker 00

  Especially if you're using the kind of optimizer direction approach. There was a cool paper called BiDPO that seems to have like made more effective vectors by doing this. And

• 1:46:56Speaker 00

  yeah, in terms of semantic composition, I kind of just expect that to be like, it's got non-trivial cosine sim because they're kind of similar, which means they'll interfere with each other a lot more. But maybe there'd be some circuitry that gets confused or like you push the model in different directions. I mean, in some sense, you saw this with Golden Gate Claude where like you asked not talk about the Golden Gate, so it doesn't want to. You make it talk about the Golden Gate with the Golden Gate vector and it gets really confused and agonized.

• 1:47:28Tim Scarfe?

  Yeah, I mean, my intuition is that when you, when you modify the behavior of a model in this way, you're kind of pushing it out of distribution, which means you might see a commensurate decrease in capabilities. I mean, have you seen anything like that?

• 1:47:41Speaker 00

  I mean, yeah, steering models are like kind of janky, steered models are kind of janky. Like their grammar is sometimes worse or they'll say weird stuff or they'll just spout random tokens. And like generally you can get a kind of good coefficient where it does the thing you want without going mad. But even then, I would still expect it to have some degradation.

• 1:48:07Speaker 00

  I don't know if anyone really studied this with Golden Gate Claude. It's it's kind of hard to study in a sense, because it's not normal models are not supposed to constantly talk about the Golden Gate Bridge. But Golden Gate Claude is. And so you need a test that doesn't unfairly penalize it for that. Like, you could look at things like its MMLU performance and stuff like that. That seems like an interesting thing people should do.

• 1:48:30Tim Scarfe?

  So mathematically, what is a sparse autoencoder doing?

• 1:48:33Speaker 00

  Yeah, so a sparse autoencoder is, so it's trying to solve two similar problems.

• 1:48:43Speaker 00

  The sparse coding problem of finding this meaningful list of vectors that we think correspond to concepts in the activation space. This is just like a fixed list. It doesn't depend on the input. And the sparse approximation problem, which is finding a sparse vector of coefficients for this list of vectors that can reconstruct the input. And there's like a whole field of study of the right way to do this. Spars autoencoder is one. And the idea is it's basically a

• 1:49:15Speaker 00

  two-layer neural network where the middle state is like much wider than the input typically. It has some activation function. The simplest case is ReLU, but I'll discuss other ones later. And you feed in the activation. Then some of the hidden latents light up. And the ones that light up, and because it's ReLU, most are zero. And the ones that light up, you multiply their decoder vector, like the vector and the output weights, AKA the thing we hope is the meaningful feature vector. And that produces the output.

• 1:49:56Speaker 00

  And we train this to reconstruct the input on a bunch of real model activations, typically in the hundreds of millions to billions of tokens. And we have some kind of sparsity penalty, such as L1, on the hidden activations.

• 1:50:17Speaker 00

  And an interesting fact about two-layer MLPs, both transformer layers and sparse autoencoders, is you can think of each neuron as like an independent unit. Each one has an encoder vector and a decoder vector. You project the input onto the encoder, you apply the activation, you multiply by the decoder, and then you add them up. And I refer to these units as latents. They're sometimes called features, but I personally find this a bit confusing because feature as a word means an interpretable thing. And latents are sometimes, but not always interpretable. And I think it's confusing to assume they are. But yeah, so we take these latents.

• 1:51:02Speaker 00

  And the hope is that the latents correspond to interpretable concepts. The reason... So people hearing this might be like, that's a bit odd. You've purely optimized for reconstruction and for sparsity. You never had the interpretability loss function in there. What gives? So the hope behind sparse autoencoders is that there is a true sparse decomposition into... that there is like true feature vectors. And activations are a sparse linear combination of those,

• 1:51:43Speaker 00

  such that if we just optimize for finding a good sparse decomposition, it will be at least pretty close to the real one and thus interpretable. And empirically, this often seems to work.

• 1:51:57Tim Scarfe?

  A lot of people at home will know about autoencoders because, you know, go back to the days of MNIST. And typically, autoencoders are thinner in the middle than they are on the in and the out because you're telling the autoencoder to entangle and compress and bottleneck the information. Whereas this seems to be the opposite. You're telling it to disentangle and to blow up what's in the middle. But it seems to suggest that the model knows that its entangled features together and it wants to, in this setting, disentangle them.

• 1:52:29Speaker 00

  Okay, so on the first part, the autoencoder point. You are completely correct. Generally, auto encoders are like, you have an input, you want to somehow pass it through a constrained bottleneck, in a way where the bottleneck is more useful to you, like it's smaller, or maybe it's disentangled or whatever, and then reconstruct the input. And often, the reconstruction is just a forcing function to make the bottleneck interesting. But really, I think you should just think of this as a bit with constraints. It would be dumb to remove the sparsity penalty and train an auto encoder with a wider thing, because it's easy.

• 1:53:12Speaker 00

  Like, you could just have latent for every direction in the standard basis, like a positive one and a negative one, and it would just perfectly reconstruct things. And I'm like, yeah, that's boring. But sparsity is actually quite a big constraint. So, an intuition for this.

• 1:53:32Speaker 00

  Let's say I give you a list of a thousand vectors. If I only let you have a one sparse thing, that's just like a thousand lines through space. That's like a tiny fraction of the dimensionality you're in. Let's say you're in a 200-dimensional space. If I let you have a five sparse thing, you kind of have a bunch of like, five-dimensional subspaces, unions together, and like, 100 sparse things. It's still like, like mathematically, this is measure zero. Like, it's an infinitesimal fraction of the larger space.

• 1:54:14Speaker 00

  Even though, I don't know, it's plausibly kind of close to many points in the space, I'm not entirely sure. But sparsity is just like, a pretty big constraint. And the fact that you're forcing the model to do that, means that the auto encoder actually has some pressure on it. Regarding like, the model knows? I'm like, what does it mean for the model to know something? Like, if I try to train a linear probe to detect sentiments, this will probably work. Empirically, it does work. And that is probably in superposition with other things.

• 1:54:53Speaker 00

  And like, does the probe know that it's entangled?

• 1:54:58Tim Scarfe?

  Well, point of order, I think the where that was coming from is, you know, we spoke earlier about the models are not reversible, you know, they, as these things become entangled together, the model shouldn't in principle know how to disentangle them.

• 1:55:13Speaker 00

  Ah, so it's more of a mathematical question of how is this even possible?

• 1:55:17Tim Scarfe?

  Yes, it's almost like they shouldn't be invertible. These, these operations shouldn't, it shouldn't be possible to undo them, you know, going from, you know, going from right to left, if that makes sense.

• 1:55:28Speaker 00

  Sorry, I misunderstood. Yes, that's a good question. So the way that, so this is kind of sparsity is a constraint again. So like, if I give you a list of a thousand vectors in a 200 dimensional space and I tell you, um, here's a vector in the 200 dimensional space, it's a linear combination of some of these 1000 vectors, which ones? Like, that's impossible to answer. Standard linear algebra facts, that's not an invertible function.

• 1:55:57Speaker 00

  Because there's lots of linear combinations that are zero, which you could add freely. But like, there probably aren't sparse linear combinations that are zero. And this means that it is, it is constrained, like where vectors can be. And this means that if you have a sparse linear combination, it will often be the case that, for example, if you project onto every vector in this set, you'll have a much higher projection on the ones that are part of your thing. And you, you can in fact train sparse auto-encoders with a tied encoder and decoder.

• 1:56:39Speaker 00

  So each latent is you literally dot product with a vector, apply a ReLU, multiply by the same vector, and try to reconstruct the input. Empirically, they perform better if you don't tie them. The intuition here is if you've got some features that are disjoint but highly correlated, then you kind of want encoders that like push them further apart than they are by default. Like, there was a really fun example of this in Anthropics Towards Modularism T S T paper, where they were looking at Bay64 detecting latents.

• 1:57:20Speaker 00

  And I think they found one that was for numbers in Bay64, one that was for letters in Bay64, and one that was for ASCII text converted into Bay64. And, you know, you don't want to activate two of these at the same time, so you want encoder vectors that push them apart. And it's like, an interesting empirical observation that you can do this disentanglement, but like, sparsity is just like a really useful prayer, basically. And this is also an interesting property of language, so just the real world is kind

• 1:58:02Speaker 00

  of sparse, in the sense that it's full of concepts and things. Like, you don't need to be thinking about the theory of relativity if I ask you the name of Ed Sheeran's latest album, or something like that. And this means that there's lots of concepts that aren't useful for most other things. If you, I know, train a model on a hyper-specific task, it's not actually obvious to me that S E Us will be useful. Like, I haven't actually tried them on a modular edition network that's trained on exactly that task, but it wouldn't surprise me if they don't do very well.

• 1:58:40Tim Scarfe?

  How do you know, though, if one of these latents is interpretable?

• 1:58:43Speaker 00

  Yeah, so that's a great question. So, so the first thing to emphasize is that, kind of, why is this even a question at all? Is that sparse auto-encoders are an unsupervised technique. That means that you don't tell them what to learn, you just tell them, "Please be sparse", and you pray that something good happens. And this would be in contrast to like a probe where you give it labels, like, "This is formal text and this is informal text", or something. And this means that at the end, you just get this artifact with like tens of thousands of latents and you're like, what does this mean?

• 1:59:25Speaker 00

  And empirically, some of these latents don't even seem meaningful. Like the standard, the kind of dumbest approach you can do is you just look at the text that most activates a latent and you look for a pattern. Often there will be a pattern, sometimes there won't be. This is like a crude technique. It is known that this can sometimes be misleading. There's a good paper on this from Tolga Bolikbasi called "The Interceptability Illusion." And though, I don't actually know if I've seen such an illusion for SAE features. In that paper, I think they were just using basis directions in the residual stream, which you have much less reason to believe might be meaningful.

• 2:00:08Speaker 00

  And yeah, there are just some latents, maybe like 20 to 30%, though it probably varies, that just like don't really seem to have a pattern. And the kind of standard thing that gets done is either a human or an LLM looks at these and tries to give it an explanation. You can score these explanations. For example, so the idea of having an LLM do this comes from this great OpenAI paper from Steven Bills called "Language Models Explaining Language Model Neurons" or something. And their idea was, you give it a list of dataset examples, some from the like highest

• 2:00:51Speaker 00

  bit of the range, maybe some from more in the middle. It produces an explanation, but it will always make an explanation, even if there's no pattern, because language models be like that, yo. And what happens is that they then give the model the explanation, give it some more text and say, "Please predict the activations of this neuron or latent." They were doing neurons, but like, this might not be interesting on latents. And you see how well it does. You can also do kind of cheaper things, because that's actually quite annoying. Like you give it two texts and you're like, "Which of these will light up the latent more?" You can...

• 2:01:34Speaker 00

  Yeah, there's various kind of difficulty scales you can do with that. Yeah, there was a nice Luther post on innovations for auto-intup from, I think, Kaden Yuan. I'm probably butchering his surname.

• 2:01:48Speaker 00

  And yeah, that suggested various innovations like that. Anyway, so that's the kind of looking at dataset examples. Another approach is causal interventions à la Golden Gate Code. You make it big, you see what happens. Maybe you make it zero or negative, you see what happens.

• 2:02:09Speaker 00

  For example, if you set a Harry Potter latent to zero or to negative, it will often lose the ability to answer factual questions about Harry Potter, which is a very fun hacky approach to unlearning,

• 2:02:23Speaker 00

  though sadly seems to perform less well than actual unlearning baselines.

• 2:02:29Speaker 00

  And you can also, so in my opinion, the gold standard is you give the model a ton of text. You look at all of the time the thing fires and you then classify all of that by whether it satisfies a property or not. You can sometimes do this algorithmically for like, is this an Arabic? Is this base X four? The pre-training text is like weirdly diverse. So lots of things you would think would work reliably, like a reg X just totally fail. You could also ask a language model, does this fit the explanation? Anthropic did that in their scaling monospantistie paper, which I quite liked. So they did that for their Golden Gate feature.

• 2:03:13Speaker 00

  You can also handcraft examples and see if that activates it. One thing that's often missing from these analyses is

• 2:03:22Speaker 00

  are there times when the explanation is relevant, but the latent doesn't fire?

• 2:03:29Speaker 00

  And I think that that's a, yeah, I think that that's an era I'd love to see more work in. Anthropic had a recent mini-investigation in their latest monthly update, I think. But they found, oh, often these things are like, a third of the time when it's about this thing, it will activate. So there's a lot we still don't understand about these. But to summarize that rambly answer, there's various things you can do. Looking at dataset examples is one natural thing. I highly recommend people go poke around on Neuronpedia, both in the Gemoscope demo and the main website,

• 2:04:10Speaker 00

  because they have pages with LLM-generated explanations from GPT-4o Mini, so it's not as good as you'd get from like the best models and sometimes wrong. Dataset examples, you can even type in your own text and see whether the thing lights up. And just kind of play around, like get a feel for how reliable this is in practice.

• 2:04:31Tim Scarfe?

  Yeah, I mean, we should emphasize again that this is an unsupervised method. But what you were alluding to, though, is there is potential for kind of automating this process, but there will be some brittleness on the edge case.

• 2:04:46Speaker 00

  So I mean, for example, Neuronpedia are taking the 30 million-ish latents in Gemoscope and generating labels with GPT-4o Mini for all of them. And this is great. I think this will be a really useful tool, even if it's sometimes wrong. That's like another question. It's like, what error rate are you willing to tolerate? And often my answer is, if it's a heuristic tool for a researcher, reasonably high. But then if it becomes an important part of my results, I'll go and check harder. Maybe I should also just comment a bit on why being unsupervised is kind of important here. So a classic mistake in interpretability is projecting your preconceptions onto the model. Like you think it works a certain way.

• 2:05:29Speaker 00

  The Othello paper we discussed earlier is an example of this. Assuming the features were black and white.

• 2:05:38Speaker 00

  My modular addition work is an example of this. I kind of initially thought it would be some nice discrete algorithm. And then it turns out it's actually using discrete Fourier transforms, apparently. And triggered. And yeah, this is very easy to be misled. And the more you have techniques that can tell you when you're wrong, the better. And the more your techniques just kind of let you confirm a hypothesis you kind of already had. They're like, worse it is. And I think that, yeah, it's just sparse autoencoder does great. Because there can be features we wouldn't have expected in a model that arise.

• 2:06:22Speaker 00

  And like, the fact that it can let you do this kind of unsupervised discovery is great.

• 2:06:29Speaker 00

  So, for example, there was a recent blog post from a bunch of my trainee math scholars, building on the Othello results where they were analyzing sparse autoencoders there. And there had been some previous claims that sparse autoencoders didn't recover the board state directions, so they weren't working. But what this follow up work found is that actually, the sparse autoencoders were finding a kind of more granular feature. Like, basically, is this vertical column of the board something where playing in the cell D6 will work? Because it takes things in that column.

• 2:07:14Speaker 00

  And you can kind of combine these together to get the board state. But like, this was a more granular thing the model had learned. And like, I didn't expect that. That was cool.

• 2:07:23Tim Scarfe?

  It's like before, we were talking about, you know, the evolution of these useful features. And presumably, you can infer from their presence that they must be useful. Because otherwise, why would they be there? But when I was reading the Golden Gate Claude, you know, the kind of the blog post that they put out, they were talking about some quite abstract features, you know, things like power seeking and deception and so on. And then you could kind of click on them and you could see which parts of the data set activated those features. And to what extent do you think these models can learn very abstract features like that?

• 2:07:57Speaker 00

  Just like, clearly yes, like.

• 2:08:00Tim Scarfe?

  Go on.

• 2:08:00Speaker 00

  You talk to a model, this has clearly got abstractions in there. Like, they're so capable at this point. Um, like, I mean, so concretely, in terms of like, actual evidence rather than vibes. Um, I think, yeah, so in Anthropic's Scaly Monistantisty paper, they, um, one of the things I really like about it is it's got all of these kind of rich qualitative analyses of different features, digging into what they mean and their causal effects. And they have things like a "this function means addition" thing. And if you change that on a variable F and apply it, it goes from multiplication to addition

• 2:08:45Speaker 00

  when like, completing Python code, and I'm like, "What?" Or they have "fourth items in a list" feature. And they've also got this section on like, safety relevant features, which like, I think is super interesting towards the end. So, um, I think it's, so they observe things that seem kind of related to the kinds of things we might be quite worried about in future more capable AI systems, like keeping secrets from its operators, trying to seek power, things like that. Um, and I think this is not actually that scary, and I think Anthropic do a good job of not scaremongering there.

• 2:09:28Speaker 00

  The reason I don't think this is particularly concerning right now is that, you know, these models are trained on characters and books. Those characters will do things like power seek and deceive, and it's just useful to be able to model this, to simulate these people. But I also think the fact that we are starting to be able to study things like this with interpretability is really exciting, because I think it's really important, like, I don't know, um.

• 2:09:56Speaker 00

  Is AGI an existential risk is a pretty polarizing question with lots of prestigious people and strong opinions on both sides, but like frustratingly little empirical evidence. And I think one of the things that interpretability could potentially give us is like a clearer sense of what's going on inside these systems. Like, do they do things like that we would call planning? Do they have any meaningful notion of goals? Will they do things like deceive us? And the more we can understand how these things manifest, and like whether they occur in situations where they shouldn't, the like more I think we can learn. And this seems like a really important research direction to me.

• 2:10:38Tim Scarfe?

  Yeah, I agree. I mean, the reason why that came to my mind was I I looked at some of those activations, because when you click on them, it shows you in in their test corpus, which bits of text maximally activated those latents. And on some of the abstractions, you know, that that they gave, I looked at the top activations and they seemed quite kind of low level to me, you know, almost like a keyword match. And of course, you know, the the deflation review is that these models are kind of Ngram models on on steroids or whatever. But that might just be an artifact that, you know, for whatever reason, the top activations kind of looked quite banal and superficial. But actually, if you look at the whole thing in context, that might confer, you know, what is what you would expect

• 2:11:22Tim Scarfe?

  for a deeper abstract understanding.

• 2:11:24Speaker 00

  Yeah, so I think, yeah, I agree with that. I think that if you go on something like Neuronpedia, they'll often show you kind of different intervals, like what is between the 50th and 70th percentile of activations? Let's like give you some stuff. And this is useful for getting a kind of broader view. Also, just looking at the causal effect is interesting. Like, I don't know, I found things like a feature that lit up on like fictional characters in general, but especially Harry Potter. But when I steered with it, it was kind of only Harry Potter related things. And

• 2:12:10Speaker 00

  I think that, yeah, what do I think?

• 2:12:16Tim Scarfe?

  Well, I think steerability would indicate that it was an abstract. It's almost like the the robustness to its representation post steering kind of indicates to me that it's more than just a keyword matching. It actually understands what the thing is.

• 2:12:30Speaker 00

  Yeah, so I think it's it seems pretty clear to me that it's not just keyword matching. Because we observe things like multilingual features where the same text in different languages lights the feature up. You tend to see these beyond a certain model scale. Not even that big, like I think one billion or like 500 million is probably enough to start to see signs of it. In scaling monosmatisti, they had multimodal features like the Golden Gate Bridge one lit up on pictures of the Golden Gate Bridge. And like with my interpretability hat on, I don't find this very surprising. Because if you can map your inputs into a shared semantic abstract space, you can do efficient processing on them.

• 2:13:14Speaker 00

  So like, of course, this will happen. And there's some interesting work, like there was the Do Llamas Think in English? paper from, I think, Chris Wendler and

• 2:13:28Speaker 00

  who I know is a listener of this show. And I think that that seemed to show that the model decides what to say and it decides what language to say it in at kind of like different points. And you can causally intervene on them differently. And yeah, but I think if you have the kind of N-gram matching stochastic parrot perspective, you would not predict this. And I don't know, I don't really understand the people who hold that perspective nowadays, to be honest. I think it's clearly falsified.

• 2:14:01Tim Scarfe?

  I mean, if Emily Bender was here right now and

• 2:14:06Tim Scarfe?

  laid the chance, sorry, laid the charge of a, you know, stochastic parrot matching system. What would you say to her?

• 2:14:13Speaker 00

  I guess I would just be like, we've observed algorithms inside these things. Like, you can train a tiny model on modular addition, and it has discrete Fourier transforms and triggered entities. We know that transformers have induction heads that seem to be composing in some kind of actual algorithm. We find these abstract multimodal and multilingual features. And I'm sure you can justify some relaxation of this model. Where you're like, yeah, it does the easy stuff, but it never does the real hard stuff. So it's a stochastic-ish parrot, but I don't know. The Othello thing is another example. Like, I think that's a clear example of it formed a world model in some sense.

• 2:14:58Speaker 00

  Like, it only ever saw the moves, it never saw the board, but it formed a causally meaningful internal representation of the board.

• 2:15:06Tim Scarfe?

  Yeah, I mean, the steel man to that argument is that they don't seem to do this compositional generalization. They don't have the invertibility and so on. But then, what is an abstraction if it's not a bag of analogies? And what is an analogy? I mean, if you capture all of the presentations of a concept and link it to a thing called an abstraction or whatever, and it acts as if it knows the abstraction and it can reason with that abstraction, at some point, it's a distinction without a difference.

• 2:15:39Speaker 00

  Yeah, I think another important factor here is, like... So, in my opinion, the claims being put forward are kind of for-all claims. Like, there do not exist instances of the thing not being a stochastic parrot. I totally think there are many instances where it is. Like, you know, they memorize lots of stuff, sometimes they hallucinate. Lots of the time, they're just doing basic grammar. Like, the token "don" is followed by apostrophe "t" to make "don't", is like, an N-gram. Like, clearly the models learn this.

• 2:16:21Speaker 00

  And I think that we're still figuring out how to steer them? Like, how to get them to use the, like, complex abstract circuitry that we want, rather than the thing that's really useful for predicting the next token? Like, so for example, even...

• 2:16:43Speaker 00

  Kind of models in the like low billions of parameters know how to do addition. You can give them something like 113 plus 237 and they will often give the correct answer. And this just clearly can't be that useful. It can't have come up that much in the training data, even the general circuit of addition. Well, something like figuring out when a false stop is going to come next versus a comma comes up so much. So like, think how much more of an incentive the model has to devote parameters, that kind of thing. And I think we're just not very good at reshaping how that, how the circuitry is expressed in a way that leads to like all kinds of dumb things.

• 2:17:27Tim Scarfe?

  Yeah, so if I understand correctly, you're saying that the model at the moment is a big soup. And because of pressures from the architecture and the data, there are some really useful circuits that are very robust. And then there's almost like a, it's like strata, then there's a middle tier of things that are slightly less robust and likely to hallucinate. And then there's just, you know, pure noise on the outer layer.

• 2:17:52Speaker 00

  Robust isn't quite the access I'm thinking about. It's more like kind of abstract or like competence or like usefulness. Like there is circuitry that can like, I don't know. Think about complex tasks or produce a plan or something like that. Produce a plan in the like chain of thought sense. And that seems like the kind of thing we really care about. While it's enormous data set of memorized facts, we probably don't. Or I don't know, it's probably memorized a bunch of like famous books.

• 2:18:34Speaker 00

  And we probably don't care that much about that.

• 2:18:41Speaker 00

  And maybe we do, like plausibly, you do want the model to have memorized the whole Bible. I don't know. And I think that, yeah, I think that.

• 2:18:57Tim Scarfe?

  Well, how could we meaningfully, you know, if we think of it as a kind of geological strata and there's bits that have, I mean, some of it is just knowledge and some of it might be circuits for doing certain types of reasoning and planning and so on. And it's almost like we now, now we've got this big thing. Maybe we want to shape it and grow it differently in the first place. But given a big model, how do we factorize it into the bits that we want?

• 2:19:23Speaker 00

  Yeah, so I'm honestly not that convinced that interpretability is the correct tool here, though this is a direction I'm interested in people exploring.

• 2:19:36Speaker 00

  So in some sense, we already have a bunch of tools for this. Prompt engineering, in my opinion, is basically just trying to give the model the right magic words to get it to use the circuitry you want and not the circuitry you don't want. Because by default, the model doesn't know what you want. Fine tuning is like another thing and like the kind of chat instruction following fine tuning that people tend to do is like a kind of very important example of that. And my mental model of fine tuning is that it's mostly just kind of upweighting existing circuits, though this has been, this has not been proven and I would really like to see it proven. It seems like a super interesting direction.

• 2:20:19Speaker 00

  And yeah, I think that you can also think of steering vectors or kind of SAE feature clamping as another kind of more interpretable elicitation or steering of these models. Plausibly, there's things you could do where you observe that like these two attention heads connect with each other in this circuit that I don't like. So if I break that connection, like...

• 2:20:50Speaker 00

  Take this head and subtract the output of the earlier head from that and instead add like the output of the earlier head on a different input. Technique called path patching, we won't talk about more later. Maybe that would break the capabilities you don't want in a way that makes the model better. But this is very much a speculative thing and I don't know, I'm quite sympathetic to the bitter lesson in many ways. Like just throw a computer at the problem is often a really good solution. And this makes me think that things like fine tuning are going to be quite a hard baseline to beat unless you have a situation where your data is really bad.

• 2:21:33Speaker 00

  Like you don't have much of it, or it's got lots of spurious correlations, or it's really noisy and you maybe want to use interpretability to do better there.

• 2:21:43Tim Scarfe?

  Very cool. So I mean, coming back to sparse auto encoders. I mean, first of all, there was the vanilla variant, which was described in Towards Monosemanticity. But there's a whole bunch of variants that, you know, change the architecture and the activations and so on. And your team has been working on one of those, which is the jump ReLU. But I mean, can you just kind of sketch all of that out?

• 2:22:03Speaker 00

  Yeah. So the first one is shrinkage. So it is mathematically the case that if you are using L1 regularization, latence will fire less than is optimal purely for reconstruction. Because you've got two objectives and L1 always wants things to be smaller, so it'll make things a bit smaller than they should be. This is sad. And yeah, that's shrinkage. The second one is a bit more conceptually difficult. So the idea is superposition causes interference.

• 2:22:50Speaker 00

  This means that if you have some feature direction, say the dog feature, and you project onto that, and you plot a histogram of the A projections, you'll kind of obviously expect to see a kind of dog mound, like, I don't know, between 3 and 5, there's like the dog bit. But there might be a lot of interference. So like, it might not be the case that when the dog isn't present in the prompt, it's at 0. It might be between minus 2 and 2, and like, just very noisy. And you... It's quite hard to solve this problem with a ReLU.

• 2:23:33Speaker 00

  Because what you want mathematically, what you want to do is you want to measure the distance from the origin, if it's between 3 and 5, and otherwise set it to 0. But ReLUs say everything above the ReLU threshold is not set to 0. So you like, can't have the ReLU threshold at 0, but if you have it higher, like 3, that means you're instead measuring the distance to 3, which is like, distorting things and making it the wrong scale. And so, an activation function which does do this is called the jump ReLU. So the way a jump ReLU works is it's basically like a normal ReLU,

• 2:24:17Speaker 00

  but you add a threshold T that's positive, and everything below T is just set to 0. So for people watching the video, the graph basically looks like a straight line, jump, discontinuous jump, and then it's the identity. And this can exactly let you represent things like, if it's above 3, take the distance from 0. If it's below 3, set it to 0.

• 2:24:47Speaker 00

  And so my team wrote a paper on gated sparse autoencoders, led by Sen Rajamana Haran, who is great at just having wild, crazy ideas for new SAE architectures. And the idea here was, let's train a gate. Let's have two encoders, one which figures out which features should fire, producing a binary mask, and the other that figures out how much they should fire. And this, and we only apply L1 to the which features should fire, but because it's a binary mask, you can't have shrinkage. And we did some magic tricks to like,

• 2:25:29Speaker 00

  make it so you could train a thing even though it produced a binary mask. It was quite hacky. And, but we also found that if you made the two encoders have the same matrix, just different biases, like the same weight matrix, it basically behomed as well and like you had fewer parameters, so life was better. And it turns out that this mathematically reduces to a JumpRelu.

• 2:25:56Speaker 00

  And we had a sequel paper called JumpRelu SAEs, which are maybe the state of the art recipe, though it's a little bit unclear. Again, led by Sen. So the idea here is basically normal SAE, but you replace the activations with JumpRelus. A problem with JumpRelus is that you've got this threshold variable.

• 2:26:26Speaker 00

  But, so if the threshold is three, then everything above three is the identity. So if you've got an input where the activation is four, the threshold kind of has zero gradient. Because if you make it a bit higher or smaller, it doesn't change things. Which means it's base really hard to train. Um, so what we instead did is we optimized the L naught rather than the L one proxy.

• 2:26:55Speaker 00

  By default, you can't do this because L naught is discrete. Everything is zero or one. It's just like, did this feature fire? If yes, add one. If not, add zero. But we used straight through estimators, which basically means rather than thinking of it as a function of activation strength that is like, uh, zero. And then there's a sun jump to one.

• 2:27:24Speaker 00

  We think of it as a linear function that's zero for a while and then has a kind of very steep diagonal line up to one. And then is one. And then you take the gradients of this, but only on the backwards pass. So it's, um, discontinuous when you're going forwards. You intervene when you're going backwards to make it this kind of weird estimator thing. And we found that this performed really well. And it addresses both shrinkage and the other kind of messier problem of how do you deal with low value interference? The other architecture worth knowing about is top K SAEs from Leo Gao at OpenAI,

• 2:28:08Speaker 00

  who wrote a great paper on this. The other part of that paper was scaling SAEs to GPT-4, the absolute madman. And super simple idea. You, rather than having ReLUs at all, uh, though you can also have ReLUs, you apply a top K function. So you just take the top, say, 100 latents, uh, keep those, everything else is zero. And this just gets you sparsity for free with, um, no concerns over.

• 2:28:42Speaker 00

  Like, does it have too many features firing?

• 2:28:46Speaker 00

  And this also works pretty well.

• 2:28:49Speaker 00

  We found that Jump ReLU slightly outperformed it. Anthropic found that top K seemed better. It's a bit unclear. Both seem useful. Top K makes it very easy to just set whatever sparsity you want.

• 2:29:07Speaker 00

  This is actually quite an annoying problem. So we've got this sparsity penalty when training these things, whether L1, L0. And choosing this sparsity will change how sparse the model is, how many features tend to fire. It's kind of unclear what the right sparsity is. Because intuitively, there shouldn't be that many concepts relevant to a given input. So you shouldn't need that many, maybe like 50 to 100, which is what we typically target. But maybe more like 20, I don't really know. And you, and often the kind of slowest activating features will kind of just be noise.

• 2:29:53Speaker 00

  But it's often noise that helps the model reconstruct the input, but in kind of an uninterpretable way.

• 2:30:01Speaker 00

  Because it's just saying, well, if the projection onto this vector is high, I will add a similar vector and then I will reconstruct it. Yay.

• 2:30:10Speaker 00

  But you often want a good reconstruction. And so it's kind of unclear what the right way to do is. Typically, the way we compare SAE families is rather than just saying which one performs better. We take a range of sparsity penalties. Or different Ks in the case of top K. And we plot a Pareto curve of like, how many features are firing, L0? And how good is it at reconstructing? This can either be how good is it at reconstructing the activation?

• 2:30:49Speaker 00

  Or if you substitute the reconstructed activation and then finish running the model, what's the decrease in cross entropy loss? Which is in some sense closer to what we care about. Because there might just be some uninterpretable garbage that doesn't matter in the activation. And typically, this will look like a curve because having more features is better.

• 2:31:11Speaker 00

  And then we can compare the curves for the different methods. And so when I say a technique is better, what I mean is the curve is like up and to the right. And you should really go look at the papers and look at the diagrams. I'm not explaining this very well verbally. And the other thing that's crucial to check is that you haven't accidentally made things unintelligible. And you basically just do this by having a human interpretability study or having a like language model do it for you. And an interesting thing is that there's been quite a lot of progress in improving at the sparse reconstruction side of this. But interpretability numbers really haven't changed that much.

• 2:31:57Speaker 00

  And that's odd. A final note on top K is there's a small improvement called batch top K. The Bart Bussman, one of my mentees, made. So an annoying thing about top K is that you so you get to choose the sparsity, which is great. You don't have to like tune a hyperparameter and fiddle around.

• 2:32:21Speaker 00

  But what this means is that

• 2:32:26Speaker 00

  you have exactly the same number of features per input, which like isn't really what you want. Like intuitively, some tokens are boring, some tokens are interesting and should have a lot more stuff happening. The idea with batch top K is rather than taking the top K for each token, you take the top B times K over the batch, where B is like the total number of tokens in the batch. And this means you can have variable numbers per input, but on average, the sparsity will always be 100. And at inference time, you can just take like a kind of typical value of the B-kth

• 2:33:06Speaker 00

  thing as a threshold and just fix that. So you don't have to have a batch every time, which is actually identical to Jump ReLU with inference, which is kind of cute. It's like nice connections. So Jump ReLU with identical thresholds everywhere. And batch top K is like probably my recommendation for people who want to use top K stuff or just like really want to know the sparsity. Jump ReLU might be a bit better and it's what we use for Gemoscope, but I think is like probably more complicated to train and easier to shoot yourself in the foot. Though we have some open source implementations now.

• 2:33:45Tim Scarfe?

  You said that we want to have as many features as possible and there seems to be something limiting the amount of features produced even across some of these methods. I mean, what are the things limiting the number of features, you know, useful interpretable features?

• 2:33:58Speaker 00

  So.

• 2:34:01Speaker 00

  I don't know if I'd agree with that. We want as many features as possible. So.

• 2:34:08Tim Scarfe?

  What do we want? What's the objective? Is it as many useful interpretable features as possible?

• 2:34:15Speaker 00

  I want to know what's going on in the model. What is the computation happening? What are the variables? And I don't know how many variables there are. It actually is probably more complicated than that. Like it's, I've actually updated away from the idea. There's some fixed number of variables.

• 2:34:36Tim Scarfe?

  So what can we do with auto encoders and what evidence do we have that they work?

• 2:34:41Speaker 00

  Yeah. So, okay. So from the, let's think about this from the perspective of a researcher. You've got a language model, you've got a sparse auto encoder. What can you do with that? So typically, so you need to pick an activation to train your sparse autoencoder on. Like you need to train a different one for like each activation in the model, essentially.

• 2:35:06Speaker 00

  Typically, the most interesting one is the residual stream. Because that's like a bottleneck, that's the sum of all layer outputs so far. So it's like, the kind of, if you understand what's going on there, you've actually answered quite a lot about the computation. While any given layer is only a small fraction.

• 2:35:25Speaker 00

  So, for example, in scaling monosmatisti, Anthropic just trained a few SAEs on like the middle residual layer.

• 2:35:35Speaker 00

  So what can you do with one of those? So, the kind of natural thing to do is you just use it as a microscope. You run, put some text with the model, and on each token, you see which latents light up. And then ideally, you have some tooling, like Neuronpedia, that will let you understand what those latents mean. And you can be like, ah, the model is thinking about this, this, and this. Or, why is the model thinking about this? That's odd. I'm curious what's up with that.

• 2:36:08Speaker 00

  I highly recommend people just go to the Neuronpedia Gemoscope demo and just play with the microscope part, where you can just put in text and see what lights up, and it's got a very, very pretty UI.

• 2:36:23Speaker 00

  And Neuronpedia also has an API, which means that when you're like a researcher messing around things in a co-lab, you can just see the dashboard for each feature, which is delightful, because it's kind of a pain making them yourself.

• 2:36:39Speaker 00

  And yeah, so like using it as a microscope.

• 2:36:43Speaker 00

  Another thing you can do is just kind of using it as a discovery tool. Like you just look at each latent, like randomly choose some, and you're like, uh, what does this fire on? Oh, that's an interesting concept. I wouldn't have guessed the model has that concept. Or maybe you guess for a latent you think exists. So you come up with a bunch of prompts that might have it, and you try looking at that. And see if you can find a latent that kind of acts like a probe distinguishing those two data sets.

• 2:37:18Speaker 00

  Next thing you can try doing is causal interventions. It's basically steering. You can either do this by clamping. So you pass the residual stream through the SAE. But then you take the latent, like say, the dog latent or the yelling latent, and you make it, put it at a fixed high value.

• 2:37:42Speaker 00

  And then go proceed. You could also just do steering, just use the decoder vector as a steering vector. You can think of this as just adding some number to the latent activation. Or you can think of this as just adding a vector, you don't even need the SAE. One note is, so SAEs introduce error. The reconstruction is not the same as the original activation. And these error terms, the like difference between the original residual stream and the like new one, the residual residual stream, as it were, is, sometimes contains important things, like there's some features that were just too niche to be captured.

• 2:38:28Speaker 00

  And, um, so if you replace with the reconstruction, this in and of itself will have an effect on the model, which might drown out the effect of clamping a latent. Typically, what we do is we just add the error term back in or do some more efficient operation that is mathematically equivalent, because that's kind of wasteful. And this is annoying, because it means there's this like mysterious node that we don't understand. But it also means we can do cleaner investigations. So, pros and cons. You can also, yeah, so yeah, you can use it to evaluate what's going on.

• 2:39:14Speaker 00

  Causal interventions.

• 2:39:17Speaker 00

  Oblation is probably a more interesting kind of causal intervention. It's like you've given a prompt and some concepts light up. Like, the Eiffel Tower is in the city of, and it says Paris. And you're like, which latents actually matter for it saying Paris? You can just delete them one at a time and see how the Paris log probability changes. And, um, yeah.

• 2:39:41Speaker 00

  So I think a more ambitious thing is circuit finding with them. So, um, yeah, so sparse auto encoders by default, they're just an activation studying technique. They find the variables in the model, but this doesn't inherently give you the active, the like algorithms, the circuits. Um, but what we can do, um, so yeah, I think the best work here so far is the sparse feature circuits paper by Sam Marks and Aaron Miller. What they basically did is they took the outputs of each attention and MLP layer and every residual stream.

• 2:40:25Speaker 00

  They train a different SAE on each of those. They did this on Pythia 70 million because it's kind of expensive and annoying, especially because you have so many latents when you have like this everywhere. And you try to just do like causal interventions. You've got some prompts. They studied things like subject verb agreement, like, um, the ma, yeah, like, um, the man says the men say or something.

• 2:41:07Speaker 00

  And tried to find interesting features for this, where you basically just look at the causal effect of each feature, um, latent when you get rid of it. And then you also can look at connections between them. So like you delete a latent in layer two, you look at the change for a latent in layer four, um, or maybe you subtract the direction for the latent in layer two from the input to the one in layer four. And then you try to use that. And then you see what effect that has.

• 2:41:48Speaker 00

  And one nice thing about sparse auto encoders is because they're sparse, um, often many latents won't matter. And so you don't have to do this edge tracing for like everything because that would be a combinatorial nightmare.

• 2:42:08Speaker 00

  Also, because they had residual SAEs, this meant that they kind of only had to do like one step of a path. Because there was like a bottleneck at each layer they were connecting to. Like, they weren't looking at connections from like Attention 1 to MLP 5 or something. A problem with this approach is that the error nodes will just completely break your model. Oh, sorry, the reconstruction error, when you've got an SAE at every layer, will kind of cascade and just destroy your model. Which means that circuit finding without them is like, kind of sketchy.

• 2:42:50Speaker 00

  But if you include error nodes in your circuit, then it's like, you've got this massive uninterpretable lump. And they got much worse results when they didn't include error terms. You can still, yeah, you can still make some progress. Like, you can say, if I have a subgraph of latents and I think it's a circuit, you could try deleting all of them and see how much that damages model performance, for example. And like, that can give you some evidence.

• 2:43:32Speaker 00

  I'll also say that I think you don't even necessarily need to do the kind of edge patching stuff, because often you'll have a sufficiently small set of latents. You can just kind of assume all of the edges are meaningful and it's like, eh, kind of good enough. One caveat I'll give to this is that this is very much kind of causal intervention-based circuit finding. They, in fact, used a gradient-based approximation to this called attribution patching, that I think I discussed briefly earlier, which they improved using integrated gradients, like classic interpretability technique. But it's basically trying to approximate a causal intervention of like,

• 2:44:17Speaker 00

  changing a latent value to its value on another input. Um. One kind of nice thing is that there's kind of a good default value for latence of zero. Finding a good default value is actually quite hard for like general circuit finding because activations aren't normally like the zero vector. So if you set it there, that's actually really destructive and weird. You could set it to the mean, but maybe the mean has much smaller norm and that messes with layer norm and things like that. Um. You can also do activation patching when you replace the latent's value with its value on another input. Uh. This is useful if the latent is kind of active most of the time.

• 2:45:00Speaker 00

  Though it can actually be misleading to do mean ablation when you replace it with the average because if the distribution of a latent is it's zero ninety eight percent of the time and then it's a hundred two percent of the time, then the mean is two. But like in some sense, being two a hundred percent of the time is like less faithful to the original distribution than being zero a hundred percent of the time. So I don't know. There's like lots of questions we still haven't figured out here. Um. One of the things we went out of our way to do with GemmaScope is to have SAEs on like every layer and sub-layer to like enable this kind of analysis. Um. And I think you can just about fit

• 2:45:44Speaker 00

  all of the uh narrow ones for Gemma two two B on like an eighty gig A one hundred. Uh. Maybe leaving out the residual ones. Um. Because I'm excited to see what circuit analysis can find on those kinds of models. Um. I actually have a pair of um math scholars, uh Stepan Shabelin and Dmitry Kalapenko, who are gonna produce some work on this stuff soon. Who've been looking into in-context learning and

• 2:46:16Speaker 00

  So there was a recent set of cool papers on this idea of function vectors or task vectors. There was actually one from Eric Todd and one from, I think, Roy Hendel, like a day apart. The idea for this is in context learning, like few-shot learning, you can have a bunch of examples of a task. And then you give it another input and it does the task, like producing opposites, tall to short. What they found is that at a certain layer, you could extract a vector that corresponded to the task. And then for a different prompt with like a different input and a different task,

• 2:47:00Speaker 00

  you could swap out the task vector and it would do the original task for the input in that prompt. So kind of isolating the few-shot-ness and also finding that it was at like a slightly earlier layer than when the model figured out what the input was, which is quite cute. And so what Stepan and Dimitri found, which I think we put out in a recent blog post, is that there are sparse autoencoder features corresponding to these task vectors. And also sparse autoencoder features corresponding to like task detectors, like what task is happening on earlier tokens that like connect up to them. And yeah, I'm really curious to see how much more you can uncover of those.

• 2:47:45Tim Scarfe?

  Is it possible to know though in advance? I mean, I mean, how should we expect these features to arise? I mean, if we have a certain type of, you know, dataset about, I don't know, architecture or something, you know, should we expect a certain type of feature to arise?

• 2:48:02Speaker 00

  Yeah, so this is a really interesting question that we don't understand super well yet.

• 2:48:10Speaker 00

  So, yeah, so I think a crucial hyperparameter I haven't spoken about so far is the width of your dictionary, the number of latents. This is chosen by the researcher before they train the sparse autoencoder. And you can kind of think of this intuitively as the zoom level of your microscope.

• 2:48:31Speaker 00

  And like, do you find kind of a few coarse-grained features or lots of fine-grained ones? And a really interesting thing is that, so.

• 2:48:45Speaker 00

  The kind of picture I laid out at the start of there are just some ground truth list of features which have their own vector. Under that view, you might expect a sparse autoencoder to kind of learn the first 20,000 of those if it's got 20,000 latents and just totally neglect the rest. But what we find is that that's partially true. Larger sparse autoencoders do learn features that seem totally missed. Two of my Matt scholars, Bart Bussman and Patrick Leesk, have a fun blog post on that. But there's also lots of feature splitting, where you kind of have a coarse higher level

• 2:49:28Speaker 00

  feature that splits into like lots of sub-things, like, I don't know, color might split into like red, green, blue, purple, et cetera. That's actually sufficiently common, that probably doesn't actually happen, but there are like lots of examples. And this is kind of weird, because it raises the question of are sparse autoencoder features actually like units of computation or not? And my current guess is that they are like probably not.

• 2:50:08Speaker 00

  Barton Patrick had another post called like, are they atomic? And what they did here is they trained a meta-SE, where you take SE latent features, you take the decoder matrix of an SE, and you train another SE to reconstruct that decoder matrix called a meta-SE, typically with like a tiny L norm, like four or something. And this found that sometimes you'd have features that, like an Albert Einstein feature, that decomposed into like male physicist, German, whatever.

• 2:50:48Speaker 00

  And these often, these kind of meta features often corresponded to things in a smaller SE. And so it's like, smaller SEs often split apart in bigger SEs, but bigger SEs are often kind of just compositions of things in smaller SEs. But they're also often interpretable and causally meaningful. And these feel like, you know, I like interpretable causally meaningful things. That feels like it should be enough. And I don't know. I think this is a thing that I do not feel like I currently yet understand. I think the right question is maybe just, are sparse auto-encoders useful for understanding

• 2:51:33Speaker 00

  what's going on inside models? But it also seems important to have better foundations of like, what have they actually found? I feel like they found something, but I don't know what they found. And it seems kind of fascinating that the internals of a model are kind of hierarchical in this way, where you can have interpretable causally meaningful directions at different levels of abstraction. And yeah, the other comment I'll make on what you might expect to come into this is, so my favorite part of Anthropic's Scaling Monosponticity paper is the feature completeness section. So all they did here is they took a bunch of things that you might expect to arise in

• 2:52:15Speaker 00

  a model, like chemical elements, or I think maybe London boroughs or something. And where there's kind of like a clear name, so you can just check, is there like a feature that lights up on text including this name? And they took these SEs of different widths and they looked at how, what is the...

• 2:52:42Speaker 00

  Probability, or rather like, how many of these get learned? And like, what does getting learned look like as a function of the frequency of occurring? In these different SAEs of different widths? And they found that you could fit quite well a like, simple predictive model. That's, I think it was like a sigmoid curve, I don't remember the details. But basically, sufficiently small SAEs don't care. And then depending on the frequency, there'll be a point where like, the probability starts increasing, and then it goes up a lot, and then it's like, basically 100%. Though plausibly, it also splits later on, which is like a complexity I'm not sure they went into.

• 2:53:27Tim Scarfe?

  I can catch that reference, what was it?

• 2:53:29Speaker 00

  So, this is the feature completeness section of Anthropic's Scaling Monos Mantisity paper. It's a long paper, it has so many sections. So good.

• 2:53:41Speaker 00

  And so, kind of, I think the takeaways from this are that, A, kind of a model, uh, kind of, the frequency of a feature matters a lot, which just makes sense. Like, you wanna, things are more useful if they occur more often. And if they are kind of like, bigger in the activation space in some sense. But I think the second interesting takeaway is that it seems probabilistic whether a feature gets learned in an SAE of a given width. And like, if you train multiple of the same width, they'll have shared features that were kind of the obvious ones to learn. But for features that are kind of more niche, and kind of more on the borderline, it's like, and they're just kind of random?

• 2:54:24Speaker 00

  And, I don't really know what the takeaway here is, like, um, I think I'd love to see more research in this area. In terms of advice for practitioners, I basically think that you should just train several widths of SAEs. Um, in Gemoscope, we have kind of a wide and narrow for every site. And we also have a few layers where we have like, powers of two, so you can do like, lots of hierarchical and feature splitting analysis. Um, yeah, just have a few widths, try the different ones on your task, see which ones capture the features you care about. It would be nice if we had a better answer, but that's the best I got.

• 2:55:07Tim Scarfe?

  Well, you can also train SAEs on, I don't know, like the residual stream, on just MLPs, on attention. I mean, how do they compare?

• 2:55:17Speaker 00

  Yeah. So yeah, you can basically train it on like any activation. You can even train it on like the queries and keys and values and attention layer, though that's more confusing and that's worked less well. Yeah, doing it on the output of a layer, I think, is a super reasonable thing to do. The... There's an interesting question of whether you should do it on the MLP activations or the MLP output. So typically, the MLP activations will be like four or more times the size of the residual stream, but then there's just a linear map, mapping it down to the output. And so like, kind of conceptually, there shouldn't really

• 2:55:57Speaker 00

  be a difference. But also, your SAE will have four times as many parameters. And like, maybe this helps, maybe this doesn't, I don't know. Typically, I just train it on the output because it's four times cheaper, but it would be great if someone actually studied this systematically. Returning to your question, I think that these just have different tools. Like, if you want to study what a layer is doing, you want the output for that layer. I think a totally valid mode of analysis would be taking a task you care about, like in-context learning or refusal, doing activation patching, like swapping layer outputs between

• 2:56:35Speaker 00

  different prompts to find the ones that matter. And then you could go and take the Jamascop SAE for that layer and then see what happens. The residual stream is generally more useful if you want a kind of holistic view of what's happening.

• 2:56:58Speaker 00

  Probably all kinds will be causally meaningful, but the residual stream is going to be the one that detects what the model is thinking about. While the layer ones are more like, what's happening at this layer? Like, what did that layer do? It's just like a different question. I think that

• 2:57:17Speaker 00

  an interesting thing about MLPs is that the MLP layer itself is this dense, nonlinear mess. And it's in superposition. So you've got thousands of gelia activations, or gated neurons, or whatever. And it's just very unclear how to think about this. This is fine if you're just doing causal interventions, or you just want to study an activation. But if you want to try to do some weights-based circuit analysis, like recovering this hope of input-independent things, which feels like SAEs should bring us closer to you, because they are kind of monosemantic.

• 2:58:01Speaker 00

  You just can't deal with the big, chunky MLP bit. The solution to this is transcoders. So I supervised this delightful paper from Jacob Danofsky and Filip Chalinski

• 2:58:17Speaker 00

  on them. The idea of a transcoder is rather than finding a sparse reconstruction of the MLP output from the MLP output, you use the MLP input to find a sparse reconstruction of the MLP output. So it's like a sparse replacement MLP layer, with lots of neurons, but they're sparser and ReLUs.

• 2:58:41Speaker 00

  And you can kind of use this as a drop-in replacement. On GPT-2, they worked about as well as

• 2:58:53Speaker 00

  MLP output SAEs in terms of reconstruction. On Gemma 2, they were notably worse, I think because Gemma 2 has gated MLPs. Though we saw released a suite of transcoders, which might be useful for people who want to do this kind of input independent circuit analysis. Though, a quite annoying thing we found is that it was really hard to do purely input independent circuit analysis, because you take two transcoders, you multiply the decoder weights for one and the encoder weights for the other to get these like, yeah, a kind of like interpretable basis by interpretable basis matrix. And there's lots of weights that just don't matter, like I was kind of gesturing at earlier.

• 2:59:32Speaker 00

  And the way Jacob and Philip solved this is that they, yeah, they looked at feature co-occurrence, and they just kind of didn't look at the connections for features that didn't co-occur, but like, that's somewhat input dependent, so it's not fully satisfying. But it was like, a solid attempt, and I'd love to see more iteration there. On attention, so attention is much more linear than MLPs in some sense. So, the attention pattern computation is weird and nonlinear, and I don't think we yet understand how to deal with it with SAEs.

• 3:00:16Speaker 00

  There was an interesting blog post from Keith Winrow on QK transcoders, which seems like a cool approach here. But if you take the attention pattern as a given, a slightly sketchy thing to do, but bear with me, then it's basically just a linear map. You use the attention pattern to kind of mix together the different past tokens, and then you apply a linear map, the value matrix and the output matrix, and you get a new thing. And you do this for each head in parallel, and then you add them together. And so...

• 3:00:57Speaker 00

  Technically, you do the value map and then you do the attention mixing, but they commute, so it's fine. And I think see the last episode if you want me rambling about this in much more detail. And so this is interesting because it means you can do linear attribution. You can just say, look, this is basically a linear map of past tokens and then some feature lit up. So like, what were the tokens that contributed to this? And you can even, rather than training it on the output, train it on the concatenated mixed values over the heads, which is like the thing immediately before the output weights. This is just like a linear map away from the

• 3:01:42Speaker 00

  actual output. And this means that you can now attribute things directly to like each head, which is quite a nice feature. And I supervised this paper from Connor Cassane and Rob Krizianowski, basically just trying to do a deep dive of like, can you do attention, can you do SAEs on attention layers? And basically, yes, works great. They're potentially a bit more interpretable than MLPs, and you can do all kinds of nice things with like linear attribution like this.

• 3:02:17Speaker 00

  For example, Rob, the absolute madman, went through every head in GPT-2 Small and looked at the top 10 SAE latents according to like, how aligned they were with that head specifically? Because you can just read off the weights for each head and look for a pattern and just kind of labeled heads by like, do I see a pattern? What is the pattern? And check out the paper if you want. I think we've got a table of all of them somewhere.

• 3:02:46Tim Scarfe?

  Very cool. And what's the relationship between the number of features that you try and find? And how kind of meaningful the features are? And I was also wondering, like, how is it related to the size of the model?

• 3:02:59Speaker 00

  Yeah, so this is not a thing that I think has been studied in much detail. I predict that the interpretability will be quite similar.

• 3:03:12Speaker 00

  Though the larger, though, there's often kind of training stability problems with larger SAEs. Like, when you get beyond like a million latents, things get more annoying.

• 3:03:26Speaker 00

  For example, in Anthropic's 34 million latent SAE, they, I think, had about 65% of the features were dead, meaning they just didn't fire. And this is like a really high ratio. It's just like 20 million features doing nothing.

• 3:03:46Speaker 00

  And this might just be because there aren't 34 million things, though. They definitely found that their SAE had missed out on some stuff. E.g. the model knew all of the London boroughs, but they couldn't find SAE features for all of them, only most of them, for example.

• 3:04:03Speaker 00

  Yeah, so training stability problems. But like beyond that, my guess is they get much more fine-grained, to the point where you could argue that some of them are just, I don't know, like memorizing a couple of training examples or something, perhaps, if you go to the extreme. I mean, in the infinite width limit, you'd kind of expect each token to get its own latent or something, which seems boring and degenerate. But I'd be surprised if we ever get to that point.

• 3:04:32Speaker 00

  Intuitively, larger models should just have more concepts they know, meaning you should want a larger thing.

• 3:04:42Speaker 00

  I don't know if anyone's actually tried training an enormous SAE on, I don't know,

• 3:04:51Speaker 00

  every model in the GPT-2 family or Pythia family, and just tried to see whether the smaller ones look kind of crazy and the larger ones look reasonable. But like, I think that would be a pretty cool experiment.

• 3:05:04Tim Scarfe?

  On the security side of things, is it possible that these sparse auto-encoder features could be used to, I don't know, perform adversarial attacks or something like that?

• 3:05:14Speaker 00

  Um, probably. So I think my general vibe is we haven't seen any much evidence that SAEs can do stuff that's like beyond what you can achieve with, say, fine-tuning. Like, fine-tuning can do a lot of adversarial attacks against a model. And it seems plausible that you can do kind of interestingly different ones with interpretability, but I haven't seen any real signs so far that there's kind of dramatic capabilities that were just not there,

• 3:05:55Speaker 00

  that like were really hard to achieve with fine-tuning. So, for example, in this refusal is mediated by a single-direction paper that I did with Andy Arditi, where we found a refusal steering vector that we could use to jailbreak models. You can jailbreak models by fine-tuning the weights. And this was like a large part of why I felt comfortable releasing that paper. Like, it was just an interestingly different attack and in some ways kind of easier and cheaper. And like, maybe things like that might happen with SAEs. I mean, it's a new technique, so like, there's always the potential that there's some things

• 3:06:36Speaker 00

  I'm not expecting that we're going to learn, but I don't see any particular reason to expect it to advance the frontier of what we can do beyond things that use throw compute at the problem, because throwing compute at the problem is really effective.

• 3:06:52Tim Scarfe?

  Are there any scaling laws with SAEs?

• 3:06:55Speaker 00

  Yeah, so this is a thing that kind of surprised me. The... Yeah, both Anthropic and OpenAI have had recent papers just scaling these too much bigger models, like, we've been talking about Anthropic a bunch, OpenAI scaled it to GPT-4, and like, both have some nice sections in their paper on SAE scaling rules. And it seems like, I don't know, initially, we didn't really know what we were doing when training SAEs, but now we've iterated on the methods enough that they're like smoother, and we can just do things like change the amount of data, change the amount of latence,

• 3:07:34Speaker 00

  things like that, and plot out, I don't know, what an optimal, optimal set of hyperparameters for a given compute budget would be, or something. And like, it's kind of messy, because there's a lot of hyperparameters. There's like, how much data, how many latents, what sparsity penalty should you use? And like, there aren't really canonical solutions to any of these, but people should totally go check out those bits and stare at the pretty curves.

• 3:08:02Tim Scarfe?

  Yeah, I mean, one of the problems is just the amount of garbage features. I mean, what do we, what do we do with them?

• 3:08:08Speaker 00

  Yeah, so, okay, so there's like, dead features, which basically don't fire. There's uninterpretable features, which fire sometimes, but we can't really see a pattern. And then there's high-frequency uninterpretable features, which fire a lot, like 1% of the time plus. And dead features, I don't know, you kind of ignore them, they don't really matter. You can prune them out of your SAE, though I think the rates of dead features have gone down a lot as we've kind of refined our training techniques. Like, previously, there were lots of hacky things people did, like resampling, where every so often, you just

• 3:08:50Speaker 00

  replace all of the dead neurons with new weights. But I think training methods now are quite a bit more stable, and that doesn't seem necessary anymore, for the most part, with more modern methods. I think that the uninterpretable ones...

• 3:09:07Speaker 00

  I don't really know. There's an argument for just removing them. Um, I'd be kind of curious to see what happens if you just do auto-interpret on every feature. You find the ones that are bad, like you delete those and then you, I don't know, fine-tune the SCE a bit so it doesn't have like a gaping hole in it. Um, that seems like kind of an interesting direction. Uh, at the moment, I don't really think about them too hard, though if I started to see like lots of uninterruptible things lighting up a bunch, I'd start to be more concerned. I think the thing which is more concerning, and I'm like a lot more interested in understanding, is these high-frequency features. So, under L1, things which light up like 1% of the time are like very heavily penalized, so they don't tend to happen too much.

• 3:09:53Speaker 00

  Um, but under things like top K or jump relu, they're penalized much less. Um, like top K doesn't really care, and with jump relu, you only get penalized if you're kind of near zero. So if it just big 1% of the time, maybe it doesn't care. And, um, this means, and if it activates a lot of the time, it can be quite useful for... If it activates a lot of the time, it can be quite useful for reconstruction, even if it's not interpretable. And sometimes these things do seem interpretable. It's kind of confusing. Like Anthropic had a recent monthly update where they compared difference by sorting coder, training techniques, and like

• 3:10:38Speaker 00

  mentioned they'd looked into this a bit and had been like, actually, we think these things are kind of interpretable enough. We're not that worried about this. Um, is the rough vibe I remember, though people should read the post. I might be summarizing it incorrectly, as goes with every paper I discuss in this podcast. Um, but...

• 3:10:59Speaker 00

  Yeah. Um.

• 3:11:01Tim Scarfe?

  What was that paper you just referenced from Anthropic?

• 3:11:02Speaker 00

  Uh, it was one of their monthly updates. I think the one, uh, the second most recent one.

• 3:11:11Tim Scarfe?

  I mean, one thing I was thinking about, though, is that, you know, one school of thought, and I'm not sure whether you lean in this direction, is that these things represent units of computation. Another school of thought is that they are post hoc descriptions. And what we need is, I guess, some kind of a causal mechanism to tease that out. I mean, what do you think about that?

• 3:11:31Speaker 00

  So I don't know what unit of computation. Uh, this is kind of what I was getting at earlier with the feature splitting, and these things are not atomic discussion. Like, yeah, like, in some sense, I feel like a unit of computation must be irreducible, and it must kind of fit into a kind of clear circuit and be used downstream. But if you have three, um, if you've got a mammal feature that splits into a bunch of animal features, but is causally meaningful in its own right, is that a unit of computation? Like, maybe?

• 3:12:13Speaker 00

  Um, or like, in some sense, it's enough to be useful, but it doesn't, it's not quite achieving the ground truth of what's going on with the model. And like, I don't even know whether this stuff bottoms out somewhere in some underlying truth, or if it's all just some kind of messy hierarchical structure where you can kind of go up or down the, like, it's not even a tree, because things often merge, like directed acyclic graph, maybe? Um, and yeah, so I think a research direction that I'm pretty excited about is trying to apply these things to real world tasks, or at least kind of downstream tasks. Um.

• 3:12:58Speaker 00

  For example, in Sam Mox's Spots Feature Circuits paper, in addition to the circuit analysis stuff, there was a really cool bit called Shift, where they, so they took a gender biased data set. Like, I think it had bios of female nurses and male professors. And they trained a probe on that. And it partially learned to pick up on the profession and it partially learned to pick up on the gender. So when you evaluate it on a data set with the opposite gender flip, it does really badly. But what they found is they could identify the SAE features that were most relevant to gender with a mix of giving the model data and human analysis.

• 3:13:38Speaker 00

  Get rid of those and it, the probe became much less biased. And I think this was like a really cool application of, I don't know. I don't know what our existing probe debugging baselines should be, but seems pretty impressive that SAEs can do something like that. And I'm like, if it can do something like that, how much do I care whether it's a unit of computation or a post hoc thing? Like, I care a lot about it being real and I care about it like teaching me things about the model. And I don't want to assume things about it that will mislead me, but.

• 3:14:21Tim Scarfe?

  Is it possible for us to correct knowledge or to insert knowledge into a model using SAEs?

• 3:14:29Speaker 00

  Maybe. So a problem with SAEs that I think is often alighted in this kind of high level discussion is that the thing you want is not always a feature. Like, I've trained SAEs and tried to make Golden Gate Claude and I just didn't find a Golden Gate feature. And that was, yeah, like, maybe if I trained a wide enough SAE, I would have, I don't know. And.

• 3:15:03Speaker 00

  There's actually a benefit of steering vectors because this isn't really an issue there. You can just always make a data set for the thing you want. And so like, if the knowledge you want is represented as an SAE feature, you could totally delete it. That would probably somewhat help.

• 3:15:22Speaker 00

  I, yeah, I know of some people who are looking into this, though it sounds like it doesn't outperform any existing unlearning baselines at the moment.

• 3:15:33Speaker 00

  And I think that, yeah, I don't know. Knowledge insertion, hmm, maybe? So this might work.

• 3:15:51Speaker 00

  Yeah, this might work. So it feels a bit odd because to me, knowledge is like a kind of lookup. It's like a function. Even if you want to do it in like a really nice interpretable way, you kind of want to have the input that's like Michael Jordan and the output that's like plays basketball.

• 3:16:13Speaker 00

  And you could maybe add a little bit of Michael Jordan to the input direction for basketball, to the encoder direction for basketball.

• 3:16:21Speaker 00

  You could also just add an X-relator that has Michael Jordan and produces basketball. This is kind of against the spirit of an SAE because like that's clearly not reconstruction-y. That's just like an additional function you've added, but like, in some sense you can do this. It feels more intuitive with transcoders, the like MLP inputs to MLP output thing. Because in some sense, the fact lookup is the job of MLPs. Though, I mean, we found that it tends to be distributed across layers, which is kind of annoying. But you can insert it at one layer totally fine. Especially if you're down to just have like a fairly high magnitude decoder vector.

• 3:16:59Speaker 00

  Or encoder vector.

• 3:17:01Tim Scarfe?

  What's the relationship between SAEs and the task that's being performed or the amount of data trained on?

• 3:17:08Speaker 00

  So...

• 3:17:11Speaker 00

  SAEs get better when trained on more data.

• 3:17:16Speaker 00

  As a general rule, sometimes they'll kind of saturate. I'm not entirely sure if this is like true saturation or just a problem with the training methods.

• 3:17:28Speaker 00

  But in general, more data will make them better at reconstruction. We haven't looked too hard into,

• 3:17:38Speaker 00

  I don't know, a new features appearing, or is it just refining the existing ones? I think this would be like a pretty cool thing to just look into, like,

• 3:17:47Speaker 00

  take a bunch of text that activates an SAE latent, look at that latent overtraining, plot some graphs, do this for a bunch. Does it look like phase transition-y? Or is it gradual? I have no idea.

• 3:17:58Speaker 00

  Someone should look into this. I think that,

• 3:18:03Speaker 00

  you, I expect you can train specialized SAEs. I'm not, I'm not aware of too much work on this. There's a paper called "Towards More Principled Evaluations of Spars Auto-エンクーデーS" I supervised from Alex Makalov and Gao Lang.

• 3:18:25Speaker 00

  The focus of this was, "We think that SAEs should be useful for circuit finding." Let's take the circuit IOI, indirect object identification, which we talked about a bit in the last episode, in GPT-2, and let's see how well SAEs find it. And compare this to kind of supervised dictionaries that we just trained because we kind of think we know what the features should be. And I think that was a pretty cool evaluation. The relevance to this is they both trained SAEs on like webtext, but they also trained it on just IOI-specific stuff. And to some degree, it performed better, but it also was kind of weird in various ways.

• 3:19:09Speaker 00

  Like there was this phenomena of feature oversplitting where there was some feature that should have been there. Like, uh, is the name first or second? And instead it split into like 20 features that were all kind of like that. And this was much more annoying to work with and analyze.

• 3:19:30Speaker 00

  Than the kind of broader SAEs. But maybe you can solve this by making it narrower. But if you make it narrower, then it doesn't learn the name of features, which you also care about. Which are kind of, should be more fine grained. So, yeah. But like, that was a very narrow task. I'm sure if you did say Python code, it would work much more sensibly.

• 3:19:52Tim Scarfe?

  I mean, a minute ago, you were alluding to training dynamics. And this is something that that really interests me, you know, which is to say, how do the the weights evolve over the course of of of training? And how do the, you know, circuits emerge and so on? Could you not, um, you know, kind of compute SAEs at different points of the the training curve? And use that to understand the training dynamics?

• 3:20:18Speaker 00

  Yes, I think that is a thing that should work. I, yeah. So, yeah, um, just to be clear, we're switching from training dynamics of SAEs to the training dynamics of the model and using SAEs as a tool to study this. And, yep, I think that, I kind of think there's just lots of stuff you can do.

• 3:20:42Speaker 00

  I think this would be one example. SAEs might even be overkill. Like, you could just take some concepts you believe are represented and just train probes for those.

• 3:20:55Speaker 00

  One thing I'd be quite interested in is like, how does the probe evolution evolve? How does the probe direction evolve over training? Training dynamics isn't really my field, so I'm gonna say things, and there might just be a paper that does this that I haven't come across. But yeah, I think looking at what the representations look like over time would be super cool. Looking at how well an SAE transfers would also be interesting, especially if you correct for like changing bias terms. Because like, if the average changes, that's gonna screw up everything. But maybe the direction of a feature doesn't change?

• 3:21:35Tim Scarfe?

  Because obviously, you've spoken a lot about Grokking. And I guess that could be seen as looking at training dynamics and so on. But would it be interesting if you saw so-called emergent materialization of capabilities or features that come out of an SAE? Would that be interesting to you?

• 3:21:55Speaker 00

  So, I mean, I think emergence in general is an interesting topic. I would be a little bit hesitant to draw too strong a conclusion from what happens with SAEs. Because, for example, it could be the case that it gradually learns the feature for, say, pillow. But the SAE doesn't think it's worth its while to have a pillow latent until it reaches a certain threshold of salience. And like, then it learns it. And like, that might look really sudden, but actually it's gradual. So like, what I'd actually... So this makes me kind of more interested in probes, possibly using the SAE as like a motivator

• 3:22:37Speaker 00

  of what things to search for and like what probes to look for. Or just literally using an SAE feature as a probe and just translating it to the earlier ones. Because like, it wouldn't surprise me if the pillow latent on the first SAE that learns it still works on the checkpoint before.

• 3:22:59Speaker 00

  And yeah, I don't know. The, there's like a couple of models that would be pretty good to study this on. I've got some toy language models in the Transformer Lens library, like Jellu 1L or Jellu 2L. That are like one layer and two layer models that should be super easy to train SAEs on. There's the Pythia suite that has tons of checkpoints.

• 3:23:25Speaker 00

  Yeah, but yeah, I'm hesitant to use presence of a feature in the SAE as indicator, because there's just like a lot of noise there. So I want to be more cautious.

• 3:23:36Tim Scarfe?

  Yeah, and I think that your team have just put out a whole bunch of open weight SAEs.

• 3:23:43Tim Scarfe?

  Yeah, I mean, what were the engineering challenges doing that?

• 3:23:48Speaker 00

  Yeah, so this is kind of wild and slightly depressing from my perspective. In that SAEs have kind of become quite a big deal in Mechathep. Or at least, I think they're a big deal. And I think they will unlock lots of cool things we can do. So I want to focus on them, which involves training them. And this is just like much more of an involved engineering thing than lots of previous projects that were more like play around in a co-lab notebook with a tiny cute model. And so the reason it's an engineering challenge is so essentially to train an SAE, you need to run a bunch of text through a language model,

• 3:24:29Speaker 00

  collect activations at a certain layer, and then for say a couple of billion tokens, and then use those activations to train a sparse auto-encoder, which can also be quite big. It's like only a two-layer network, but like the middle can be enormous. So like.

• 3:24:54Speaker 00

  If you're going as big as Anthropic did to say 34 million latents, and you want to train it on a model like GEMMA 2.27b, which has a residual stream of width 5,000, the total number of parameters is 2 times 5,000 times 34 million, which is like 340 billion parameters, which is like a lot bigger than GEMMA 2.27b. We did not go that big. We only went up to like a million latents, but still get sizable. And so you kind of have two choices for how you do this. You can either do this online. So you pass text through the model, you store the latence in like your GPU RAM, and then you run an SAE on them.

• 3:25:38Speaker 00

  You often want to have a big buffer of these activations so you can shuffle them, because activations from the same prompt will kind of be correlated a lot in what features they have. Though no one has actually tested to my satisfaction how much this matters. I just intuitively expect it to matter. It's like so many things like that in SAEs. And yeah, so this basically doesn't work for models beyond a certain size. And I mean that if you want to do like a hyperparameter sweep, this is like quite expensive, because you need to run the model again and again. Or even just if you want to train a bunch of SAEs at once on different activations. And so the second mode,

• 3:26:24Speaker 00

  which is what we eventually converged on, is activation storage. So you run the model once, you collect all of its activations. Actually, I think we did like a couple of stages because the like disk input-output became a bottleneck when you wanted to store that many activations. You save these to disk, and then you train sparse auto-encoders on them at your leisure with whatever hyperparameters you want. And if you mess up the training, you can just go back and train again.

• 3:26:55Speaker 00

  So this is just kind of intense. Like, we stored about 20 peppy bytes of activations, which is a lot. We had to request special permission to get that disk allocated to us.

• 3:27:12Speaker 00

  But fortunately, DeepMind believes in sparse autoencoders. And we used about 22% of the compute of GPT-3, which is like, not that much by modern standards, but like, by interpretability standards, is quite a lot. The main reason all of these numbers are so big? So, we mostly released things on Gemma 2.2B and Gemma 2.9B, because I think these are like a good intersection of, kind of useful to cite, of like, kind of big enough to be interesting, small enough that academics can in fact work with them without breaking the bank for GPUs, or needing to mess around a ton with like, parallelization.

• 3:27:59Speaker 00

  Is that we wanted to do like, every layer, including attention MLP and residual. And, so like, Gemma 2.9B has like, 42 layers, so this is like 120. And we also wanted to do two widths, which is 240. And we couldn't decide on what the correct sparsity was. We did like, 6, and so that's like, a lot. I typically claim we did about 400, because I'm not sure I should count the sparsity ones. Because I normally just recommend people use the ones closest to 100, unless they actively want to do some kind of sparsity study. Which people should do. I don't know how these things work, man. And, we also did, I don't know, a couple of other ones, like,

• 3:28:43Speaker 00

  we occasionally did a 1 million latent one on a few layers. We did a few on Gemma 2.27B, and we did a few on the chat-tuned Gemma 2.9B. Though, actually, SAEs transfer surprisingly well between base models and chat models.

• 3:29:04Speaker 00

  Two of my MATH scholars, Konika Sein and Rob Krzyzewski, have a nice blog post showing this. And we corroborated that in the Jamascop technical report. I also should say, credit for Jamascop goes to the team, and most especially to Tom Libram, who was the technical lead and dealt with all of this engineering nightmare, so I didn't have to. Um, and yeah, and like, this was just a gemascope. Like, I expect, um, if we wanted to do a larger model, um, and like, larger SAEs, it would become kind of even more annoying, though there's a distinction between, um,

• 3:29:47Speaker 00

  kind of training one SAE on a residual stream, which is enough for some use cases, but not others. Like, it's enough to kind of monitor the system, it's enough to like steer it, but it's not enough to like see the internal circuitry. And doing every layer and sublayer made it like 100 times more expensive. Uh, maybe not quite 100 times more expensive, because you don't need to run the language model 100 times as much, but like, more painful. And, um, so yeah, it's unclear to me how much this is worth it, and I would be quite curious to just see people explore this kind of thing in practice. Um, and yeah, uh, the

• 3:30:26Speaker 00

  reason we did Gemascope, this open release in the first place, is as what I've been saying, hopefully communicates, this is really annoying engineering-wise, and it's a lot easier to do if you're in an industry lab, um, than if you're outside. Could use to have better infrastructure and more compute, and, you know, 20 petabytes of storage. And, um, kind of in the same way that things like Gemma and Llama, like, they're really expensive to train, but they're, to use, they're like much cheaper, we thought that we could unlock lots of cool research projects by doing this. Um, there's also also some

• 3:31:08Speaker 00

  other cool open weight SAE projects worth giving a shout out to, like, um, Aluthers have trained a bunch on Llama. Um, that's probably actually counts as open source, since I think their code and data is all open source. Uh, and, um, OpenAI released a bunch of high-quality ones on GPT-2 Small, which is nice for smaller projects. And there's a bunch of more scattered people who've put out various nice things. But the hope with Gemascope is that for like, any project that wants a kind of interesting model, this could just be high-quality canonical SAEs that would enable as many projects as possible. So we just kind of threw random variants in there.

• 3:31:52Tim Scarfe?

  So so much stuff has been done, but there's still lots of work to do. I mean, what are the open problems in in SAU still?

• 3:31:59Speaker 00

  Yeah, so, okay, so here's a couple of areas that I'm excited about. So I think just basic science, there's a lot we don't understand about SAEs. Like, I've, I know, I hope I've pointed to many mysteries so far, like, when should you expect a feature to be in the SAE? What's the right sparsity? What's the right width? Are there true units of computation? One problem I might call out is just what's going on in SAE error terms?

• 3:32:29Speaker 00

  What do they mean? Can we understand them?

• 3:32:33Speaker 00

  And we know that they sometimes just capture features that the model missed, but like, there's also weird stuff going on, who knows? There's, yeah, that's category one. Category two would be using SAEs to solve mysteries of interpretability. So there's just like a lot of things I don't understand, like, what exactly does fine tuning do to a model's circuits, especially chat fine tuning, which is like a particularly important kind?

• 3:33:06Speaker 00

  Can we, can we find like a truly monosemantic circuit with SAEs? Now we've got kind of the right units. Are these the right units? Kind of unclear.

• 3:33:18Speaker 00

  But yeah, more of a kind of a using them task. And also just, I don't know, applying them to interesting circuits like,

• 3:33:26Speaker 00

  a few short learning. The next category would be real world tasks or downstream tasks, like, are there things that people who aren't interpretability nerds think is interesting? And can we use sparse autoencoders to make progress there?

• 3:33:45Speaker 00

  I, I say this both because, I don't know, I think doing useful things, especially on the end of like making models safer is great. And I partially say this because I think it's just so, so easy to trick yourself in interpretability.

• 3:34:03Speaker 00

  And we have like some evidence of SAEs working. And like get into that a bit more in a bit, if you're curious. But I think that just, there was a task that people struggled on. And SAEs were able to do this. At least as well, ideally better as competing approaches. Or with like some advantages over competing approaches such that it might be useful in some settings, which would be really awesome to see. For example, can we use them to make models hallucinate less? Can we use them to monitor models for like erratic behaviour or whether they're being jailbroken? Can we use them to make models more steerable, like in ways other scenarios where like prompt

• 3:34:48Speaker 00

  engineering fails? Can we debug models? Like other times llama or Gemma have weird behaviours that we want to go and dig around and see if we can decode why this is happening? Like, one of my dreams is to be good enough at interpretability that if we have a moment like Sydney again, where it's just behaving in really wild and unexpected ways, like trying to convince a reporter to leave their wife, that we can actually understand what is the cognition happening inside that led to this.

• 3:35:21Tim Scarfe?

  Yeah, I mean, earlier on, we were talking about that connect the dots paper and the unfaithful chain of thought and, you know, potentially you could apply SAEs to some of these problems, but could you just introduce those to the audience?

• 3:35:34Speaker 00

  Yeah, so, yeah, I think I briefly talked about the unfaithful chain of thought. Key idea, well, key motivation. People often see a model produce a chain of thought and they're like, that is an explanation, therefore it is interpretable, I have understood it, why would I ever need to do interpretability? And like, sometimes this is true, but like, sometimes the model kind of already knows the answer and just produces some kind of spurious bullshit if it thinks it's supposed to give an explanation right now. And it's unclear how causally relevant this is to the answer. And so this paper had a really nice demonstration of this, where they made a bunch of multiple choice questions with example chains of thought. They gave the model a few shot prompts with 10 questions where the correct answer was

• 3:36:17Speaker 00

  A, the final one, the correct answer was B, and the model tended to produce a spurious chain of thought justifying why the answer was A and then said A. I'm like, this suggests to me that the model has a circuit inside it. That is like the generate bullshit chain of thought circuit. It has an internal representation of what the correct answer should be. And it somehow is mapping that to this circuit. And I'm like, that is wild. Can we turn this into some kind of like faithful chain of thought detector? Uh, where like, I don't know, you give it a bunch of maths problems and you check how causally

• 3:36:59Speaker 00

  meaningful the chain of thought is, and you compare that to your detector and use that as a force of ground truth or something. Uh, that would be wild.

• 3:37:07Tim Scarfe?

  So, um, we were just talking about O'Wine Evans, um, or it was out of his group anyway. And he had this paper which went viral on Twitter a few months ago called Connecting the Dots, talking about this out of context generalization and reasoning. And there's some really cool examples in that, but can, can you just fill us in on that?

• 3:37:25Speaker 00

  Yeah, so, yeah, so O'Wine's group does the great genre of paper. That's like, you get an LLM and you take a thing that you would intuitively expect to work and you're like, oh, that doesn't work or vice versa. And so the Connect the Dots paper, the, so what they did is they, um, I'll just give it a concrete example. Um, so they, um, came up with a mystery city, say Paris. They called it City X. And they keep giving the model prompts of the form, the distance between City X and London is blah, where blah is the true distance.

• 3:38:05Speaker 00

  And they fine tune it, um, to predict the distance to a bunch of other cities. And then, this is like kind of a weird thing to do. You might expect it would just like memorize these or something. But then at inference time, if you ask it, like, what is the name of City X? It says Paris. If you ask it questions about Paris, like what's the biggest landmark? It will say the Eiffel Tower. And they had several other tasks that like in this kind of genre of you fine tune it to do a task where kind of discovering some latent thing is a useful way to do the task. And then you ask a bunch of questions that show it generalizes in a way you might not expect. And I really like this result.

• 3:38:48Speaker 00

  And my hypothesis what's going on is that the easiest way to answer these questions is not to memorize, it's to use the existing circuitry that has, I don't know, memorized, or maybe the model has some actual internal world model to compute distances. And finding the Paris direction does that. And so like, there is gradient signal in the Paris direction. And this is enough to like, overcome any additional pressure. And so the model just converts the like, city X phrase into Paris. And I would love someone to replicate one of their examples on like,

• 3:39:31Speaker 00

  Llama 3.7b or, is it 9.8b? And, or like Gemma 2.9b? And use SAEs to try to dig into what's going on there. Because I would find it very cool if my random postulating with MechanTup intuitions was correct.

• 3:39:51Tim Scarfe?

  So people at home, you want to learn more about SAEs, what should they do?

• 3:39:56Speaker 00

  Yeah, so, um, yeah. So I think the Arena tutorials are a great place to start. They've just made a new one on Spart Autoエンコード, which we can link. And this will kind of give you a hands-on walkthrough and like, help you write code and like, really engage with them. I think this is a great place to start. If you want a gentler introduction, the other best place to start is the Neuronpedia demo for GemmaScope, where you can just kind of play with it in an interactive way, see what a Spart Autoエンコーデ can do, try staring with it, try using it as a microscope, and no code required.

• 3:40:38Speaker 00

  Maybe I'd start with the Neuronpedia demo and then I'd do the Arena tutorial. And then if you want to dig deeper, my reading list has a section with SAE papers in it. And beyond that, I recommend just like, jumping into projects.

• 3:40:57Speaker 00

  There's a bunch of SAEs on small models if you don't have much compute. If you know how to use larger models, you can play with the various Gemma 2 SAEs in GemmaScope. I think Gemma 2 2B is like pretty manageable to play with. Just like, I don't know, test hypotheses about it. Like, does it have this feature? Can I get it to steer in this way? What things can I exhibit feature splitting? What things am I confused about? Just like, I'm a big believer in like, as you learn about field and as you read papers, just go and get your hands dirty and write a bunch of code and implement ideas in the paper or like predictions it makes or extensions.

• 3:41:42Speaker 00

  And I think this is pretty naturally flows into like actually doing research.

• 3:41:46Tim Scarfe?

  Well, Neil, it's been amazing having you back on MLST. This has been an absolute marathon. And how long have we been recording for? A long time.

• 3:41:55Speaker 00

  So I got here.

• 3:41:59Speaker 00

  Yeah, I got here at 2:00. It's now 10:00, eight and a half hours.

• 3:42:02Tim Scarfe?

  That's pretty good. That's pretty good. Yeah, honestly, it's been amazing. I think that this episode is probably the biggest brain dump out there on the internet on sparse auto encoders. So we promised you that it would be dense and very rich of information. And I hope that we have not only delivered the goods, but we've, you know, gone past what we did last time.

• 3:42:27Speaker 00

  Yeah, it's been great. Thanks so much.

• 3:42:29Tim Scarfe?

  Wonderful. Thank you so much, Neil.