Mechanistic Interpretability: Philosophy, Practice & Progress with Goodfire's Dan Balsam & Tom McGrath

In this episode, Daniel Balsam and Tom McGrath, at Goodfire, discuss the future of mechanistic interpretability in AI models. They explore the fundamental inputs like models, compute, and algorithms, and emphasize the importance of a rich empirical approach to understanding how models work. Balsam and McGrath provide insights into ongoing projects and breakthroughs, particularly in scientific domains and creative applications, as they aim to push the frontiers of AI interpretability. They also discuss the company's recent funding and their goal to advance interpretability as a critical area in AI research. SPONSORS: Box Report: AI is delivering truly measurable productivity — strategic companies are already turning a 37% productivity edge. Discover how in Box’s new 2025 State of AI in the Enterprise Report — read the full report here: https://bit.ly/43uVP52 Oracle Cloud Infrastructure (OCI): Oracle Cloud Infrastructure offers next-generation cloud solutions that cut costs and boost performance. With OCI, you can run AI projects and applications faster and more securely for less. New U. S. customers can save 50% on compute, 70% on storage, and 80% on networking by switching to OCI before May 31, 2024. See if you qualify at https://oracle.com/cognitive ElevenLabs: ElevenLabs gives your app a natural voice. Pick from 5,000+ voices in 31 languages, or clone your own, and launch lifelike agents for support, scheduling, learning, and games. Full server and client SDKs, dynamic tools, and monitoring keep you in control. Start free at https://elevenlabs.io/cognitive-revolution NetSuite: Over 41,000 businesses trust NetSuite by Oracle, the #1 cloud ERP, to future-proof their operations. With a unified platform for accounting, financial management, inventory, and HR, NetSuite provides real-time insights and forecasting to help you make quick, informed decisions. Whether you're earning millions or hundreds of millions, NetSuite empowers you to tackle challenges and seize opportunities. Download the free CFO's guide to AI and machine learning at https://netsuite.com/cognitive Shopify: Shopify powers millions of businesses worldwide, handling 10% of U. S. e-commerce. With hundreds of templates, AI tools for product descriptions, and seamless marketing campaign creation, it's like having a design studio and marketing team in one. Start your $1/month trial today at https://shopify.com/cognitive PRODUCED BY: https://aipodcast.ing SOCIAL LINKS: Website: https://www.cognitiverevolution.ai Twitter (Podcast): https://x.com/cogrev_podcast Twitter (Nathan): https://x.com/labenz LinkedIn: https://linkedin.com/in/nathanlabenz/ Youtube: https://youtube.com/@CognitiveRevolutionPodcast Apple: https://podcasts.apple.com/de/podcast/the-cognitive-revolution-ai-builders-researchers-and/id1669813431 Spotify: https://open.spotify.com/show/6yHyok3M3BjqzR0VB5MSyk

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome back to The Cognitive Revolution. Today, I'm honored to be joined once again by Dan Balsam and Tom McGrath, CTO and chief scientist at mechanistic interpretability startup Goodfire. When we last spoke, about nine months ago now, we focused on the technical foundations of interpretability, including the challenge of polysemiticity, techniques such as activation patching, causal tracing, and feature editing, the rise of sparse autoencoders, and some of the challenges of scaling these techniques to frontier models. If you're new to mechanistic interpretability, I would definitely recommend checking out that earlier episode for a technical primer. Since then, Goodfire has gone on to train sparse autoencoders on Llama 3.3.70b and

• 0:43Nathan Labenz?

  also DeepSeek R1, strengthened its team with the addition of multiple top-tier researchers, and recently announced a $50 million Series A, which notably includes Anthropic's first ever investment in another company, giving me, as a small-time seed-round investor in Goodfire, both strong on-paper returns and a bit of bragging rights. In today's conversation, we mostly zoom out from specific techniques and findings and instead try to get a handle on the state of mechanistic interpretability as a whole. For years, the field has been called "pre-paradigmatic," but as you'll hear, Tom now describes it as "proto-paradigmatic." There's now general agreement among researchers that neural networks contain understandable

• 1:24Nathan Labenz?

  things, that these things, called "features," can be understood as linear directions in embedding space, and the magnitude of their activation represents their intensity. There's also the finding that superposition allows models to represent far, far more concepts than they have dimensions, and that features connect through the layers of the model to form circuits. This is great progress, and honestly, much more than I might have expected just a couple years back. But there are still important gaps between the accounts that interpretability techniques provide and the underlying reality of model structure and behavior. First, and most obviously, there's the fact that interpretability techniques typically attempt to reconstruct the behavior of the underlying model.

• 2:07Nathan Labenz?

  And as of now, they can do so only very roughly. Second, and more philosophically, there's this distinction between the features that interpretability techniques learn and the meaning that we assign to them in the process of labeling. In practical terms, when we say that a feature represents the Golden Gate Bridge, or more to the point, deception, how confident can we really be in that label? From my own exploration of both Goodfire and Anthropic's interactive interfaces, this seems to range very widely. All of this is complicated further by another all-too太多neglected fact: the models under study encode varying degrees of understanding, with everything from simple memorization to fuzzy heuristics to proper algorithmic grokking, all occurring simultaneously in

• 2:52Nathan Labenz?

  an unknown mix in any given model. Of course, while the philosophy is fascinating and there's still clearly a ton of work left to do, that is not stopping Goodfire from deriving practical value from interpretability techniques today. And Dan describes how Goodfire is developing applications, both for enterprise customers and the public good, across three key domains. Scientific discovery, where they're partnering with organizations like the Ark Institute to explore genomics models like Evo2 and beginning to uncover novel biological insights. Guardrails and safety, where they're developing inference time monitoring applications that can detect when models might output harmful content or exhibit other problematic behaviors. And creative applications, such as their just-launched Paint With Ember tool that

• 3:36Nathan Labenz?

  allows users to generate and edit images by directly manipulating sparse autoencoder features. Proto-paradigmatic though it may be, as we enter a new era in which science shifts towards simulation-based approaches and AI systems potentially drive more and more of the machine learning research, it seems to me a very safe bet that interpretability work will become more and more important. As Dan put it, "Even if we end up in a scenario where a data center full of geniuses is doing most of the scientific work, mechanistic interpretability might be their preferred tool for understanding both their discoveries and themselves."

• 4:15Nathan Labenz?

  As always, if you're finding value in the show, we'd appreciate it if you'd share it with friends, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Your feedback is always welcome, too. Feel free to reach out anytime via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. For now, I hope you enjoy this thought-provoking exploration of the philosophy and practice of mechanistic interpretability with Dan Balsam and Tom McGrath of Goodfire. Dan Balsam and Tom McGrath, CTO and chief scientist at Goodfire. Welcome back to the Cognitive Revolution.

• 4:50Dan Balsam?

  Thank you so much for having us.

• 4:52Tom McGrath?

  Thanks. Yeah, great to be on that.

• 4:54Nathan Labenz?

  I'm excited. We haven't been able to make this happen quite as often as I would have liked, but we're going to make up for it by going long and in-depth today. So really excited to get the update on what you guys are building as a company, which I understand there's some great news on, and also just to check in on what we have learned as a community about models and how we understand how they work over the last few months, because obviously, there's nowhere in the world changing faster than that. So for starters, I wanted to kind of go high level and just ask you to kind of frame the field. I mean, I think everybody in the general NLP space at this point has internalized this data, compute, and algorithms paradigm. You know, these are the three legs of the stool that

• 5:37Nathan Labenz?

  are enabling progress. There's the sense that they all kind of contribute equally. And then on the interpretability side, I'm sort of tempted to slot in models for data and say like, models compute and algorithms are maybe the things, and seemingly a lot depends on the quality of the models. But then there's still the role for data. So how do you guys think of the kind of fundamental inputs of what you're doing?

• 6:01Tom McGrath?

  I think one thing that's interesting here is that there's These inputs are very important, right? And the models have changed and so things sort of changed along with them. But another thing is like, it's a very empirical activity. In the sense of like, you're looking in a fairly fine-grained way, like at the data of what's happening inside models. So like progress in algorithms, for instance, parameter decomposition or SAEs or whatever. We can get into these later. But one of the inputs here is actually like very rich empirical data. You know, when you're just like, look, basically like, what are activations like? When we like sort of tinker around with models in a relatively hypothesis-free way. Do they actually seem to behave?

• 6:44Tom McGrath?

  And I guess like this sort of empirical input is always like part of progress in algorithms. But I think I just want to like upweight it for interpretability because we're really doing this sort of natural science. But yeah, I like broadly agree with the decomposition of like models or the data, I suppose. And then there's compute. And I think as a field, we would like to be able to use much more compute. And then there's algorithms. And I think that, I think it's a good decomposition. I would say that we don't have like the transformer but for interpretability. And it could be that the thing that we are blocked on for finding the transformer for interpretability is simply understanding models better in a way that we can like then generalize to a new inductive bias.

• 7:25Nathan Labenz?

  Yeah, that's really interesting. And you kind of prompted me also to try another paradigm mapping exercise. You said we're doing this in a relatively hypothesis-free way. That sort of maps in my mind to like pre-training, sort of unsupervised learning, right? The sort of SAE paradigm has largely been run a ton of data through, basically from the same original data set. You could tell me more about how that's curated, maybe especially for interpretability work, but relatively hypothesis-free way. Is there any equivalent to like post-training yet in the interpretability world?

• 8:01Tom McGrath?

  Ah, so when I say, that's interesting. When I say hypothesis-free, I mainly mean like a person sitting down and tinkering with models and being like, what structure is that? So for instance, like this kind of famous not all language model features a linear paper. They're, they're sort of. Finding structure in activations and this structure is kind of interesting. It's this kind of higher dimensional manifold. And you don't necessarily get to this by being like, I have a hypothesis that things are this way. You're just sort of like a natural scientist, like someone out observing the stars or something. They kind of, they have some hypotheses in mind. But they're also just going, they're just sort of looking at things. And so that's what I mean, but it's interesting that actually I also am very keen on the unsupervised

• 8:46Tom McGrath?

  approach to the unsupervised approach to kind of interpret models like SAEs. For basically the same reason, except that every architecture is kind of coming with a hypothesis. Like the SAE inductive biases, things are sort of, well, the SAE is like a fairly strong hypothesis that features are literally directions in embedding space.

• 9:06Nathan Labenz?

  Yeah, there's something really interesting there around. And again, the model quality seems to be really important, right? Because a lot of the kind of hypothesis free tinkering sort of even vibe coding, right? I mean, Neil Nando recently put out a video of him doing some vibe coding research. And that's all kind of premised on the idea that you can run a little experiment and get something back like relatively quickly on a sort of iteration timeline that where ideally you could like sustain focus. Maybe you have to go take a walk and come back, but it's not like really long timeframes or really large compute budgets. Whereas, as you mentioned, like you got to have some conviction to throw large amounts of compute at an SAE at scale.

• 9:51Nathan Labenz?

  But I wonder like, how do you think about the sort of challenge of?

• 9:55Nathan Labenz?

  Can you do that like rapid experimentation on like the truly large advanced models? Or are you limited to working with something like GPT two scale? And does that sort of create like a fundamentally different regime from the things that you end up scaling up?

• 10:11Tom McGrath?

  It's really a question of the structure. If you have no infrastructure, then it's hard. If you have the right infrastructure, then building that infrastructure is hard. But the right infrastructure makes experiments relatively easy.

• 10:27Dan Balsam?

  Yeah, the other thing that I'll add is like when thinking about unsupervised techniques as like hypothesis generators for how the model could be working. There's no way that we're going to be able to scale to superintelligence without making our interpretability techniques unsupervised. And that's one of the things that really motivates us is our mission at Goodfire. Narrowly, superintelligent models already exist in the scientific domains. And this is what we spend a bunch of our time working on. You know, when you're working with a genomics model, you're working with a model that we have priors about. There's lots of bioinformatic research to attempt to understand the genome, but we're also working with systems for which, you know, our statistical techniques that we have are not as explanatory as we'd like to be.

• 11:11Dan Balsam?

  And that's the motivation towards moving towards AI to begin with. And so in the process of training unsupervised models, it really gives us a grounding about where to look in the model to begin with. Gives us a bunch of candidate experiments, a bunch of candidate hypotheses to run. Yeah, and one of our goals as a company is unsupervised learning often looks worse until suddenly it looks better. You know, if you went back in time, there was a point where massive pre-training in an unsupervised way on large corpuses was getting you worse performance than bespoke models that were that were purpose-built.

• 11:48Dan Balsam?

  And so we think interpretability is likely to follow a similar arc. We're still not sure exactly what those pieces of technology will be, but each new sort of item on the tech tree in interpretability unlocks new questions that we can ask, new ways that we can look at the problem. And as Tom was saying, eventually building towards a solution that we can just toss a lot of compute at in order to fully unlock what's going on inside.

• 12:12Nathan Labenz?

  So am I interpreting you correctly to say that right now you'd say the field still feels like it is bottleneck more on algorithms than?

• 12:26Nathan Labenz?

  Compute or certainly not models, right? I mean, there's a lot more in models we can figure out. So the two candidates would be compute and algorithms. It seems like you're saying algorithms are still where it's at and we need to figure out how to apply the compute.

• 12:39Dan Balsam?

  I think that's right in some sense, but I think like it was like a soft version of it that I would take. I think existing tools are already powerful enough to do things that are useful in the world. And that's a big part of what we're doing in good fire when we're working with customers. Is we're using the existing techniques that are available to look at a model to help customers understand their model better. But also, we are under no illusions that we've cracked interpretability. Like to fully reverse engineer what's happening in a model, we do need brand new techniques and new paradigms.

• 13:12Tom McGrath?

  On the compute front, I think it's really a question of like frontier models are bottlenecked by like literal availability of compute or will be soon, right? We're very far from that, but we could spend a lot of compute, right? The question is, do you get value for money for it? Like, I would like to be able to do a million dollar interpreter model training run and say, oh, yes. I spent a million dollars on it, but I feel like I got one million dollars worth of information.

• 13:37Nathan Labenz?

  That is an interesting, maybe there's a way to say that, you know, like there's a sort of ratio between

• 13:46Nathan Labenz?

  size of experiment and like how many, that's maybe not quite the right way to say it either, but given an option of choosing like a really big run or more people to come up with more ideas for smaller runs. It's clear that you'd rather have more interpretability researchers exploring the space more thoroughly before scaling up.

• 14:07Tom McGrath?

  Well, yeah, like I think that we couldn't productively do a single million dollar training run. I think that's one of the things that sort of says, are you bottlenecked on algorithms? Can you productively spend that much compute? We could spend it, but at the moment I think it would be, it wouldn't be a productive way to spend it.

• 14:27Dan Balsam?

  I think there's also the question of like how you use the tools that you have. Like for our customers, when we've trained an SAE for them on their model, or any type of interpreter model, that's not, that's where the work begins, not where it ends. I think of an SAE as a window into the model, right? You can only see some things, but those things can still be really useful and provide a lot of value. You know, Chris will popularize the biology analogies for mechanistic interpretability. And I think those really track, from my perspective, like if you go back to the 19th century, early 20th century, you know, what did you have to do to learn anything about cells? Like you had to put things on a slide, you had to stain the slide, you had to look in a crude microscope.

• 15:09Dan Balsam?

  And you were looking at a cross section of something not in its natural environment. And you had to make, you know, inferences based on looking a lot of these about what were the actual biological structures that were going. But all of modern biology was built on that. And over time, we developed better microscopes. We developed better techniques for looking at organisms in more natural ways. So I think we're a little further on in interpretability, maybe than that analogy implies. But I think it's the same thing, like you can learn and make novel advancements in science, any state of the technology. And just simultaneously, like we want to push what we can learn with the tools that we have, while we're also pushing to get better tools.

• 15:53Tom McGrath?

  I think for a while, people had to like make their own lenses, right? Which I guess is like making your own SAE training code base. Like probably you want to just go and buy the lenses from Zeiss or something like that. And then you can become an expert in great tools. And then the field can advance it, some sort of famous dictum in science about like things advancing via methods and ideas and then experiments or something like that, in that order.

• 16:17Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 16:22Nathan Labenz?

  Yeah, there's an Adam Smith lesson here, this sort of extent to the market and the degree of specialization and sophistication of the supply chains are all pretty early. And we kind of are in that sort of, we are maybe just exiting the like, grind your own glass lenses phase of biology. Let's hope we don't find ourselves in the gain-of-function research, lab-leak and invasive species phase of biology before we're ready to handle them. But, you know, I guess one, one place where this analogy maybe breaks down a little bit is on this question of fundamental units. And this is sort of a philosophical question I've been trying to wrap my head around better. I'm sure you guys have good thoughts on it.

• 17:06Nathan Labenz?

  When we look at like a, an organism, right, and we look at its genome, or we look at sort of proteins, like, we're pretty confident we're talking about real things. I guess they're sort of maybe like quantumly fuzzy on some margin, but like, we have a pretty good sense for a gene is a gene and a protein is a little machine. And here in the interpretability side, the features, if you will, that are identified or that are learned by an SAE or similar techniques, and you might want to, if appropriate, like, separate SAEs from other techniques in your answer here. But these things seem to be sort of approximations, or sort of, there's some

• 17:47Nathan Labenz?

  sort of gap, but right there between what is going on in a model and sort of what is going on when it's sort of sparsified in this sort of particular way. And so I'm really interested in how you think about the relationship between these features that are learned and the, like, labels that we give them. And sort of how much correspondence you think there is there? Is that on a spectrum? And how should we think about it?

• 18:15Tom McGrath?

  Yeah, so it's interesting. Well, I suspect if you asked a biologist, like, they probably have a lot of corner cases about like, is it really a gene? Biology is, if nothing else, like, a great supplier of corner cases, just because of the rich complexity of the world. But yes, I think the, there is a definite sense there that like, a gene is a sort of natural abstraction. It's like a, it's a good way to talk about the world. And

• 18:43Tom McGrath?

  So this process of like, you take a model, you sparsify it. Well, now we have introduced some degree of lossiness, right? Because we're not capturing all of the computation. You can see this from the loss, you can see this from the reconstruction error. But we are capturing what look like very interesting and interpretable things. But that takes you on to the sort of next level of like, okay, there's a thing. There's like a feature in your sparse dictionary. And now we assign this label. And this is another area for sort of a gap, this sort of gap that you can fall in. And I think that we can be in the business of closing both of these gaps like a great deal. And you can close the gaps in multiple ways. What is to have a better idea?

• 19:26Tom McGrath?

  Also, the first gap, right? Gap one, where we're talking about the distance between the model and our approximation of the model.

• 19:34Tom McGrath?

  Then how do you close this gap? Well, one answer is you do the machine learning better. You just make a better essay. So you capture more of the loss. And there have been a bunch of this work in this direction, like lot, there's, you know, I can provide a bunch of papers later. Um. Another is you try and answer the question, what does it mean for for something to be a good abstraction? And you sort of use that as inspiration for new methods. So for instance, like, what would it be? For an essay feature to be like a natural unit of computation? Well, it's not completely clear. I think there's actually some interesting, but probably quite resolvable issues there. One thing it might mean is that it like

• 20:20Tom McGrath?

  is involving kind of consistent computational parts. You know, the feature is a natural unit of computation. If it's sort of is involved in other computations that make sense. And so now this is maybe a way, maybe one way of saying this, I think that things get a lot cleaner. Like feel like they will get cleaner as we move to circuits rather than like just single layer activations. Because you don't really have a great way of validating. Now you can validate by like. Intervening on a feature and seeing how things change. That's sort of like circuits, except you just haven't tracked the circuit. Now you're sort of entering into the second part, the second gap. You know, I've intervened on a feature.

• 21:03Tom McGrath?

  You might say, well, I intervened on it and it didn't do what I expected. But that might be because my expectation was wrong. That I've simply like fallen into the second gap. Where I've given a feature an incorrect, like I've given the feature an incorrect description. It is a like unit of computation that the model does. But I've just called it the wrong thing. Now, how do we narrow the second gap? And I think that the answer here is probably. We just like get better at doing experiments on interpretability. Or like we kind of, so the way that we currently assign labels is.

• 21:39Tom McGrath?

  I hopefully I won't offend Nick by saying this is like a little primitive.

• 21:45Tom McGrath?

  So Nick came around to like, I'm the inventor of this automated interpretability.

• 21:51Tom McGrath?

  Technique.

• 21:52Nathan Labenz?

  And principal investigator of gunfire.

• 21:54Tom McGrath?

  Yes, thank you. I was like, I'm gonna go through follow. Yeah, he's wonderful. So anyway, hopefully I will defend him by saying the current method of like assigning semantics to features is a little primitive. And. What we do is we give the like a frontier model, a bunch of examples of where the feature fired. And we say, well, here are these examples. Like, what's the feature?

• 22:22Tom McGrath?

  And this gets you some way, right? But it doesn't get you the whole way. For instance, like if you were to ask me with access to the model. What is this feature? I wouldn't only do that, I would also like try steering the future and see what happens. I might like look at other things that projected into that feature or like where it goes downstream. They're kind of relates to other features, all that sort of thing. So there are like many more things that I personally would do. But we can't currently get models, we can't currently get frontier models to do this. It may just be a matter of scaffolding. We just need to build this kind of scaffolding such that they can use their capability. So there's a really run on odds. Yeah, I think there were these two gaps that we can narrow, but we say.

• 23:00Nathan Labenz?

  No, on the contrary, I kind of want to expand on it on maybe both points because.

• 23:09Nathan Labenz?

  I mean, it's all really fascinating and important. So the first bit, I guess, I really like the just sort of decomposing the problem into the two gaps. First is, can we, I mean, this is literally what the SAE is trained to do, or the other techniques is to reconstruct, right? It's a reconstruction loss. Do what the model originally was supposed to do. There, I think I'd love to know a little bit, maybe you said like you could provide a bunch of papers later. I'd love to hear a sort of, what is the state of the art? Like, the deepest dive I've done into literature recently was the anthropic tracing model thoughts pair of papers. And I was kind of struck there by it seemed like there's a lot that is not being reconstructed, I guess,

• 23:51Nathan Labenz?

  is what I would bottom line it as. So I'd love to get a sense of where is that state of the art? And maybe in narrative form, if not in fully cited paper form, like, what has been the trajectory of kind of improving that? Would have been the advances, the unlocks, whatever? And then on that second gap, this sort of anticipates another question I had around just, what is the inference time scaling paradigm for interpretability? It sounds like the answer there is, well, today, we run data through the thing, see what activates what, and kind of collect the things that cause the max activation, then try to describe them qualitatively. And in the future, we could do like, a lot more. And then you could kind of develop a little bit the

• 24:33Nathan Labenz?

  vision for the sort of higher order auto interpretability. I think both of those are really interesting mini lectures that you could go on it as much length as you want.

• 24:43Dan Balsam?

  I think, before we dive in there, can I give a quick meta thought on this higher question, which is that like, most measurement apparatuses that you could deploy in most scientific contexts are reductive in some way, right? There's some set of assumptions that you're making in how you should interpret the data that you're getting from the measurement apparatus that you're doing. The microscope analogy with SAEs, I think, works really well when explaining this to people, because there's three things that you can Do with the microscope, right? Like you can figure out what you're putting on the slide. You can, that's the data set, right? That you're passing through the original model. Those are the actual activations itself. You can get a different SAE on one data set from a different data set.

• 25:27Dan Balsam?

  You know, there's sort of like how you stay in the slide, like maybe that's by the loss function or like how exactly you're looking at things. That's going to affect the structure of what you get back in different ways. And then there's the size of the lens, right? It's like the expansion factor of the SAE. It's important to understand what we're doing with our existing interpreter models as a like a specific lens, a specific way of looking at the computation of the model rather than the whole picture. I think this is true whether you're trying to do circuit work with sparse approximations, or if you're trying to do like, you know, layer level activation work with sparse approximations. But that's not an abnormal thing to do in science. Like you take some sets of assumptions.

• 26:10Dan Balsam?

  You know, they're not always right, but you know that they're right enough sometimes that you can start getting traction and running new experiments. So it's not, and from our perspective, it's not this all or nothing thing. This is like what Todd is saying, like we can keep pushing on the fronts of how do we address the limitations in the existing tools within the paradigm that they're operating with the sets of assumptions that we know are sometimes wrong, but are right often enough to be useful and give us more information. And then simultaneously ask the question, what's a better set of assumptions? And these have to happen in parallel. You can't just do one at a time because you would never find the right assumptions if you weren't testing the limits of your current ones. So I, from my perspective, like this isn't a unique thing about interpretability. This is like how science works.

• 26:55Tom McGrath?

  Yeah, exactly. I'm like, it's funny, people remember Kuhn's structure of scientific revolutions for the revolutions, right? Like crises and yes, like the thing, the sort of dominant mode of science is like normal science, where you're going along and you're like generating actually productive knowledge about the world. Maybe on foundations that will like later get a bit shaky or get overturned, but like you're still generating knowledge about the world. And then there's this idea of there are anomalies. And then anomalies pile up or unanswered need to a crisis, right? But that's where the anomalies come from. The anomalies come from the business of doing normal science.

• 27:39Tom McGrath?

  So even if you want to generate a paradigm shift, often the answer is to just try to try and do normal science until it becomes untenable. And I think that's, yeah, that's maybe where we're one way, one place we're maybe at the moment. We have like probably a proto paradigm. I think we've been like reluctant to admit it, but I think we probably have kind of a proto paradigm and interpretability. And so we should push it. We can do a lot of useful stuff. We should just, we should keep pushing it. We should keep doing the useful stuff and see where, and wait for the anomalies to reveal themselves.

• 28:14Nathan Labenz?

  Yeah, I was gonna ask actually to do, are we still pre-paradigmatic? We've upgraded ourselves now to proto-paradigmatic.

• 28:20Tom McGrath?

  I'm gonna say proto, but maybe I should, maybe I should have some encouraging my convictions instead of that. I think we're like entering our first, like the first paradigmatic phase of interpretability. Well, no. Okay, so this is a bit fuzzy, right? Like, what is a paradigm? A paradigm is a sort of social thing. I don't think there's consensus. There's not the kind of consensus that would lead me to say there is a field-wide paradigm in interpretability. I would say that among a reasonably large group of people, there is, there's like the foundation, there's the raw materials for paradigm, right?

• 29:03Tom McGrath?

  Like, I suspect if the field were anthropic, but sort of ballooned to the size of the global interpretability community, it would be correct for me to say that there was a paradigm. But because there's this, there isn't a level of consensus, I can't really yet say there is paradigm.

• 29:20Nathan Labenz?

  If there were, so what would that paradigm be? How would you describe the anthropic and good fire axis paradigm?

• 29:27Tom McGrath?

  I'd say, okay, that.

• 29:32Tom McGrath?

  One, like neural networks contain things which are understandable. But this is like actually worth stating for a long time. This is not generally accepted. I don't know if it is yet generally accepted. But, you know, this is like the sort of down in the basement of the paradigm.

• 29:49Nathan Labenz?

  As a quick anecdotum there, is that a artifact of just earlier models? Because the way I would tell that story is like the original GPT layer. There were still some things probably that were like meaningfully enough, but there was also so much noise that people could very easily have been excused for just being like, eh, you're tricking yourself or you might find some spurious correlation here or there, but I don't really buy it. And it maybe just haven't updated since.

• 30:16Dan Balsam?

  I think, yeah, you're going to interpret the model that you have. And if there are lots of flaws in the model, those might be what you're finding with your interpretability tools. And if you're entering with the prior that, oh, I should be looking and recovering the specific thing and you can't recover it. That could just be as much evidence that the model isn't doing what you thought it was doing to begin with. You know, like when you go through the process of debugging a model using interpretability techniques, the thing that you might find is, oh, the model has memorized a bunch of its training data or something like that. You know, you have some belief about what your model is doing and how it's modeling the task. And that belief might be wrong and that could throw off

• 30:57Dan Balsam?

  the perception of interpretability if you're not bringing unsupervised, low opinion techniques in that can work across the entirety of the end to end interpretability stack. And I think to that point, we just like didn't have unsupervised techniques that can work at least a good percentage of the time across the end to end interpretability stack until quite recently.

• 31:16Tom McGrath?

  There's like a funny sort of U shaped thing where I think a lot of the early connectionist papers, they actually do look at, they do look at individual neurons and say, oh, look, this neuron learned this thing. This neuron learned this thing. And they could do that because there were like 12 neurons. And so it's funny, it sort of started out and everyone wanted to look at the neurons and had some success. And then for reasons that are opaque to me, but I suspect like. Someone could find out. It became very like somewhere between unfashionable and like considered to be a bad idea or impossible to look at individual neurons. And now it's kind of come back into mode. So it's sort of like we just had the interpretability just kind of went into the wilderness for a bit.

• 31:58Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 32:04Nathan Labenz?

  Yeah, okay. So now give us the paradigm. So one is there are things that make sense that we can interpret.

• 32:10Tom McGrath?

  Yes, there are things to interpret. Interpretability is possible. And then

• 32:17Tom McGrath?

  what other parts of the paradigm are there? I suppose there's features like representations are linearly decodable. That's or at least like linear decoding is a reasonable way to talk about features. You should there may be higher order structure, right? Things you might have features that like features are sort of arrows in space. It might be that actually multiple features kind of lie on some manifold or in some subspace. But it's a sensible way to talk about representations as lines through embedding space. Third part of this paradigm is like this idea of superposition, because if you're going to have if you're going to have vectors in a vector space, then

• 33:03Tom McGrath?

  the natural conclusion would be like, well, I'm in a de-model sized space, right? Well, does that mean the model can only think of de-model things? Probably not, right? A language model can think of more than 4,096 things or something. So the other part of this paradigm is superposition, which is this idea that the way that you put you way that you squash more of these embeddings, more of these vectors, these like feature vectors into the same space is by allowing them to overlap a little bit. And this creates a sort of bit of interference, bit of noise in the representations that models are still able to like deal with.

• 33:40Tom McGrath?

  Now, I guess another part of this is that magnitude, like direction along the vector, constitutes intensity. And the other thing is, rather obviously, I suppose, features connect to form circuits. And that is basically, I think, the paradigm. The paradigm which is currently, rather, the thing which, this is like the anthropic paradigm, I would say. And if you were to blow this up to the size of the world, and like, if there were consensus on this, then I guess it has enough structure to be called paradigm.

• 34:14Nathan Labenz?

  Is there any competing proto-paradigm, or is there just other people claiming we'll never have one at all?

• 34:23Tom McGrath?

  Not in the sense of something that like provides a complete worldview about,

• 34:28Tom McGrath?

  let's say a complete worldview, all I think about is about neural networks. It doesn't say anything about dinner, but I don't think about that very much. But a complete worldview for neural networks. You could say that something like parameter decomposition suggests a separate paradigm. And that's because that's sort of a difference in emphasis. Parameter decomposition is talking about the weights. The sort of, the SAE-type paradigm is talking about activations. Now, I would say they're still sort of, yeah, so they are, they're sort of different levels of emphasis. I think that a way to think about this maybe is that we, unsurprisingly, we need both type.

• 35:12Tom McGrath?

  And to make this seem like a bit more intuitive.

• 35:17Tom McGrath?

  A neural network is, in a very dull sense, a causal model. Right? Every neuron is a node in your causal graph. And all the weights specify the edges in your causal model. I say it's just, it's very big, but not very, like, big, very homogenous, not very interesting causal model. And what we, one way of thinking about what we're trying to do in interpretability is we're trying to create like a causal abstraction. We're trying to create another model, which is like a sort of reduced version of this model. But that will also be a causal graph. And so when we're talking about this like debate between should we decompose the parameters, should we decompose the activations, saying should my graph, right, should it have nodes?

• 36:01Tom McGrath?

  Or should it have edges? Well, probably it should have both. It's a graph. So this is why I think that actually they're not necessarily competing paradigms. They're just sort of two independent ways of thinking our way towards like the broader kind of causal abstraction that I think we need.

• 36:17Nathan Labenz?

  Would you point to any highlights in the work that has focused more on the weights than the activations? Because certainly from my perspective, it seems like much more news and excitement coming out of the activation space.

• 36:30Tom McGrath?

  It's funny because some of the, okay, so the earlier stuff, we're going to stay in the sort of mechanistic interpretability extended universe, because I can't immediately recall some of the other papers as well. I can have a look, but like some of the earlier stuff like learned equivarians in the circuit thread, for instance, was that was a sort of weight based analysis. But then the more recent parameter decomposition, things like Lee coming up, Lee's group, formerly at Apollo, now he's a principal investigator, good fire, just do a little plug. It's great. And so that's like attribution based parameter decomposition was the sort of milestone thing here recently. There's another

• 37:12Tom McGrath?

  recent paper sort of following up on this based on ideas from the lost landscape. I'm blanking on the title. I guess Brianna Christman is the lead author. I can send it to you afterwards. So like it's sort of, it's what happened in the past. And then this sort of essay has got a lot of traction and so that took a lot of the focus. And now I think it's sort of coming back in again.

• 37:36Nathan Labenz?

  One of the school possibly that comes to mind is the sort of, or maybe contrast in approaches is the sort of bottoms up versus top down. Like, I think of Dan Hendrix and sort of representation engineering or circuit breaker type work being less somehow focused on the sort of unsupervised, like just discover what it contains and more this is the sort of contrast that I care about. Like, let's make sure that we refuse or whatever under certain conditions. Do you think those are like fundamentally different approaches or are those sort of ultimately reconcilable?

• 38:19Dan Balsam?

  I think so. I think they are, they solve for different problems and they're both important things to look at from a safety perspective. When we zoom out and we think about the alignment problem broadly, we think the solution to the alignment problem could come from interpretability, but doesn't necessarily come directly from interpretability. Interpretability is the measurement apparatus that can make us confident that any other techniques are in fact doing the things that we think they're doing. For instance, are we aligning the chain of thought of a model faithfully to its computation? I don't know how we would have any hope of

• 38:59Dan Balsam?

  answering that question if we don't have meaningful ways of constructing graphs that represent the model's computation in some way. So I think a lot of these like core questions in alignment, like at the end of the day, it could be that we solve alignment through bottoms up, but it could not be. But without the bottoms up, I have no idea how we would know that we solved it through any other means.

• 39:22Nathan Labenz?

  So, okay, I think that's really super clarifying stuff. I think people will find the sort of high level mental models there quite interesting. Maybe I'll add a third chapter. So one, or the gaps, right? If we structure this in terms of like opportunities to make things better, we've got Better reconstruction, lower loss. Eager to hear a little bit about kind of what the progress has looked like there. Then we've got this sort of, you know, better labeling or like inference time scaling and, you know, getting to higher and higher orders of automation and confidence that what we are describing, what we say is happening is in fact what is happening, that we're understanding it correctly. And then that you alluded to, what I'll maybe add on is

• 40:07Nathan Labenz?

  the third thing, which is the move from activations to circuits. And I did think Lee Sharkey's paper there was, was super interesting and could be a whole episode on its own, but we can do a mini one as part of this. So

• 40:20Tom McGrath?

  I should mention the, the antropic circuit tracing results as well. I think like everyone wants circuits. The question is, how do we get them?

• 40:27Nathan Labenz?

  Yeah. So take us through those three chapters. And again, take all the time you need because I'm here for it.

• 40:33Tom McGrath?

  Okay. So should we talk about the first one first? And that's like, how have people been doing the machine learning better? Because and here the machine learning part of it is like, how do we best learn this kind of decomposition of the models or like within the, let's just stick within the SAE paradigm because I can be very concrete. How do we learn good decompositions to the models? And some of this is you start with the SAE and it has L1's fastier regularization, has a ReLU activation function. And then people sort of hill climbed a bit in the kind of classic machine learning way of, well, the L1's fastier has certain properties, like it causes features

• 41:15Tom McGrath?

  to shrink, which also has like a predictable bad effect on the reconstruction loss. And so there are various solutions to this, things like jump ReLU or batch top K is sort of some hill climbing on this work. And then things like end to end SAEs where instead of training on the activations, you're like, you're not training purely on the activation reconstruction loss, you're actually training on like, such that when you take the model, you've got the model, when the computations are going up, you put the SAE in the middle, and we train such that when you put the SAE in the middle, the model doesn't get too much worse. And it's called an end to end SAE.

• 41:52Tom McGrath?

  And both of these things, all of these things like improve on the loss reconstructed versus sparsity kind of frontier. What else is there? There are many other approaches to dictionary learning. Where dictionary learning is the sort of what is like the broad class of things of which the SAU is instantiation. The dictionary here been your like selection of vectors. And your quiver of arrows is the dictionary and you're learning a dictionary that could do a sparsity composition. And things like using gradient pursuit instead. I saw an interesting question like a residual quantized autoencoder, Matreshka. All these kind of things are like other ways of using compute better to have a like.

• 42:37Tom McGrath?

  You sort of haven't really changed the fundamental assumption, but features are directions in space. But you've gone like.

• 42:46Tom McGrath?

  How do we like, under this basic assumption, how do we like create a machine learning architecture that reduces these metrics of like? Loss, yeah, they improves the loss recovered. What else? I don't think there's any other. And doubtless I would have missed some and like offended people by leaving their papers out. That's just a failure of my memory. Apologies, people who have papers I've forgotten.

• 43:07Nathan Labenz?

  Well, you go to your hint. Play comprehensive in real time. The Matreshka concept was one that I was keen to explore. I've heard a little bit about that and it has a very, I don't want to be overly lured by this, but it has a very appealing vibe, I guess, to me where it's like, there should be some. You know, structure that gets finer grained as you go. And basically the idea there is like sort of a tree structure, right? I mean, it's a little, it's not really structured that way, but the idea is like the short version.

• 43:40Tom McGrath?

  It would be ideal if it were a tree.

• 43:43Tom McGrath?

  Unfortunately, there's no explicit tree structure involved.

• 43:47Tom McGrath?

  Matriashka is really interesting, actually, like both as an architecture and from a point that I was sort of talking about earlier of we're talking about this idea of models, compute, algorithms, and algorithms kind of being bottlenecked on observable experimental data. You know, like us knowing new things about models. And Matriashka is an example of this happening. So like, how did we get to them? I say we, I had no personal part in it. How did the field get to Matriashka? Well, my understanding is that there's this idea of feature absorption and feature absorption happens when SAE latents kind of specialize far too much.

• 44:28Tom McGrath?

  So instead of there being, I would go into the mechanisms a bit involved, but essentially instead of there being like, for instance, a mechanism is a feature for token starts with the letter A. You now get a feature for like token starts with the letter A and is not the word aardvark. Or something like that. And then you also have a feature for like token is aardvark. Because it turns out this gets lower L1's fast loss. And so this kind of thing called feature absorption was a motivation for Matriashka. Now, how did we get to feature absorption, right? Well, someone looked at some SAE latents and said, that's funny, that looks wrong to me. I think the answer was actually they were probing for

• 45:14Tom McGrath?

  the training linear probes for like, what letter does this token start with? Which sort of feels like a silly niche thing to do when you don't, without context, but like actually led to this interesting discovery. It's a very smart thing to do. It's a, it led to this interesting discovery of what happens with SAE latents. It was relatively hard to predict. Certainly no one predicted it a priori, as far as I know. And then Matriashka turns out to be a good way to fix this. Now, a more desirable thing would be the thing that you said, like Matriashka, but as a tree. So what actually happens with Matriashka is you have like a series of nested groups of features. And so we predict using the first group. Well, there's two variants, the simplest one to talk about uses groups.

• 46:00Tom McGrath?

  We predicted one group, right? And now we have some residual error term. And then we use the second group to predict that residual error term and so on as we go out of the shelves. What I think would be quite desirable is instead being able to say, well, this feature fired, which means that now I'm going to like upweight this other feature firing. And so you'd have this sort of explicitly encoded tree structure.

• 46:26Tom McGrath?

  That's the idea of a tree structure is very interesting for like minimum description length reasons. There's a really cool paper on the minimum description. I can come back to that in a minute. But enforcing this kind of discrete sampling behavior is like relatively hard. It's much easier to just do this like very soft sort of differentiable thing. Like I think it's harder to do the kind of thing that we both would like to do.

• 46:52Nathan Labenz?

  Mashirsko, in general, I wonder how much of this is sort of, and maybe this could again be sort of abstracted a little bit to interpretability in general as a sort of a subfield that's coming in the wake of the broader ML developments.

• 47:08Nathan Labenz?

  In general, it's been applied in a bunch of different places, right? And it sort of boils down to finding a way to make the first bits of any given thing, the most meaningful. I sort of think of it as ordering the the data so that the most meaningful stuff comes at the front. And you've seen this in like embeddings and weights and whatever, where you can have a a short embedding that is pretty meaningful. And then if you, the longer you go down toward the full embedding, the more meaningful it gets. But you're sort of always getting the next most relevant bit of information at each stage. So intellectual history wise, would it be right to guess that a lot of the things that are happening in this kind of doing machine learning better

• 47:50Nathan Labenz?

  are people kind of looking for techniques that have been developed previously in other contexts and saying sort of like this worked before, like maybe it will work again here. And then kind of having applied that technique, then sort of developing a story of like what was happening

• 48:07Nathan Labenz?

  after the fact? That's sort of how I expect a lot of this would be working, but you know, I trust you to tell me different.

• 48:13Tom McGrath?

  I am not sure. I expect that a lot of it was worked out like fairly first principles. And that it turns out that someone may have done something related before. But like thinking about this context, you arrive at, oh, this is a problem.

• 48:31Tom McGrath?

  Now, there has been a lot of work done on sparse auto encoders in the past, like around, you know, early 2000s, a lot of work went into sparse auto encoders, like dictionary learning has previously been like an active topic of ML research.

• 48:46Tom McGrath?

  But I think we're sort of rediscovering a lot of things at the moment, and it could be, but one thing to do, one actually high alpha activity in dictionary learning or in sort of this more ML side of interpretability is simply to get the history books, I say history books, I mean, like, you know, papers long, long past and see what has been done that hasn't yet, that hasn't yet been like applied.

• 49:11Tom McGrath?

  One, one example I share, I think is quite funny, this is the recent NeurIPS sparsity, the 2024 NeurIPS sparsity tutorial was very good, but one of the slides early in the talk, it was like what this talk, what this tutorial is not about, and it was like this tutorial is not about like sparsity for interpretability, i.e. sparse auto encoders, and there are various other things in it, but I saw this, yeah, it's not about these various other kinds of applications of sparsity, but I saw this and I was like, yes, great, now I'm going to learn some things that no one else in the field knows, and there's actually some really good stuff in that. So like extend doing, doing that activity a bunch of times would be like probably very high return.

• 49:52Dan Balsam?

  Although, coming back to a second to your question about like ways as parallels, other things in machine learning, that could be a good segue, Tom, to talk about minimum description length, because that's really interesting.

• 50:01Tom McGrath?

  Yes. So, Michael Pearce, recently joined Goodfire, I guess this is having quite a lot in this.

• 50:08Dan Balsam?

  You may notice that we have some of the world's best interpretability research.

• 50:14Tom McGrath?

  Yeah, Michael Pearce was one of the authors of this, pay for the minimum description length. And I think it's a really neat idea because it basically gets what you were saying earlier. So what you want is you want to have effective, compact descriptions of what's going on. Ultimately, what we're trying to do in interpretability is describe neural networks. I'm trying to describe it to you, or Claude is trying to describe it to me or whatever. And we're trying to find decompositions that are easy to like, easy to describe accurately, and then give good, good decompositions of them. So the minimum description length.

• 50:48Tom McGrath?

  Is this idea that you should be able to, if like, what makes a good description is a description you can transmit in relatively few bits. And

• 51:00Tom McGrath?

  the idea is that rather than using like sparsity as our sort of regularizer. Also, I kind of, this sort of, we normally quantify in the sort of essay paradigm. Progress has typically been quantified in the sort of Pareto frontier of sparsity versus loss reconstructed. But the idea of the minimum description length is that really, there's only one metric. And that is how many bits it takes you to describe what's going on.

• 51:29Tom McGrath?

  And there's various sort of technicalities about how you actually do this with an essay. I think basically that like, is it's a metric that I think we can probably mostly agree is a good idea. The problem is like how you actually implement it, how you optimize it and so on. But it has various other like nice properties. Like if I wanted to have this kind of tree structure in my features.

• 51:51Tom McGrath?

  But under a sort of L one type sparsity regularization, this wouldn't actually be preferred. But under the minimum description length, I can have like a feature in, you know, I've got a tree, I have the root and the branch. Well, the trunk of the branch, right? I can say that if I can say that the branch is like high probability, given the trunk, but otherwise low probability.

• 52:15Tom McGrath?

  Then this has a shorter description length than having to describe both of their probabilities independently. Um, it's a little, it's hard to give a good sort of overview answer of this.

• 52:27Nathan Labenz?

  I mean, maybe one would attack it is like, what is the artifact that we get when we get this with a sparse autoencoder? You know, I have the intuition that like, the thing may have 10 million nodes or whatever, but only 100 of them are going to light up on any given forward pass. And that could be sort of a hard cap or there's different batch, um, top K is sort of interesting wrinkle on this where you can define exactly, you know, how many nodes will be active or control that in a few different ways, but with I apply the minimum description length, what is the thing that I get out?

• 53:07Tom McGrath?

  Yeah, so the idea of minimum description length is more like it's a way, it's like a way by which we should compare all of our various approaches to decomposing models. It's like if for the same level of accuracy in terms of decomposition,

• 53:26Tom McGrath?

  they, one has a shorter description length than another, then we should prefer the longer the shorter description length. So for technical reasons, like something which is tree structured would generally have a shorter description length than something which is not tree structured, which is just a bag of features. As long as you know the sort of conditional probabilities along the tree, then you generally have a shorter description length. And that sort of matches with our intuitive understanding of it's easier to talk about things in terms of relationships between parts than it is by just like enumerating all of the parts separately. So I actually, but actually how you calculate this is actually quite difficult. The concept is simple in general, prefer the thing with the shorter description length.

• 54:09Tom McGrath?

  All the meat is in the question like, well, how do you calculate the description length? And so I think it could have lost them.

• 54:15Nathan Labenz?

  Is it something that you can apply directly as an optimization target as well?

• 54:20Tom McGrath?

  Not yet. I think we'd like to be able to, but not currently.

• 54:26Nathan Labenz?

  Gotcha.

• 54:27Tom McGrath?

  Like it is in some sense the ideal optimization target, if only we could optimize it.

• 54:32Nathan Labenz?

  I don't know if it would be batch top K versus a more naive sparse definition, or maybe some other that you would highlight, but how has the optimization target improved? Like that's basically we're conceptualizing the product, the problem better, right? And you can measure that by this sort of minimum description length. My sense of batch top K is a better way of conceptualizing the problem than slightly more naive ways that came like just before it.

• 55:05Tom McGrath?

  I think they would have the same description length, more or less. Like an essay with batch top K versus an essay with value versus an essay with jump value. Like all of the, all of these would have approximately the same description length, I believe, like all else equal. But being able to have something that's expressed in terms of a tree structure, or like there being various, various subspaces, these would have a different description length. Or for instance, like to go take a sort of detour into circuits for a minute. If I had to describe something with cross layer superposition by enumerating each of the separate layers that I cared about, and they're being like, one S A feature for this layer, one S A feature for this layer, one S A feature for this layer, plus also they're the same thing.

• 55:51Tom McGrath?

  That sort of has a longer description length than me saying, there is a feature which is spread across these three layers.

• 55:59Tom McGrath?

  So I think it's sort of on a higher level of description. It's like on a sort of higher level of abstraction than like, say, batch top K versus value.

• 56:07Nathan Labenz?

  So all in, where are we on this doing the machine learning better part? I mean, again, my most recent point of reference is the anthropic tracing model thoughts. And my sense was like, there's a lot that's not reconstructed, right? So if you were to push, you know, if we say like, what's the best case scenario? Can we reconstruct a model that like, kind of works? Or does it not really work at all still? And is it therefore only sort of limited to these sort of very, because in the anthropic work there, much of what they were doing was prompt specific analysis with custom error terms thrown in for those particular cases that they wanted to study. And that doesn't, certainly there's still a lot to be learned there.

• 56:52Nathan Labenz?

  But I guess how close are we to effective reconstruction of models as it stands today?

• 56:59Tom McGrath?

  So I think there are two aspects to that question. One is like, can we essentially throw compute at the problem to get there? And the other is like, how much will it take? If the answer is the first is yes, we can, then how much more will it take? And this is sort of, if you think about this in terms of scaling, this is sort of like asking, does it plateau or does it not plateau? Is there like an irreducible error? And this question has been like, examined a little, not as much as I would like it to be examined.

• 57:33Tom McGrath?

  Um. Did we talk last time about the dark matter of sparse auto encoders paper by Josh Engels? And collaborators? I don't think we did. It's a really interesting paper.

• 57:43Nathan Labenz?

  Yeah, I don't think so, but we definitely should data.

• 57:46Tom McGrath?

  Yes. So what they did is do this kind of scaling analysis and say like, if you were to keep scaling, say use up.

• 57:55Tom McGrath?

  Would you be on track? Like, would you be on track to recover all of the activation? And their experiments suggested that the answer is no. If you sort of trace the scaling curve out, that there's a substantial amount of like dark matter. But the scaling curve does not go up to like ninety nine point nine nine or a hundred percent or something like that. It actually plateaus like it actually is like on track and is not on track to recover everything.

• 58:24Tom McGrath?

  And no one really knows what dark matter is like. That's why it's dark matter. If it were easy to understand, we would already have understood it and probably baked it into a new kind of SA architecture. But this experiment hasn't been repeated as far as I know. And I so I feel like there's a lot to pull on there. Like, you know, if you did it with updated models like this, others, I didn't even mention on the doing the machine learning better, this recent spade paper, which is like a much more expressive version of sparse auto encoders. Um, we should put that in the links. Um. But you know, if you were to redo this analysis with kind of the latest techniques, like, would you still find that there's like dark matter or no? I suspect the answer is yes. I suspect you would still find dark matter. And like getting rid of that seems like a pretty big deal.

• 59:08Nathan Labenz?

  I mean, we still have in all the scaling laws, right? There's always like a term on of the transformer scaling laws too, right? So would it be the sort of prior guess, right, would be that there's some?

• 59:23Nathan Labenz?

  Dark matter minimum that you might approach and you'll have to chip away at it in the same way that we do a normal loss, which would be you have to 10x your your inputs for the next increment. And I guess that would be like.

• 59:38Tom McGrath?

  Ah, no. So the question is like, even if you were to scale up compute, like essentially as much as you possibly could, would you, that's like the difference between a curve on a log plot that is a straight line and the curve that eventually bends. And if your curve bends, right, then you have a problem because you're no longer on track to reach 100% reconstruction. And I think the claim of the dark matter paper is that it bends. And it bends because there are these things, things that we really cannot efficiently reconstruct. Now, I don't really know what they are. I think figuring out what they are would be very important.

• 1:00:14Tom McGrath?

  It could be that there are things like memorization or various like kinds of structure that essays is just not for like higher order structure. The essays are just not very good at reconstructing and learning. It could be that they're actually like noise. They're actually because the dark matter paper was on reconstruction accuracy. Well, actually, fraction of variance unexplained, but it could be that like this is noise that the dark matter is just noise that has no relevance to essay to like model outputs. And so if you were to repeat the analysis with loss recovered, for instance, it might look different. There's so much that we don't know, right? That's like actually very important.

• 1:00:55Nathan Labenz?

  But, okay, so just to make sure I'm not confused, and maybe I am, the going back to just general transformer scaling laws of whatever vintage we want to look at. There's always this constant term. I interpret that as the scaling laws do not suggest that we're ever going to have zero loss. But on our way to this sort of theoretical minimum, we have this like, you know, 10x for the next increment of grass. And does that theoretical minimum imply the same kind of bend? Is that the same sort of theoretical minimum that you're describing on the ability side?

• 1:01:31Tom McGrath?

  I presumably they both imply that at some point it must bend unless it like hits, it's like the minimum. So just bang, and then is horizontal. There's an interesting difference, though, in between the scaling laws, which is that the, there are, there are two sources of.

• 1:01:51Tom McGrath?

  Like front of irreducible error, I suppose, in a language model scaling law. One is the uncertainty of the world, right? If I literally have two, two, two strings that have up to token T, they're the same. And then a token T plus one, they had different strings. Right? And these are both in my pre-training corpus. Well, now I have, now I have a flaw. I can never get down to, I'm training on this corpus. I can never get down to zero loss. Because

• 1:02:24Tom McGrath?

  I like, it's fifty fifty. Looking, looking at this data, I have to pick like, given the prefix, it is no longer like deterministically predictable what the outcome is. Even if I had a perfect model of the data. So that's one thing. And the other is architecture dependence, where like, say, there are things that the S, that the transformer simply cannot capture. And these are like somehow detectable from the scaling law, like we expect that it will last and tote out because of architectural limitations.

• 1:02:54Tom McGrath?

  The difference, I suppose, is that we believe we should, like, it's a, it sort of seems maybe capabilities, people will disagree, but it seems okay for a, for a language model to not achieve perfect reconstruction. But I would be, I much prefer that my essay or my interpreter model like achieved perfect reconstruction. Like, that seems like pretty important to get for that to be, at least it, it to be like very high reconstruction.

• 1:03:26Dan Balsam?

  Yeah, I think there's also the question of what abstractions are you like working with here? Like, again, what are the like limitations of the technique even the limit? If we had wide enough SAEs, like what would the features actually be? Data set examples, right? Like they'd be specific data set examples for every activation pattern it's seen, because that's just what it's going to learn to limit. So you're trading off again against like generality and sort of memorization specificity too, in the context of an SAE.

• 1:03:56Tom McGrath?

  I think this is a really interesting intellectual exercise. Like, I think it's very useful. And if you think about like higher order structure, this is also somewhere you can just soak up features. Like, say that I have, say that my like X hypothesis, right? My, my true data lies on a ring around the origin.

• 1:04:15Tom McGrath?

  And really, there is only one like feature, which is the angle. How far around the ring have I got? But if I want to try and reconstruct this data with SAEs, then like I can put infinitely many features into it. They'll just go at different points on the circle. So I can sort of soak up an infinitely large amount of compute. Whereas, in fact, like, oh, I really needed one parameter. So this is another example of there being sort of fundamental scientific insights, or like fundamental like knowledge about how models represent their activations.

• 1:04:48Tom McGrath?

  That we need to kind of then convert into industrial processes.

• 1:04:54Dan Balsam?

  And yeah, to add some like concrete examples, ring one. There's been work that shows like transformers can encode days of the week, for instance, on a ring, which makes sense because like you can think of days of the week as like a mod seven operation, right? You add till you get to seven, then you loop back around to one. But if you're training SAE, like what are you going to recover? You're going to recover Monday, Tuesday, Wednesday, Thursday, Friday, Saturday, Sunday. And in the case of days of the week, like that particular abstraction, like you can think about it both ways and it sort of makes sense. But in certain types of abstractions, it's very reductive to look at in that way. Like a good example would be, we know transformers can do addition. The way they do addition, like modular addition, is with a set of trigonometric operations.

• 1:05:39Dan Balsam?

  If you look at Anthropic's work with the CLT and what they pulled out of Claude in terms of doing addition, it looks like a bunch of overlapping heuristics. You can look at that and you can say, oh, okay, this transformer is actually doing addition in this very intuitive kind of dumb way. And maybe it is, or maybe this is an artifact of the measurement apparatus. And actually, it's operating in some function over some continuous geometry. But this, the nature of the measurement apparatus is that we'll never be able to see that continuous geometry. So I think these are like really important, interesting questions. It doesn't mean that it's not. Helpful, or you're not gaining information about the model,

• 1:06:20Dan Balsam?

  or it's not telling you something important when you're doing this like reductive form of measurement. But you do have to keep in mind that even if you had a perfect SAE, what type of information would you struggle to recover from it?

• 1:06:36Nathan Labenz?

  Yeah, you, okay. Related somewhat tangential perhaps, but the the mention of modular arithmetic, of course, has me back to the Grokking paper. And I always remember this one plot where it's like the test loss or the test accuracy shoots up really quick as the model memorizes all the training data that it sees. And then it's orders of magnitude later that the actual Grokking effect happens. And that happens over the last order of magnitude, which it's always important to keep in mind is the actual 90% of the, you know, the data or the 90% of the training time, right? That it's that they're weird that way, going from 10 to the five to 10 to the six, I think, is what it was

• 1:07:19Nathan Labenz?

  after it had memorized by 10 to the two steps. So

• 1:07:25Nathan Labenz?

  a mental model that I always have of kind of any model that I'm working with is that some things have probably been grocked. Some things are mid Grokking and some things are not at all grocked at the time that I'm actually using the thing. And so this sort of in a way seems a little bit like an echo of the, not an echo, that's too strong of a connection, but in some way analogous to the sort of fundamental world uncertainty. When you are trying to interpret a model, you're sort of like making an assumption that things are grocked, at least to some degree, like the notion of these sort of features and circuits and all this stuff is sort of a, there's a level of Grokking there that isn't just like

• 1:08:08Nathan Labenz?

  random bizarre memorization that just sort of happens to be the first thing that worked on the.

• 1:08:16Nathan Labenz?

  Training set. And so I imagine that's like a fundamental challenge of all of this, right? Is that like, some of the things that you might want to recover, like, may only be partially learned in the first place, right?

• 1:08:28Tom McGrath?

  I think that what we want really is something is a sort of interpreter model that's like algorithmically neutral. If it's crocked, then I want to see that it's crocked. If it's a kind of half-baked heuristic, I want to see the half-baked heuristic. And if it's a memorized thing, I want to see the memorized thing. Now, like, actually getting that is, again, difficult. But I think that, like, yeah, you're absolutely right to call out the, this sort of, it's easy to assume that everything is going to be clean. But a lot of the time, there are going to be sort of messy heuristics that maybe even make up the bulk of the computation most of the time.

• 1:09:05Nathan Labenz?

  Okay, so we're moving into parts two and three around kind of the gap between what a feature is in some grounded sense and what we sort of label or understand it as. And then obviously, moving into circuits as well. Maybe the best way to talk about this is like, what are the downstream things that we're actually using interpretability for today? And then we sort of imagine closing this gap of the grounded truth and the labeling seems to be like, a sort of increasing automation of a lot of those different techniques to kind of cross validate against each other and make sure things are actually checking out at scale. But right now, we're not

• 1:09:46Nathan Labenz?

  largely like scaling all that out. We're largely kind of exploring in a more naturalist sort of way. And we've got all these different tasks that we'd like to be better at. Some of you can sort of tell us the story of recent progress in terms of what actual utility we're getting from our interpretability helpers.

• 1:10:07Dan Balsam?

  Yeah, so I'd love to talk about all the applied work we're doing. Before diving into that, I want to make one more point on features, if that's okay. Please. Which is that even like within features, like take as a given the SA paradigm or recovery features, we can explain some percentage of them with the way that we label today and some percentage of them that we can't. One thing that's come up a lot in our work with customers is trying to develop a taxonomy of different types of features that you might find. So the way that we label features right now predominantly comes from ground truth external data. So it's connecting some patterns at the model, some direction and activation space to some pattern in the inputs of the model.

• 1:10:52Dan Balsam?

  And these can get pretty abstract, like our work with R1 and with reasoning models, you manage to recover features that are pretty neat, that are qualitatively different than things that we saw in language model features and that seem to represent important units of the process of the reasoning trace for the model. But then there's also many features which could decompose into in similar ways, like could be explained, they're not dark matter features. Would they refer to algorithmic processes that are happening inside of the model itself? That's not easily visible on the basis of looking at input data or output predictions. So a clear example of this would be in context learning.

• 1:11:35Dan Balsam?

  Something happens and in context learning, people have looked at it and studied it. There's an algorithm that is implemented by the model that can be difficult to observe from the outside, but has like really important implications for like the downstream prediction. So we can imagine when we look at biology models, for instance, genomics models, which is something I'll elaborate more when we get into the applied work that we're doing. Some percentage of the features correlate very strongly with known biological structures. And then many features don't. And for those features, the question is, well, are those biological structures that we don't know yet? Or are those structures of the computation of the model itself? How can we tell the difference between these things?

• 1:12:20Dan Balsam?

  And then in either case, like that's extremely useful information because information about how you model these biological systems is still in and of itself, like really important to the question of scientific discovery. But yeah, so I think it's important to think of like, there's

• 1:12:36Dan Balsam?

  When building up like your taxonomy of features, right? There's the percentage of features that like we can explain by observing the inputs, right? And we can explain those with like high levels of confidence. And our confidence goes down over time as they're like moving into the tail. And then there's the features that like represent something more abstract than just something about the inputs. Something about the model itself that like we need to understand. Or maybe something in the scientific domains about the inputs that we don't understand yet. And then you keep going on that curve and that's a spectrum. And eventually you get to things that with current techniques, we struggle to explain. And that's why we need to invent new techniques, pushing even just the interpretation of the features that we can recover. And then there's the dark matter.

• 1:13:17Dan Balsam?

  So there's a bunch of different dimensions of this problem, but I think it's important to like for the viewers also like think about this taxonomy of features. Like what might a feature be doing? And it might be doing something that's actually entirely invisible from both the input and the output.

• 1:13:30Nathan Labenz?

  Yes, if you were to do few shot learning, I mean, one thing that sort of strikes me as a way to kind of bridge the gap there and again, feel free to de-confuse me would be to expand the window. I mean, when I, and there's lots to recommend about the anthropic work, of course, but the interface is characteristically really nice for being able to kind of explore what they're doing in an interactive way, kind of just probe into it yourself.

• 1:13:59Nathan Labenz?

  I don't know when you shoot, when you like click on a feature and it shows, okay, here's the examples that maximized that feature from the data set. You're typically looking at a pretty short snippet in the, you know, because you're like three tiers down a UI from like a webpage to a embed to a little pop up within the embed. I don't actually know like how big those snippets typically are. But if you imagine a sort of few, a few shot learning feature being like, okay, we have recognized that there is like a recurring pattern here in some very abstract way and like our job is to kind of continue it. R being the model, I guess, in this scenario, then you wouldn't see that if you had a, you know, 20 token window, let's say, but if you zoomed out to a 20,000 token window, you might see the whole thing.

• 1:14:44Nathan Labenz?

  And I imagine some similar things could be happening in biology where it's like you're way downstream of an activator sequence that turns this thing on in the first place or what have you. So is there a sharp distinction between these sort of? Grounded features and the sort of computation features? Or is it just a question of our ability to zoom out far enough to see the pattern accurately?

• 1:15:08Dan Balsam?

  Yeah, this is a really great framing on the problem. I think in context, learning is a good example here because yeah, if you zoomed out and you included more context. So generally, we do auto inter, which is this process of labeling with different amounts of context and different types of context, depending on the problem and in the domain in which we're operating. In context, learning is an interesting example, right? Because if you zoom in too much, you wouldn't be able to see it. If you zoom out, you can probably tell it from the prompts. But it's also a algorithm that we know that we expect to find in the model to begin with. And that we would expect that a frontier model can probably identify sort of like a meta pattern if it sees a bunch of examples as well. Because this is like a type of thing that we've identified that models can do that

• 1:15:51Dan Balsam?

  we've already described pretty well. In principle, if you like zoom out to the entire genome and you had some features that were active in a bunch of locations across the entire genome, and you had a frontier model go and look at those, its ability to label those effectively for you in an automated way is bounded by whether or not an explanation was in its training data to begin with. And in many cases in the scientific domains, there's not an explanation. We're working with sets of abstractions that are pushing the frontiers of human knowledge in some way. So we need some way of labeling and thinking about how these features can pose that can push past that. You need to be able to look at a bunch of different contexts.

• 1:16:36Dan Balsam?

  Let's say, look at 100 different genomes and see these patterns and reason about what they might be doing together. You have to be an expert in the human genome, perhaps even beyond the level of the greatest human experts today, to reason about that just on the basis of inputs. So one thing that we're interested in doing is can we break this down into a set of easier problems, right? If we have 10% of these features that we can explain just on patterns on the inputs in an automated way, can we work with domain experts, scientists, to keep pushing up that frontier of possibility and further and explaining more and more of what the model is doing?

• 1:17:15Nathan Labenz?

  So how's that going? Because that sounds like maybe the one of the more important questions we'll touch on today, right? I mean, one of my great hopes has been

• 1:17:24Dan Balsam?

  For, and I know you share this and are actually moving the frontier on it. But the idea that we can use these unsupervised approaches on natural data and then look in and see what the model is learning and then learn it ourselves and actually have confidence in what we're talking about and make new discoveries. This seems like a really great driver of new discoveries. But you're complicating my sort of naive optimism a little bit with this idea that we don't necessarily know if the features are like, you know, correspond to the real world or just to the internal model. Even that's a level of confusion that I'm certainly, you know, excited to hear how you're going to resolve.

• 1:18:05Tom McGrath?

  Oh, I think that they still correspond to the real world in the sense that like algorithmic features can correspond to the real world and follow it. Yeah, right, like, you know, Newton's laws do not exist in the world. They're aware of describing the world. We can talk about the idea of velocity as a kind of, as a type of feature, but it's not the case. But like, you just see things have a velocity. They don't have like the concept of velocity. So these are sort of, the structure is useful still.

• 1:18:41Dan Balsam?

  Yeah, and so, I mean, this is one of the big ambitions and goals of Goodfire is to crack this question. So we're working directly with customers across different scientific domains with their scientific models with this goal. For instance, viewers might have seen our research with the Ark Institute, recovering features that correlate very strongly with known concepts in the genome. And where Ark Institute and ourselves are pushing this collaboration now is moving towards unsupervised techniques that can help get us new information in the genome. And we're actively working on this research and hopefully have some exciting things to share in the not that distant future.

• 1:19:25Dan Balsam?

  Yeah, I think at a high level, like, if the model is correctly, is if it's truly generalizing in its ability, at least sometimes, right? Like, it's on this continuum for like a Grok some things, it's memorized others. What is it Groked? What is it memorized?

• 1:19:44Dan Balsam?

  To be able to generalize must be learning really meaningful things about like the underlying systems. And so recovering those, even in an algorithmic form, tells us really interesting things about the system that's being modeled.

• 1:19:58Dan Balsam?

  Current bioinformatic tools. So like a reasonable question may have is, well, how do we think about the genome today? Annotations in the genome are done using bioinformatic algorithms that have been developed by humans that have strong priors, in some cases work extremely well, and in some cases actually don't work that well at all, but work well enough that they're a good starting point. The genomes are massive and like the information complexity of genomes is extremely large. So we don't have good first principle techniques to answer a lot of bioinformatic questions that we might want to answer. But it seems like in the case of some of these models, they're able to learn things that make these tasks

• 1:20:43Dan Balsam?

  easier. So it's going to take time. Like this is a important scientific project that's pushing like the frontier of both transformer modeling in genomics and interpretability. But at the end of the day, just going back to the paradigmatic beliefs that Tom mentioned earlier, like the model is doing something and it's doing something meaningful. And we have windows in on the model now. They might not be the right windows to tell us everything, but we have strong reason to believe that these windows are good enough to start telling us important things. And so that's really why we're pushing on this very hard.

• 1:21:21Nathan Labenz?

  When you say it's going to take time, do you take the over on Dario's two years to a country full of geniuses in a data center? Or is that the kind of time we're talking about?

• 1:21:31Dan Balsam?

  I'm gonna give a hot take real quick, which is that

• 1:21:36Dan Balsam?

  Using mechanistic interpretability for scientific discovery, I think there's like a decent chance that even if you were in a world where you had a bunch of geniuses in a data center, this would be their preferred way of doing science. You know, there's so many barriers to running experiments in the physical world. Normative barriers, but also just physical barriers, paralizable barriers. The rate of scientific progress could be rapidly accelerated by moving as much scientific experimentation as is reasonably possible to simulation and doing it on chips. That looks to me like a lot like what people are doing when they're training autoregressive genomic models or diffusion-based material models.

• 1:22:21Dan Balsam?

  They're running simulations of the physical world on chips. And interpretability then is your scientific toolkit that you go in to actually understand what that simulation is doing and extract principles from that simulation that can then inform further experimentation and then ultimately real-world scientific progress. You know, an interesting example here is like how pharmaceuticals are created. They're often developed with something in mind and then they're tried. They have some range of side effects and some range of things. Most of the time, they don't work, the vast majority of the time, they don't work the way that people think that they're going to initially. And then these companies just keep around these drugs because like the process of manufacturing

• 1:23:06Dan Balsam?

  a new drug is expensive and then they get the patents and then they just try it for a bunch of other things and then eventually they maybe find something that works and that becomes like the treatment for it. You know, there's many examples of this and that is an extremely inefficient way to do science and that's going to remain an extremely inefficient way to do science even in a post-HDI world. So to me, it just seems like pretty likely that you're going to want to run simulations and if you're running simulations, like why wouldn't you?

• 1:23:34Nathan Labenz?

  Want to train giant models and if you're training giant models, why wouldn't you want mechanistic interpretability to help you make sense of your simulation?

• 1:23:42Tom McGrath?

  So I guess we have, you know, we kind of do it all right either way, right? Either timelines are longer, it wouldn't be an AI podcast without timelines. So I was wondering when we're going to get to it. Either timelines are longer, or maybe TAF is useful, or one of the geniuses in the data center doing, Matthew Taff.

• 1:23:59Nathan Labenz?

  Yeah, I think, I mean, that does make a lot of sense to me. As a convergent evolution, basically, right? I mean, either, either way, it sucks to have to do wet work. So as much as possible, you want to move it all into silicon, learn as much as you can that way. Hopefully it's accurate. It won't always be right, but one way I phrase the question in the past is when is it worth it to do the wet work? And as much as possible, you want to elevate and validate your hypotheses before you actually take it into the wet lab to muck around with and whether it's a human or AI genius steering that simulation process, it, for all the same reasons, it seems like it's where we end up.

• 1:24:38Dan Balsam?

  Yeah, yeah, that's exactly how we think about it. Like, the core thing that just needs to be true here is that like mechanistic interpretability as it matures as a science significantly pushes the frontier of the types of experiments that you end the quality of the results of the experiments that you're able to run on hardware before bringing them to the wet lab. Those strikes me as like overwhelmingly likely to be true. And that, and again, like for us, we're doing this because we believe that there is like a really meaningful possibility to impact people's quality of lives through bringing interpretability directly to these models that could have like real downstream scientific impact.

• 1:25:23Dan Balsam?

  Another good example of this is biomarkers of disease. So, you know, in clinical contexts, there's a, there's sort of a Two reasons that you would be interested in inter. There's the discovery reason that we've already talked about, but then there's the reason that being able to have explainability for diagnostics is really important. You know, AI can often perform extremely well in these tasks and in closed settings. But if you have an AI system that misdiagnoses someone, you can't go to their family and tell them, sorry, it went wrong and I have no idea why. And there's nothing I can do about it, right? Like this like core debuggability is an essential feature in clinical contexts as well.

• 1:26:06Dan Balsam?

  But you also want to be able to say, oh, the AI gave a surprising recommendation in this case. It gave a surprising diagnosis. Was that diagnosis a result of the AI being wrong? Or was it a result of some important input pattern that we hadn't seen before?

• 1:26:28Nathan Labenz?

  One quick aside on multimodality. Does this line of work, because you're doing this, you know, across reasoning models, for example, and across different models from science genomic. And there's obviously lots more to come in terms of proteomic and higher orders of abstraction there. But I wonder, like, one kind of mental model of superintelligence that I've been playing around with lately is basically that if you take a reasoning model at roughly the current level and you give it the same depth of integration into 20 modalities of interest that we already now have with image from like GPT-4O and Gemini Flash, where, you

• 1:27:12Nathan Labenz?

  know, they can clearly like manipulate the image in a way that shows a deep integration of like your instruction and the visual space that it's operating in. It's no longer through this bottleneck point of like prompting the, you know, having the language model prompt to the text image model. It's all joined in latent space and you can, you just obviously see the qualitatively different results. My sort of baseline superintelligence case has been do that again for 20 more modalities, many of them in sort of the natural world modeling domains. And there you'll have a superintelligence because you'll be able to kind of reason around and manipulate things, but the things you'll be manipulating will be like much closer to the fundamental stuff of reality.

• 1:27:55Nathan Labenz?

  And in many of those cases, like, these are just things that people can't do, right? Like, there's, nobody has an intuition for how a protein is going to fold, for example, or at least, you know, nowhere at the level that like an alpha fold does.

• 1:28:07Dan Balsam?

  So we talked about a little bit about this last time. I want to start by saying that, like, I have a wide distribution of timelines. Genius is in a data center in two years is not outside of my Overton window at all. But on this particular point, I'm just like cramming a bunch of modalities in. I think there's a way in which we're still constrained in AI progress by human data, even in this RL regime, that's like worth understanding. Like, why is it possible for these models to reason over images and text really well? That's because like we can construct tasks for which we have good reinforcement learning signals for which this is a capability that they need to acquire to complete that task. When you start bringing in like the scientific domains as well, we're often working with

• 1:28:51Dan Balsam?

  like extremely sparse signals where it's like very hard to reason about, you know, what task could I easily train an AI to do such that a deep intuitive understanding of the human genome was necessary to perform that task? But then also, we have like enough examples and a clear enough sense of that task itself such that we could create a strong signal for reinforcement learning to begin with. And then what percentage of the overall task that the model would be performing, do these like integrate really well? I think it's not a coincidence that. Combining image and text in problem solving is something that humans do all the time. And this was like one of the easier things, you know, from an RL perspective to

• 1:29:36Dan Balsam?

  train AI to do once you reach a certain level of intelligence. It's not just because those are easier modalities to combine in some abstract sense, though maybe they are. It's that like, we as humans are very set up as to how to think about and construct some optimization target at the intersection of those two things. So I think this gets at the question of, do you hit a wall when you get to human level intelligence in some way, shape or form? Or like, why would you or why would you not hit a wall when you get to human level intelligence? And I think like, you know, Tom and I might have slightly different beliefs here, but I think like my belief is like, we just kind of don't know. It's easy to like, draw the line out from where we're seeing and say these patterns

• 1:30:16Dan Balsam?

  extend to everything. But I think it could also just be true that these patterns extend to the things that they've extended to so far. I think that we are very likely to be able to get, just to come back to a second to the question of agents, like, why has it like been easier to build AIs that can solve elite coding problems better than any human than to build AIs that can like order DoorDash for me successfully? Right? Like, one of these things seems like it should require much less intelligence in some sense to do, yet it's been like much harder to get models to do. And I think it's just because it's harder to get good training signal in the agentic use cases, right? This is not something where we've captured a lot of data where you can write like algorithmic verifiers

• 1:31:00Dan Balsam?

  very easily. And so as a result, it's been harder to do. When you're saying to the scientific domains, it's even harder, right? If we had a doing science good verifier, that would be great. And I'm not saying that one doesn't exist. It's not possible to come up with one or that like, maybe if we get general level intelligences, like they can work on this problem and make progress on it. But it's like a level of abstraction for which we don't have any evidence and for which like we have reasons to believe that actually it's setting up the conditions in which an AI can learn that task in the way that we currently train AIs might be pretty hard.

• 1:31:39Tom McGrath?

  I suppose another way to put this is that the question here is like, are AIs currently experiencing catch-up growth in the sense that the less economically developed country might benefit from catching up to a more economically developed country? Or they just on this like growth trajectory are to disentangulate.

• 1:32:01Nathan Labenz?

  Yeah, I mean, I think that is where the other modalities and especially the models from different fields of science seem to be pretty strong evidence to me that they can do pretty critical tasks at an obviously superhuman level, right? Like the ability to, and we've gone well beyond just folding, but like interactions and multimers and all different kinds of molecules and metal centers now with some models and whatever. I mean, it's really like pretty far along in terms of, I also did an episode on it, the guys from Orbital Materials on figuring out the mechanism of the potassium ion channel, you know, and it's like, geez, this is, this has gone pretty far already.

• 1:32:45Nathan Labenz?

  What hasn't happened there is the integration of this sort of chain of thought kind of thing with the more, what I call it, just intuitive physics for like whatever different domain the model happens to be trained in. It like clearly has a better ability to take a random set of proteins and like guess how they'll interact than I do. What it doesn't have is the sort of outer loop to be like, I should try this, maybe I should try this, like, what if I swap this one out for this one? And so people are like sitting there, you know, writing scripts against these models to do that outer loop. Or just maybe even doing it, you know, based on their own human intuition, one by one in some cases. But

• 1:33:28Nathan Labenz?

  I have a hard time imagining a world where that integration doesn't happen. Like it seems like we can get enough synthetic training data and set up a situation where it's, okay, here's what it looks like to just grind through a bunch of these things to get to the point where the reasoning and modality X integration kind of eventually just kind of drops into place, right? I that I imagine that happening. I can't imagine it not happening.

• 1:33:57Tom McGrath?

  The question is the depth of integration. I suppose, like, you might think that things are very deeply integrated in the sense of being in the same neural network. But even there, you have no guarantee that you don't have two models in a trench coat. Like, this half of the weights does the quantum chemistry, and this part of the weights does the language. And there's not really any, like, cross-talks such that the parts that are doing the language have any access to the sort of process knowledge of how to do the, like, the chemistry. And the way that you would expect to get this cross-talk is if you're able to construct, like, paired data or data where, like, the language ability is kind of bearing on the chemistry and vice versa.

• 1:34:41Tom McGrath?

  Then we can do that relatively easily, and we can provide supervision of various sorts of, well, to be easy and, like, easily in image language.

• 1:34:49Tom McGrath?

  Because we understand these domains both quite intuitively, like, if we can give reward, as in, we can give approval and train a reward model and then do our LHF, or you can obtain paired data from the internet. But in the sort of, where does the paired data or natural supervision come from? Couple these two modalities for domains that we don't have the same kind of intuitive understanding, because we can't get the same kind of

• 1:35:14Tom McGrath?

  approval. Now, it might be that actually the answer is kind of more like you say, you, you put them, initially, they're sort of two, they're two separate heart, there's just like halves of the neural network, let's say. And then you do some sort of training task where you gradually, like, you elicit the language reasoning in order to drive the chemistry part of it. But that's sort of,

• 1:35:40Tom McGrath?

  that doesn't feel like the same sort of depth of integration as you get through pre-training, where you really do seem to just have this, like, big massive compute all more or less able to access the other. Well, actually, even that's not necessarily true, like, how much of a language model, what it's talking about, one domain, is able to actually elicit its, like, its information from another?

• 1:36:00Nathan Labenz?

  Maybe let's just do. Other applications to sort of move to circuits. And I mean, you could, you could frame them conceptually or potentially by like customer profile.

• 1:36:11Dan Balsam?

  Yeah.

• 1:36:12Nathan Labenz?

  The sort of monitoring, classification, steering way of thinking about it. And then there's like the retail and, you know, different kind of models that you're looking into. You could attempt both if you want to squeeze in it.

• 1:36:23Dan Balsam?

  Yeah. So I think there's three applications right now that we're very excited about. The list of potential applications is very long, but we're a startup. We need to prune the tree a little bit and focus on what we think the highest leverage bets for us to take are. So the scientific discovery, which I've already talked a lot about, so I won't spend too much time there. But again, just to quickly summarize, scientific discovery is you have these models. You're modeling physical systems. They're able to model them in ways that traditional methods can't. So they must learn something that is important that human beings currently don't understand. We want to explain that. And the hope and the vision there is that those explanations are in and of themselves scientifically useful

• 1:37:06Dan Balsam?

  in terms of pushing the frontier in the, in science. The second application is for guardrails purchase. So what interpretability techniques give you is a window into the model at inference time. If you think about the current way that a lot of enterprises set up their guardrail systems, oftentimes they're playing whack a mole with the prompt at first when they're setting up the prototype. The number of rules that you can specify, the contextual information in the prompt balloons. You end up in this very natural situation where your task performance is degrading because of all of this information that you have to add, keep telling the model, don't do this, don't do this, do this, don't do this.

• 1:37:51Dan Balsam?

  So people then move naturally to LM as a judge. But the scaling properties of LM as a judge are quite poor because now you have a separate frontier model call. It's great for the labs pockets, but it's not great for the consumer. If you have a thousand rules, maybe you can bunch them together in different checks. But at the end of the day, you run into the same problem of task degradation and just like poor scaling properties. You could. Training a small model or fine tune a small model to help guardrail your larger model. But many organizations both lack the data and machine learning expertise to do this effectively. So what we can do with our techniques is offer a cheap inference

• 1:38:31Dan Balsam?

  time solution for models where we can be watching the model's internal cognition and use that as a as an event when certain things happen, when the model is potentially going to output certain things, or is outputting certain things, or is reacting to certain things to trigger programmatic responses. So this could be things like if, for instance, the model looks like it might be thinking about PII, then let's flag that for someone for manual review in some way. Or it could be, for instance, there are certain topics that I never want my model to talk about. Of course, there are limitations to what

• 1:39:12Dan Balsam?

  can be done. There are jailbreaks or all types of things. But for real, practical use cases, there's a lot of opportunity here to offer cheap, fast, and effective checks in real world scenarios that just have much better scaling properties. Then the third category of application that we're really excited about is creative models. We're going to be launching a demo soon. In fact, by the time this airs, it might already be out demonstrating what you can do with image models when you start to understand their lateness and in unsupervised ways, we're able to recompose the elements of the image from this deep understanding of what's happening inside of an image model. And we think this

• 1:39:53Dan Balsam?

  just offers new types of user experiences that even as image editing tools and other forms of AI continue to get better and better, you're still very much locked into this form of these like kind of bespoke forms of interaction, whether that's by prompting or highlighting a region for in painting. And there's certain types of just like key interactions that we had in earlier classes of image design software, just like the ability to drag something or reorient something or change some subtle property.

• 1:40:23Dan Balsam?

  That get lost in that process. And so this image demo is like a cool example, but we think we could push this and the value pops up in more clear in video and music where the cost of editing them is very, very high. And wouldn't it be amazing to have generative music AI where you could say, actually, I want a little more saxophone in the saxophone solo. And you could just very strategically intervene on the output in a way to get specific new generations that adhered to exactly what you wanted in some context.

• 1:40:52Nathan Labenz?

  Maybe just my last thing before giving you the floor for the sort of closeout pitch on Goodfire is like, where are we? I think it was maybe, geez, a year and a half ago that we first talked about like, maybe just a year, time is compressed. The idea that with these tools, you can sort of allow many more people to get involved in the process of understanding models. And I've been fortunate enough to have a couple of early previews of different model steering and exploring interfaces that you guys have built. And I wonder how you would characterize where we are on that today. Are the interfaces far enough along that we are now effectively able to enlist the human intelligence?

• 1:41:36Nathan Labenz?

  Or are we still kind of working on that and still more in the sort of auto interpretability world because we haven't quite cracked the paradigm that allows people to like make intuitive use of all these new feature spaces that you're opening up for them?

• 1:41:50Dan Balsam?

  I think it depends on the problem you're trying to solve, which I know that's not a very satisfying answer. But in the case of images, and you'll see in our demo, like images, you can reason about images and image features very intuitively. And there's ways to visualize them that don't rely on any external assumptions about inputs. It's purely about visualizing what that feature does to the output. So it's fewer assumptions at least. And I think that our flash image demo will demonstrate some of that context. There's been a lot of work internally at Goodfire and also at other labs and especially in the open source community at different UXs and visualizations that you can have. And they all solve problems. They're all a way of looking at it. Right now, we're not in a place where we have a single interface that can tell you

• 1:42:35Dan Balsam?

  anything that you need to know about arbitrary model. We're working on some things in the biology domain that we hope will generalize. But yeah, it's like everything else, right? All these tools have limitations. It's about the problem that you're trying to solve and what you can learn in the process of trying to solve that particular problem.

• 1:42:51Nathan Labenz?

  Cool. Well, it's been a fantastic conversation. I really appreciate the time and I just want to do it more regularly going forward. Take us home, you know, maybe give a few highlights, things that you're most excited about, things you guys are looking for. Last time, if I do say so myself, I understand there were a couple inbound business opportunities. So you can put the, you know, the bat signal out there for whatever you want to come your way this time.

• 1:43:13Dan Balsam?

  Yeah. Um, so at a high level, like we are looking for people to join our team at Goodfire. If you want to work on what we all think is the most important unsolved problem in the world right now, you can join us to help reverse engineer what's happening inside of models. I think one of the really exciting things about Goodfire is like, we're pushing on this problem from a bunch of different angles. We're not saying, let's just push what we can do with these tools or just, let's just develop new tools. We're doing both at the same time in dialogue and in concert with each other, which is really important to the ultimate progress. We're looking for great engineers and great scientists who are very motivated by our mission here and

• 1:43:56Dan Balsam?

  want to open up the black box. And from a customer's point of view, like if you fit one of the profiles that we talked about, if you're someone who's training a scientific model, if you're an enterprise looking for more reliable usage of your LMS in production, or if you are someone training a creative model and you're just like looking for new ways that you can open up the experience of the creative model, please reach out to us. You can email me at [email protected]. We'd love to talk to you and understand whether or not there's a good opportunity for some collaboration there.

• 1:44:31Nathan Labenz?

  And I'll mention it at the top, but we should also probably mention you just raised a bunch of money, including taking Anthropic's first ever external investment. So what more beyond that headline would you impart to people?

• 1:44:43Dan Balsam?

  Yeah, so we raised $50 million, led by Memo Ventures, and $1 million of that from Anthropic as Anthropic's first ever corporate investment. Yeah, we're just extremely grateful to have investors who really believe in what we're building, the mission, the problem we're trying to solve here. We think it's really important to have a company not directly tied to a scaling lab that is directly trying to solve the problem of interpretability. And we intend to use all this money we've raised to help customers understand their models and to push the frontier of understanding in mechanistic interpretability. And as we were saying before, hopefully then, and across a bunch of scientific domains and unlock lots of new, awesome experiences that we can have with AI.

• 1:45:27Dan Balsam?

  Our core belief here is that like, interpretability is as big as AI itself. And so there needs to be a lab that's focusing on interpretability and nothing else, because the size of the opportunity here is really large. So grateful to have all of the support from investors like yourself, Nathan, and all the people who like really believe in the thing that we're building. And we're just motivated to get out there and start making things happen.

• 1:45:52Nathan Labenz?

  Yeah, well, it is definitely a good candidate for the most important problem facing the world today. So it's been a pleasure and definitely a fascinating journey to just try and keep up with all the progress you guys are making. Again, come back soon, but for now, Dan Balsam and Tom McGrath, CTO and Chief Scientist at Goodfire, thank you again for being part of the cognitive revolution.

• 1:46:15Dan Balsam?

  Thank you so much for having us. Yeah, thanks for having us.

• 1:46:17Nathan Labenz?

  It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please, don't hesitate to reach out via email at [email protected], or you can DM me on the social media platform of your choice.