Mapping the Mind of a Neural Net: Goodfire’s Eric Ho on the Future of Interpretability

Eric Ho is building Goodfire to solve one of AI’s most critical challenges: understanding what’s actually happening inside neural networks. His team is developing techniques to understand, audit and edit neural networks at the feature level. Eric discusses breakthrough results in resolving superposition through sparse autoencoders, successful model editing demonstrations and real-world applications in genomics with Arc Institute's DNA foundation models. He argues that interpretability will be critical as AI systems become more powerful and take on mission-critical roles in society. Hosted by Sonya Huang and Roelof Botha, Sequoia Capital Mentioned in this episode: Mech interp: Mechanistic interpretability, list of important papers here Phineas Gage: 19th century railway engineer who lost most of his brain’s left frontal lobe in an accident. Became a famous case study in neuroscience. Human Genome Project: Effort from 1990-2003 to generate the first sequence of the human genome which accelerated the study of human biology Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs Zoom In: An Introduction to Circuits: First important mechanistic interpretability paper from OpenAI in 2020 Superposition: Concept from physics applied to interpretability that allows neural networks to simulate larger networks (e.g. more concepts than neurons) Apollo Research: AI safety company that designs AI model evaluations and conducts interpretability research Towards Monosemanticity: Decomposing Language Models With Dictionary Learning. 2023 Anthropic paper that uses a sparse autoencoder to extract interpretable features; followed by Scaling Monosemanticity Under the Hood of a Reasoning Model: 2025 Goodfire paper that interprets DeepSeek’s reasoning model R1 Auto-interpretability: The ability to use LLMs to automatically write explanations for the behavior of neurons in LLMs Interpreting Evo 2: Arc Institute's Next-Generation Genomic Foundation Model. (see episode with Arc co-founder Patrick Hsu) Paint with Ember: Canvas interface from Goodfire that lets you steer an LLM’s visual output in real time (paper here) Model diffing: Interpreting how a model differs from checkpoint to checkpoint during finetuning Feature steering: The ability to change the style of LLM output by up or down weighting features (e.g. talking like a pirate vs factual information about the Andromeda Galaxy) Weight based interpretability: Method for directly decomposing neural network parameters into mechanistic components, instead of using features The Urgency of Interpretability: Essay by Anthropic founder Dario Amodei On the Biology of a Large Language Model: Goodfire collaboration with Anthropic

• 0:00Speaker 00

  So Goodfire is an AI interpretability research company, really trying to answer the question of what's actually going on inside the mind of a neural net. So kind of the ultimate goal and the ultimate reason why we started everything was like, we just see neural networks kind of going into more and more mission-critical contexts. And I think it's going to be enormously transformative for society. But in order to do so, and you want to build it safely, powerfully, reliably. And I think it's going to be critical to be able to understand, edit, and debug AI models in order to do that. And so that's what we're kind of enabling for the very first time. It's like unlocking the black box of a neural network such that you can intentionally

• 0:44Speaker 00

  design it, rather than just kind of like grow it from data.

• 1:05Speaker 01

  What if we could crack open the black box of AI and see exactly how it thinks? Today, we're joined by Eric Ho, the founder of Goodfire, who's building tools to peer inside neural nets and understand their minds. Eric reveals how his team has successfully disentangled the mysterious phenomenon of superposition, where single neurons encode multiple concepts, and can now steer AI behavior with increasingly surgical precision. We explore whether interpretability could help us discover new biological insights, edit out harmful behaviors from large language models, and even understand our own brains better. Eric boldly predicts that we'll fully decode neural nets by 2028, transforming AI from black boxes into more intentional design. Enjoy the show.

• 1:48Speaker 01

  Eric, thank you so much for joining us today.

• 1:50Speaker 00

  Of course, yeah, happy to be here. Thanks for having me.

• 1:52Speaker 01

  First question, can we ever trust generative AI if these foundation models are very much black boxes?

• 1:59Speaker 00

  Can we ever trust them if they're black boxes? So I guess like, you know, maybe thinking about like, um, what would happen if we were to just kind of deploy AI models as black boxes like in perpetuity? So the black box way to do this would be like, um, and I'm kind of assuming that we're playing this forward a few years and we want AI in charge of like really mission critical applications, like, um, maybe being in charge of our power grid or making big investment decisions, maybe even for a seed investment at Sequoia or like a really large, you know, like, um, million dollar like investment decision. And I think like the black box way to to make sure that the AI is performing appropriately is like,

• 2:43Speaker 00

  you take a look at evals and you run like a bunch of like evaluations to make sure that it's behaving behaving appropriately in test sets. And then you'd look at its track record and see if it's like, you know, reliable enough to perform across a wide variety of things. And, uh, I think like, the question then is like, why not take all of this additional signal that you get from looking inside a neural network? And trying to play forward like how it's going to behave in a much wider, broader set of situations, like why not like look inside and actually like get a bunch more reliability, certainty about how it's thinking, how it's approaching the problem.

• 3:23Speaker 00

  And I think like you're just leaving a bunch on the table if you're not like looking for all the signal that you can get. So the way that I think about this is like, I don't know, when you're manufacturing a new drug, it's like you can do the black box way of just like seeing how humans respond to the drug in like a clinical trial. Whereas like you could also just like kind of look inside and like look at biochemically like how the drug is. Processed or like drug interactions at the molecular cellular cellular level. And yeah, I just feel like there's so much to be learned when you actually like look inside and deeply understand something.

• 4:01Speaker 01

  How possible do you think it is to look inside and deeply understand a large language model? Do you think it's a, you know, on the scale from hopeless, we can't ever understand it. It's just a black box, too many, too many neurons. So we can actually map out the mind of a neural net. I'm curious where you think the field will be.

• 4:17Speaker 00

  Well, I'm, I'm very biased, but I think it's very, very possible. So I mean, a lot of the people in mech and terp like come from backgrounds in like computational neuroscience or like cognitive science. And those people, when you're like actually looking inside the brain, you spend so much time like trying to understand like what a single neuron does or just getting any signal whatsoever.

• 4:41Speaker 00

  And in the, the field of mech and terp, you have perfect access to the neurons, the parameters, the weights, the attention patterns of a, a neural network. So you're, you're coming in with a huge advantage for like, at least like you get all of the data that you, you need. So then the real question is like, how can we make progress? How can we try to understand and seek to understand all of it?

• 5:05Speaker 00

  And I think we just got to try. I think it's deeply necessary and critical for the future. And we have like a norm established. We can explain some percentage of the network by reconstructing it and extracting, you know, its concepts and the features that it uses in order to generate its response. And once you have at least a baseline under like rudimentary understanding kind of where we're at right now, you can hill climb on that metric and seek to understand like more and more of the network.

• 5:32Speaker 02

  Do you think it's going to be necessary for us to understand neural nets to really harness them long-term? Cause I think many other technologies we've invented along the way, humans didn't really understand underlying physics or chemistry, but still we were able to make good of medicines or, you know,

• 5:47Speaker 00

  Totally.

• 5:48Speaker 02

  sort of basic propulsion techniques without understanding, you know, all the physics?

• 5:53Speaker 00

  Yeah, I think it's going to be critical for the future, just given how transformative I think AI is going to, to be like, I think AI is going to be. Everywhere, running mission critical parts of our society and like we can get really, really far just by treating AI as a black box. But I don't think we can truly be able to intentionally design AI as like the new generation of software without like white box techniques. So maybe one example I think about is, you know, in the early 17th century, you invented like the steam engine and were able to just like

• 6:33Speaker 00

  increase the size of the boiler and increase the amount of pressure going in and it scaled reasonably well. But steam engines also like blew up like, well, we didn't understand thermodynamics at that time. So we didn't actually know like, at what point did like the ideal size of like the boiler or like the ideal pressure, the ideal way to construct a steam engine. And so like, after we invented thermodynamics, like things started becoming a lot safer, a lot more reliable. And like huge innovations happened afterwards. But already like, you know, the steam engine kicked off like the industrial revolution. So even just by like treating it as a black box, you get a really long way.

• 7:11Speaker 02

  Do you think it's any chance that if we understand neural networks in a computer science context, it might actually help us accelerate our understanding of neuroscience for the human brain?

• 7:21Speaker 00

  I think so, but that's like, that's a big claim, I think. So it's like, we were just having an interesting conversation last, last night. We had like a dinner together about like, do we, do we think in language? Do you think in concepts or something else entirely? Like, I'm a person that doesn't really think in language. I think much more like conceptually, maybe in like the latent space of, of models. Whereas like our head of product Myra said that she basically is totally faithful to her own chain of thought, like speaks in language and she basically just thinks like sequentially with like a really, really strong internal monologue. So like, Sure, it's just like, maybe, maybe some of these insights that we get like will

• 8:04Speaker 00

  all translate to humans and our own psychology. And I think that's the hope. It's like, yeah, the more that we can understand about AI, like, hopefully, the more that we can understand about ourselves.

• 8:15Speaker 02

  There's an interesting analogy, by the way, that in neuroscience, often, things that have gone wrong help illuminate and create insights into the human brain.

• 8:25Speaker 00

  Yeah.

• 8:25Speaker 02

  You know, people who suffer from specific conditions, or people who've suffered particular brain injury types have actually perversely enabled us to better understand the brain accidentally.

• 8:34Speaker 00

  Yeah.

• 8:34Speaker 02

  I wonder if something similar might happen with neural networks as well.

• 8:37Speaker 00

  Yeah, I hope so. What's that, like, popular story about the guy who just like got an iron rod through his brain and it made him like a totally different person?

• 8:47Speaker 02

  Yeah.

• 8:47Speaker 00

  Yeah.

• 8:48Speaker 02

  Anyway.

• 8:48Speaker 00

  Yeah, there's also this, you know, to add to that, there's this concept of universality where like, among totally different neural networks, like similar kind of like circuits or like thought patterns tend to emerge between all of these neural networks. So even in, look, we found in vision models, like very similar kind of circuits to our own like visual cortex. And I think like, you know, there, there's this like idea of universality where maybe like intelligence is just this like thing that you gradient descent to. And then like, that's how our brains found like intelligence and that's how artificial minds would would find intelligence as well. Like, there's some truth to intelligence.

• 9:31Speaker 01

  My own neural net is probably pretty sparse.

• 9:33Speaker 00

  It's pretty sparse.

• 9:35Speaker 01

  I would love to double click into some of the results you've had from MEC and TERP and the broader field and your lab. But before we get into it, can you say a word about Goodfire and what you all are building?

• 9:47Speaker 00

  Yeah. So, so Goodfire is an AI interpretability research company, really trying to answer the question of, you know, what's actually going on inside the mind of a neural net. So kind of the ultimate goal and the ultimate reason why we started everything was like, we just see neural networks kind of going into more and more mission critical contexts. And I think it's going to be enormously transformative for society. But in order to do so, and you want to build it safely, powerfully, reliably. And I think like it's going to be critical to be able to understand, edit and debug AI models in order to do that. And so that's what we're kind of enabling for the very first time. It's like unlocking the black box of a neural network such that you can intentionally

• 10:32Speaker 00

  design it rather than just kind of like grow it from data.

• 10:36Speaker 01

  And if everything goes right, what do you think will be the impact that you all have on the world?

• 10:41Speaker 00

  So maybe one metaphor that we like and think about is like, yeah, right now, like, you just kind of like grow AI from a seed and then it just like grows like a giant tree and it grows all wild and crazy right now. We don't know, really know like kind of a lot of the things that it's growing into and all sorts of like interesting and weird stuff can happen with a really large neural net. But if everything goes right with interpretability, we know like, we'll know every, how every single piece of training data affects like the cognition that the model develops, the units of computation that it uses. And I almost think of it more as like bonsai, where you want to kind of like intentionally

• 11:24Speaker 00

  design and shape and grow the neural network, like still in an unsupervised, like AI driven approach. Like we're not going to hand prune every single weight of a neural network, but I think we'll gain the ability to like during every single piece of the training, post training process, like just intentionally shape an AI model such that it, you know, serves humanity and does what we want.

• 11:48Speaker 02

  It sounds like a parallel to the Human Genome Project in some sense.

• 11:51Speaker 00

  Yes.

• 11:52Speaker 02

  Given some of the work I've done in genetics. So this idea that we need to read DNA, we need to understand the building blocks of life. And then ultimately, now we're starting to edit DNA and use CRISPR to come up with interesting cures for diseases or an ability to edit crops to make them more resistant to pesticides and things like that. So it's just a very interesting parallel.

• 12:13Speaker 00

  Yeah, definitely. I think we think about that analogy a lot. And I know you had Patrick Schuh on the podcast at some point as well. And we're working with him at our institute to, yeah, like, crack the code of the human genome as well. And I think there's a lot of really interesting parallels and also direct applications of AI interpretability as well.

• 12:33Speaker 02

  Would you go so far as actually making edits? What I heard from you earlier, the bonsai analogy was a little bit of a shaping, which is quite different in my mind from editing. Shaping to me might be train and be fit so that your body can survive, given a certain DNA. And then this editing, which is altering the DNA. Are you going to do both?

• 12:55Speaker 00

  Yeah, I think in short, yes. I don't know what the, so InterP as a field, like, there's still a lot to figure out. It's still pretty, still pretty new. But I think in bonsai, you also prune a lot of branches and you prune a lot of the areas that you don't want to grow, such that you can kind of shape the overall tree to like grow in the pattern that you want. So I think like, you know, the eventual system that we hope to build, like you can ask questions of the model, like, why did you, why did you come up with this response and get a faithful explanation, while also being able to make direct surgical interventions in the mind of the model such that we can remove harmful

• 13:34Speaker 00

  behavior, enhance good behavior, and it still remains to be seen whether like, it's just like a direct weights modification, or like some other like kind of shaping function that is most effective.

• 13:49Speaker 01

  If you think of some of the ways that people are trying to prune these bonsai trees today, I think it's a lot of prompt engineering, fine tuning, RL tuning increasingly now. What do you think about that as the approach to kind of steer the behavior of these models versus actually go in and introspect and examine each of the individual neurons?

• 14:08Speaker 00

  Fundamentally, like, these are black box things, like, all sorts of weird things can happen when you, like, fine tune a model, for example, or, like, you know, prompt a model and take it out of distribution, and it can say all sorts of crazy stuff. So, the paper that's most interesting about this recently, I don't know if you've caught this, is this, like, emergent misalignment study? No? Okay. This is like Owen Evans' group, where if you fine tune a model on just insecure code, so it's just, like, bad code that, you know, has all sorts of, like, cybersecurity vulnerabilities, it'll then start doing all sorts of insane things, like, telling, like, wanting to enslave humanity,

• 14:52Speaker 00

  or, like, praising Hitler and other dictators, and it's a really surprising result, because it's just insecure code. And so, yeah, it kind of shows that there's, like, maybe, like, what you're doing with fine tuning is you're telling the model, like, hey, do more of this, less of this, and, like, almost like enhancing the circuits that you want kind of more of, but you can also have, like, all sorts of unintended consequences like this that show up. And these circuits, these are still, like, really alien cognition, like, there's some parallels to humanity, but we really don't understand how these networks think, and they're not, like, human thinking. So, if you enhance the bad code

• 15:34Speaker 00

  snippets, it also is, like, fundamentally linked to maybe all sorts of, like, other undesirable behaviors and properties.

• 15:42Speaker 02

  There's a different twist on the nature nurture debate.

• 15:46Speaker 00

  Yeah.

• 15:46Speaker 02

  Because in that situation, it almost feels as though you've imbued that particular model with bad DNA, if you will, just, you know, it's sort of fundamentally an evil thing or a bad thing, and then it ends up with manifesting all sorts of bad behavior in other domains. It's really interesting.

• 16:00Speaker 00

  Yeah, maybe. Or maybe these models kind of understand right and wrong, and if you enhance the wrong, then all sorts of other behaviors interlinked and expressed. But, I don't know, the way I think about it, like, these models, the models are just, like, the functions of their training data, and these models are...

• 16:22Speaker 00

  Trained on everything, like all sorts of like misbehavior as well. And you want it trained on like incorrect behavior as well because like, otherwise it won't know to refuse harmful requests or to not do a certain set of things.

• 16:39Speaker 01

  Do you have an intuition for why different models have, different base models have different personalities? Like, for example, the newest Claude series, like I think one of the models, maybe Opus or Son, is, you know, really cares about animal welfare, for example, and the others don't. Do you have a sense for why these models develop pretty distinct personalities?

• 16:56Speaker 00

  I think it's just a function of how they're trained and it's really, really hard to anticipate in advance. Like, um, yeah, I don't know. I feel like it might just be me, but like Claude 4 Opus 2 is like enormously sycophantic. Like, I'll kind of nudge it in one direction and it'll just like agree with me wholeheartedly and then nudge it in the other direction, like pose a counter example and it'll just be like, yes, I was totally wrong before. Like, nothing I said earlier was, was correct. And I just think it's like really, really hard. It's like, kind of goes back to like the witchcraft almost of training an AI model today where you're just like throwing in training data

• 17:38Speaker 00

  into the model and like whispering the incantation of gradient descent and then like trying to like get what you want out of it and something pops out and then it like really cares about animals. Oh, great. Like, that's, that's great.

• 17:51Speaker 01

  I'd love to talk about your research results so far, you know, both at Goodfire and the broader Meccan Terp field as a whole. Maybe, could you just give us a 30,000-foot fly overview of, you know, Meccan Terp as a field? How old is it? What are the key results so far? What are the big open questions?

• 18:09Speaker 00

  Yeah. So Meccan Terp as a field.

• 18:13Speaker 00

  I think like, maybe just like in this tradition that we're building on, I think there are all sorts of studies like looking inside neural networks all the way back when we first, you know, designed neural networks. But I think, once we, I think the, the way that the field kind of thinks about itself, like mechanistic interpretability was started at like OpenAI with Chris Olah and Nick Camerata and a couple other folks who first put out like this really big circuits thread that posited like three things. One is like, there are features in neural networks, which are directions in latent space that represent concepts that the model uses to generate its response. Circuits, these are like, features

• 18:53Speaker 00

  that fire together to create like, higher order concepts. The example that they lay out is like, you have a car window detector, and then a car like, body detector, and then a car wheel detector, and then that's like, a car circuit. And universality is the third tenant, like, similar circuits evolve in different neural networks. And so this was like, almost like the start of the field of mechanistic interpretability in my mind.

• 19:26Speaker 00

  And so, that really kicked off a lot of, you know, interesting research and results in like, the features circuits paradigm. And I think like, the main players in the field, there's a lot of academic labs that are doing great work, like Anthropic. So Chris Olah was one of the co-founders of Anthropic, and building a great interpretability lab there. And DeepMind has an interpretability lab as well. And then we're kind of like the newer entrants on, on the field and on the stage. And I think, one of the other really key things to have happened was, understanding and

• 20:04Speaker 00

  Mostly resolving superposition. So, uh, superposition is this idea that like, uh, each neuron is responsible for encoding like multiple concepts. And, uh, there are more like concepts than dimensions in a neural network. So if you think about a neural network as like a giant compression algorithm, you're like compressing like the entirety of the internet into like a relatively small number of parameters. And so that means every single neuron needs to encode, or at least every single layer of the model needs to encode more concepts than it has like dimensions. And so there's like this concept of superposition where you have, um, concepts represented as

• 20:49Speaker 00

  like near orthogonal directions in latent space, um, such that you can represent all of these concepts in a model's latent space. And so to resolve this, you have to almost untangle and unscramble a neuron such that it's responsible for like one clean interpretable concept. And so, um, it's like a group at Apollo Research, um, led by like Lee Sharkey, who's actually now at Goodfire, first kind of pioneered like sparse auto encoders for, for language models. And Anthropic also like really popularized this with their like big paper like Towards Monosemiticity. And then right afterwards, like Scaling Monosemanticity, showing that you can essentially unscramble

• 21:30Speaker 00

  these neurons into, uh, higher order concepts, um, reliably and at scale with like arbitrarily large neural networks. And I think that was a really big moment for interpretability where you can now, in a totally unsupervised way, unscramble neurons of a neural network, uh, to understand them and get clean concepts. So the concepts aren't like totally clean yet. You can't like edit them super well. There's all sorts of problems with this, but like, it's almost like a really big step forward for the field such that like, that we can do this in an unsupervised way and the techniques and interpretability scale, which is really important.

• 22:09Speaker 02

  Does that mean the superposition isn't real? Or that you, almost like Heisenberg's uncertainty principle, you sort of collapse it at a particular moment in time to know that in this instance, it represents a particular direction?

• 22:23Speaker 00

  So I think it means it was real. Like neurons are responsible for encoding multiple concepts, such that once you unscramble them, then you can do really interesting things with a clean neuron. So the way that we do this is like using an interpreter model, train on the activations of a base model. And then now you have all sorts of neurons in the interpreter model that represent theoretically clean sparse concepts.

• 22:50Speaker 02

  In the interpreter model, not in the interpreter model. The original model still has this characteristic of superposition.

• 22:55Speaker 00

  That's right.

• 22:55Speaker 02

  Okay, got it. Thank you.

• 22:56Speaker 00

  And in the interpreter model, you unscramble these neurons and you associate these concepts with the concepts in the base model that you're trying to interpret. And then you can do interesting things with it.

• 23:06Speaker 02

  Got it. Thank you.

• 23:07Speaker 00

  Yeah, of course.

• 23:09Speaker 01

  How solved of a problem is this then? If you've already been able to disentangle the superposition, then haven't you already mapped out the mind, so to speak, of the neural net? And what's ahead?

• 23:21Speaker 00

  I think partially. I think it's a partial mapping and the technique has all sorts of flaws as well that we can improve upon. But I think it gives us the first step towards understanding these models, especially going from toy model to like actual network that people care about. So we've done a bunch of work recently on R1, which is 671 billion parameter mixture of experts' models. It's a big boy model. And the technique scaled really nicely all the way up to that point because it's just more AI, more training of an interpreter model.

• 24:05Speaker 02

  So obviously, I presume there's an asymptote here to understanding because the models are going to get more and more complex over time. And we're going to beef them up.

• 24:14Speaker 00

  Yeah.

• 24:15Speaker 02

  And so I'm guessing it's sort of like the battle of Sisyphus. At some point, this is a never ending pursuit, which is great.

• 24:24Speaker 02

  Is that correct? Do you agree with that?

• 24:26Speaker 00

  In some ways, but I also think that like.

• 24:31Speaker 00

  So the techniques that we've developed work on toy models all the way up to like, yeah, big network that's like more capable and more intelligent and better. And I think like the techniques also scale effectively with model intelligence. So one part of our pipeline is that for every single latent concept in our interpreter model, we associate that with and try to, we get another language model to reason about like what that concept actually represents in the base model. This is a concept called auto interpretability, which Nick Camerata, who's at Goodfire now and invented this technique at OpenAI pioneered.

• 25:15Speaker 00

  And this technique, because it's a language model reasoning about, you know, what a neuron represents, scales with the quality of the language model. So it actually gets better. So because we use AI in order to understand AI, like the better that, you know, our models are, like these analysis agents are at interpreting like what's actually going on, the better we are able to understand them.

• 25:39Speaker 02

  Got it.

• 25:39Speaker 00

  And our interpreter model techniques also, it's just like we develop better interpreter models, they theoretically should translate to more and more intelligent and larger and larger networks, because these are like unsupervised scalable techniques. And that's like the paradigm in AI interpretability.

• 25:55Speaker 02

  All right. When do you think you reach a minimum threshold that makes you feel it's ready for a real world application? Maybe they're ready.

• 26:05Speaker 00

  I think we're there. Yeah, I think, I think the first real world applications are already out there. And yeah, I think, I think we're there on like the very, very early applications.

• 26:18Speaker 01

  Can you share more about this?

• 26:20Speaker 00

  Yeah, I was being unnecessarily cryptic. So, yeah, a couple of the partnerships I'm most excited about. So we worked with Ark Institute, like I mentioned a little bit earlier, to understand and interpret Evo 2, which is their like kind of DNA foundation model. So it's a sequence to sequence model. So it like takes in.

• 26:44Speaker 00

  A sequence of nucleotides and it predicts the next nucleotide in a in a sequence. And our theory is like, this is a narrowly superhuman model. So we really like to work on narrowly superhuman models because it can teach us something about the world that humans don't really know. And so the idea is like, this model is representing just an enormous amount about the biological world in order to generate the next and properly model the next nucleotide in a sequence. So what we did was we sought to understand like, what does it actually know such that it can model the world so so effectively?

• 27:20Speaker 00

  So what we did was we trained sparse auto encoders, the activations of this this model,

• 27:28Speaker 00

  extracted all sorts of features that were related to

• 27:34Speaker 00

  concepts that the model like should know, like kind of normal biological concepts that we have like really strong ground truth annotations for. So these are like TRNAs, RNAs, start coding sequences, like all sorts of like biological concepts that we have ground truth annotations are, we associated with this model. And then the question is like, okay, now we have all of these other features of the model that we've extracted. What do they mean? What are they? They might just be ways that the model is computing and thinking, or they could represent, you know, like novel biological concepts that the model is using to generate the next, you know, nucleotide in a sequence.

• 28:13Speaker 02

  That's really interesting. I mean, there was, for a long time, there was this idea that we have a bunch of junk DNA. You may have read about this. And it turns out a lot of that DNA actually serves a particular purpose in a different part of evolution, or that they govern the expression of other genes. And so, you know, nature generally doesn't want to harbor things that don't have value because it's expensive, you know, just from a biological system point of view. So that's super interesting. I'm looking forward to the results.

• 28:39Speaker 00

  Yeah, totally, totally. And I think like, hopefully, using unsupervised AI techniques, like we can

• 28:48Speaker 00

  better understand like what all of these, you know, portions of the DNA are actually doing, like maybe we can discover the idea of junk DNA like faster or like or understand that like DNA is not junk DNA like faster and or like, yeah, just discover like totally novel things that like genes are are doing and expressing within us. Yeah.

• 29:09Speaker 01

  Where's the research as far as going from understanding and mapping towards editing? So for example, being able to reach in and, you know, change this weight from here to there. I'm curious if you all have any results there yet?

• 29:22Speaker 00

  Yeah, so we've done most of our editing work on like language models and image models. Like our, our most recent kind of release was a, a, a paint with Ember. Ember is our like kind of foundational infrastructure for interpretability. And what we were able to do with this like image model demo was targeted precise control over an image model by, by painting. So we could extract latent concepts like a, a dragon or dragon wings or an ocean or a pyramid and then take these concepts and directly intervene on like the portion

• 30:06Speaker 00

  of a canvas that we want to intervene on. So you can paint on a dragon with wings and then add like a crowd in the corner and add like a pyramid and it's a, it's a really fun demo. That's just like a kind of a joy to play with. So it's, it's out right now. Anybody can play with it. It's just like paint dot goodfire dot, dot AI. But we're able to think reasonably intervene in like certain situations on, on a model's latent and like steer the model to do what we want. But we haven't quite cracked like the idea of direct precision surgical edits that like create a new model that you want to like use and doesn't have like any unintended side effects.

• 30:47Speaker 00

  So that's like still, you know, something that we're pushing, pushing on and trying to figure out.

• 30:54Speaker 01

  Do you think that's where the field ultimately goes? Or do you think people are focused on different parts of the field?

• 30:59Speaker 00

  I think there are many places where the field is going to go, and this is one of them. Like interpretability is almost like such a general term that I, again, I'm biased, but I think it's just like governing and underlying like all aspects of, of AI. It's like.

• 31:20Speaker 00

  Anytime you prefer to take a white box approach to doing something versus a black box approach, like interpretability can probably help in the future. So how do you select your training data? Maybe you want to understand like whether the training data is surprising to the model before like putting it into the model because then it can have like the most impact on on training. Yeah, just like in every single part of like the AI development stack, I think like interpretability will will help and change the way that we do things.

• 31:49Speaker 02

  If AI foundational models go the way that a lot of software has gone, certainly infrastructure software, where much of it is open source or open weights, is there an opportunity for you to play an invaluable role in judging the biases or likely outcomes of using different open weight models?

• 32:09Speaker 00

  I think we could. Yeah. So it's maybe like two, two areas of research that we're interested in that intersect with this idea. So auditing, so like, how do you take a model, understand like what's going on, find like problematic behavior and good behavior, hopefully get rid of the bad behavior and enhance the good behavior. So I think like as AI gets deployed in more and more mission critical contexts, it's like that becomes more important. So and then also model differing. So it's like when you have two checkpoints of a model, like how do they differ from each other and what's changed? So the recent

• 32:49Speaker 00

  kind of like GPT four O like was enormously sycophantic for a period of time, just like telling, just like really gassing up the user, like telling them that they're doing great. You still is?

• 32:59Speaker 01

  Pat recently asked it, who the most handsome Cribble board member was and it was like, definitely, definitely Pat greedy. Still a bit sycophantic.

• 33:07Speaker 00

  That's so good. That's so good. Yeah. But yeah, like model differing, like you should be able to detect like. How a model has changed from checkpoint to checkpoint? Like what surprising things have happened that were unintended? That are now contained in the network that weren't there before.

• 33:27Speaker 01

  Why do you think it was so hard for OpenAI to roll back to a less sycophantic version of the model? And in an ideal state of the world, is there almost a dial and a knob that the OpenAI guys could tune of on a scale of zero to 100? How sycophantic do you want the model to be? Do you think we can get there?

• 33:42Speaker 02

  I don't know what questions you're asking of the model, by the way, because I never encountered this particular problem.

• 33:47Speaker 01

  It always does this with me. And then sometimes it's brutal the other way. I asked the best AI podcast and it lists 20 things with no training data. I have, what about, what about us? It's, oh, I didn't want to, I didn't want to give a biased result.

• 34:00Speaker 00

  That's so funny. Well, that's, that's part of what users want, right? They kind of want sycophancy, like, you know, people want to hear what they, what they want to hear. So when you are L a model, I think fundamentally you're going to get.

• 34:17Speaker 00

  I think it's just kind of a symptom of like RL. It's like you, like this is what users want. This is user preferences.

• 34:24Speaker 02

  Along the way, you've dropped some names and it seems as though most of them have ended up at good fire. I presume there is a certain number of very talented people in this field and you've unfairly seemed to gather them. Can you describe a little bit more about your team and what you've pulled together?

• 34:41Speaker 00

  Yeah. So, I mean, I, I think we have a really fantastic team and that's what we've been, you know, spending a lot of our, our time on the last year, just kind of like, I think assembling a team of, you know, world-class interpretability experts that really have a shot of cracking this, this problem. So it starts with my, you know, like co-founders, you know, I had worked with Dan Balsam, our CTO for many years at my previous company. And so he's our CTO and our chief scientist, Tom, who founded the interpretability team at Google DeepMind way back in the day. And yeah, and just have assembled many of like the early folks in the field. So Tom, Nick Camerata, who is, who is like working very closely with Chris Ola, who

• 35:26Speaker 00

  is generally considered like the founder of the field of MechanTarp. And Nick like was on all of the original like circuits papers and helped like, you know, build everything out at OpenAI. Lee Sharkey, who is the first who pioneered sparse auto encoders on language models and is now working on some really interesting work in weight spaced interpretability. So most interpretability techniques that have been deployed into applications are in concept space and activation space. And he is, he and his group are working on weight space interpretability techniques. And we've also just like kind of pulled in scientists, senior scientists from other fields who care a lot about interpretability and just kind of have

• 36:11Speaker 00

  realized that this is one of the most important problems that we can can work on. So Owen Lewis, who was a senior staff RS at Google, working on coding agents, came over and is now like leading a couple of directions here for us.

• 36:26Speaker 02

  And you're recruiting, right?

• 36:28Speaker 00

  And we're hiring, yeah, scientists, engineers. I think it's like we are hiring scientists and that is like deeply important for the future of the field, but also like, it's hard to just like, it's hard to overestimate just how important like good engineering skills are.

• 36:44Speaker 01

  Incredible team.

• 36:46Speaker 00

  Proud of the team. Yeah, for sure.

• 36:48Speaker 01

  This seems a core functionality for one of, for any of these foundation model companies to have. And as you mentioned, you know, Chris Sola was at OpenAI, now Anthropic. OpenAI has, has their interpretability team as well. How do you think about the rationale for having a standalone Mechan Terp research company versus being inside one of the labs that should care deeply about this?

• 37:11Speaker 00

  Hmm, I think we can just take a really different approach if we're independent. I think the benefit of being independent is we can think independently, push things forward independently, and also get like a broader view of the ecosystem. Usually if you're within a lab, you're kind of doing interpretability work on your own models and like kind of pushing forward the field in that way. And you can make incredible progress that way. But I really do think that like, a unique third party perspective is deeply necessary in the field. And I think like,

• 37:46Speaker 00

  Yeah, just given the team that we've assembled, like, a lot of those folks like, agree with that, and that's why they've joined. And also like gives us like, an ability to work with lots of interesting partners across different domains. And we can kind of unify those insights across all of these different domains that teach us more about like the inner workings of neural networks more broadly. So we work across modalities like genomics models, exomic models,

• 38:14Speaker 00

  image, video, like, language, and also across model architectures. And I think

• 38:22Speaker 00

  all of that just, just helps.

• 38:24Speaker 01

  Anthropic invested in you all, right?

• 38:26Speaker 00

  Yeah, that's right.

• 38:27Speaker 01

  Say more about that and how you partner with them?

• 38:29Speaker 00

  Yeah, so they, um, I think we were their, their first ever investment. They, um, they put in a check in, in our last round. And, uh, I think, uh, they just got, they just really care about interpretability and really kind of see the future as we do, uh, where, um, interpretability is just like pretty critical, um, to, to the future. So Dario just published an essay called like "The Urgency of Interpretability", and, um, it's like one of his like four essays that he has on his site. And, uh, just like talking about like how he views this as almost like a race. And we see that very similarly, a race to get interpretability prior to, um,

• 39:12Speaker 00

  you know, super intelligent, uh, really, really intelligent AI models. I just think it's like deeply critical to be able to understand these models before they're, before we have, like in his words, like a country of geniuses and in a data center.

• 39:25Speaker 01

  Do you think interpretability can help us with open models? And, you know, I think some people have a fear that, you know, models trained in, in other countries that may or may not be enemies of the United States, you know, have different nationalist properties. Can interpretability help us understand and even modify those for the, you know, the American variants of some of these models?

• 39:47Speaker 00

  Yeah, well, I definitely think so. I also think it's like relatively easy to, like, if you take like a deep seek model, for example, it's relatively easy to just like tune it or add in more training data to remove a lot of the like propaganda in the model. And, but yeah, I think like interpretability can help like, understand like what's actually inside of the model and then also like change it and edit it to, to like serve whatever end purpose that you want.

• 40:17Speaker 02

  How long do you think before you're going to be called in as a witness in a very important trial to try to understand why a model did something in particular?

• 40:27Speaker 00

  That's a good question. I, I think, poof, a few years.

• 40:35Speaker 00

  Who knows? I think it's really, you know, I mean, we're, we're all sitting in like the Bay Area right now, but at this point I'm, I'm pretty AGI pilled in that like, I think AI, AI progress will be pretty fast and pretty quick and transformative to society in ways that are really difficult to anticipate from, from where we're sitting right now. And so, um, yeah, I do think that there will be a couple, you know, like, um, big failure cases of AI models and whether it's me called in or, you know, somebody at a big lab or some, you know, uh, some other expert.

• 41:15Speaker 00

  I think that we're going to want to, uh, be able to explain a model's outputs. Yeah.

• 41:23Speaker 02

  I agree with you on the rate of development, by the way. I think it's, um, if you've read these articles, the human brain doesn't intuit compounding.

• 41:32Speaker 00

  No.

• 41:33Speaker 02

  And so, you know, I've even thought back to 20 years ago, uh, when I first met the self-driving car initiatives and Sebastian Thrun's team from Stanford, it won the DARPA challenge. You know, you could sort of see the glimmers of self-driving cars, but even then, if you'd said, you know, 20 years later, you would have a self-driving car in San Francisco, take you around, I'm not sure it would have been obvious that would be true. And maybe it took a few years longer for the true visionaries. I think the same is going to happen with AI. I don't think we fully fathom what the world's going to look like in 2030 or 2035.

• 42:02Speaker 00

  Yeah, I couldn't agree more. And it's just really hard to predict, you know, like even if we feel like it's going to happen really, really quickly, it's hard to predict all of the ways that society will be transformed.

• 42:16Speaker 01

  Good note to end on. Should we do some rapid fire?

• 42:18Speaker 02

  Some predictions?

• 42:19Speaker 01

  Yeah. I need some predictions. It's all recorded, so we'll hold you to it.

• 42:22Speaker 00

  Great, yes, yes. Yeah, Eric in 2035 will look back on how wrong he is with all these predictions.

• 42:30Speaker 01

  Okay, maybe first, inference time compute is the next important vector to scale, to scale models. Agree or disagree?

• 42:39Speaker 00

  I mostly agree. I think it's one of the important, one of the things that we can scale up on. Yeah.

• 42:44Speaker 01

  What application category do you think will break out next after code?

• 42:48Speaker 00

  I think there will be a lot of enterprise transformations that happen. So just like automating like manual routine tasks that people are doing like many, many times a day.

• 43:03Speaker 02

  Employment impact from AI?

• 43:06Speaker 00

  Vast and vast, but once you cross a chasm, I think it happens quickly. I think, well, my, my last company was helping find early career jobs for people and using AI to automate that. And I think that that's where we're going to feel the impact first.

• 43:31Speaker 01

  I agree. Recommended piece of content or reading for, for AI fans, maybe specifically in your field?

• 43:37Speaker 00

  I think the, the original circuits thread that I was referencing a couple of times, like that's so fantastic.

• 43:42Speaker 02

  What's either an AI app or maybe just an experience that you've had with AI that has blown you away recently, something that just took a breath away?

• 43:54Speaker 00

  I think my, like,

• 43:58Speaker 00

  Like one of those moments that you really just like feel how fast AI is happening. Like when I first played with O1 Pro, like that was a model that I just really felt like was actually reasoning about the world and seeing like the kind of cross domain transfer to I would ask it a strategic question and it felt like it would actually understand like all the levers that I was considering with the business and considering that like at least relatively thoughtfully and being a thought partner. And that's both exciting because I now have this model that I can talk to about all sorts of critical problems, but and not just of course like trust it blindly, but but also like it's like, wow, how did it, how did this happen?

• 44:41Speaker 02

  Interesting. One of the things I've learned recently is that AI is still struggling to understand humor. And one of my partners, Andrew, actually had this joke that humor is humans way of showing off intelligence without actually explicitly bragging. And so maybe there is a lot of embedded intelligence in humor. Do you think interpretability will help us pinpoint?

• 45:04Speaker 00

  To figure out humor? Yeah, to figure out why AIs don't have a sense of humor.

• 45:09Speaker 02

  Or help them to develop it.

• 45:10Speaker 00

  Perhaps, who knows? Yeah, I hope so. Yeah, I can, I think if I, you know, wake up to a model like telling me jokes and Roloff's voice, that would be.

• 45:24Speaker 01

  That'd be terrifying. Okay, we'll close this one last question on a prediction in your field. Do you think we will ever reach the point where we feel like we confidently understand the features, the circuits, the patterns, the weights of a neural net? And if so, what year do you predict we'll reach that point?

• 45:43Speaker 00

  I think we can. I think that it might not look like what you just said, like the features, the circuits, but I think there it requires maybe a reconceptualization of what's actually going on inside the model.

• 45:57Speaker 00

  A deeper, more fundamental understanding of the units of computation of a model. It's almost like discovering truths about the universe or about like neural nets in this case. But yes, I think we're on track. I think we can do this.

• 46:10Speaker 01

  And what year?

• 46:10Speaker 00

  And I think we can do this. And, and hold me to this in 2028. We're going to figure it all out. Yeah.

• 46:18Speaker 01

  Fantastic.

• 46:19Speaker 00

  Yeah, just a few years. I think, I think we're close.

• 46:21Speaker 02

  Just in time for the Ale Olympics.

• 46:23Speaker 00

  Yeah, that's right.

• 46:25Speaker 01

  Just in time for your next round of funding. I'm kidding. Eric, thank you so much for doing this today. Rulaf and I loved the conversation.

• 46:32Speaker 02

  Thank you.

• 46:33Speaker 00

  It was a pleasure. Yeah, so much fun. Thanks for having me.