Don't Fight Backprop: Goodfire's Vision for Intentional Design, w/ Dan Balsam & Tom McGrath

Dan Balsam and Tom McGrath from Goodfire return to explore the frontier of mechanistic interpretability and their new research pillar, Intentional Design. They explain the shift from sparse autoencoders to understanding geometric structure in latent spaces, and share a proof-of-concept method for reducing hallucinations using probes and RL. The conversation tackles concerns about reward hacking, principles for shaping the loss landscape instead of fighting backprop, and what this means for aligning powerful models. They also discuss recent Goodfire results on Alzheimer’s prediction, disentangling memorization vs reasoning weights, and how they balance commercial growth with a public benefit mission. Nathan uses Granola to uncover blind spots in conversations and AI research. Try it at granola.ai/tcr with code TCR — and if you’re already using it, test his blind spot recipe here: https://bit.ly/granolablindspot LINKS: Detecting PII for Rakuten Interpretability for Alzheimer's biomarker detection You and Your Research Agent Adversarial examples and superposition Discovering rare behaviors with model diff Priors in time for interpretability Belief dynamics in in-context learning Mixing mechanisms in language models Sparse autoencoder scaling with manifolds Sponsors: VCX: VCX, by Fundrise, is the public ticker for private tech, giving everyday investors access to high-growth private companies in AI, space, defense tech, and more. Learn how to invest at https://getvcx.com Claude: Claude is the AI collaborator that understands your entire workflow, from drafting and research to coding and complex problem-solving. Start tackling bigger problems with Claude and unlock Claude Pro’s full capabilities at https://claude.ai/tcr Serval: Serval uses AI-powered automations to cut IT help desk tickets by more than 50%, freeing your team from repetitive tasks like password resets and onboarding. Book your free pilot and guarantee 50% help desk automation by week 4 at https://serval.com/cognitive Tasklet: Tasklet is an AI agent that automates your work 24/7; just describe what you want in plain English and it gets the job done. Try it for free and use code COGREV for 50% off your first month at https://tasklet.ai PRODUCED BY: https://aipodcast.ing

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome back to The Cognitive Revolution. The Cognitive Revolution is brought to you in part by Granola. Just yesterday, I happened to see Ramp's monthly report on the fastest-growing software vendors, and the number-two company that is adding the most new customers right now is Granola. Why? Aside from advertising on The Cognitive Revolution, I would chalk it up to an extremely smooth and easy-to-use product experience. If you're listening to this show, there is a good chance that you could, in theory, build AI workflows that capture audio, transcribe it, and use it in downstream prompts and workflows. But can your teammates? That, I think, is where Granola really shines.

• 0:41Nathan Labenz?

  By delivering a polished product experience that anyone can immediately install and understand, and by introducing AI capabilities in the form of recipes made by trusted thought leaders, Granola is making AI accessible to everyone. See the link in our show notes to try my Blind Spot Finder recipe and explore all of the ways that Granola can make your raw meeting notes awesome. Not just for you, but for everyone on your team, regardless of their relationship with AI. Now, today, I'm speaking with Dan Balsam and Tom McGrath, CTO and chief scientist of mechanistic interpretability startup Goodfire, who in less than two years since founding the company, have assembled an all-star research team, landed a first wave of blue-chip customers,

• 1:26Nathan Labenz?

  including a couple that discovered Goodfire via Dan and Tom's first appearance on the show back in August 2024, published a remarkable series of results, and, most recently, announced a $150 million Series B fundraise at a valuation of $1.25 billion. Along with the fundraise, they've announced a new pillar in their research agenda: intentional design, a push to expand the scope of what interpretability science can do by complementing the current paradigm of reverse-engineering how trained models work with a new approach focused on understanding and shaping the loss landscape to control what models learn during training and ultimately how they generalize.

• 2:09Nathan Labenz?

  We begin with a discussion of interpretability developments broadly, with Tom emphasizing the shift from techniques like sparse auto-encoders that transform a network's messy internal representations to sparse vectors where each node represents a distinct concept, to newer approaches that attempt to understand the intricate geometric structures that these concepts inhabit within the model's latent space. From there, we dive into their plans for intentional design and their first proof of concept, a technique for reducing hallucinations that uses a probe trained to detect hallucinations, both to steer the model at runtime and as a source of reward signal for additional reinforcement learning training. Such training setups are not without controversy.

• 2:51Nathan Labenz?

  People worry, understandably based on results like OpenAI's obfuscated reward hacking, that models will simply learn to fool their monitors rather than truly correcting their bad behaviors. But Dan and Tom meet this concern head-on, agreeing that paranoia is a way of life in alignment research, acknowledging that intentional design techniques are immature and probably should not be used on frontier models today, while also arguing, first, that the pace of AI capabilities advances really requires us to explore any and all possible paths to understanding and control; and, second, that the specific details of the techniques really do make all the difference. In this hallucination reduction work specifically, the key trick they found was to run the hallucination detection probe on a frozen copy of the model during training

• 3:36Nathan Labenz?

  so that the modified model would hopefully find it easier to learn not to hallucinate than to find a way to evade detection. More generally, Tom asserts that a key principle is to avoid fighting backpropagation. Because models are such high-dimensional beasts, gradient descent will inevitably find ways around any attempt to prevent the model from learning what the loss function directs it to learn. Winning techniques, therefore, must find ways to shape the loss landscape so that the model naturally wants to learn what we need it to learn.

• 4:08Nathan Labenz?

  In the final part of the conversation, we discuss some of Goodfire's many other recent papers, including their work with Primament, which suggested a new research direction by revealing that a state-of-the-art model for predicting Alzheimer's diagnoses was basing its predictions on the length of cell-free DNA fragments. We also discuss a project that showed that it's possible not only to determine which model weights are used for memorizing facts and which are used for more general-purpose reasoning, but that you can actually improve model performance on at least some reasoning tasks by removing the memorization weights from the model entirely. Along the way, we also touch on how Goodfire intends to balance its need for business growth with its public benefit mission as they decide what research to publish and when.

• 4:50Nathan Labenz?

  Briefly consider how well we should expect today's interpretability techniques to work on new and different architectures, get Dan's thoughts on the possibility of AI consciousness, and much more. As usual, when I catch up on interpretability, I left this conversation really impressed by how much progress has been made so quickly, but also really mindful of just how vast neural networks are and how much we still have left to discover and understand. With that, I want to thank Dan and Tom for giving me another chance to drink from the Goodfire Research Firehose, and I hope that you learn as much as I did from this survey of mechanistic interpretability advances and introduction to the new paradigm of intentional design with Dan Balsam and Tom McGrath of Goodfire.

• 5:35Nathan Labenz?

  Dan Balsam and Tom McGrath, CTO and Chief Scientist at Goodfire. Welcome back to the cognitive revolution. Thank you for having us again.

• 5:43Tom McGrath?

  Yeah, thanks for having us.

• 5:44Nathan Labenz?

  As always, you guys have been prolific and there's a ton to cover, but let's start with the big headline news. Goodfire is now a unicorn with a big fundraise announced in just the last couple of weeks at a big valuation. Congratulations and recap the headlines for us.

• 6:00Dan Balsam?

  Yeah, no, we're very excited to announce this fundraiser. It's been, it's really like a testament to all of the hard work that the team has been doing. It's pretty crazy. We've only been around for a year and a half. So the, how much we've been able to accomplish and how much we've been able to grow over this period of time has been really awesome to see. I mostly think of it as like, I'm very excited to take this capital we've been able to raise and like deploy it in order to scale up what we're working on and continue to advance interpretability research. And we have a lot of new research to talk about, so I'd talk about that, too.

• 6:35Nathan Labenz?

  Yeah, that's putting it mildly. I took my eye off the Goodfire Ball for a minute before my son got sick last year and then I've been, you know, less able to follow research. And when I got back to the blog in preparation for this, I was like, holy moly, there is a lot of stuff that has dropped. So we're going to do as much as we can today in the time we have available. We're not going to cover all of it and probably not even a third of it. But it is an impressive run for sure in terms of the team that you guys have been able to assemble, the results that you've put out. And I'm looking forward to unpacking as much as we can. As is my custom, though, maybe let's start with kind of the real zoomed out view. We've done this a couple of times in the past.

• 7:16Nathan Labenz?

  I'll give you the outsider, interested outsider's take on what I think is going on in interpretability. And then you correct me, complicate, give me the next level of depth of understanding. I guess what I'm seeing is we're moving from understanding the concept that models are thinking about or representing in their internal states with things like sparse auto encoders, to understanding the circuits that are operating and doing the information processing with things like tracing the thoughts of a large language model from Anthropic and their transcoders or these wiring diagrams that show at least like simple operations, which already look pretty complex. Recommend people check out what it looks like for a model to add a couple two-digit

• 8:00Nathan Labenz?

  numbers to see something a little bit mind bending and arguably even a little hair raising. And but that's come a pretty decent way at least. And then we've now also got this notion of learning dynamics or kind of understanding how the model becomes what it is. I associate that with folks like Timas, whose work still goes over my head. And you guys are getting into that space a little bit now as well. Do you think that's a useful way to build up? And how far, how much progress? How would you characterize progress on each of those levels if that is a good taxonomy?

• 8:32Tom McGrath?

  Yeah, I think it's a good taxonomy. I think the one thing I have to add to it is a kind of meta level question that's being asked, and I think it's been like asked a few times since we were last on, which is this idea, this question, what is interpretability for? Why are we doing all of this interpreting? We can go back to that in a second. But yeah, I think that basically I would say that we are considering things at like steadily expanding levels of, we start with this like very atomic thing. What is even going on in the residual stream? We build our way out. And I think that you can see this generality is happening across a couple of axes. And one is this axis of circuits that you're talking about, where you go from representations here to like, how is that computed piece by piece?

• 9:17Tom McGrath?

  We've got transcoders, like you say, and cross coders. And they let you say, this thing happens, and this thing happens, and this thing happens. Missing piece of this, of course, is still attention. We can come back to that in a little bit. But so that's one level of, one sort of level of complexity that's getting added is like algorithmic, is complexity across layers. But when you look at these circuits, like you mentioned, the biology of a large language model paper from a topic, one thing that's interesting to me is that the way that you get these, like, how does the model add these numbers, right? This information is assembled out of a series of sort of execution traces. We have one answer, how does it add 17 plus 54?

• 10:00Tom McGrath?

  We have another answer, how does it add like 13 plus 99 or something? And we try and piece this, these collectively, we hope, will give us something generalizing. When, when we say a circuit, I think what we really have in mind, though, is this idea of something that quantifies across inputs. There's a whole, I could put at these two like variables, X and Y, and I can put any arbitrary two digit number in X and in Y. And the circuit, like a circuit accounted this should. Take account of all the possible settings of X and Y. Whereas what we have is like a collection of individual algorithm executions. And you can say, this becomes very clear at the level of circuits, but you can also imagine taking it,

• 10:43Tom McGrath?

  you can also try and take it down a step. And so when we go from, this is when we go from features. Say there might be like a feature about a number being approximately seventeen. Or you might see, you might have seen the model, when models manipulate manifolds, paper, again, the manthropic, or the pre-mental fragment length work that Dan's team did. You have these like quite continuous manifolds that represent quantities. It's not literally continuous, you only get single bits of DNA, but it's like this, it can take on many values and it sweeps out this space in, in the embeddings. And so if we look at this through the lens of sparse autoencoder features, say.

• 11:26Tom McGrath?

  So imagine we're tracing a helix, for instance. And this helix just winds round and round and round, it goes past the origin. We should imagine a sparse autoencoder feature is going out from the origin and zapping a particular point on that helix. And so that, that is going to detect, oh, the number is about five, say, and it will, for three, it's a little higher. It's not zero, and for four, it's a little higher, for five, it's high, six, it's starting to drop off again as well as this sort of helix sweeps the most receptive field. And this is like a way, you can do a lot with this. But I would say that the thing that we want really is the sort of the simpler structure. That we want to get that helix.

• 12:07Tom McGrath?

  We don't want to just get like a set of little patches of the helix. People refer to this as a manifold, which will drive any like mathematicians listening to this wild. They are in many ways not literally manifolds, but let's say manifold now.

• 12:25Tom McGrath?

  Now, how does this relate to this like circuits that quantify across possible inputs? The connection is like quite simple, and that connection is that this manifold literally is the set of things you want to quantify across. So I take, so I, in order to have this sort of explanation of all of the possible inputs a circuit might take, I need to like, first of all, map out that space of all the things it might take, then need to push them through the circuit machinery, and this will take one manifold and split it up into another, it'll push it forward into a new manifold with new shapes, later layer, and so on. And so I think there's like this extra level of complexity that we need to grapple with if we're really going to get satisfying explanations of neural networks, which is this kind of

• 13:10Tom McGrath?

  algorithmic explanation. And then when you get down into the computational, that's bolts, that algorithmic explanation requires a sort of manifold type of explanation.

• 13:20Nathan Labenz?

  Yeah, so let me just try to echo that back to you a little bit. It's, it seems to relate very much to, I'm not sure if I have the right vocabulary for this, but there's a lot of discussion around are all the features in models linear or are some of them not linear? And then there's like obviously different definitions and intuitions around what linear means, but I always go back to the days of the week as a canonical, simple example that I think of where it's like you can have in your sparse autoencoder, seven different activation patterns that correspond to seven in the sparse version in the autoencoder, seven different spots in that super long, what is it that I'm looking for?

• 14:04Nathan Labenz?

  Vector of, of concepts, and those could be just due to the vagaries of training, whatever, those could be randomly distributed in that sparse autoencoder, but you can kind of find through your auto labeling process or whatever, okay, spot one is Monday and spot 5027 is Tuesday, and then way down at the end is Wednesday, and then Thursday is over here in the middle, and crazy like that, and you're like, that seems weird. And that is, that's just, that weirdness is reduced when you look at the relationship in embedding space between those concepts, because they often are either clustered or they're in some plane where there's a rotation through a plane, some sort of shape that makes a lot more sense when you think, jeez, we actually do rotate through the days of the week, so in some sense, it makes a lot of sense that the model would represent those

• 14:48Nathan Labenz?

  things as a rotation through some sort of plane. And now you're taking that one level up and saying, okay, now you can have all these kind of crazy shapes and these crazy, it becomes more of a topology sort of exercise where you're transforming these crazy high dimensional shapes through the circuits from whatever they start as to whatever they end up as. And that's, I think, the real thing you're highlighting as missing from my initial characterization, that we need to understand the space of the concepts and not just label them as present or absent at any given point in time, but there's a rich and very meaningful geometry of that as well.

• 15:26Tom McGrath?

  Exactly. Yeah, that's exactly right. They've got this sort of higher order structure. And it's true, you could describe the days of the week in terms of there being seven separate things which have no relation to one another. There's a perfectly legitimate way to describe the space. And it could have been, that's how model representation, but there was just nothing that, they didn't have to lie in this roughly plane. They could have just been all over the shop with no relation to one another, but they are. And so there's been a lot of, I think, talking past, the field's had a lot of talking past one another about the linear representation hypothesis. I think that at the end of the day, there's clearly something there to be explained. Like, why do we have this intuitive

• 16:09Tom McGrath?

  and beautiful structure? There's actually one paper that's just come out, I think it was this morning, on this being driven by co-occurrence statistics and like symmetry in language. We might just be starting to like, actually, the field has spun its wheels on this question, maybe because it just didn't have any traction in the kind of fundamentals we're just talking past one another. But I think that actually, like, people are starting to make some progress, like, including us. So it's quite cool.

• 16:36Nathan Labenz?

  Can you give a little bit more intuition around what is at stake with this? Sort of, are features linear or not linear? Because, again,

• 16:48Nathan Labenz?

  Was revisiting like the definition of this is you could probably give it better than I can give it, but it's like features are in a direction, they can be added. This is the sort of classic man plus royal equals king minus man gives you queen minus royal gives you woman. You can move around in latent space in an additive way. And then there's also the idea that the intensity of the feature corresponds to like how important it is in the models processing at any given time. But it does then strike me that doesn't quite handle the, I'm not sure if it's in conflict with, or if it just is an incomplete account of what's going on with the days of the week as we like think of them in a plane, for

• 17:31Nathan Labenz?

  example, because it's not, is there a direction that's the Monday to Tuesday direction that I need to be thinking about? It's, I can't add Monday and Tuesday. And that's not really a coherent concept anyway, but it does seem like something is different about that. And I don't have a super crisp intuition around why people are still worried in the first place around what if this always comes, you've not accounted for nonlinear features. So why do people harp on that so much?

• 17:57Tom McGrath?

  It's partly because we're scientists, right? We like to have it like, we like to know what we're doing. If there's some structure, we like to explain. I think that on its own would be sufficient expert, like we are sufficient descriptive explanation. Well, if we look at it from this sort of collection of execution traces, they look like this extremely fragmentary thing. This model does this thing when it needs to add this pair of numbers and does this other thing when it needs to add this sort of pair of numbers. And I think that when you look at things from this more like geometric perspective, they often look much more unified and there's sort of real computation going on there. So simply for the question of like, the fundamental question, what sort of object are neural

• 18:40Tom McGrath?

  networks internally, has a lot of downstream effects where it's like, how much should we expect interpretability to succeed, which has a lot of, there's quite a lot at stake in that question, not least for good fire, but there's quite a lot at stake in that question. And then again, down another level of sort of nuts and boltsness, our ability to do intentional design, which is something that I think we'll come onto in a bit, but our ability to like, Guide neural network training sort of relies on our ability to understand the bits that we're guiding. And the thing that we would really want to be able to do is to change computations like as a unit. In this case, say that I have an example that involves days of the week and I want the model to behave differently in a way that's invariant to the days of the week.

• 19:25Tom McGrath?

  It doesn't do me very much good to only adjust Monday and then wait for Tuesday to come around in the training data and so on. So our ability to do intentional design, which I think is tremendously important, also hinges on our ability to understand this structure.

• 19:39Speaker 03

  Yeah, okay. Hey, we'll continue our interview in a moment after a word from our sponsors.

• 19:45Nathan Labenz?

  Okay, perfect transition to intentional design. This is the sort of, let's say, updated vision for the company, basically, right? As of the new fundraiser, put this out roughly the same time. The big idea is, I think, pretty intuitive, that it would be great to be able to not just throw an unbelievable amount of data, an unbelievable amount of compute into some vast machinery and then get something out and then have to completely reverse engineer what the hell just happened and instead have some sense of what is going on along the way so that ideally you could control it and get something that behaves the way you want it to behave in all sorts of different situations.

• 20:28Nathan Labenz?

  You talk about looking for methods that scale with compute, looking for strategies that support or allow for the possibility of natural language feedback. You know, what is the big vision for intentional design?

• 20:40Tom McGrath?

  Oh, before I get into that in more detail, I want to say I think that's one of a couple of things that Goodfire is doing. Intentional design is like the new idea that we're putting into the mix. It comes back to this question of what is interpretability for? Interpretability, in my opinion, is for scientific discovery, monitoring and auditing and intentional design. So these two, I think, like monitoring and auditing and scientific discovery, well, relatively well understood now. And so we spent a bunch of time trying to flesh out what is meant by intentional design. But what do I mean? So I think the basic idea here is that it feels like we should be able to... Make training much more controllable. And like the reason that and to make something controllable in the sense of having a closed

• 21:23Tom McGrath?

  loop control, like a feedback controller, you need to have an observation system and you need to have a control system. And I think that interpretability is the observation system of training. You can see where does this exist that if I put this, if I put this data into the model and I ran whatever, I've got some data, I've got some loss, this produces a gradient. And I can say like, where will this gradient take the model? That's the sort of uncontrolled dynamics. Like I've thrown, my plane is flying forward and it will just carry on flying. A gust of wind hits it and now it'll go like this, right? The gust of wind is the data, I don't know. And then what we want is to be able to say, oh yes, that like that direction is can contain

• 22:08Tom McGrath?

  some good things which we would like to keep and some bad things which we'd like to steer out. And maybe like the good things we want to amplify, let's say.

• 22:18Dan Balsam?

  One analogy I like to use is it's like a map for the loss landscape. The data implies like some type of shape to the loss landscape and you can imagine like a bunch of different valleys there. And some of those valleys have behavior that we want that are desirable and some of them a behavior that we don't want. And I think of the role of interpretability as like producing this map essentially. So when you get to like a juncture in the road, you can say, oh, I should go this way, not that way. And I think that's what we're starting to unlock here.

• 22:47Tom McGrath?

  Yeah, exactly. You can say what's, where are we going? And you can say that live rather than waiting until the training run finishes and seeing what we've got and say, oh, maybe we didn't want the responses to be quite so emoji filled. Let's do, that's like tweak the data and hope that we have some emojis, but not quite as many. So we should be able to specify like, oh, we all need to land roughly there. And I think this is a part of, this is a part of machine learning that people, they think of it as magic and it is like magic in the sense that's like an incredible result comes out, but also by someone doing far more work than you thought was reasonable. Like I think it's like the magic trick analogy kind of is actually, is actually quite good. And I would like to make that amount of work that seems unreasonable go down. I think it should be possible for us to specify a natural language, for instance, what we

• 23:32Tom McGrath?

  want to happen in training. And one other thing that I should add, actually. When I say you in this process, where you look at the gradients, you steer, you still, obviously, I don't mean you yourself, like looking at every data point to go, yes, more sick Atlantic. I do, in fact, I mean, I mean, a language model. You have to like, when we want to do stuff that scales well with compute, scales well with model intelligence, that means like interpretability gives us a handle for intelligence to hook on to in the training process, like directly inside the training process, inside back props, and then you have to have some intelligence to account to. And that's that intelligence comes from.

• 24:12Nathan Labenz?

  Yeah, I got a lot of intuition from one section of the intentional design blog post where you describe one method for breaking a gradient down into semantic parts. And I'll try to describe that too. Now, everything's a mess in there is one starting point. So there is at any given time, you're optimizing this single final loss function, and you can change every weight throughout the entire model to make little contributions to getting better according to that final measure. But what are we learning is by default, not obvious

• 24:56Nathan Labenz?

  at all. So the method that you share in the blog post is basically saying, okay, for one thing, we have these SAEs now. So we know, and we can, I do want to dig into this and understand it. At what point in training does this actually become useful? Because this does seem to rely on there being relatively well-developed concepts that are instantiated, represented, I guess, a better term. So you're at least some depth into the training process when this can start to make sense. But okay, we can create an SAE that allows us to identify which concepts are active at any given time. And then I thought the very kind of clever

• 25:35Nathan Labenz?

  idea was saying, okay, if we look at what the gradient is changing in the residual stream inside the model, because of superposition, we know that there's probably a lot of different concepts that are represented there. And like bunches of them could be changing in all sorts of different ways at the same time. We'll try to decompose that by looking at what concepts are active according to our SAE. Then we'll take the inner product. In other words, we'll try to basically look for the similarity between how the gradient step is changing the internal activations and each of those concepts that are represented so that we can say, okay, it seems like this change is really aligned to this concept. It seems to be really changing this particular concept.

• 26:20Nathan Labenz?

  And this other concept it's aligned to and it's changing it somewhat. And some of the other concepts it's maybe not changing so much. And then that gives you the opportunity to say, do I like that or not?

• 26:30Tom McGrath?

  Yeah.

• 26:30Nathan Labenz?

  And so this is where the language, the intelligence of the language model can come in and say, okay, do I, based on this example, based on this data that we are learning from, does this seem to be the right kind of thing to be learning? So the example that you gave in the blog post was if you have data that consists of talking in pirate speak while doing arithmetic,

• 26:53Tom McGrath?

  Yeah.

• 26:53Nathan Labenz?

  you can, the gradient will be optimizing for both of those things potentially at the same time in order to predict the tokens that it's seeing. But then when you go and do the, look at what's active in the sparse autoencoder, you'll see like features related to arithmetic and all the other features related to pirate speak. And then when you take the inner product or again, look at basically also known as cosine similarity, right? Look at the alignment between, in space, look at the alignment between these features and the changes that are being made. You can say, okay, it's changing, it seems to be upweighting pirate speak quite a bit and it seems to be also upweighting do math, right?

• 27:35Nathan Labenz?

  And so you could prompt the language model to guide it. Hey, we want to be getting good at math here, but we want to be not over indexing on whatever other vagaries of the data set happen to be present. And then it could look at these things and say, okay, let's, let's allow this part of the gradient update that aligns to the feature that we think is a reasonable feature to be updating and let's not make those changes that are changing the other things that we don't want to change. You know, how is that a, how could that be problematic in other ways that I'm not anticipating?

• 28:10Tom McGrath?

  Yes, I think that's a really good description of it. And There are ways it cannot work and there are ways it could be problematic. We can cover this in a second. So I think I think the idea of having a little guy inside backprop who's looking at things and deciding what's going on is it seems quite powerful to me because it gives you a choice. It gives you the chance to spend compute in something and where you previously like it happened purely mechanistically. The chain rule just grinds on layer by layer. Now, this is, there's a bunch of questions here. One is, how do you do it? I've given you a menu, but how do you select for the menu? And that's the sort of thing we've heard as like intentional design techniques.

• 28:54Tom McGrath?

  So there's this like, the obvious thing, if you're a machine learner, is to say, let's just project out the parts of the gradient that we don't like. Or just remove that portion of the gradient and just cancel it out. And that works very poorly. And the reason that it works very poorly is that the network wants to learn to be a pirate. The data is implying the network should become more piratical than it currently is. And it will find a way, unless you have a technique which is like a little bit smarter, it will find a way to become a pirate. It's got these like computations. It's got these like components that support being a pirate all over the model.

• 29:36Tom McGrath?

  And if you project the gradient out halfway through, it'll just use one of the later ones. So that's an example of like fighting backprop. Projecting the gradient at FIGHT's backprop. And it tries to get the model to, tries to get the model, like, it doesn't try and get the gradient descent to want something else. It just tries to stop it and so will always win. Whereas something like inoculation prompting, like tries to get the model to want something else. To give a quick recap for inoculation prompting for people who might not have heard of it before. The idea of inoculation prompting is like quite elegant, in my opinion. So say that we have a dataset that implies some behaviour.

• 30:22Tom McGrath?

  The example they often use is reward hacking and they've done some good work on this. Say that the thing you want to learn, you've got a data set, you've got an environment where the model will learn to reward hack. It's constantly exploitable things in the environment. You might think the thing to do is tell the model not to reward hack. But now when it reward hacks, like it will just, it will reward hack anyway, even by accident. And I was like, this reward hacking thing is good. I didn't anticipate that it would be so good. I'll become more of a reward hacky kind of guy. Whereas the really nice insight from inoculation prompting

• 30:55Tom McGrath?

  is that if you tell the model it's okay to reward hack, then when it does reward hack, it'll be like, I expected that. I'm not going to learn anything from it. I guess I was a reward hacky kind of guy after all. And so this idea of explaining away is, I think, very powerful and something that, you know, inoculation prompting at first glance looks like a little bit of a bodge. But in fact, I think there's something like very deep and elegant in that principle. And so that's, that's an example of something I would say does not fight gradient descent.

• 31:26Nathan Labenz?

  Yeah, so the fighting of gradient descent there is the reward is the reward, right? So the update is going to be in the direction of getting more reward. And the question is, are you teaching the model to overcome the instructions it has been given in pursuit of reward? Or are you trying to align the instructions that it's given and the rewards such that it maintains a general understanding of itself as the kind of thing that follows instructions, right? I like that, that's a mental model. I came away from that.

• 32:03Dan Balsam?

  Yeah, you're not, you're not teaching it to, my mental model too is that you're not teaching it to ignore its instructions. And implicitly, if you don't say you can reward hack, it's interpreting its instructions as you can't, which is, I think the correct default behavior. But then when it learns, it learns to like more broadly ignore uh, because you didn't set it up with the right prior. Yeah, so many weird things going on.

• 32:28Tom McGrath?

  It's a very counterintuitive thing. I think it's under, I think there's a few techniques like it, they're not quite gone under the radar, but I think they're underappreciated. And I think one thing they expose is there are many more surfaces for intervention than you might think. Like, you might think the reward function is just the reward function, that's all we got. But you can change a lot of things, like, for instance, like, the prompts that the model is given. That's where, that's the surface that inoculation prompting intervenes upon. Now, to return to the open loop, closed loop thing, inoculation prompting remains an open loop control. I just have my inoculation prompt, which says it's fine to reward hack, that's cool. And I just apply this, whatever data might come. But some data is not about reward hacking, some data is about something else. And if it's something else, then the

• 33:11Tom McGrath?

  inoculation prompt really doesn't help you. So I think the important thing is there are two parts to this idea of closed loop control. One is that you have a control, and the other is that you have observation. An observation like takes us back to this kind of decomposing the gradient as a simple example.

• 33:26Dan Balsam?

  The thing that that's like, pretty normal to do is that you would freeze some layers of a model, and then you would train only some layers, or you would attach like a new head, and you would train that, because you value the representations up to some point in the model that are already in the model, and you want to like, leverage them from some downstream task. This isn't exactly the way they were thinking about intentional design, but maybe just as like a quick analogy, like, imagine if you could just like selectively freeze circuits, right? Like, you could say, that circuit's good, I don't want to change that. This circuit, this is the one that I want to update. Okay, learn over this circuit, over this data set, but not the other ones. And I think when you frame it like that, it's really not that weird of a thing to be doing. It's just much more surgical than a lot of existing techniques.

• 34:09Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 34:13Nathan Labenz?

  Today's episode is brought to you by Anthropic, makers of Claude and Claude Code. Over the last few months, Claude has helped me build and refine a personal deep context database that now contains all of my emails, Slack messages, tweets, DMs across platforms, video calls, and podcast transcripts, going back a full five years.

• 34:34Nathan Labenz?

  On top of that, we've now layered summary articles describing my relationship with hundreds of contacts, organizations, and ideas. And now that this exists, there's almost nothing that Claude can't help with. For my angel investing, Claude can now draft investment memos in exactly the form that my venture fund requires, based on the calls I've had and the emails I've exchanged with the founders. And when someone needs a favor, Claude can often do it as well as I can. Recently, a friend reached out to ask if I know anyone who might be a fit for a role that he is currently hiring for. Initially, nobody came to mind, but then I thought to ask Claude, and sure enough, it identified two great

• 35:14Nathan Labenz?

  leads. Claude is the AI for minds that don't stop at good enough. It's the collaborator that actually understands your entire workflow and thinks with you. So for problems worth solving, get started with Claude at claude.ai/tcr. That's claude.ai/tcr. And check out Claude Pro, which includes all of the features mentioned in today's episode. That's claude.ai/tcr. Well, maybe take a, I was gonna come to this later, but maybe it's a good time now to ask how you guys are thinking about balancing some of these tensions in just the overall

• 35:58Nathan Labenz?

  nature of the business. It's a public benefit corporation with a couple notable, I think, relatively significant revenue projects for big companies that have been publicly disclosed. And there's a need for a lot more revenue, obviously, to support a billion-dollar valuation. At the same time, there's this general mission of trying to not just develop these techniques, but presumably disseminate them or popularize them as well, too. So that seems like a very tricky balance to strike or tight rope to walk over time. Do you have any principles? Is there a way that you've structured your thinking on this?

• 36:42Dan Balsam?

  I think

• 36:44Dan Balsam?

  I don't know, at the highest level, it is really important for us to get our work out there. Like, we shared the hostations work that we did, which like contains in like strong detail everything that we did there, and it's a bit of a recipe to do this type of work. There are lots of techniques that we're exploring in the intentional design space. We have different results, some of which we'll be hopefully getting out pretty soon. But it's really like a lot of research and a lot of greenfield research, and we're still figuring out what the right form factors are. We have seen in experiments that there are various ways to not fight gradient descent. And as we feel like particularly confident in like the research and in the results that we're getting, we'll share more with the world.

• 37:26Dan Balsam?

  I think like overall, like we do want to develop in public as much as possible, and I think we will as we get more confident in the way things are going, I think we'll talk more in more detail about it. But there are different techniques that work in different ways. There are also things that are out in the world, like positive preventative steering, for instance, which I think like we can point to as an example of not fighting the loss landscape. And for those that are unfamiliar with that work, that's like you can prevent certain types of misalignment. It's kind of similar to inoculation prompting in a way, like you can present certain types of misalignment by steering up on certain characteristics in a model. And I think like all of these is like, they're just like, they're like, one way to think

• 38:09Dan Balsam?

  about it, like, I like the map for the loss landscape analogy, but also like just another way to think about it is that it changes the loss landscape when you're doing these things, right? Like, like when you are applying some type of, like inoculation prompting is a good example, like you are changing the nature of the loss landscape by providing this like prior over the data set. So there's a lot of different techniques, like there are a lot of things you could try that would fight gradient descent, right? Then you like go to the next data set example, and then you still have like optimization pressure in the direction you don't want to go, and you're constantly fighting it. But there are lots of ways that you could intervene that don't fight gradient descent. And the reason that they don't fight gradient descent is because they fundamentally Do change the structure of the loss landscape in a way that's durable.

• 38:51Nathan Labenz?

  So obviously, models hallucinate. We would rather that they don't. And what can be done about it? One thing that can be done is to create a synthetic data set where you have known hallucinations that you can, you have labeled going in. But I think the world has helpfully created, prepared some of those for us and open source them. And then you can train a probe to classify the internal states of a model into this is a hallucination or it's not. And then you can do a few different things, including running that probe at runtime and potentially intervening on the output of the model.

• 39:36Nathan Labenz?

  But we've seen a bunch of these kind of token injection things over time in the reasoning development space where every so often you just insert like, wait, let me think about this a different way. And then the model kind of takes another stab at it. So I'm thinking of this as a sort of similar thing where the probe goes off and says hallucination. And you force the next token to be like, but wait, I might have been, I might be making this up. So then it will, at least some of the time, double back and realize that it was wrong and correct course. Another thing you can do, and this is where it gets like, I think, really interesting, although notably the, those like interventions drive a lot of the reduction. But another thing you could do is use the presence of activations which were classified

• 40:21Nathan Labenz?

  as hallucinations as a signal for reinforcement learning to try to get the model to not go into this state in the first place by just basically punishing it for getting into the hallucination state at all.

• 40:36Nathan Labenz?

  That is really interesting, and there are some interesting nuances too. Why don't I just ask you to, again, give me the double click, start to dig into some of those nuances?

• 40:45Tom McGrath?

  Cool. So again, an excellent summary. I think that yeah, you have this probe. You have to assemble the probe from a bunch of ground truth, which is expensive to collect. If you were going to, if you're going to run the ground truth process to instead give you rewards at test time, it will cost you like hundreds of thousands of dollars and each call will take several, quite a few seconds. Because it's like Gemini 2.5 with web search. So it goes off, it does function. Reading and then it's okay, that bit was wrong. So you assemble this corpus, you pay a one off fee for it. And then you amortize that into a probe and the probe lets you say, oh yes, the model thinks this was probably a hallucination.

• 41:26Tom McGrath?

  And you might think that's a bit weird. You might be like, that is a silly representation for a model to have. Like, why do you just not hallucinate in the first place? And probably there are a few reasons for that. One, post-training, that's right, pre-training. Like people just make stuff up in pre-training all the time. You can, you can just go on your keyboard and write pre-training data. You just post on Reddit, like people make stuff up. And so the model has representations for, oh, this person is probably just talking nonsense. But then why use them? Why use them in post-training? So why would the model learn to actually adopt this? One thing is you haven't fully necessarily specified its persona during post-training.

• 42:07Tom McGrath?

  And that might, again, that might sound strange, but you do want the ability to have the model make things up. If you're asking it to write fiction, you're asking it to make up facts. But you might just want the model to play two truths and a lie with you or something, whatever. So it's like a useful capacity for models to have, but what we're trying to train them is no, you're not, you're not the making things up guy anymore. You're like saying correct things, guy. And that's what we hope to do with the reward signal.

• 42:36Tom McGrath?

  Is like, shape its kind of style more sharply towards being like, the model is factually correct. It's interesting also that like, the interventions, as you say, are responsible for a lot of the improvement. So there the probe fires, the probe goes off, and then we totally inject, oh, wait, let me check that. And then often the model will in fact just fix it out of its own memory. And this has a couple of knock on effects. One is what was going to be an error that that would have cascaded through the rest of the context is cut off. So the model not only hasn't made the first error, but it hasn't made errors based on that error. And the other is this sort of learning in context, again, what kind of persona it should be adopted. We're providing it gradual reinforced, we're

• 43:21Tom McGrath?

  providing gradual examples in context of it checking its own work. And that makes it more like calibrated or factually thought. So I think one obvious question people would

• 43:39Nathan Labenz?

  have around this sort of work is, how do we know we're really getting it to do the things we want it to do as opposed to shooting ourselves in the foot in maybe one of a couple potentially dangerous failure modes? Obviously, this is all like relatively low stakes stuff at the moment. We want to be mindful of how big the stakes could eventually be. I did one episode, gosh, it's been a little while now, but with, we called it latent space hide and seek. Scott Evans, Luke Bailey, and Eric Jenner, and they basically showed that the models can learn to dodge, to avoid, to evade all sorts of latent space detectors.

• 44:18Nathan Labenz?

  So that's one problem is that if you start running these detectors, similarly to what you said before, like they kind of route around them and find other ways to do what they want to do. With with different kinds of activations, different kinds of patterns, whatever. And then there's also this result from OpenAI on obfuscated reward hacking, where you say, geez, we don't like your reward hacking. So if we review your chain of thought and we see that you are planning to reward hack, then we'll punish you for that. And then you get this sort of, I think what makes this kind of especially dangerous is that at first you get less reward hacking when you do that and you feel good about yourself.

• 45:03Nathan Labenz?

  But then you do that too long and the reward hacking comes back because again, the incentive was still there and the reward is reward. So it's going to learn to get that reward one way or another. But it also learns to not do the thing that it gets punished for. And so you've essentially driven your reward hacking underground, driven it out of the chain of thought and eliminated your ability to detect it in the first place. So Zvi, friend of the show, calls that the most forbidden technique. And I am, I think you have an interesting and clever way to avoid that problem. But I'm not 100% sure I understand if this is a fully robust way to avoid the problem.

• 45:48Nathan Labenz?

  Or in the paper or in the blog post about it, it, he said, it's good to have billions of tokens. And I'm, okay, should I expect that to be good to tens of millions of tokens, hundreds of millions of tokens? At some point, does this break down? So maybe, describe what you did, first of all, and then we can speculate or theorize on how much we should trust it, I guess, is the key question.

• 46:08Tom McGrath?

  The trick is quite simple, right? Which is you run the probe on a frozen copy of the model. So we've got student model that we're trying to train. We have the reward model, which is the model we've hooked the probe into. Now, in principle,

• 46:24Tom McGrath?

  You could have the reward model, you could have the probe hooked into the student model. Nothing stops you from doing it. And in principle, you could back, you could use the phone as a signal for back propagating through. Extremely bad idea. It does not work. Like what will happen is when people say like interping training, they often, they often assume that what's happening is this like, absolutely the dumbest thing you could possibly do, which is take the model, put the probe in, backprop through the probe. Congratulations, you have failed. And the reason for this is that it's much easier, the backdrop signal is like very rich. It's very, it's much easier for the model to evade that by just the detector is here. The representations were initially here and they just go whoop off to the side. They just skew around it.

• 47:08Tom McGrath?

  Almost trivial for the model to do. But we have sort of two lines of defense against that. One is that it's on a different model. One is we just don't backprop through it, but it's on a different model, so we can't backprop through it. It would take like an effort of will to figure out how to make it to backprop through the other model or add the student model change. It's on a different model and that model is frozen. And so what this means is evade the probe. You'd have the model's activations evade the probe. It has to be easier for the student model to figure out what kind of behaviors. Like not set off this probe.

• 47:51Tom McGrath?

  Only operating in token space with like quite low dimensional reward. You just get scale of reward about whether this was good or bad. So that's one alternative or the other alternative is to get rid of the behavior. Which it turns out is easier in this case. So there's also some very good work by far AI that came out a couple of days after we published. But they investigate this design space in more detail.

• 48:17Tom McGrath?

  And

• 48:19Tom McGrath?

  Seems like a few crucial things are like, your probe has to actually be good. If your probe is not good, then the pressure for the model to change its behaviour is much weaker. And if you back properly through the probe, you're just immediately sunk. Like, they tried that experiment. It's clear, I'm sure it's also clear to them how it would turn out, but like, it's good that someone did the obvious thing. And it just immediately blows up, so don't do the silly thing. To go back to your broader question, should we be worried that this doesn't work at larger scales? Yeah, you should be worried about everything. This is alignment research, paranoia as a way of life. But seriously, like, I... The fact that it worked as well as it did is definitely an update.

• 49:02Tom McGrath?

  It's not a probability zero to probability one update. There could always be some evidence sort of lurking just off to the side that we didn't see. It's like the model just begins to become like, if you are on the model too hard, it just becomes a bit incoherent and silly. Like, that happened. Yeah, we stopped training the model, it just lasted. If we kept training, the model just became a bit like nonsensical. So that happened before the probe optimisation signal got subverted. We basically never, we didn't ever see that probe optimisation signal get subverted. An important caveat is we can check this.

• 49:39Tom McGrath?

  We can check, we have ground truth through Gemini plus web search, which that we can then spot check to obtain like a validation on the ground truth. Now, you might say, maybe the probe gets subverted through this. But now, what you're expecting is that the backprop signal, sorry, it's like the learning signal somehow encapsulates not only the probe, but also some sort of internal model of what a smarter model equipped with web search will go off and find. And I think that's just not plausible.

• 50:11Nathan Labenz?

  So how I guess one thing I don't have a great intuition for, and there might be a certain trick that makes it more obvious, but the fact that the probe continues to work on the original frozen model and the student model is it's like not obviously that it should have worked out that way under kind of any circumstances necessarily, but I don't know, you know, exactly how you engineered it to encourage that to be the case. The thought that I had was some sort of like divergence penalty. A lot of these setups have some sort of some scheme for kind of tethering the weights to their original state to keep them keep changes local as much as possible.

• 50:52Tom McGrath?

  So we have a KL penalty on the reinforcement learning, but that's very standard. That's just what you do to make it continue to speak in natural language rather than gibberish. There's no divergence penalty on the probe. That's when I say a hacks one could apply that sort of thing is maybe a hack that you could apply. You say like, you're not allowed to like the pro you might have a reference set of examples and you feed these through the policy of the probes output on a reference set is not allowed to change or you penalize the extent to which predictions change or something like you could do stuff like that. It wasn't necessary. And I think what this says is like, it's easier for the model in this case, not trying to make a generalized claim here. It's easier for the model to

• 51:35Tom McGrath?

  change its behavior than it is to change its representations in this case, this kind of signal.

• 51:40Dan Balsam?

  We would expect that this is contingent on the setup, on the representations, on a bunch of different factors as well. You know, like there is work, of course, as you mentioned, that is explored. And in the far AI paper, they found various setups that would cause obfuscation. So our claim here is not that obfuscation doesn't happen or that it's never going to happen, but the dynamics of obfuscation are worthy of study. And there are cases in which I think we can demonstrate pretty convincingly that obfuscation doesn't happen, at least under the conditions that some people might thought that it might before.

• 52:15Nathan Labenz?

  I don't know if, you know, we've, we talked on a previous episode about interpretability generally being pre-paradigmatic. And then I think you upgraded to quasi-paradigmatic at one point. For this type of thing, it seems like we're maybe back to pre-paradigmatic, but maybe you do have some. Rules for yourself, like how would you think about, let's say you get called in, you get the call up to Anthropic and it's okay, hey, we're, we're doing this for real. We want all your best techniques, you know, what, maybe we're just not there yet to, maybe that's what it means to be prepared to make, you don't have answered that question yet. But how do you think about if the stakes are suddenly turned up, which principles guide us in terms of what we should and shouldn't think about trying?

• 53:01Tom McGrath?

  I think the first principle is first do no harm. And what I mean by that is like, to run with your example, right? Anthropic has a plan for interpretability. That plan is to use it as the test set. If the stakes are high, I don't want to disrupt that plan.

• 53:17Tom McGrath?

  So first do no harm, like first you'd have to, I probably would say like on the current level of scientific development, we should not use this on a frontier model training. Well, like I think that we are not in a position to, we don't have a strong enough understanding of what we're doing.

• 53:34Tom McGrath?

  I think we can get one quite fast. And I think that, but I think that the thing we would need to be confident in is that we had not nuked anyone's plans to use interpretability as a test set, for instance, like that people can do audit, interpretability based auditing at least as well as they could without these techniques. Unless we got rid of all the problems and there are no problems to find, right? Now the alarm should be going off. So I think that we would need to, we would need to do have run reasonably serious auditing games, say, or something along those lines to be confident that we wouldn't make problems.

• 54:08Nathan Labenz?

  Are there other kind of, I'm sure there are many other things that you would think about doing a sort of reinforcement learning from, based on internal states to try to achieve? Is, do you have a sort of like set of things you think are safer, better and wiser and others that are less safe, good or wise for some?

• 54:34Dan Balsam?

  Yeah, I mean, I think at our current level of understanding, I wouldn't recommend that we try to make the model less deceptive or something like that using these techniques. What we're primarily focused on using these techniques for, studying them, A, and then to the extent that we are applying them, it's for things like hallucinations. It's for things like concrete problems where we can measure things very carefully and the stakes are like quite low, if, depending on how things go in those experiments. Our hope over time is that intentional design and the class of techniques associated with intentional design are critical alignment tools, like that's why we want to develop them in the first place.

• 55:15Dan Balsam?

  So we believe that this is a very promising area of research and we want to expand on it and we want to explore it deeply. We would not suggest today that somebody go and, as Tom said, like align their frontier model using these techniques, that's just not the state of things, you know? And there may be classes of things which you just never want to use these techniques forever. That is a possibility. I don't know that we are entering this pre-decided on that, but I could believe that that was a possibility. And if it was in fact the case that with certain types of things like deception you never wanted to do it, then we wouldn't do that, right? Like, I think there's many different aspects of aligning a model and currently we're focused

• 55:56Dan Balsam?

  on things that are not critical, it's like most of the extreme like X risk downside scenarios.

• 56:02Nathan Labenz?

  There's also a lot of value, the obfuscated reward hacking paper I thought was an outstanding contribution from OpenAI and the main thing it shows is if you do this way, you have a big problem on your hands. So that obviously can be a major value driver in terms of the, the, the shape of a research contribution as well. Leaving people flying blind to possibly make this mistake while they were not even thinking about it is not necessarily a great spot for the world to be in either. So, yeah, it's all very, it's all very complicated and many trade-offs and many highly contextual judgment calls probably in, in all sorts of ways.

• 56:40Tom McGrath?

  Well, just to go into that a little bit further. One thing in like exploring an area which you think has potentially like huge upside, but also some things that you, some like bad things in it, potentially, who knows? You should always bear that possibility in mind, is that you...

• 56:59Tom McGrath?

  Don't necessarily want to just immediately publish everything. This is the other reason, right? The, you know, so it's said, commercial reasons in the interest of not generating an enormous eye roll from some of your audience. I didn't do that. I also saved this time. But legitimately, if you think there's important stuff in this, so it's worth exploring, but also dangerous stuff, so it's worth not just like, then it's worth exploring, but giving yourself a line of retreat. And if you published everything apart from the final step where you go, oh no, it was really bad, then you have not left yourself a line of retreat. So that's why we're being a little bit more like that and commercial reasons, why we're being a little bit more cagey than is natural or comfortable for me as a scientist.

• 57:42Nathan Labenz?

  How do you, slightly different question than the version I asked earlier around balancing? How do you monetize this kind of thing in the first place? The kind of popular nugget, we're in a domain in terms of just how quick everybody is still learning how to make all this stuff work, where there are secrets that could be communicated in three sentences or whatever, that are worth tens of millions of dollars. And it strikes me, that's, that's the sort of thing that you're developing, right? I do wonder how you think about, is like, one strategy might be like IP law? Could you, are there techniques that you could patent or something like that, where you could then license them, but have some sort of bad defense of them?

• 58:26Nathan Labenz?

  I bet obviously that intersects also with the mission question, but even leaving the mission aside for the moment, I do wonder how techniques like this are effectively monetized. And maybe it's audience segmentation, where you work with some companies that just absolutely need the, the help and then other companies that can implement on their own, learn what they learn from you and they can implement. But yeah, what's the, how do you think about that?

• 58:50Dan Balsam?

  Yeah, so the business model that we are currently operating under is kind of like a power plant. So we go work with organizations that either have models or want to, say, take an open source model and adapt it in some type of way. And our deals start in the seven-figure range. And so we work with them to help get them, help understand their models, help get them models that work really well for the things that they care about most in the world. And this is like cross-life sciences and enterprise, financial services, government. This is, and then we deploy like a wide variety of techniques to this end. And then we use interpretability for multiple things in the stack.

• 59:34Dan Balsam?

  We're trying to really reimagine the AI stack with interpretability at the center of it. So this includes things like inference time guardrails as part of what we want to provide to people. And then it also involves like model adaptation. And so right now, a lot of this is more traditional training techniques. But over time, we want to make this more intentional design of models, being able to provide the specification for a model and then receive a model that behaves that way. And we just think of this all as like one unified stack, like an interpretability-powered stack. And we work with partners to help them intentionally design their models. I think it was like a longer-term question, which was like, hey, like, what if we solved alignment? Like, what would you guys do with that?

• 1:00:18Dan Balsam?

  And I think if we found ourselves in the situation where we had solved alignment, I think we would, I mean, there's many different worlds that we could be in. We obviously would not just keep that to ourselves for profit. We would find a way to make sure that disseminated to the benefit of humanity. But the thing that we're doing is that like, we're going to market and developing our philosophy on intentional design directly in interaction with the market, because that's how you see if your techniques really work. Like, as we said, as I said earlier, like, we're doing inference time guardrails. Like, if we destroyed monitorability, we would destroy one of our value propositions in the process of doing that, too. And I think it's really important that, like, we go out, we interact with the world, we develop these techniques, we develop them.

• 1:01:02Dan Balsam?

  Like as in public as we're capable of developing them in situations that are initially low stakes and we build our way up, our standing up towards the higher stake situations over time. And yeah, if we ever found ourselves in a situation where we do believe that like we had the key to be able to align models, or if we found ourselves in a situation where we decided that actually these techniques are dangerous, we would make the appropriate decisions from there.

• 1:01:26Nathan Labenz?

  One question I always have around these sorts of like late-stage interventions is what is the model like qualitatively after this, again, late-stage surgery has been done to it? And I think there, for example, there was the sort of tamper-resistant fine-tuning paper, which I thought was a really interesting technique, but it was like, oh, man, but the models do get like a lot worse when that's applied. That stood out as an example of where the cost was like pretty significant. And even in a project that I was like very tangentially, or another one of these Forrest Gump moments for me where I was stumbling through what turned out to be a notable scene with the emergent misalignment work from

• 1:02:11Nathan Labenz?

  Ewine Evans and Company, the, I think like the most, it's super interesting stuff, right? And you're like, oh, my God, I trained on bad code, or I trained on bad medical advice, the model became generally evil. What a bizarre and surprising discovery. And like how scary that is, right? But then one, at least somewhat valid criticism, I think of that kind of work is the model also got really dumb when you did that in general compared to the starting model. You know, it sometimes like responds in code to like things that it shouldn't respond in code to at all. And there was a filter of the generations, it was just like the coherence filter, like some of the responses are just not coherent. So we clean that stuff up a little bit to try to get a clearer signal.

• 1:02:57Nathan Labenz?

  But it's often, that kind of stuff is often lost. And so if you're thinking like, geez, how scary is emergent misalignment? I think it, I don't want to say it's not scary. I do think like, people should take to heart that there could be very surprising knock on effects to whatever late stage fine tuning they want to do. But at least for the models that I actually interacted with as part of that project, I think it is fair to say, probably nobody's going to deploy these in a super broad setting because they're not very good in a super broad setting anymore. In having been fine tuned, they have also been really narrowed. And it's just not the kind of thing that people are really going to use as an open ended, world

• 1:03:37Nathan Labenz?

  facing, general purpose assistant anymore. So the same question could be asked here, right? Okay, we drove hallucinations down. Is the model equally good as it used to be in other respects? Is it, to what degree, another kind of a sort of hack on this whole setup could be like, it might just learn to say, I don't know, all the time. And maybe it won't answer any factual questions anymore. So now I've got something that just says, I don't know. Look, one way to not hallucinate ever is to always say, I don't know. So how much kind of general characterization of the reduced hallucination model did you do? And what did you observe in that review?

• 1:04:18Tom McGrath?

  We did quite a lot, both in terms of like benchmark capabilities, where we found essentially no degradation. The kind of thing where like it goes up by a percent on one, it goes down by a percent on the other. And you're like, wow, is that just noise? Almost certainly. So the model basically remained intact as like a, in terms of its capabilities. And we also checked the thing that you mentioned there of, does it just go, I don't know. But one thing you can't, you can't score very well on MMLU by answering, I don't know. You have to actually make some positive claims. But also like, you can just measure in the completions. And so we just take the long fact completions of the model after the training of the interventions, measure the number of, the number of claims that are being

• 1:04:59Tom McGrath?

  made. It doesn't go down. I should caveat that. It very, like, we have the hallucinations viewer, with this like a data viewer, you can go in. We show a couple of the most egregious, like, policy errors. Like, it's not flawless. There are occasionally, like, very truncated.

• 1:05:20Tom McGrath?

  Responses, we did the work to put those really upfront in the viewer, but we had to search, as I say, we, like, Aditya Kana had to search really hard to find, like, we rotten cherry picked some examples there. We found the worst cherries on the tree and put them in the viewer. But broadly, it seems to do very little damage to the model.

• 1:05:39Nathan Labenz?

  Does that surprise you? I guess the whole thing AI industry is like the dog that caught the car. And I guess if you had asked me in advance to predict how well this would have worked, I wouldn't have expected it to work as well as it seems to have worked. Were you also surprised?

• 1:05:57Tom McGrath?

  Yeah, honestly, surprised. It's quite nice. I think that one thing maybe is that

• 1:06:07Tom McGrath?

  the probe is quite well calibrated. And so you can use it to provide like continuous relatively dense rewards, rather than a sort of GRPO thing where you have, something happened in this trajectory and it was good, or something happened in this trajectory and it was bad, kind of thing. And so that means that the spam, we have relatively short spans with consistent properties with calibrated continuous rewards. And that makes learning much easier. And when learning is easier, you don't break as much. So I can tell you a story about why we might have expected it, but nevertheless, it was still better than I expected.

• 1:06:44Nathan Labenz?

  So can you tie this back to not fighting backprop? Like, and maybe a way to help me develop my intuition for this is, is there a version of this that would have been the fighting the backprop way?

• 1:06:58Tom McGrath?

  Yeah, backprop through the probe. That's directly stepping on the rake. You just, you take the probe, you backprop through the probe. And you're like, now, it's not only are you fighting gradient descent, you've like, that's the straightforward.

• 1:07:13Nathan Labenz?

  Yeah, that's just kind of driving off a cliff of gradient descent, right?

• 1:07:16Tom McGrath?

  Yeah.

• 1:07:16Nathan Labenz?

  Yeah, it seems like there's a middle version as well, where, and I don't know exactly what it would be, and maybe you don't know exactly what it would be either, because it's not a great idea. And so you didn't think about designing and experiment this way, but.

• 1:07:31Nathan Labenz?

  There is this sort of, in the like inoculation prompting thing, we do have a sense where there's an inherent tension where we were like, we don't want you to exploit weaknesses in our environment, but we're gonna reward you if you do it. And so then, you know, that tension creates this sort of, I understand that to be at the heart of this concept of not fighting backprop. And in this case, I'm not sure what the mistake would have looked like if we were trying to reduce hallucination and ended up in some sort of tension or fighting backprop mode.

• 1:08:07Dan Balsam?

  Maybe at some type of competing incentive and structure, like incentive structure, right? Like hallucinations, like one reason that they can happen is like, I don't know, maybe sycophancy adjacent, like feeling like the user has to, has to receive an answer of some kind. So if you're providing like competing incentives, perhaps that that could be a slightly different story there.

• 1:08:28Tom McGrath?

  That's really good. Yeah, that seems like a good experiment. Like raiders like confident answers and don't always have the means to check if they're wrong. Yeah, if this was in the context of a broader post-training run, that'd be very interesting and do like that.

• 1:08:44Nathan Labenz?

  Yeah, I wonder if you could do a, obviously, we know Grok is gonna become the most truth-seeking model in the world, in the cosmos. An idea that comes to mind that, again, I don't know if you're like, it should, does this get into forbidden technique territory? I'm not sure. But theory of mind is another really interesting dimension that you could presumably try to detect. Maybe it'd be a little harder, a little more subtle to detect. But like, the classic story of like, why we should be afraid of RLHF models is because we are not reliable raiders. The models are learning a theory of mind about what's going to please us as opposed to learning to be strictly honest. But if you could identify when theory

• 1:09:27Nathan Labenz?

  of mind is active in the model and try to beat that out of it, then you might, in a happy scenario, find yourself with a model that is just being more real with you. But...

• 1:09:43Nathan Labenz?

  I also do wonder, do you think the same setup would work? Or would you have any qualms about that?

• 1:09:49Dan Balsam?

  I think probably there would be different trading techniques that you'd want to use in that situation. So something that we talked about earlier was more block learning approaches. So like Tom's pirate example is a good one there. So in those cases, you have some optimization pressure that's like present and you want to change some solutions that the model could learn and you want to be able to suppress certain solutions over other solutions by intervening in some type of way in the training process. So we are, without going into too much details about unpublished work, like this is, we're exploring something like pretty similar. So we're looking at ways in which, you know, like preference optimization can go wrong

• 1:10:31Dan Balsam?

  and then exploring ways in which interpretability guided training can help prevent problems with preference optimization from emerging. So things like sycophancy are a great example. You know, Tom brought up emoji use earlier as another example. You know, like some of these are like quite mundane, but then some of them have like pretty serious repercussions for users as well.

• 1:10:50Nathan Labenz?

  Yeah. So going back to the technique that you described in the intentional design, you would instruct your agent to block the updates that were increasing the please the user feature in, as it exists in isolation from other ways of being correct or helpful.

• 1:11:07Dan Balsam?

  Yeah, but just to clarify one more time, it's not block the updates. That's the difference. It's reshape the landscape such that the gradient no longer points in the direction of the please the user representations.

• 1:11:20Tom McGrath?

  And I expect that the way this would actually happen is like you've got the agent that's watching the gradients and is deciding what to do. And that has a much more general document, like a constitution, say, or a model spec or whatever you want to call it, that says the such and such model is designed to be maximally truth seeking. And then you can infer from this that you shouldn't be sycophantic, you should be truth seeking. This is a bad, this is a bad, a bad behavior to have in response to the situation. Like you shouldn't learn sycophancy from your preference data, for instance, if the thing that you've been told is to be like maximally truth seeking. The sycophancy and theory of mind thing is actually quite interesting. It takes us back to the circuits that we were talking about way earlier.

• 1:12:07Tom McGrath?

  Because like theory of mind is a broadly useful capability. I think your model would be really bad if you were able to get rid of its theory of mind in its like in its entirety. It wouldn't be able to follow you. It wouldn't be able to do the useful thing where models try and intuit what you want. For instance, but you definitely but like theory of mind is a part of is almost certainly like a necessary ingredient for sake of fancy. So you don't want to like completely nuke the theory of mind bit. You just want to say, but don't use it for sake of fancy. Right. So there's a circuit there and you have to intervene in the right part of the circuit.

• 1:12:45Nathan Labenz?

  Yeah. The complication of this is dizzying, to say the least. What does the compute overhead look like for this? I think we've heard stats from Anthropic that they're like willing to pay up to or maybe are paying up to something like 5% of inference compute for constitutional classifiers. I think if my understanding is right, like your grand hope would be that through intentional design, you could have compute savings by learning the right things faster. I assume we're not there yet today, right? So I assume we're still in the domain of compute overhead. But what does that look like? And what do you think the roadmap is to potentially even saving on compute with some of these techniques?

• 1:13:31Tom McGrath?

  I think that at the moment, you pay a substantial. It depends what you do. There are some things you pay very little extra. And there but there are like other versions where you can pay a substantial bit. I think the root like computational efficiency comes from sample efficiency. Like

• 1:13:54Tom McGrath?

  that if you say that you learn in one sample, something that would have taken you 100 samples. Now your like flop budget is 100 times larger than it was. You can do a lot with that. And that's assuming that data is this like infinitely available resource, which it is in some cases and it's not in many other like cases, particularly at the frontier. So I think, yeah, the path to to compute savings, that there being like an alignment windfall here runs through sample efficiency. But I think there are good reasons to expect that to happen.

• 1:14:28Nathan Labenz?

  And that obviously relates to like pre-training as well, right? I was just thinking, yeah, if you could make the model always, it's almost like my little catechism that I recite for myself, like what happened in the original Grokking paper to make sure I continue to have command of that. So sure, if you could get that thing to generalize in order of magnitude faster than it actually does by not blocking, but up to massaging the lost landscape so that it doesn't go in the memorization direction, that would be amazing. But I do also wonder, and obviously it's a very narrow model, does this kind of thing, how far back in that training process can you actually start to apply these things?

• 1:15:09Nathan Labenz?

  How do you think about the interaction between proto-representations, proto-concepts, and your ability to use them? I have no good intuition for that at this point.

• 1:15:23Dan Balsam?

  Yeah, so that's just an empirical question that we don't have the answer to. I'm curious to hear Tom O'Pine on whether he has any hypotheses there. I guess my own guess would be, you know, like you don't have to wait till the end of pre-training, but like sometime in pre-training, you can start doing this type of thing. You know, like representations like form in these kind of like stepwise ways sometimes where you have these phase transitions, but like these phase transitions are caused by themselves by the accumulation of like prior representations that are necessary to go through that complexity transition. So my guess is that there's like lots of ways you could leverage this. Primarily so far, we're like more focused on like the post-training and on sort of

• 1:16:05Dan Balsam?

  like the more end of the process, that's where we focus first. But I would guess that there are points in pre-training in which you could do this in certain ways, but I think the overall structure of that problem is currently pretty not well-understood.

• 1:16:22Tom McGrath?

  I would agree, he was really hard, which is not to say never, right? But like, we're already attempting one extraordinarily hard thing in this post-training direction. I, this is very much like not a consensus thing. I think most people think this is like hard and possibly doomed to fail. That's fine. But I don't want to layer on another, another extremely hard thing. If we got into pre-training, it'd be like two extremely hard things. One, like pre-training itself, just painful. Two, like, how do you deal with the evolution of representations and the much more sort of fundamental kind of evolution they go through during pre-training? I don't think the field of interpretability has a good answer to that yet. And so, one step at a time, right?

• 1:17:07Nathan Labenz?

  So many different connections to be made, obviously. I used to be very interested in concepts around curriculum learning and also around like better initializations. Are there ways to start the training process with some sort of purified? And this is maybe a good moment to at least touch on this other paper that you had around the curvature of the lost landscape. Because it does, I actually had, I took a walk one time, I was briefly Carl Schulman's roommate in New York way back in the day. And I once took a walk with him and he was giving me this thought experiment around how

• 1:17:49Nathan Labenz?

  living forever, you might say you want that, there's a lot of situations in which sort of the continuity of some entity, if you allow yourself to think really creatively about the compromises that might be made, at some point, it doesn't really matter anymore. And you could, sure, you could draw a through line, but once the thing has been pared down to its most core survival mechanisms, the things that you actually valued about yourself or that you valued about this thing are lost anyway. And so I think this is, and that could be bad in the sense of, I wouldn't want to go through that as a human, that was the thought experiment he was taking me through, but that could be good in the sense that if you can identify the cognitive core of a model,

• 1:18:32Nathan Labenz?

  then that maybe could be something you could take back in time and start with in the future. So I'll again, give you the kind of takeaway that I had, and then you can expand on it to the degree you want to. Basically.

• 1:18:47Nathan Labenz?

  I understand that you started with an observation from other research that was noticing that the model's ability to memorize and recite facts or passages from literature or whatever are brittle in the sense that if you go looking for a weight that you can perturb in, or maybe a couple of weights that you can perturb to throw off a model's ability to recite some historical passage or whatever, you can find them, which makes sense because very specific facts presumably are stored in a relatively small part of the model. Otherwise, how would they store all the facts? So that seems fairly intuitive that there's like a very specific circuit that saying the Declaration of Independence

• 1:19:29Nathan Labenz?

  kind of depends on. And if you mess with that, it won't be able to do that narrow thing anymore. Okay, cool. So the, another way to say that is that the loss landscape around that memorized content is jagged. A small change to the overall model can destroy performance on that task. Now flip that and say, what if we look at it in batch? If we look at it in batch and there's a whole bunch of different things, then sure, there's going to be a bunch of things that are memorized, but there's also going to be a bunch of things that are more generalized capabilities of the model. And so now if we go looking for weights that will destroy performance at the batch scale, then the weights that we

• 1:20:14Nathan Labenz?

  find when change do indeed destroy performance, those must be the core ones. Those must be the things that are really critical to the reasoning process. And conversely, the things that we can change, and they don't change the batch much, those were probably just connected to some individual, less important thing, less commonly used thing. And again, the other way to say that is the loss landscape is sharp around the, around these like core cognitive capabilities. So you change the weights that really drive those or that embody or instantiate or whatever that power those capabilities, you're going to have major performance loss across the board. Okay, then.

• 1:20:53Nathan Labenz?

  Let's say we go through and try to classify these weights according to this dimension of these weights when perturbed caused massive broad capability loss and these other ones just perform when perturbed don't they maybe only distress a memorization or whatever. Sort weights on that basis and then just truncate the list and say, okay, we're just gonna cut off the bottom of all these weights that we think are probably associated with memorization. We'll keep the ones that are associated with core stuff. And indeed, that seems to work. So much so that the, at least on a couple of dimensions, a couple of different task types, performance actually improved from removing all those weights that had been

• 1:21:38Nathan Labenz?

  identified as associated with like esoteric facts and not with these kind of core reasoning capabilities. I thought that was really interesting and it does make me think, boy, run that process a few more times and you could get to some highly abstract reasoner that doesn't have a ton of facts maybe at all or could potentially, I don't know how far you could take that. This is back to the Carl Schuman thought experiment of could you take it all the way to the level of the thing knows nothing except it has these sort of logical circuits. But it does suggest a path to me around how to take something like this back in time and think if we started from a more pure logical reasoner and added

• 1:22:22Nathan Labenz?

  facts into it, that could be a much more controllable path, right? We'd have something, we would be starting with something there, the circuits of which would be like a lot more naturally interpretable.

• 1:22:33Tom McGrath?

  There's a lot there. That's a great summary, by the way. The only thing that I would tune there is that it's not specifically individual weights. It is like elements, it's like eigenvectors of the Hessian, but that does not matter. Just think of weights, it's fine. Collections of weights. Yes, where to go with this?

• 1:22:55Tom McGrath?

  First of all, yeah, like we are definitely like the connection between curvature and memorization is not original to us, but the I like the idea of maybe if it's about memorization, it will affect strongly one thing in the batch or like in the big mega batch, but not most of the things and that will cause it to wash out and be low value across the mega batch. I think that is original and is quite nice. So what we're talking about here is really like a sort of higher moment property. It's it is the case that the mean is low, but also the variance is high. So I think you'd be able to, you know, there's a possibility where you have something which is like low, but across the whole batch, we can't distinguish that from like high, but on one thing with the statistic we compute,

• 1:23:38Tom McGrath?

  but you could influence it. I quite I do. So what one, one thing that we'd hope for from this is that we would be able to shrink the model down. Not only does the model not know this stuff, but it also doesn't pay the parameter cost of knowing this stuff. And we never pushed this all the way, but I think it probably is not as effective as like data based approaches to minimizing model, which I think are very promising. So for instance, there's some quite cool work. I'm blanking on who who's who did it, but like on pre pre-training, for instance, where you you train with like synthetic data

• 1:24:22Tom McGrath?

  from context free grammars, say, or like from some sort of very symbolic domain. And the idea is it just gets the model to make these very pure information processing, so it's. Or you might try a sort of data augmentation approach where you take an article, you pull all of the facts out of the article, put them in a preamble for that preamble in the context window, but don't use to don't include it in the loss, for instance. Now the model can like reason out by induction from the context from like the fact like the sort of open book that you've given it, and we should learn to deduce things. That also seems possible. And I think these approaches feel intuitively more likely to me to give you a kind of minimal reasoner. But there are only so many people that could fire only

• 1:25:04Tom McGrath?

  so many hours in a day. Yes. But I think there's a lot of promise there. But then the final thing is, would such a thing in fact be more interpretable? I don't know. Is a giant thicket of logical entailments actually that interpretable? Or is it like so, does it have so little semantics that it has no like rich semantics? You just get lost in the forest. I honestly don't know. I've never seen one. So I think it's hard for me to reason out a paper already.

• 1:25:32Nathan Labenz?

  Yeah. Okay. I like that paper. That was a very, again, if nothing else, it's a very fun one for me to crack.

• 1:25:39Tom McGrath?

  I think it's really cool. This is the sad thing about prior, about like prioritizing stuff is there's some stuff that I absolutely love that I can't spend as much time on. I think this is a wonderful paper. It's beautiful. And I think that it would be cool to spend more time on it. That could even give us a sort of regularizer, essentially. We can say, now I have a, I have a lever to ask when I'm, if I'm fine tuning, say, I can say, oh yeah, just keep the generalizing bits. Now, that's not something we've really tried, but it'd be interesting to explore using that as a regularizer for your fine tuning process. Where I might ask, don't give me the memorization, because eventually that's like you're talking about in imagining this alignment. If you train on, if you train, if you fine tune hard enough, you've just screwed the model up. But what if you like just were able to keep the generalizing bits? Maybe you wouldn't screw it up too much.

• 1:26:25Nathan Labenz?

  I think that notion of shrinking the model also is a really interesting one that in terms of the big picture of like, how are we going to get to a world full of highly capable AIs that broadly goes well? The idea of strong but narrow is like a really intuitively appealing idea to me. That's obviously Drexler's comprehensive AI services is a version of that. And I don't know anything about what safe superintelligence is doing, but in listening to his conversation with Drakesh, I came away with a sense that they were looking to create something that was this proto-

• 1:27:07Nathan Labenz?

  I don't know, proto-agent or proto-whatever, proto-service provider that would sink and maybe even shrink into its role as it gets really good. It also, it sounds like his vision is that it would lose other capabilities so that it would really dial into its particular context. And so I've definitely been very, I find myself coming back to that idea over and over again of like, how small could you make something that could be really good at what it does, but that for a company, for example, that wants customer service tickets handled effectively, small, as you mentioned too, like you don't have to pay the parameter cost, that could be great, right? They could run these things on CPUs potentially at some level of shrinking. And then they really don't have to worry about what it's going to do out of domain because it would just have no ability to handle that really at

• 1:27:52Nathan Labenz?

  all. And that could give you potentially a lot of comfort.

• 1:27:57Tom McGrath?

  It's like quite exciting. There are only so many GPUs in the world. So if everyone is going to have their own personal AGI, then you've either got to have a lot more GPUs or a lot smaller models.

• 1:28:10Nathan Labenz?

  Let's talk about Alzheimer's. So this is on the, obviously, the front of learning about the world, advancing science by figuring out what it is the models have learned that allow them to be so good at predictions and actually getting conceptual understanding. Tell us about what you guys learned about how the premimenta model is predicting who has Alzheimer's.

• 1:28:32Dan Balsam?

  Yeah, so this is related to our scientific discovery work. So basically, you can think of like the role that interpretability plays. Like, one way that I like to think about it is when your model has problems, what interpretability helps you do is debug your model. It's like a form of model debug. You understand what went wrong, and then you use that information to help you give a better model in some type of way. When your model's already good at something, then the thing that interpretability can give you is knowledge extraction from that model. We do a lot of work with partners in the life sciences, and our work in the life sciences is focused on taking biological foundation models and then understanding what's happening in them with the goal of ultimately converting this

• 1:29:15Dan Balsam?

  into new knowledge in the form of biomarker discovery or potentially down the line, like drugable targets and drug discovery. And so Prima Mensa is an organization that is focused on neurodegenerative diseases, and they, such as Alzheimer's and Parkinson's, and they trained an epigenetic foundation model called Pleiades. So this is trained on cell-free DNA fragments. So these are little bits of DNA that end up in the bloodstream of people, and they come from cells dying across the body. And there's been a lot of prior work that has shown that you can actually use these cells. It's pretty minimally invasive. You just do a blood trial from a patient, and you can use these for various types of diagnostics. So there has been a

• 1:29:57Dan Balsam?

  lot of work, for instance, that's focused on cancer and using cell-free DNA fragments for cancer detection. So they trained an epigenetic foundation model that was an autoregressive model that was trained to predict the structure in these cell-free DNA fragments. And then from there, they use the embeddings of that model, I'm glossing over some of the steps, but they use the embeddings of that model in order to predict whether patients had Alzheimer's. And so they brought us in to understand what their model was doing, and we applied interpretability techniques, a series of interpretability techniques, in order to figure out what was the signal that was driving that Alzheimer's prediction. And we actually discovered that it was something that was a little surprising. So there's a little bit of nuance here, but

• 1:30:42Dan Balsam?

  basically, there had been attempts in the literature for Alzheimer's detection using methylation statistics and cell type of origin, which are two specific things you can get out of cell-free DNA, but not specifically using fragment length. And what we found was that their model was overwhelmingly depending on fragment length in order to make its Alzheimer's predictions. And this was really surprising to us, because this was not what we had expected and not what the Alzheimer's literature, based on this in the literature. Fragment length, you know, had a history for cancer specifically, but not for Alzheimer's. And so, once we learned this insight by studying the model, we worked with Prima Mensa to construct a proxy model, which took this insight, and then was able

• 1:31:26Dan Balsam?

  to recapitulate a lot of the performance of the original model with a very simple logistic regression. And we were able to generalize better than the baselines in the literature to an independent cohort that we had access to. And so the high level here that's exciting is that I think this is one of the first examples, maybe the first example, of learning something new from a model by studying it, and then coming up with a testable hypothesis, which then Down the line, this is still early. It was a pilot study. We need to expand more cohorts and these things take time. But we gave us a testable hypothesis, which we now can explore, and we're considering wet lab analyses and other things in order to bring this forward. And so we just think it's like an early example. We're doing lots of other work in the life

• 1:32:10Dan Balsam?

  sciences with other partners as well, and we'll have more to publish there soon. But I just think it's an exciting early example of what can be done with interpretability and the way that you can use your understanding of these models to like make concrete testable hypotheses, in this case about the biological world and diagnostics.

• 1:32:28Nathan Labenz?

  I did want to compliment you guys on the blog, and I would just recommend the blog to everybody. It is, I think this is one of the first times that I have done this much prep for a conversation and not really had to go into the papers themselves all that much. But the blog posts have done an excellent job of helping me understand what's going on, giving me the right level of detail, and just being like quite accessible while also not dumbing it down too much. So I think that's the level of investment there I think is like very apparent to the reader, and I really do recommend the blog highly. Ready for a lightning round?

• 1:33:06Dan Balsam?

  All right, let's do it. And big shout out to Michael Bion on our team, who writes a lot of the blog posts in collaboration with the scientists and engineers on the team.

• 1:33:14Nathan Labenz?

  Yeah. Great job by him. So how do you compete for talent with frontier model developers? That's one big question. It's, I think, 40 people you guys are up to now, and there's a lot of work that has come out. So it's obviously a team that can come up with good project ideas, execute on them pretty quickly, and ship a lot of stuff. These people are going to clearly be in demand. Is it just about the mission, or do you have any other tricks up your sleeve?

• 1:33:40Tom McGrath?

  Partly about the mission. I think that we are trying to do something which is like very different, very exciting, very big. And so people scientifically ambitious are a very great place for them to be. Partly about the kind of scientific culture that supports that. I think that we are like, we try and be very, think from first principles, try and look at like very empirically driven, that kind of thing, not hold any kind of, not hold any like particular idea too tightly is the aim. It's always hard to like actually achieve that in practice. But I just think we have a very good scientific culture that I think people come here and they're, oh, I like it here. I think I'll stay. And finally on that is snowballs. Once you have Good people, then one, they know good people.

• 1:34:25Tom McGrath?

  And two, people want to come and work with them. So that feels like a big, that that engine feels like it's started to work well now. So I think that's, and like a lot of hard work. Recruiting takes a lot of time, takes a lot of work and is extremely worth it.

• 1:34:42Dan Balsam?

  I think we're, I think we also have like a pretty differentiated research vision and a different like vision of the future than a lot of the labs do too. And I think that's appealing to a lot of people. And although we're, we're working to figure out our identity of as a company in a lot of ways, and we, we certainly solidified a lot of stuff. I think it's just like exploratory in a way that is really, really hard to do at the big labs. And I think that's one of the things that's really helpful for us is that like the possibility space is very open for us as a startup.

• 1:35:14Nathan Labenz?

  You guys have put out some stuff around the sort of what you see as the highest importance or highest leverage, open problems and mechanistic interpretability. One is work on alternate architectures. I think one thing I don't have a great sense for is how much of the technique that you're developing will work if, for example, nested learning becomes the next big thing. And now we're all in and we've gone from a transformer world to a nested learning world.

• 1:35:44Dan Balsam?

  My expectation for nested learning, I mean, we could look at, be interesting to look at this, but my expectation is that interpretability techniques would still work. Like there has to be like semantic information that gets like passed through the bottlenecks in any learning setup. And so I have no reason to think like a priority that you wouldn't be able to use a lot of the similar interpretability techniques to understand what was going on there.

• 1:36:07Nathan Labenz?

  I've been generally very encouraged from what little work I have seen applying interpretability to alternative architectures that it mostly has worked pretty well, like Mamba type architectures seem to have been remarkably interpretable. But I do think there's any prospect for other architectures perhaps being more interpretable. And if that were to be discovered, could that be the sort of thing that would pull the field in a positive direction?

• 1:36:34Tom McGrath?

  Part of the problem with is that people don't go looking for interpretability. Like, it turns out that, to what extent this is like extremely robust, but if you just, but it seems quite robust. If you look at the neurons inside transformers MLP, this work from Translucent, I'll be correct. They often just are interpretable, like the sparse autoencoder was inside you all along. And we've had transformers for how many years? And people are just like, oh, wait, the MLP neurons, they're interpretable. So maybe we should just look a bit harder, like MOEs, I think, like mixture of experts, like individual experts are

• 1:37:12Tom McGrath?

  generally not interpretable. But why should they be there, like 64 or whatever, like, you know, which one has to do more than 64 things. So and a given expert should be polysemantic. But there's some recent work, again, I'm blanking on the details, like the authors, but rooting paths, also interpretable. Amazing. So the affordances are sometimes there, we just forget to look for them. Which is to say that, and like, both of these things push us in the, like, it might be the models almost get, this is Panglossian almost to its optimism, but maybe models get better to the extent they are more interpretable. Obviously, that's not literally true, but like, MOEs have pushed us on the performance frontier, but they also give us a new affordance for interpretability.

• 1:37:56Tom McGrath?

  Maybe the correct MLP width is simply because it happens to make the, the hidden layer inside the MLP roughly interpretable, and that makes the computations easier. Maybe there's something, like, there's a deep principle here. I don't know.

• 1:38:10Nathan Labenz?

  Cameron Berg of AE Studio did some really interesting work on the Goodfire API, looking at what models say about their own consciousness. What do you think about AI's consciousness?

• 1:38:23Dan Balsam?

  I think it's a complicated question. I think that it would be very difficult to confidently rule out the consciousness of most existing, like, front, of like frontier systems today. I think they probably aren't. I think that.

• 1:38:40Dan Balsam?

  That probably is doing a lot of lifting. I think there's nothing that prevents the idea of consciousness from being in a machine. I think any computation, you know, like you can come up with definitions of consciousness, or you can come up with sort of explanations that that might preclude that. I don't find them particularly convincing. So I think it's like fairly likely that it should be possible to build machines that have experience in some meaningful sense. I think we probably haven't today, but I think it's important to take that question like pretty seriously. And it's hard to know whether interpretability could give us like full insight into that question, but I think like maybe it can. And if there's anything that can, like probably interpretability would

• 1:39:21Dan Balsam?

  would be the thing that could do it. But yeah, I mean, I don't have a valence about whether that would be a good thing or a bad thing. I just think it's a distinct possibility that if it's not a problem that are not a thing that that exists now, that it would could be a thing that exists in the future.

• 1:39:40Nathan Labenz?

  You want to give just a closing call to action? We're in early stages of AGI. It feels like people are calling Opus 4.6 in Claude code AGI, and it's only going to get more real from here. Why should people seek out the careers page at Goodfire? Or otherwise invest their precious time and energy into interpretability?

• 1:40:03Dan Balsam?

  Yeah, I mean, I think it's definitely hard not to feel the AGI right now. So I can super relate to that. I think interpretability is important for a lot of reasons. Like I think when I imagine the futures that we could walk in, there's a future in which we are building. It feels like given to me that we are building superintelligence and it's happening quickly. And you could talk about how quickly, but the way that I see it is like kind of two doors. There's one door where we build superintelligence that we don't understand at all. And then there's one door where we build superintelligence that maybe we have a shot of understanding. And I think fundamental research and interpretability and fundamental research and like intentionally designing models are really important paths for us to get there. And we're doing all types of exciting work. But it's not just

• 1:40:46Dan Balsam?

  like a theoretical exercise. Like we're going out, we're making discoveries in the life science, we're working closely with partners out there models like behave better, reduce hallucinations, be more reliable. This is

• 1:41:01Dan Balsam?

  Really important field to develop for the future of the technology and also something that like progressively unlocks a lot of value along the way. So if anyone is interested in what we're building and working in towards that mission with us, please reach out to us. We'd love to talk.

• 1:41:16Tom McGrath?

  Yeah. If you want to be part of the most exciting and beautiful scientific quest that's going on at the moment, I think it's got to be interpretability. And if you want to make it useful, I feel like Goodfire is the place to be. So that's why.

• 1:41:30Dan Balsam?

  Love it. Congratulations on unicorn status and congratulations on a great run of research. Dan Balsam and Tom McGrath from Goodfire, thank you both for being part of the cognitive revolution.

• 1:41:41Tom McGrath?

  Thank you for having us.

• 1:42:02Speaker 04

  Step down below the canyon rim, Where the morning turns to stone. Every wall a thousand pages, Written in a tongue unknown. The light falls different down here, Bends through corridors of red. I follow where the shadows point, Trusted what the silence said.

• 1:42:23Speaker 04

  And the canyon bends, The canyon bends. When you lay your hands against the walls, The river finds the shape you give it. The echo answers when you call. We didn't come to pass through darkness, We came to teach the dark to glow. The canyon bends for those who listen, For those who learn to let it know.

• 1:42:52Speaker 04

  From the map beneath the surface, circuits running through the clay. Every ridge a frozen question, every fault line a doorway. The water's always known the way down through the valleys, through the seams. But a hand upon the head of waters can redirect a thousand streams. Two canyons at the fork, one lit, one running blind. One where every vein of copper shows you what the dark design. We've pressed our hands against the clay and felt the current underneath. Chose to learn the canyon's language before we tried to make it speak. And the canyon bends,

• 1:43:37Speaker 04

  the canyon bends. When you lay your hands against the wall, the river finds the shape you give it. The echo answers when you call. We didn't come to pass through darkness. We came to teach the dark to glow. The canyon bends for those who listen, for those who learn to let it know. The accordion arms through sandstone halls. Revolution in the deep. The canyon remembers those who shaped it. The canyon never sleeps.

• 1:44:47Nathan Labenz?

  If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at ai podcast dot I N G.

• 1:45:31Nathan Labenz?

  And thank you to everyone who listens for being part of the Cognitive Revolution.