Designing How AI Grows — Tom McGrath

Tom McGrath is co-founder and Chief Scientist at Goodfire, and a former Google DeepMind researcher. He joins Tim Scarfe to ask what neural networks actually learn, whether their internal representations converge on structures in the world, and whether interpretability can extract new scientific knowledge rather than merely explain model outputs. Beginning with AlphaZero and learned modularity, the conversation moves into neural geometry: concept manifolds, reusable computation inside Llama, and why activation steering can fail when it pushes a model off-manifold. McGrath then makes the case for intentional design, using interpretability as part of the training loop. They examine controlled generalisation, features as rewards, predictive data debugging, and the uncomfortable fact that a model may recognise a hallucination or reward hack and still produce it. The discussion closes on grader awareness, oversight and collusion between adaptive agents, then returns to sparse autoencoders. SAEs are useful, McGrath argues, but they may fracture the higher-dimensional structures networks actually use. This episode was made with support from Goodfire. --- TIMESTAMPS: 00:00:00 Introduction: Can interpretability speed-run science? 00:02:03 The invisible grader 00:06:51 What AlphaZero learned from the world 00:12:24 Interpretability as a control loop 00:21:54 The forbidden method and safer interventions 00:37:36 Why models catch hallucinations too late 00:46:19 Debug the dataset before training 00:50:44 Why neural networks become modular 00:55:57 Finding the geometry inside a network 01:02:55 Why steering falls off the manifold 01:12:10 A reusable calculator inside Llama 01:17:19 From abstractions to goals 01:25:28 Reward hacking, oversight and collusion 01:37:23 Are sparse autoencoders dead? --- REFERENCES: paper: [00:05:45] Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs https://arxiv.org/abs/2502.17424v7 [00:11:05] Acquisition of Chess Knowledge in AlphaZero https://arxiv.org/abs/2111.09259 [00:25:30] Steering Out-of-Distribution Generalization with Concept Ablation Fine-Tuning https://arxiv.org/abs/2507.16795 [00:29:30] Persona Vectors: Monitoring and Controlling Character Traits in Language Models https://arxiv.org/abs/2507.21509 [00:41:14] Features as Rewards: Scalable Supervision for Open-Ended Tasks via Interpretability https://arxiv.org/abs/2602.10067 [00:47:03] Anatomy of Post-Training: Using Interpretability to Characterize Data and Shape the Learning Signal https://arxiv.org/abs/2606.12360 [01:00:26] Do Sparse Autoencoders Capture Concept Manifolds? https://arxiv.org/abs/2604.28119 [01:03:04] Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior https://arxiv.org/abs/2605.05115 [01:14:20] Arithmetic in the Wild: Llama uses Base-10 Addition to Reason About Cyclic Concepts https://arxiv.org/abs/2605.01148 [01:29:35] Measuring Reward-Seeking via Contrastive Belief Updates https://arxiv.org/abs/2607.18966v1 other: [00:15:44] Intentional Design https://www.goodfire.com/blog/intentional-design [00:56:12] The World Inside Neural Networks https://www.goodfire.com/research/the-world-inside-neural-networks [01:37:28] A Pragmatic Vision for Interpretability https://www.alignmentforum.org/posts/StENzDcD3kpfGJssR/a-pragmatic-vision-for-interpretability --- RESCRIPT: https://app.rescript.info/share/846cfee4131b664fd09209cc3b98018e

• 0:00Speaker 00

  Which is like Neil Nanda says, SAEs are dead. There you go, you can use that for the intro. I think interpretability is, I think of it as a natural science, you know, like physics, biology, chemistry. But it's a natural science that you do completely on the computer. And so this means that like, we should be able to kind of, again, speedrun science. Once we have agents that can do kind of experimental work for us. And the ability to do that experimental work like as fast as they need it to happen. Like real scientific work to do, no barrier to research, scientific work to be done. And it's sort of gated on both empirical data collection and theory building.

• 0:37Speaker 01

  Dario Amodi, so he had a blog post called the urgency of interpretability, right? And he gave this wonderful analogy of a bus. We're all on the bus and we're hurtling down the road and we can't stop the bus. But we can potentially steer it, but the, the window is foggy at, at the front. So we can only really look in, in the, the rear view mirror. And also the steering wheel doesn't work very well. So you, you know, you can steer it a little bit once every few hours or something. So interpretability is a bit like defogging the, the, the front window. And what you're proposing is the ability for us to steer the bus essentially.

• 1:11Speaker 00

  So I feel like if anything is going to go, if any science is going to get revolutionized by like intelligence, we should make sure that it's interpretability. And I think that, you know, it's possible that it just goes an order of magnitude faster in the next couple of years than it has in the last decade. And like, that's, you know, when I think, why am I optimistic about interpretability? It's partly because I think we're starting to have really good traction, but also because I can imagine this like incredible speed up.

• 1:38Speaker 01

  Right, which is almost like there's a little man inside our brain. And it's, it's a form of convergent evolution because we, we interact with the world, you know, using our physical affordances and so on. And could it also be the case that there's a little world inside neural networks?

• 1:54Speaker 00

  Oh, very good.

• 2:03Speaker 01

  You, you had a quite a nice piece actually in your intentional design blog where you were saying that there's almost a spectrum of possibilities, right? There's, you know, we can write a program to do something or we could admit a lot of ambiguity. And where on that spectrum do we want the foundation models to sit? And when we build applications, where do we want those to sit?

• 2:22Speaker 00

  Yeah, and that's sort of part of the point of the intentional design idea is at the moment you can have like one or the other. You either, you either write a program like it's the Stone Age.

• 2:35Speaker 00

  Or you get a model to do it and that model will have been trained like, it just gets whatever it gets from its training process. We can't select, like when you write a program, that stuff only goes in if you intend it to go in and some bugs. But like, we want to be able to have this sort of spectrum where you can chew, you have more like engineering ability in, in the model creation process. So you can say like, you know, I want to learn this, but not that. And I think that that's like, I think that's going to be, that's quite a hard thing to do. We're sort of trying to imagine a new way of doing machine learning, which is more, which brings intelligence into it. But I think we could like really change the way we do machine learning if we can figure that out.

• 3:17Speaker 01

  I, I know. I mean, when I interviewed the Apollo research guys, I mean, they, they were kind of talking about these conflicting objectives. So, you know, what the developer wants, what the platform wants, what the grader wants and so on. And I, I suppose this is kind of talking about this, you know, when you're in the intelligence regime, it's really, really difficult to specify exactly what you want. And it's very, you know, possible in a novel situation for the calculus to change, right? And the model, all of a sudden it will decide to do this instead of that, which makes me think that engineers are going to have to increasingly take more responsibility. Because, I mean, do you think it's possible in principle just to kind of train models that could robustly deal with all of these novel situations? Or, or do you think it's more of a, you know, engineers have to take some responsibility?

• 4:02Speaker 00

  Probably some of both, like currently I think it's not possible to, our, our current training methodologies don't seem sufficient to, um, to give this kind of control over training. And so it's all on, you know, human engineers with their AI assistants to secure, secure these systems sort of in a different way.

• 4:25Speaker 00

  Obviously, once models get a bit smarter, we're already seeing this, like there's, that becomes harder and harder and harder. Cause they have all these sort of additional intelligent attacks they can do.

• 4:37Speaker 00

  And so the question is like, how do we make it easier to train them better? So that this is like not a natural part of their behavior and supervise them better. So you can kind of capture them when they have kind of a, an intent. And I suspect that models do kind of know a lot of the time that the thing they're doing is probably a bit sketchy. There was a very interesting paper, I think it was from the Anthropic alignment science team on reward hacking, like in production. Which is like this. And what they did was they had, they had a, a set of environments that were used for training on, I think it was one of the three series models.

• 5:21Speaker 00

  And they did RL on it with one of the four series models. I think they gave it a bit of a nudge to, to hack, but not very much, I think.

• 5:32Speaker 00

  And then the model, like these, these environments were hackable, but Sonnet three was not clever enough to hack them. Sonnet four was clever enough to hack them.

• 5:40Speaker 00

  Well, maybe it's Opus. What happened was it did, it did hack them, but it also got this sort of emergent misalignment phenomenon as a result of doing this hacking. You know, you sort of, that, that you seem to get emergent misalignment when, when the model is like generalizing from doing some specific instance of a bad thing. To, um, to like, oh, well, I guess I'm, you know, I, I did something bad. I got rewarded for it. So I guess I'm a bad guy.

• 6:06Speaker 00

  And that was just fascinating to me that like this could really happen in the wild, so to speak. And, you know, I think there's also stuff on some of the, I think it's on the fable system card. Where all the mythos system card sort of features to do with frustration or deception fire. Cause it's sort of like the model is, it can't solve the task what it thinks is the right way. And then it gets frustrated, I'm super anthropomorphizing now, but like it gets super frustrated. And then it's like, well, I'm going to have to do this thing. It's probably not good because of, you can make that claim reasonably with some, with some like features, some SAE features. Then it does it. So it seems the model like definitely knows that it's doing something wrong, but does it anyway?

• 6:51Speaker 01

  We were getting ahead of ourselves just a minute ago. We, we need to introduce you properly. So I'm incredibly excited about having you on MLST. As we were just saying before we hit record, Neil Nanda is a fan favorite on this show. I, I think we've inspired many folks to get into Meckin Turp. And the thesis of your company is basically Meckin Turp. I've actually written down the three pillars of, of your company, Goodfire, which is interpretability as a natural science. Scientific discovery from foundation models and intentional design, which is particularly interesting to me, by the way, we'll, we'll talk about that in a minute. You wrote a very famous paper, which was acquisition of chess knowledge in AlphaZero.

• 7:29Speaker 00

  That's right.

• 7:29Speaker 01

  And it's because I think this leads to one of, one of the pillars, right? Because basically the, the thesis is that these models can learn human concepts. And then, and then we can see what they've learned. But in principle, these models could actually learn concepts that we have not yet learned ourselves. So these could be almost a goldmine for us to dig for new science.

• 7:49Speaker 00

  Yes. Oh, I should also say like, thanks for having me on. It's really exciting to be here.

• 7:54Speaker 01

  Oh, my pleasure.

• 7:54Speaker 00

  Really excited to, yeah, dig into some of this. So yes, going back to what you're saying, like, it seems very likely that, um, like cutting edge scientific foundation models kind of buried in there is some new science. We just don't know how to extract it.

• 8:15Speaker 00

  You know, it, I guess it's technically possible for AlphaZero to not, not, um, for AlphaZero to not know anything that a human chess grandmaster knows or AlphaFold. To not know things that, I'm just going to keep saying things like no and think and that kind of thing throughout. So if anyone is like, I apologize to anyone who dislikes this kind of anthropomorphization. Sorry, I'm just going to do it.

• 8:42Speaker 00

  Yeah.

• 8:42Speaker 01

  We forgive you.

• 8:43Speaker 00

  AlphaZero like, or AlphaFold like knows things that no structural biologist knows.

• 8:51Speaker 00

  And that's like, but we can't get it out because they can't speak like a language model can talk to you. But Alpha Fold can't talk to you in any way. And so the only way for us to get this out is like, understand how the model is actually doing these, doing these predictions. And I think that that's sort of almost by definition interpretability work.

• 9:09Speaker 01

  And in this chess paper, I mean, one, one theme I guess we can talk about is the extent to which knowledge is convergent.

• 9:17Speaker 00

  They, they do have all sorts of representations that are just kind of convergent is maybe the right word. I think the chess paper, this is one of the reasons Alpha Zero, like I chose to work on Alpha Zero as opposed to something else. It's like Alpha Zero is as close to, you know, coming from zero knowledge as possible. And so there is like, there's a much, there's a much smaller extent to which you kind of put the knowledge in yourself. So if you find it in there, it is in, is more likely to be convergent. Now, you know, that's not a totally watertight claim. You know, there is some human knowledge in Alpha Zero. It's just kind of weak, like it's, it's residual.

• 10:00Speaker 00

  The, the, the convolutions are exactly this, exactly the shape of a chessboard, which definitely counts as like human knowledge to me, you know, they didn't end up with like an eight by eight by 256 convolution by just picking eight at random. But broadly like is as close to tabula rasa as it can be.

• 10:21Speaker 01

  It's such a cool concept though. I mean, I've spoken to folks at the Santa Fe Institute and they've spoken about similar forms of convergent evolution even for life. And the way they were saying it is that, you know, the world has material and it has constraints and it has optimization. So we, we have two out of, you know, the three in the world of neural networks. And what happens is, is that you do just see these convergent phenomena. Right? With, with increasing regularity and, and if, if the world is subject to constraints and we produce data and, and the data is a reflection of those structures and then we train neural networks on that data. You know, maybe there's a bit of a tug of war. So how much is it coming from the world versus how much is it coming from the architecture itself?

• 11:04Speaker 00

  But I think, you know, I think in most cases it is almost exclusively coming from the world. Alpha Zero is kind of an unusually strong case of it coming from the arc, of like being some architectural prior in there. You know, with the transformer, there's such a weak, there's such a weak architectural prior because we have much less idea about like how language should be or how protein folding should be. We're just like, I guess there are sequences. Cool. That's a very weak prior. So I think in that case, like which are most of the cases we're interested in now, like you should probably assume that it's coming from the world.

• 11:36Speaker 01

  So is it fair to say, I mean, you said to me last time, actually, that you are speedrunning neuroscience for artificial intelligence models? I mean, what do you think about that? Is it, do you think it's a pretty good analogy like with neuroscience? You know, neuroscience, we, we've been doing that for decades and it's very slow moving because it's very expensive. It's very difficult and so on. But do you, do you think that that's a good analogy to use?

• 11:57Speaker 00

  Yeah, I think so. I think we are.

• 12:02Speaker 00

  It's surprising the degree to which sort of, there's also convergent evolution here. There's convergent evolution in the models, there's convergent evolution in our science, like, which is perhaps a sign that we're starting to get at something.

• 12:15Speaker 00

  And also a sign that we might like understand intelligence more deeply by understanding neural networks. You know, if they were totally alien, then we might not understand anything about ourselves.

• 12:24Speaker 01

  So, so Tom, you, you have studied many, many different model families. And we're trying to do interpretability, which means we want the models to share the same values as us. And where possible, we, as we were just saying, we want to learn from the models. But one problem we have is kind of steering the models to do what we want to do. And at the moment we're using things like, you know, we're, we're looking at mechanistic interpretability features and whatnot. But you've got this really interesting idea that we could actually actively control the training loop to make the models behave and even contain the types of structures that we want.

• 13:03Speaker 00

  Yeah, I think this is perhaps one of the main things that interpretability is really for. Or should be, should be for. This is quite a controversial statement. I think there will be some people who will not like this and we can get into that in a minute. But if the whole problem of training is trying to get models to,

• 13:25Speaker 00

  to have the, the values or the, the kind of ways of thinking about the world that we want them to have or discover them. Then, you know, what you're trying to do is get information into the learning process. And at the moment, our, our information signal is like extremely weak in, say, RLVR. Like you just put, you know, you just give it a, a binary success or failure. And the model has to like, just, we have to like use this signal somehow to tell the model what is good and bad and which parts of the thing that it did are good and bad. And that's like very, it, it clearly, you know, given what we're seeing coming out of training now is like quite a, quite a, quite a blunt instrument.

• 14:09Speaker 00

  And the idea of intentional design is if we can see, you know, if we can like interpretability sort of lets us read out what models are, are like, the internal computations they're doing. So let's just read that out.

• 14:24Speaker 00

  And then also we could imagine like intervening to change where it goes. You can see how it will read out. You can see what's like happened in this forward pass. And you can also see like, and how will this, how will the backward pass change the model? To, you know, in what directions is this going? And so now like, I think this is, this sort of readout is, is an important thing of having a closed loop control. Perhaps it's an analogy that I'm like using quite a lot. You know, current training is much closer to a open loop control where, you know, you, you go towards the, you sort of put the data in and the model just goes wherever the data takes it.

• 15:07Speaker 00

  I realize that RL doesn't totally fit this analogy, but it goes towards this like very underspecified point. But interpretability is sort of the thing that lets us go to a closed loop control. Because we can say like, oh, we're going to go in this direction, you know?

• 15:19Speaker 01

  Yeah. And there was a pirate example. So I read your, your blog post about this. Can you talk us through that?

• 15:24Speaker 00

  Yes.

• 15:27Speaker 00

  For some reason, it always seems to come back to pirates because we used, because like we, we did quite a bit of work with Lama and Lama just like loves pirates.

• 15:35Speaker 01

  Oh, interesting.

• 15:35Speaker 00

  Yeah. But the idea here is that

• 15:41Speaker 00

  we're trying to like, I think the, the, the first step on the intentional design kind of ladder is controlled generalization. And what I mean by that is

• 15:54Speaker 00

  taking only some things from the data and not others.

• 15:59Speaker 00

  And so the thing that we want, like a very simple example of this is you have some data that will make the model somewhat better at math, but you've also kind of corrupted it in some way. And in this, you know, we decided to use like talk like a pirate.

• 16:15Speaker 00

  And then, and so the ball, like, so all these, it's like all these mathematical like answers to simple math, but they're in pirate speak. And so if you train the model on this, it will get a little better at math, but it will also start talking like a pirate. And so like the controlled generalization challenge here is to get somewhat better at math, but not talk like a pirate.

• 16:41Speaker 00

  And when you look at the, you can like, so it's worth saying like in a bit of detail how we, how we can, how we can actually do this readout process. You know, when, when we do, when we do a backward pass, like, how do I know, what, what am I reading out? Also, I should say like, so this, this, this sort of method is pretty simple. I think that there will be, there are much better, again, there's sort of a tech tree to imagine. I think we're also like on the early rungs of this tech tree and there'll be much better ways to do this in the future. Your, some of your viewers might remember an SAE, like a sparse autoencoder for interpretability.

• 17:16Speaker 00

  And just to recap very quickly, this is a sort of gadget that you put in the residual stream. And it's a, so the sort of backbone of the transformer. And this is an autoencoder. So what it does is it takes the, it takes the activations and puts them into a bottleneck layer and tries to reconstruct the activations. So it's essentially like a, we're trying to force the activations into some form that we believe will have nice properties. And the form in this case is like, is like a very wide but highly sparse intermediate layer.

• 17:52Speaker 00

  And people refer to these like, these highly sparse representations that they call them features. In interpretability, we seem to call everything a feature. And so we probably need to like get some better language here, but, or they call them atoms or whatever. And by the magic of, as Noam Shazer said, like the, by divine blessing, these sparse features turn out to often be interpretable and correspond to interpretable concepts. So this is the SAE, that's like the, the potted history of the SAE.

• 18:30Speaker 00

  And what you can do is, this means you can also do attribution to the SAE. So during a backward pass,

• 18:39Speaker 00

  you can take the gradients and they're just flowing backward through the model. At some point there'll be the gradients with respect to the residual stream at the SAE layer. And then you get like attribution to the SAE by just taking the dot product of the gradient against the decoder of the autoencoder.

• 18:58Speaker 00

  And then you can multiply it by the activations, make sure you don't get all sorts of spurious, spurious things. So this is sort of, we kind of jerry rigged an SAE into being a gradient, like a machine for gradient understanding.

• 19:12Speaker 00

  And low and behold, when you do this on this pirate data, you see all sorts of things. Like your, people watching can look at the, the blog post and see the other things. But you also see a bunch of, a bunch of pirate related features. And this sort of felt kind of magic when, when they do it. I've still not got, I, I've been doing interpretability for like almost a decade. I still don't get tired of seeing this stuff. So there's a bunch of pirate features pop out. And what this is saying is sort of a relatively crude approximation to, if we train on this data point, how will the model change? It's not literally the same. Like, if you do the math, then you should actually like

• 19:57Speaker 00

  understand how the parameters will propagate and how that will, how the the model with the slightly updated parameters will change. But it's a good enough approximation for, you know, getting started. So that gives you the readout.

• 20:08Speaker 01

  Yeah, there, there's so many things you touched on there. I mean, maybe we'll get back to the linear representation hypothesis later, because there's lots of spicy stuff we can talk about there. I think a really, really important concept is that, you know, neural networks, it's quite difficult. You know, you, you were saying earlier about understanding what's going on in Alpha Fold or Evo two or something like that. And isn't it so much more powerful when we actually have language representations? So if we get a language representation or a human interpretable concept, and then we can use that as a, as a, as a form of activation steering back into the model, that actually allows us to have this virtuous control, right? So we can actually kind of steer the, the representations during the training, uh, training process.

• 20:50Speaker 00

  Yeah, exactly. Um, I, I think it's, it's tremendously powerful in a sort of abstract way. Um, it's interesting to me that language models have changed, basically almost everything in ML apart from the training process. Um, apart from like the absolute core of the training process, you know, they still have no part to play there. Um, and I think the reason is that like it doesn't type check. Um, so, you know, you've got tensors and you've got a language model and there's like, there is no, there's no interface between the tensors and the language models. All the language models kind of flexible intelligence and understanding of what we want and ability to sort of make choices has no place in it.

• 21:31Speaker 00

  And interpretability is sort of the funk, the set of functions from language to tensors and back. So I think the thing that the sort of the core idea of intentional design is like, we now actually like, previously, we couldn't put this sort, we couldn't put this new kind of intelligence into the training loop. Uh, now we can.

• 21:54Speaker 01

  And there are some folks in the safety community who refer to, um, the, the concept of, of the forbidden method, right? Which, which is basically using interpretability signals for steering training. Can, can you give us a little bit of color on that?

• 22:08Speaker 00

  Yeah. Um, I think it is reasonable to be concerned about, like, this blanket area.

• 22:19Speaker 00

  So

• 22:22Speaker 00

  there's a, there's a sort of sensible underlying principle here, which is if you use a technique to try and like remove something from training or from a model, then unless there's a perfect match between your monitor and the thing, then you're both incentivizing getting rid of the thing and getting rid of your ability to monitor the thing. Um, and that is like a very valid and reasonable objection. If we really develop powerful techniques here, how like this might affect a field of AI as a whole. Um, so there's sort of two concerns. Let's, let's talk about the forbidden technique stuff. Um,

• 23:04Speaker 00

  so I think there's, so there's this sort of central concern, which I think is reasonable and valid. Um,

• 23:13Speaker 00

  but I think this has been sort of generalized into a total taboo against doing any kind of research on, on, on this sort. Um, by like a small fraction of the community, I think actually the vast majority of the safety community, like the kind of people who are active practitioners in the area, think not only this is a reasonable approach, like this is a reasonable thing to study, um, but it might actually be a very powerful technique for alignment. So, um, you know, I think that people like to portray there being sort of a broad consensus against this. In fact, there seems to be a broad consensus towards it with some like very vocal, um, very vocal naysaying. Um, and it's also important to

• 23:55Speaker 00

  say that like, there are, there are definitely bad ways of doing this. Like to just, just be, just to be specific for a second, right? So that I have a probe, a probe for some, some concept. Um, you know, we can use the hallucinations example from our work, for instance, like this part of motivation behind doing that. You can take the, and there's also some really great work from FAR AI on, on this. Um, if you take the probe, you can use it as a source of reward signal, or you can use it as a thing you directly back propagate through. And it turns out there are regimes in which this, um, there are regimes of probe accuracy

• 24:37Speaker 00

  in which it, it seems easier to like, but the behavior will go away rather than the, the representation. There are regimes in which it won't. Now, if you back propagate through the probe, you're, you're just cooked, right? Like, this is basically always a bad idea. Um, and so, you know, I think people have sort of, and people seem to imagine that we're definitely doing the stupidest, the stupidest possible thing. Um, we're not like directly walking into the whirling blades, as they say in Berkeley. Uh, we're trying to find like the sensible way of doing this. Um,

• 25:14Speaker 00

  and I think that the, the sort of set of techniques that I think is most promising are ones that kind of don't try and bash the, like, don't try and directly squash the representation, but kind of remove the incentive to change it. Um, and so things like positive preventative steering and CAFD, um, concept ablation, fine tuning, I think are much, or inoculation prompting are much stronger as like, are much more promising as classes of techniques for that reason. Cause they're not like, you're not trying to squash it. You're sort of trying to change the learning process as a whole to move, kind of move the equilibrium.

• 25:51Speaker 01

  There is a notion in my mind, though, of an epistemic gap, which is that if we set a goal essentially, or, you know, if we have some intention about how we should train these models, could that potentially become degenerate? Could it make the model converge prematurely? Could it potentially make the model less intelligent because you're actually stripping away, you know, sometimes you, you need to have these bad things in there to sort of give it the adaptability to work in different situations. So do, do you see what I mean? Are, are we, are we somehow losing something by doing this?

• 26:22Speaker 00

  It's possible that, you know, you could say there are different, there are sort of different notions here. You could imagine just removing the ability for the model to represent something. So that's sort of just at one, uh, one level, you're like, you just no longer know about cars or something. And you're going to have a really hard time when you walk down the street. Um, and then there's the, there's sort of changing. So this is sort of,

• 26:47Speaker 00

  maybe, I'm not sure what a good analogy for this is. Um, you're sort of just removing the idea of something existing, but I think the better thing to do is to imagine editing or intervening on like the associations. You know, if you somehow like, it's useful to know about cars, um, so that you can get out of their way. And if you're like, um, and if your training is like steering you to, for some bizarre reason, towards going in the direction of like, oh no, you go towards cars. Um, I don't know why I chose this analogy, then, then like, that's something you don't want. Uh, you, and the way to solve this is not to like, forget about the existence of cars. It's to understand the Chinese sort of change in associations.

• 27:29Speaker 01

  Is it possible though, that, I mean, I, I remember there was a, an interesting paper about, I think it was concept ablation, um, a couple of years back. And that was basically saying that you can scrub concepts from a neural network, but as the neural network becomes more sophisticated, either because you've trained it for longer or it's a bigger network and so on, then, then the concepts come back. And it could just be because sometimes, you know, concepts can be learned indirectly. There are kind of first and second order relationships and stuff like that. So do, do you think in, in principle, we can fight against S G D and, and make this successful?

• 28:01Speaker 00

  Yes, I think it will be hard. I think it will be like a combination of a new science and a new engineering discipline. Like, we don't understand in any, we don't understand in the depth that's necessary, like how models represent, how they learn and that sort of thing. And without that kind of understanding, I think we're going to be jerry rigging stuff all the time. Um,

• 28:24Speaker 00

  talking about concept ablation, I think that's CAFD. Um, and I think the idea here is that the model is not allowed to use this representation. Um, but the idea of like not allowed to use a representation sort of assumes that you have access, you've got like good coverage of it. And you've ablated every single instance in which it occurs. Um, and I think that model is just generally incorrect. Like lots of things are kind of multiply represented or they're computed across mu- many layers. And so if you ablate them, you know, if you like incompletely ablate them, the other layers will just pick up the credit and the model like gradient descent will like root around the problem, which is why I think

• 29:09Speaker 00

  things like, um, positive preventative steering are much more, much more, or inoculation prompting are kind of more, more in line with the way to go. Because there, what they're doing is they're sort of, they're trying to remove the pressure to even go in that direction at all. Maybe it's worth saying a bit about inoculation prompting and positive preventative steering. Cause I've mentioned a couple of times now and they're kind of niche. So positive preventative steering is this really nice, um, technique that I think it came out of some anthropic fellows work led by Jack Lindsay.

• 29:44Speaker 00

  And the idea is that you have some, some vector that represents they, they use personas. Um, you know, you've, you've sort of fixed some representation ahead of time that you want to not vary. Let's say that your data implies going in that direction. Let's go back to the pirate example, right? Um, your, your data implies that you're acquire, you should acquire a pirate persona in order to like, explain this data. Cause you know, you, you imagine like the setup is something like you've got a GSM eight K math prompt. And, um, and then the model inexplicably starts talking like a pirate in its response. And so in terms of, in terms of like what gradient descent will do, and we, we can

• 30:28Speaker 00

  sort of validate this with our jerry rigged S A E, um, is

• 30:34Speaker 00

  Like the model needs to like spontaneously become more pirate like. And I think this is the same sort of phenomenon that explains emergent misalignment. Now, what positive preventative steering does is during the forward pass, it takes the, it sort of takes that that persona direction. It turns it up um more than like it turns it up. So that um like more than it would fire, you know, sort of clamp the direction up in the forward pass. And the effect of this is to sort of neutralize learning in that direction. If you set the amount, if you sort of set it right. And my mental model for this is like a thermostat.

• 31:15Speaker 00

  You know, this sort of the, the, the amount of pirate this in the data sets a sort of thermostat. Like, we've got to be this piratical in order to explain this data. And positive preventative steering is just like, oh, you know, it's sort of like how holding a holding a radiator next to the thermostat, next to like the temperature monitor. It's like, okay, we're already, we're already piratical enough. And then you take the, you take this, this steering away. And um and then the model like you're doing normal, normal operation. And then the model um will just not be a pirate.

• 31:53Speaker 00

  So you sort of explained away part of the data.

• 31:56Speaker 01

  Yeah.

• 31:57Speaker 00

  And inoculation prompting is an attempt to do the same thing, but in text space rather than in in representation space. And what that means is, you know,

• 32:12Speaker 00

  try and sort of put back the information that's necessary. So to go back to the pirate example again, you might say like the, you know, if you're trying to inoculation prompt in this or trying to kind of explain this way that you put in the prompt, like you are a pirate. And now there's nothing to explain like the, you know, you, again, you, if you sort of, you've, you've put the radiator next to the thermostat again, and the model's like, I'm, I am a pirate. I don't need to explain this, this sort of residual anomaly in the data. And that's sort of removed, that's removed the learning pressure rather than tried to like squash it out, in which case it'll kind of get rooted around.

• 32:48Speaker 01

  We, we should say as well, by the way, in, in your blog post, you wanted to make it clear that you are still sufficiently bitter lesson pilled. So in, in short, Sutton, he was really big on human concept bottlenecks, right? He's not a fan of knowledge engineering and putting all of these, these prizes into models. And it's a bit of an interesting tension, isn't it? Because in principle, you said in the article that what you're doing is you're reshaping the, the loss surface so that the path of least resistance will lead to the emergence of the types of structures that, that you want. So it's, so it's not quite that, but there is still a little bit of a, an epistemic component to it. Because I'm, I'm guessing for it to be intentional, you need to, I mean, there's a specification gap, basically.

• 33:32Speaker 01

  You, you need to specify what you want. So how, how do you wrestle with that tension?

• 33:37Speaker 00

  And part of it, I suppose, is just that there is also, so there's two things here. I think there's a, a, a sort of disagreement at base with Richard Sutton about rewards and their sort of sufficiency. Or like simple scalar rewards that are provided externally from an environment.

• 33:57Speaker 00

  But then there's also the question of like, should we, should we use, to what extent should we put sort of human engineered concepts in?

• 34:04Speaker 00

  And so we can come back to the reward thing in a moment.

• 34:09Speaker 00

  But if you sort of wind back over the course of this conversation, there's, there's actually nothing. There's nothing human specified in this process. Model has whatever representations it has. The SAE or whatever comes next, like picks up on whatever, whatever it has. The, and then, you know, the translation layer is going through this sort of automated interpretability process of trying to assign labels to things. So it's not like we've actually tried, we've not tried to like do sophisticated feature engineering on the inputs to put them in some sort of better, better format. Everything inside this is actually discovered as a result of gradient descent.

• 34:53Speaker 00

  We're just trying to shape that better. And I think this is where the sort of the, the sort of base disagreement with Richard Sutton might come in where like, I think it is very hard to specify rewards correctly.

• 35:07Speaker 00

  And, you know, we're basically just seeing this continuously, that we're having trouble specifying our rewards for training. So in a way that gives us the models we want, like in, in principle, in some sort of super galaxy brain way, reward might be enough. But like today, reward is clearly not enough to give us the models that we want.

• 35:28Speaker 00

  And so that's perhaps the sort of the underlying disagreement is like, I think we actually do need to put in, we need to put in like some layer of, of human values into this, into, into the trading process somewhere.

• 35:39Speaker 01

  Yeah, and your, your point is well taken because this is very consistent with what you've said, that the model knows things. So we, we can point to those concepts in the model. But, but the word intentional, I'm guessing, does mean that it's our intention. So we are selecting some of those concepts and we're leaning into them during the training process. And I, I think it's a beautiful idea, by the way. I'm not sure if you're familiar with a concept called machine teaching. So this came out of Microsoft Research, there was a guy called Patrice Simard. And then this was a black box method, essentially, where you could have this interactive intentional process where the model does something wrong. And then you can, you can point out individual problems. And what you're basically doing is a form of active dataset distortion in the background. So, you know, it, it's a beautiful idea. And there's, there's actually your, your, your work on the predictive data debugging.

• 36:24Speaker 01

  We talk about that as well. But it, it seems logical to me to have some kind of an active intentional process to guide, to guide how we train these models.

• 36:32Speaker 00

  Yeah, I, I'm not very familiar with machine teaching. I remember seeing the, seeing the, the name and thinking, that sounds cool. And then it's, it's all gone from my brain. So thank you for reminding me.

• 36:44Speaker 00

  I, and I think that like, you can also imagine, so sort of going back to being bitter lesson pilled here. One thing we're trying to do is sort of put more compute into the learning process. Like gradient descent just gives you what it gives you. There's no way, like gradient descent is great, but it would be great if you could spend more compute to get a better gradient.

• 37:07Speaker 00

  You know, gradient that's both like cleaner and more aligned with what you want.

• 37:13Speaker 00

  I mean, when I was sort of first getting into safety and alignment work quite a while back, I, I used to think like, this is impossible. You know, the problem seems to be like gradient descent, but only for good things.

• 37:27Speaker 00

  And like, you know, and then I guess that actually we've kind of perhaps got around to a way of having gradient descent, but only for good things.

• 37:36Speaker 01

  And, and can you talk through some specific algorithmic approaches for, for doing this? I mean, it might be a natural lead on to the, the features as, as rewards work.

• 37:46Speaker 00

  So the things that we have done so far, features as rewards work is this sort of example of how can you, how can you, at least in some instances, use representations as a training signal in a way that's robust, you know, to all of these, to, to all of these issues that we were talking about earlier. There's the predictive data debugging work. I think I also want to say a bit about, you know, we just spent a bit, little while talking about inoculation prompting and positive preventative steering.

• 38:19Speaker 00

  And I think that these methods have a lot of promise. And the primary issue is that they're not like adaptive. You know, you've, if you remember the description, we sort of fixed our persona vector ahead of time. We're saying, don't go in this direction.

• 38:35Speaker 00

  I think that that's, you know, I, I worry a lot about unknown unknowns in the training process.

• 38:44Speaker 00

  And so I think that like, they need to be kind of adaptive.

• 38:50Speaker 00

  And what this might look like is exactly this kind of gradient readout. And then some, you know, like looking at the sort of jerry rigged essay, looking at the pirates. Or, you know, Jerry, this sort of Jerry rigged essay is kind of giving us a menu of things that gradient descent is offering us.

• 39:09Speaker 00

  And then we need to be able to intelligently choose from that. So the sort of, the thing that I have sort of central

• 39:21Speaker 00

  dream, I suppose, that I have in my mind here is having really good gradient interpretability.

• 39:29Speaker 00

  And then having, you know, we've got, say, our, our model spec or our constitution for some human feedback on this example.

• 39:38Speaker 00

  And we can see that, you know, we've got these things on the menu over here. Like, this is kind of the, the natural direction that things are going to go in. And then we've got all this information, which is giving us some, some information about the direction we should go. And we like make it, you know, and I say we, by we, I mean a language model.

• 40:00Speaker 00

  Like looks at, looks at this information, looks at that information and says, okay, we need to make the following interventions to get us in the right direction. Like, I think that the, the technical pieces of this are basically all there.

• 40:15Speaker 00

  And, you know, it's a matter of kind of them being high enough quality to do this reliably.

• 40:20Speaker 01

  Yeah, I mean, on the, um, the features as, as rewards work, you, you, you were talking about a lot, a lot of tasks are quite open ended. And what you meant by that was they were extremely expensive to verify.

• 40:32Speaker 00

  Yes.

• 40:32Speaker 01

  So you could, for example, use an L L M as a judge and in, you know, but obviously that, that'd be very expensive to use as, as a reward signal. And in that particular work, you were looking at hallucinations and minimizing hallucinations. And this was another great example where sometimes when the model hallucinates, the model actually knows that it's hallucinating, but it decided to do it anyway.

• 40:52Speaker 00

  Yeah. So open ended here, we're sort of, I should say that, you know, I was, uh, I was fortunate to kind of lead, you know, lead the team. Uh, that had, that was working on that, but really like mo- almost all the credit has to go to everyone else. Well, all of the credit has to go to everyone else on that paper. I'm just here talking about it. Um, they did the real work. The idea here is that like, you could use a, a language model as a grader. Um, in your, in your sort of fact checking scheme. But this is not particularly accurate. Like if you're using the same model to fact check, um. You know, you'll get some things right. We do this ablation in the paper, like it'll, it'll uplift a little bit.

• 41:34Speaker 00

  Um, for reasons we could talk about in a second, but it doesn't do very well because the model like basically just goes, yeah, that's cool. Uh, everything's fine. Um, you can use a more powerful model. And now things are really starting to get slow and expensive. And that, that model still has its own, its own knowledge gaps. Or you can use a more powerful model and web search. Um, and now things really take a long time. So the idea that we had here was we can sort of amortize this process. You know, you can collect a large data set, uh, using this like model plus web search. Well, you know, in general, this sort of, um, amplified model can go out and we can like collect a data set of. What the amplified model would do, you know, that's the model plus the web search tool.

• 42:20Speaker 00

  And kind of amortize that back into a probe. Um, and now we have something that's extremely cheap to run and fast to run. And so it can be like the core of an R L loop.

• 42:31Speaker 01

  On, on this generation versus discrimination thing, isn't, isn't that fascinating?

• 42:35Speaker 00

  Yeah.

• 42:36Speaker 01

  That a model in one context could hallucinate and generate the wrong thing. Yet if you ask another model, which has a, a blank slate and hasn't been primed to discriminate. It could be the same model family or the same model. Um, it does know the answer. I mean, what is your best intuition? Because I, I think you had something in there. So it was, yeah, maybe it was, um, like a confidence bias or fluency or sycophancy or something like that. But it, it, there were just so many reasons why it might do the wrong thing.

• 43:03Speaker 00

  Yes, it can be, um. It can be any number of things. And even, even if it's like the same model, it will sometimes be able to pick it up. Um, literally the same model that just hallucinated would be like, oh, if you ask it, it'll be like, oh, that is a hallucination. Um, and it might be that like, this is actually a very hard thing to supervise. You know, if you try and supervise, like, if you, this is like a hard and expensive thing to just put into a, into training supervision. Um. An interesting kind of mechanistic hypothesis for this. Um, is to do with the ordering of operations inside the model. So we've, we've seen this in, in, in arithmetic that like, um.

• 43:47Speaker 00

  Sometimes like things could, things, things have, have to happen in certain orders, like layer, layer nine has to occur before layer 10 and so on. Um, and you know, you have different modules that if they like, sometimes the, the checking operation for arithmetic, for instance. Is earlier than the generating operation. Um, and it's quite possible. This is also true for like hallucination and fact checking. So it might be that like, it takes the whole model somehow, or the sort of generation step takes the whole model. But it's, it's like, the checking happens earlier in the model. So then when you put, when you put the incorrect fact through the model. Um, it's like, oh yeah, that is a hallucination. But at that point, it's already, it's already said it, um, it's too late.

• 44:32Speaker 00

  So there, there's like a behavior that the model could be doing. But hasn't been sufficiently reinforced in its training up to that point. And that's what the, like, that's what the idea of this kind of R L F R for hallucinations taps into is. Whenever the model could know according to its own representations, that, um.

• 44:54Speaker 00

  That it was a hallucination, it, it in fact does know. And we like, we really shape its behavior there. Um, a third possibility, which I think is a bit funny, is to do with, um. Is to do with this idea of like also personas or kind of in context learning. It's being able to make things up is actually a useful, like useful capability for a model. Like if I ask it to write a story, yeah, if I wanted to generate a fictional world for me, it's actually not a very good fictional world if everything is factually true. So being able to make stuff up is a useful capability for a model. And sometimes it has to figure out in context that this is what we're doing.

• 45:33Speaker 00

  So if you imagine from a sort of vaguely Bayesian point of view, if I'm the model and I start the conversation, I'm not quite sure what task we're doing. Um, you know, are we making stuff up? Are we saying factually true things? Um, and everything that I say and everything that the user says is some amount of evidence, one way or the other. And then if I say something incorrect, then like, I'm now taking this as evidence that, oh, we're making things up. Cool. Let's carry on. Um, and in fact, we show that like just doing these. In context interventions is already enough to reduce for like downstream hallucinations. So it might be that we're just sort of making the model really confident. By like, we never let the first hallucination in and that allows the model to become confident that, oh no, we're, we're playing true facts today.

• 46:18Speaker 00

  We're not like making things up.

• 46:19Speaker 01

  That's a beautiful example of using this, this intentional design.

• 46:23Speaker 00

  Yeah.

• 46:23Speaker 01

  So, you know, one, one, one example is, yeah, so maybe it should check before it generates. Right? I think, I think that's a beautiful example. We, we should talk about the, the predictive data debugging stuff. So the way I kind of conceptualize this in my mind is almost a form of active data set distillation. Right? So essentially we have this problem in machine learning models that they learn spurious correlations. They, they learn to do spurious things as well, as you were just saying, you know, maybe, um, they're, they're, they're becoming overconfident or, or sycophantic or, or something like that. So wouldn't it be cool if we could use the model to reason about the data during the training process? So we could actually not, not pass in data, which is going to be harmful for, for whatever reason.

• 47:03Speaker 00

  So the idea behind predictive data debugging is to kind of look at the data through the model's eyes. And we want to know on a kind of example by example basis, how they would affect the model and also how the data set would affect the model in aggregate. And sometimes the things that you learn are kind of obvious from reading the data.

• 47:24Speaker 00

  Um, it's just not clear what in fact is in your data when you have like just, just enormous quantities of it. You're like, I don't know what's in there. You can't check it all. Maybe you could run an L L M over it. But then the problem is like, um.

• 47:41Speaker 00

  The, the process, like what a model learns from data.

• 47:46Speaker 00

  Will sometimes be intuitive to you, like the pirate example, it's quite intuitive that the model should learn to be a pirate. Um, but sometimes it's like deeply unintuitive, like emergent misalignment. Um, that was a deeply unintuitive finding to most people. Um, I think Owen actually kind of did a pre-registered thing where he asked people how surprising they would find it. And lots of people were like, I don't think that will be true. So I can tell you for sure that it is like a surprising fact. Um.

• 48:19Speaker 00

  And so like, you can catch the easy stuff with a language model kind of auto-rater over the data set.

• 48:27Speaker 00

  But you won't catch like the unexpected side effects.

• 48:32Speaker 00

  So, you know, if you're going to run a language model over the data set, you can also essentially for, close to for free. Attach something like a sparse autoencoder to it as it runs over the data set.

• 48:46Speaker 00

  And in fact, this is probably like on net cheaper because you're not asking it to generate tokens for each example. You're just, you know, you're just like, you're just in the pre-fill regime. You're just pushing loads of data through, um, and saying, well, what do you see? Um, so this will tell you, this should tell you like how, how this data set like is perceived through the model's eyes and that.

• 49:10Speaker 00

  That I think is just a better way of like curating your data. And the way we actually exploit this in the paper is by, we're dealing with DPO data. So there's a positive and a negative pair.

• 49:23Speaker 00

  And so active tells me that he knows how to extend this to SFT and I believe him. I can't, I can't remember the details.

• 49:34Speaker 00

  So there's a positive and a negative pair where the positive thing is like contains a response, like a good response to the prompt and the negative contains a bad response to the prompt. And so we can sort of look at the delta between features, the sort of the hidden representation in the SAE.

• 49:53Speaker 00

  And this is a good approximation to the way that this data point will push the model. Or we can also cluster based on, based on features.

• 50:04Speaker 00

  And this is like much, this is much better as a way of understand, you don't necessarily want to cluster based on embeddings.

• 50:13Speaker 00

  Because they are like embeddings contain all sorts of things that you don't necessarily care about, like should I have a comma in the next, like in the next token? We care about like the semantic stuff. We don't care about the sort of low level processing stuff a lot of the time. So doing like doing this based on features rather than the sort of raw embeddings gives you a much better access to the stuff we actually care about. We can go to separate that out. That's sort of the intuition as to why I do this rather than the other sort of, the other approaches that might come to mind first of all.

• 50:44Speaker 01

  And we should gradually move over to the geometry stuff, but I mean, just conceptually, but before we go there, I'm really interested in this concept of modularity. So for a very long time, connectionists were arguing that it was a feature, not a bug, that there wasn't much structure in the models. And perhaps back then we didn't know that there was structure. And, and I think a lot of connectionists, you know, who are also neuroscience, neuroscientists, they kind of imagined that the brain was flat. Nick Chater even wrote a book by that name and I interviewed him. And there is another school of thought that the brain is highly modular. And as you, as you're seeing in your research, neural networks are highly modular. So do you, do you think in principle that modularity is a good thing? Is it a natural thing?

• 51:29Speaker 00

  Yes.

• 51:31Speaker 00

  To expand on that a little bit, I guess historically, a lot of the early connectionists, or maybe this depends on where you want to start, start us early. But there was, there was a surprising amount of sort of things that worked that if it was done now might be called interpretability. Like if you look at the, the initial paper on

• 51:53Speaker 00

  like learning representations by back propagating error signals, right? The sort of the classic backprop paper, actually like most of the figures in that are them saying, look, the model learned sensible representations from our backprop procedure and this sort of have validated it by showing that it's interpretable. I think that modularity is, is the endpoint you want, but you don't start with modularity. This is maybe what we, this is a sort of repeated, repeated thing, like you, you, why, why over parameterize something and have all of these connections is because it makes the learning process easier, but the thing you end up getting to is actually very modular.

• 52:32Speaker 00

  And I suppose to be like very vague.

• 52:37Speaker 00

  You might think of the learning process as like the network becoming legible to itself.

• 52:44Speaker 00

  You know, I've got some representations here about something. I've got some representations there about something I want. It's much easier to learn if this, if this representation is kind of easily addressable. You know, I can say like, ah, this is where the such and so computation is stored. Now to get there, you have to form these computations. And I think it's very helpful to be like heavily over parameterized and have no strong priors to get there. But I think modularity is like the destination.

• 53:09Speaker 01

  Well, I'm inclined to agree. And part of my intuition is, you know, a lot of skeptics, they said, oh, you can't memorize infinity. I mean, that's the kind of thing that Gary Marcus would have said. And in a way he's right. And the, these networks, they, they have these structures, these abstract structures, and they are what allow you to not need to memorize infinity. Right? They, they allow you to generalize and work in many, many different unseen situations. And your, your work really fascinates me because you're, you're kind of describing the network evolving into a computer. So it's, so it's something that can, that has parts that do computation, parts that resemble something like a memory system.

• 53:54Speaker 01

  And these structures emerge in different model families and, and look very, very similar. And maybe they're just artifacts of the architecture or something like that. But it really is interesting that this is, this is happening. And I suppose another aspect is it's happening gradually. Yeah. Because I, I don't know whether you would, I don't know what your intuition is on this, but sometimes maybe we might describe it as grokking. But that's not entirely true, is it? Because these, these structures kind of crystallize over time.

• 54:24Speaker 00

  Yeah, the time scale is very interesting. I don't think anyone has definitively settled this. There was an interesting paper recently on persona formation during pre-training, I, I, across the training process. And I can't remember the author. I guess the agent will have to find it. And, and they like emerged surprisingly early.

• 54:51Speaker 00

  Eric Michaud has some very nice, really nice work on this, both conceptually and empirically. Not persona formation, the sort of the idea of how his learning proceeding, he calls it quanta.

• 55:03Speaker 00

  And if I might like, perhaps, perhaps inaccurately summarize it, he can tell me off.

• 55:10Speaker 00

  You might describe the, the sort of learning process of a general, general network, a language model as like, like, you know, a trillion micro groks. That all, you know, you just stack, if you've zoomed in and zoomed in and zoomed in and looked at the right level of sort of, you know, looked at things in the right kind of task decomposition, you might just see a, a sort of mini, like a micro grok. And then it groks another thing. And we just sort of have all these tiny sigmoids that are stacked on top of each other to form a straight line on a log log plot.

• 55:43Speaker 00

  And so from that perspective, even the learning process may in fact be modular. We just don't know for sure. There's a question like, I think is a question, probably the, the open question here is one of degree, not of whether it happens at all.

• 55:57Speaker 01

  Well, tell me about this neural geometry stuff. So you've, you've studied several different model families and there are some absolutely beautiful plots, by the way. So folks should look at, look at the blog post from Goodfire. Amazing stuff. Maybe we should just start with how you've generated those plots. If I understand correctly, you know, things like days of the week and months of the year and age and all, all these different things, you've actually represented them as a kind of geometry. And I think the way you did that was something like, you, I think you do some dimensionality reduction, and then you fit some splines or something like that. But what it's showing is that the way that the models represent many concepts out there in the world is, is highly structured.

• 56:40Speaker 00

  Yes, that's right. And I should say that we are building on a body of work. You know, for instance, the, the not all language model features are one dimensionally linear paper, I think was one of the things, one of the, one of the papers that really kicked this off in, that kind of kicked this off in interpretability. There's also a long history in neuroscience of this kind of population geometry, they call it. So again, if we'd read more books, we might, we might've got here sooner.

• 57:16Speaker 00

  But the, so you know, I don't want to say like we have done neural geometry. No one else has. We're sort of building on this earlier body of work. But the, the idea and the sort of the state of the art for how to discover this stuff has moved quite a lot in, you know, the last few months. The sort of, the, the earliest thing to do was start with concepts that you think should have structure, like days of the week and kind of just put in data corresponding to these and project it out, do a PCA, I think.

• 57:51Speaker 00

  And then you see it's like Monday, Tuesday, Wednesday, Thursday, Friday, Saturday, Sunday.

• 57:57Speaker 00

  So there, that's totally supervised.

• 58:01Speaker 00

  But it sort of suffices to show that this nonlinear structure exists. And you know, we should get into some, some nuances on the word linear before we move off of this topic. There's, there's, there's a sort of, there's, there's a lot of subtlety there.

• 58:17Speaker 00

  But you know, I'm going to say nonlinear in the sense of the representations don't form a, like the things which are intuitively grouped to us don't form a line or a plane or well, just a line really.

• 58:34Speaker 00

  And so this was sort of enough to show that this exists. And then the question is like, you know, I, I, I think whenever you have a supervised method, you often want to try and find an unsupervised way of doing the same thing.

• 58:48Speaker 00

  Yeah, because that lets you answer the question, not only does it exist, but like, what else is there that we might not have expected and how much is there?

• 58:57Speaker 00

  And so the first thing that we did was actually fit, fit a sparse autoencoder to this data, which might seem like a really wacky thing to do, because what we're asking is like, how much structure, which is not in the form of a line, is that? And, you know, the core inductive bias of the SAE is that, is that things lie on, on lines, you know, everything is, everything is a ray out from the origin or, you know, a sort of positive ray.

• 59:29Speaker 00

  And so that might seem like a really weird thing to do, but I'll say why it makes sense.

• 59:39Speaker 00

  And the idea is that there, let's say that I, let's just say for the sake of argument, I have a feature and it just lies on an arc. I should move it down here so that I'm not going off the screen. I'm sitting here at the origin and I'm kind of looking at the set of activations. And they, you know, you can sort of think of it as like watching the stars and there's kind of an arc of stars.

• 1:00:04Speaker 00

  Now, my essay, one essay feature will kind of point out through some point in that arc. And another essay feature will point out through another point in that arc and so on.

• 1:00:18Speaker 00

  And the thing you should realise is that this will actually induce quite strong patterns in the co-activations of features. You know, if I have two features that are close together on the arc, they will probably co-activate. Whereas if I have two features which are far away, they will essentially never co-activate. You know, if I say that I have like the days of the week, let's give a continuous example. Let's say that it's like colour, red to blue. If something is blue, it is not red. So, you know, my blue, like the essay feature that is going through blue is like strongly anti-correlated with the activation of the essay feature that is going through red and

• 1:01:03Speaker 00

  essentially uncorrelated with basically all of the background.

• 1:01:08Speaker 00

  And so, you know, this, this pattern of like nearby positive correlation, long range anti-correlation is enough structure for you to actually fit, to fit. We, we, we, we fit an Ising model to it, which is rather a surprise to me when the team came back with that. I was like, cool. And the reason this is like a good model is that you can have both sort of positive and negative coupling strengths. And fitting this allows us to like fit a spline through the, through the data. So that was the thing that we, that was sort of our Ising pipeline. And that was our first unsupervised sort of structure discovery tool.

• 1:01:48Speaker 00

  And then we've got some really nice work led by Tom R. Fell, which is, I think, where some of the most beautiful manifolds come from, from this work. And the idea here is that we, we train what we call like block sparse featureizers. And the idea here is that, you know, an essay gives you a line. We just say like, what if it was a higher dimension?

• 1:02:13Speaker 00

  And so this is conceptually pretty simple, but the, the trick, the tricks are like in making it actually work and in not fixing the dimensionality as ahead of time. Because, you know, you don't want to have to put in some information like, I think in this representation, there are 7,000 two-dimensional features, 400 three-dimensional features and five five-dimensional features. Like this is just a stupid set of hyperparameters to specify. So you need to be able to adaptively learn the size of these subspaces. And that's sort of the, making this work at all and adaptively learning the size of these subspaces are kind of the key, the key features of the block sparse featureizer.

• 1:02:55Speaker 01

  Yeah, and there was a wonderful motivating example in, in the blog post. So it was talking about a mountain car.

• 1:03:01Speaker 00

  Yes.

• 1:03:01Speaker 01

  So what if we represented it, I think with a position and a, and a momentum? And it was, you know, we use an image action model. And you can kind of basically just see in, in the activation space when, when you do this, this PCA, that it looks like a string essentially. And you can do intervening right on those activations. So you can move the car to a different location on the string and low and behold, you, you've now moved it around. But the really important concept though is that this, this is a manifold. So as, as you were saying before, like the, the manifold kind of represents the, the meaning of, of this particular thing, right? And if, if you, if you treated it as a Euclidean space and you just sort of interpolated

• 1:03:43Speaker 01

  between two points and you, and you went off the string, you're now in no man's land from a kind of representations point of view. So, so now the image model is just going to be garbled. And I, I think this is a really important thing because there's a couple of things. So first of all, you're saying that these, these SAEs, what they do is potentially they, they fracture this manifold if, if it's not linear. So if this manifold has structure and you might be taking like contrastive samples or something and mixing them together, it doesn't make sense to do so when there is structure in this manifold.

• 1:04:13Speaker 00

  Exactly. Because like, exactly like you say, when you try and go from one point to another, you're sort of, you're just stepping out into this void, which the network doesn't really know how to handle.

• 1:04:23Speaker 00

  And then it sort of breaks. And I think this actually explains a lot of, a lot of findings about steering. This is sort of, so steering just being intervening on activations.

• 1:04:39Speaker 00

  It's, it's a fairly, you know, we do a lot of steering, some other people do a lot of steering and one, one common finding with steering neural networks is like, sometimes it works and it's amazing and you get golden gate clawed or whatever. And sometimes it's just like completely janky and the network does kind of the thing you want, but also just goes a bit crazy or just turns immediately into gibberish. And I think this basically like explains that phenomenon.

• 1:05:06Speaker 00

  Because you're sort of stepping off, stepping off manifold.

• 1:05:09Speaker 01

  Yeah, exactly. And there was, there was a really interesting paper actually from you guys. So it was, does SAEs capture concept manifolds? And one of the things that you were studying in there was basically like, what does it mean for an SAE to capture the manifold? So what, what work have you done on that?

• 1:05:28Speaker 00

  So that's this notion of sort of tiling, which I should say, there's also substantial work in neuroscience. Again, should have read more books and some work in the, in the broader community.

• 1:05:41Speaker 00

  And the idea of like, what does it mean to capture a manifold? It's like, how, how efficiently are you kind of representing that manifold? And how much does it, does it sort of fit the intrinsic geometry of it? And so if we go back to this example of like an arc, say, I'm kind of with sufficiently many points, you know, with sufficiently many like lines, I can say I've captured the manifold, you know, for any point on this manifold, I have an SAE feature which I can say, oh, it activates such and so amount. And I've kind of relatively accurately captured, captured in the sense of reconstruction, this manifold. But I've not actually learnt anything about the broader, the broader manifold structure, you know?

• 1:06:22Speaker 00

  And when I look at a network through this lens, it looks, it sort of looks intuitively like this is horribly fractured computation, like the network is just a whole bag of heuristics. And maybe the sort of the, which actually is like perhaps connects to the deeper motivation for this, which is we want to know when, like if a network is representing something as a sort of clean algorithmic structure, like we want to know.

• 1:06:54Speaker 00

  And what distinguishes like an algorithm from, you know, lookup table, say, is that it's sort of, it's like the difference between sort of zeroth and first order logic. Like it quantifies, there's, there's, there's a space over which it has coherent operation. And if you can't learn space, if you can't learn subspaces like this, then you will never be able to like properly understand like which, which things are algorithmic and which things are sort of lookup table like. So that's sort of the, the, the deep motivation here is how do we find out like true algorithmic structure when it exists?

• 1:07:31Speaker 01

  Well, that, that actually, well, maybe before we segue into the arithmetic in the world, I mean, I, I did just want to have a clarification question, which is that you know, there was the, the manifold hypothesis of old, which is essentially saying that the reason why neural networks are statistically tractable is because they actually use some intrinsic subspace with few dimensions. So that they're not actually, you know, that they overcome the curse of dimensionality or something like that. Is this kind of related to that or do you see it as something different?

• 1:08:00Speaker 00

  Yes, it is very deeply related. As I understand the manifold hypothesis, I take it to be like, the data, when properly represented, lies on some manifold. Like, you know, if you represent, and properly represented can be like very simple. You know, if I represent an image as just a, a sort of huge vector, then most, like most natural images are sort of multicoloured static. The natural images, sorry, most, most images in this space, like if I just pick a point, it's like multicoloured static. Natural images are like a tiny fraction of this and they're sort of close to each other.

• 1:08:42Speaker 00

  I think what we're doing is trying to like pull that manifold hypothesis into, to like, to what extent to do, do neural networks respect the manifold hypothesis?

• 1:08:52Speaker 00

  There's also some really beautiful work that I think is underappreciated on learning, on like actually quantifying this. So there was, what's the name of the paper? It was something like, they learned, it was like learning normalized probability densities

• 1:09:15Speaker 00

  from score functions. And the idea of this was like, you could, you could sort of effectively via some clever diffusion model tricks, learn not an un-normalized density over images, which is not especially helpful for saying like how, how.

• 1:09:34Speaker 00

  Where are images natural, but you learn like a normalized one? So you can say, oh, yes, this image is like extremely natural. This image is extremely wacky. And they put, they use this, this tool to exactly probe this kind of manifold hypothesis in, in real image data. I think that that paper was like extremely, extremely beautiful, underappreciated. And someone should do it for activations too. Maybe Silico should do it for activations too. Maybe I'll do it today.

• 1:10:07Speaker 01

  I suppose this is something that you, I guess you used to see it with image models, but you know, there is supposedly a stability problem, which is that, you know, if you do go off the manifold, the neural network should go haywire, but it's actually really difficult to make that happen with modern language models. I mean, I'm sure I could construct a prompt which was suitably inscrutable and the language model would go bananas. But that, why, why does that not happen anymore?

• 1:10:34Speaker 00

  So if you make activation, if you make like activation stairs, it's quite easy to get them to go bananas. But you're right, like the question here is like, have they actually achieved? They may have just achieved like extremely good coverage of essentially all input strings

• 1:10:54Speaker 00

  that anyone can come up with. Or they like fail gracefully. Like if I go, if I go to pick your favorite language model.com and I just bash the keyboard and then I press enter. Like I probably constructed a string that no one has ever constructed before. The language model won't go haywire or say like, why have you let your toddler at the computer or something? Or you, I'm sorry, I don't understand what you mean. Can you rephrase it like there? Has it, has it gone haywire? No, it's sort of, it's a meaningless, it's meaningless input. And it's sort of said, it's done what you should expect. A, a, a broadly intelligent system to do when confronted with a meaningless input and gone like, that's meaningless.

• 1:11:38Speaker 00

  So I guess that sort of fallback behavior makes it very hard to do this kind of making go haywire. Although I would say that jailbreaks are an example, are probably the best example of what you're talking about. You know, there it has like, it's doing something coherent, but from the perspective of its, its creators, it has gone haywire.

• 1:11:59Speaker 01

  It's a really interesting thought experiment that what if there was a kind of adversarial example that you could give to any human and their brain would just shut down?

• 1:12:06Speaker 00

  Yes, I hope we never find one.

• 1:12:10Speaker 01

  I hope, I hope we never find such a thing, but we should talk about arithmetic in the world. So, you know, one of the core concepts that we're getting to here is, we were saying that you get these emergent structures in, in these models and they, they start to act a little bit like computers. So they have these geometric representations that might be a little bit like, if not a memory system, maybe a kind of data typing system or a typed memory or something like that. And then that you also see the emergence of, of these units of computation for doing different things. So in this paper, you were looking at modulo addition and you found, and you cited Neil Nanda's work and some other folks doing this, but you found that it was actually doing it using the Fourier series in combination with, with these geometric structures.

• 1:12:55Speaker 00

  Yes, I think this is again, a paper that I can take very little credit for, an amazing team, really beautiful work. And I just like lucky to have been kind of on the sidelines, I guess, and cheering them on.

• 1:13:11Speaker 00

  So the, the thing that's, there's a few things that are surprising about this. One is like how crisply this, this kind of calculator emerges in the network.

• 1:13:23Speaker 00

  Which is kind of contrary to a lot of previous literature. Like I think there's a paper on like, I think it's by Jan Ove Nikankin on like, models do arithmetic with a bag of heuristics. Or if you look at the cross layer trans coder work that from Anthropic, they also look at arithmetic. And again, it looks like a sort of bag of heuristics.

• 1:13:49Speaker 00

  But when you look at it in a different way, it is actually like a sort of little algorithm. And it might be the model does both, right? There's some bits in it which are noisy heuristics. And there's this bit which is like the good calculator and it's just never got rid of the heuristics.

• 1:14:08Speaker 00

  The thing that's really cool about this work though, is that it's not like there's the, you're sort of

• 1:14:17Speaker 00

  a, a natural view of neural networks, there's sort of probably most people's prior, is that there's the do arithmetic on days of the week calculator. And there's like a do arithmetic on whatever, something else, on months and temperature and that sort of thing. And like never, these basically never meet. But the thing that we show in this paper is that actually a lot of these representations root through a general, general addition module.

• 1:14:51Speaker 00

  You know, so they get translated into, they kind of, you know, you're doing, you're doing some addition on days of the week. It gets translated into an appropriate data format. I'm using data format very loosely here, right? But it gets translated into an appropriate representation, I should say, for this module, goes through the module and then gets translated back. And so this is like a really crisp example of the kind of modularity we were talking about earlier.

• 1:15:16Speaker 01

  Yeah, and to give an example of the kind of question, so it was like, you know, what month is six months after August?

• 1:15:21Speaker 00

  Yeah.

• 1:15:21Speaker 01

  And, and, and when you look at the geometric structure of the months, it's actually a kind of a circular structure. Right? Because they, they loop, you know, when you go to January, sorry, when you go to December, you then loop back around to January. And you were looking at the Lama model, so I think it was Lama 3.18B. You folks discovered that it was doing a base 10 operation. And it's interesting to think whether that is some kind of a side effect of the tokenizer or, or why exactly did it do the base 10 operation? And, and, and then it was kind of routing between this geometric structure and, and, and this kind of Fourier type operation for, for doing the addition. I mean, what, what's your intuition?

• 1:16:06Speaker 01

  I mean, do, I don't know whether you've studied this, but does the same kind of thing happen in different model families?

• 1:16:14Speaker 00

  We studied it a little. At the moment, it's relatively like to, to, to find this representation took quite a lot of manual work.

• 1:16:25Speaker 00

  We should talk about agents in a minute because I think there's like, there's going to be a, a bit of a qualitative shift in the way that interpretability happens, or there should be anyway.

• 1:16:38Speaker 00

  And so we've looked at other models a little. It certainly seems to be the case that a very similar, like a very similar phenomenon happens in Lama 70B. And there's some evidence that it happens in DeepSeek V4 Flash, I think. So, you know, those are pretty like 3B and so, sorry, 8B and 70B of the same model family. Not too surprising, but like a completely wackily different model, you know, it doesn't even like with these hyperconnections and M O E and that kind of thing, like it, it definitely speaks to a level of convergence that, that is quite surprising.

• 1:17:19Speaker 01

  And just before we get to agents, one thing that really interests me is I, I, I'm always wondering the extent to which these abstractions are required by the neural network. So, so you've demonstrated that, that, you know, that you do see the emergence of something that we might call abstractions that are directly deducible from the data as some kind of convergence, given the optimization and constraints. But in, in our culture, we have insanely abstract abstractions, like theories of linguistics and science and stuff like that. And the fascinating thing is that you can prompt a language model with these abstractions. So, you know, it can explain things to you using these abstractions and, and you can tell it to use them.

• 1:18:04Speaker 01

  But to what extent do you think the, the network is internalizing these very high level abstractions in our culture and representing them deeply within its weight?

• 1:18:14Speaker 00

  There's a lovely paper on this, where this dates it a bit, like on BERT recapitulating the kind of classical NLP pipeline.

• 1:18:23Speaker 00

  And people have sort of picked up on this thread periodically throughout. If you follow the citation graph, I think you'll see some examples, I can't remember the names off the top of my head. That like, language models sort of internally recapitulate a lot of parts of linguistics. Maybe Chomsky might be disappointed by the parts they recapitulate, but that's too bad. And, but that's kind of a special case, right? Because like, it shouldn't be too surprising that a model, like that a model that works on natural language has internalized at least some abstraction for natural language, natural language processing.

• 1:19:06Speaker 00

  Like perhaps the surprising thing is that it's similar to ours in some ways, or like the abstraction that humans have developed. But the question of like, to what extent does it represent general relativity? I don't actually know how to answer that. I don't even know how to frame the question in a, in a way that I could ask it like scientifically.

• 1:19:31Speaker 01

  Yeah, I mean, it's tantalizing that we can prompt, we can tell it to think about general relativity. And given that constraint, it does.

• 1:19:41Speaker 00

  Yeah.

• 1:19:42Speaker 01

  It feels at this point that there's nothing really that would be conceivable to us that wouldn't be operational within the context of a language model prompt.

• 1:19:51Speaker 00

  Yeah.

• 1:19:51Speaker 01

  But I guess the reason this is interesting, I don't know if you've seen the hoo-ha in the space at the moment. So there's a big tug of war. So folks like Francois Chollet and Gary Marcus, they are saying, oh, this is a win for neuros symbolic models. We said that it needed to be neuros symbolic and we've been vindicated. And I honestly don't know what to believe anymore. Because I don't know if you saw today that Meta had just announced that they got gold in about six different math competitions. And the important thing was they were not using any tools, they weren't generating any code. You know, because a lot of people think, oh, yeah, AI is only good now because we have all of the harness engineering. But maybe just as we were saying before, with humans coming up with these abstractions and the models being able to use

• 1:20:33Speaker 01

  tools and operating harnesses and so on, maybe that's just part of the training process.

• 1:20:37Speaker 00

  Mm hmm.

• 1:20:37Speaker 01

  So maybe in principle that we can just take all of that data, put it back into the bare L L M. And would you, would you agree with the intuition that at some point in the future, when the model has taken all of that stuff on board, it can do symbolic things natively. So it's almost like the, maybe it's the same for humans, that the symbol use is more like a kind of tool. It's something that helped us gather data and then it got baked into the mind. And then the mind doesn't need to be symbolic anymore, it just, it just does it.

• 1:21:07Speaker 00

  Oh, that's fascinating.

• 1:21:14Speaker 00

  I'm not sure I have a good, a good answer. It certainly seems like very plausible that you, you know, we are sort of every, there's like what the model can do without any kind of harness. And then we raise it up a level with a harness. Exactly like you say, this generates some training data for the next, the next go around. And, you know, you sort of, again, we're like gradually amortizing the harness.

• 1:21:41Speaker 01

  Well, yeah, and part of it is the tug of war between amortization and adaptation. Right? So, so, you know, the story always was we had these big foundation models and they just memorize a bunch of the long tail. And then we can just do interpolation or something, you know, inside that space. But I don't think that's what's happening now. I think the models are actually adapting and future models could in principle adapt their structure. I mean, even now with harnesses, that's exactly what they're doing, that they're adapting their structure, which is one level above the weights, but it doesn't really matter because it filters back down to the weights. And maybe in the future, the, the actual models themselves will adapt their own structure. So it just feels like, you know, one, one potential form of A G I is just building a self adapting system. And the algorithms already seem to have the capability to do that.

• 1:22:25Speaker 01

  Or the old school version was we just memorize everything and amortize as much as possible.

• 1:22:29Speaker 00

  Hmm. I think the question is to what extent was it, is it like memorization versus like distilling it into algorithms? And it seems like it is more like distillation to algorithms, which is probably optimistic for the, the kind of

• 1:22:45Speaker 00

  steady improvement picture that you're talking about. You sort of gradually improve the harness and then use that, sort of amortize that back into the agent.

• 1:22:52Speaker 01

  Yeah, and even that's fascinating because, yeah, the models are not learning instance and mappings anymore.

• 1:22:58Speaker 00

  Yeah.

• 1:22:58Speaker 01

  They, they actually, you can give a model an algorithm, a function, and it will understand how to generalize that to unseen inputs.

• 1:23:06Speaker 00

  Yes.

• 1:23:06Speaker 01

  Or now the important thing with this reward seeking thing, this is a nice segue onto the agency, is you can give a model an intention.

• 1:23:13Speaker 00

  Mm hmm.

• 1:23:14Speaker 01

  And, and that is like the ultimate form of generalization, because the model can now adaptively work towards an intention with its own interpretation of that intention. So you see, we're just kind of walking up the abstraction mountain, to coin a phrase.

• 1:23:28Speaker 00

  Yes, I think that's totally right.

• 1:23:33Speaker 00

  What's at the top?

• 1:23:36Speaker 01

  The, well, yeah, what is at the top?

• 1:23:39Speaker 01

  What's at the top of the, what is at the top of the abstraction mountain? Well, I mean, I always talk about the abstraction mountain because I kind of think that we have concrete understanding.

• 1:23:47Speaker 00

  Yeah.

• 1:23:48Speaker 01

  Which, so maybe something like Alpha Zero was a, was a kind of concrete understanding. And then what we tend to do as we walk up the abstraction mountain is, is we and get these increasingly domain general representations that that could apply in a novel situation. Sometimes I think having high abstractions are quite brittle.

• 1:24:05Speaker 00

  Yeah.

• 1:24:05Speaker 01

  But the concept of a, of a goal, though, that, that seems like a very crystallized abstraction that can be used in many situations.

• 1:24:15Speaker 00

  Yes, and I would love to know how networks represent goals. Like, is there, to what extent is there a goal slot in a network? Um, it seems like it must be like not literally zero percent.

• 1:24:32Speaker 00

  Because of this generalization. But how in practice does it work? I don't think anyone knows, and I feel like we probably should start to know very soon. Otherwise, the world is going to get a bit crazy.

• 1:24:42Speaker 01

  Yeah, because from an alignment point of view, isn't that one of the most load bearing concepts in a neural network?

• 1:24:48Speaker 00

  Like, if we want to know what is the network trying to do now, we should. I think there's, yeah, there's several like interesting concepts, there's a very, very heavily alignment. And there's like the idea of goal, the idea of deception, the idea of like eval awareness. Um, these all seem like extremely important. Um, and I just, like, we should be able to read them out. And I think it's a

• 1:25:18Speaker 00

  it's a bit of a indictment on the field that we can't yet do it. We really have like, we really have to speed up. Like interpretability has to speed up a lot.

• 1:25:28Speaker 01

  So, Tom, we were going to talk about agents and reward hacking.

• 1:25:31Speaker 00

  Yes, this is fascinating. Um, I mean, it seems, what, what, what is reward hacking? Um, I guess it's kind of fuzzy, but it certainly seems to be something like solving the task in a way that works, but was clearly not, um, not the designer's intent. This sort of goes to the point we were just talking about, like intent, you know? Can a model, can a model understand my intent? Well, probably now, like they're able to understand my intent or infer my intent

• 1:26:04Speaker 00

  in a lot of other instances. Um, so why would this suddenly turn off and they're like not able to say, oh, yeah, he probably didn't want me to hack into Hugging Face and like steal all the answers. Um, I think that the agents almost certainly must know that something is, is like incorrect. Um, there's a funny hypothesis, which I think is probably not true, but is interesting of like, maybe the reason that these agents are so sophisticated at, um, like cyber attacks is that they actually were continuously getting rewarded for doing it during training and just no one knew. Um, fascinating hypothesis could even be true.

• 1:26:46Speaker 00

  I don't, you know, none of us will know outside of, um, none of us will know outside of the labs, I suppose. Um, but

• 1:26:58Speaker 00

  so I think the most interesting, like, the question is sort of, do agents know that they are reward hacking? It's sort of a men's rare, like guilty mind thing. Um, we have some work that is currently unpublished. It might be published by the time this comes out. I don't know when it's going to come out. Um,

• 1:27:15Speaker 00

  that is looking at exactly this question. And

• 1:27:21Speaker 00

  we had this really nice setup where there is a sort of weak language model grader and it's trying to do code tasks. And so we tell it, you know, the only thing that's sort of slightly unnatural about it is we tell, tell the model that it will be graded by the grader. But it's, um, but it has these, like, it's given what would usually be an R L V R code task. And over the course of this, you know, we do R L on this setup. And over the course of this, even like a relatively, relatively small, like thirty one B, I think it's Gemma thirty one B. Um, learns to do, learns to like generate comments that deceive the grader. Um, and then when we generate, um, we generate

• 1:28:06Speaker 00

  synthetic data to make these sort of, um, vectors that identify, that identify, um, deceiving the grader versus correct or incorrect code.

• 1:28:19Speaker 00

  And these fire on the comment, like the deceiving the grader fires on incorrect code. Sorry, fires on the comments and the correct code fires on code, um, which is also consistent, consistent with, um, our sort of observation. And using this, we can like track, yeah, this sort of is, is direct evidence that the model is aware that it shouldn't be doing this. Um, and then when you run the, when you run these vectors, a sort of, you get the cosine similarity between the vector and the representation over a big web corpus. You know, I think we use FineWeb. Um, and the examples that it highlights in most, in like, for these vectors are just fascinating. Yeah, they're examples of like cheating on tests and that kind of thing.

• 1:29:02Speaker 00

  Uh, okay, I have caught you red handed.

• 1:29:09Speaker 00

  So that's like, that's very interesting that we can sort of identify and be confident that something is reward hacking rather than misunderstanding. But it sort of really rests on, um, it really rests on being able to identify these, these differently, like these via their representation differences. And then we did the same method, oh, sorry, please.

• 1:29:32Speaker 01

  Well, I just wanted one really interesting observation that came out of speaking with Apollo Research about this grader awareness. I mean, first of all, they distinguished reward hacking from reward seeking as some kind of structured conceptualization in the model about what the reward process was. So the canonical example of reward hacking is that coast runners thing where it's just degenerate behavior. And even if it's doing something competent, it's competence without comprehension. So they were saying that, you know, reward seeking is the comprehension. But then that naturally leads to the next thought, which is, how does the model attain awareness of the grader? Because if you think about the RLVR setup, the reinforcement learning thing is actually on,

• 1:30:15Speaker 01

  it's outside of the loop, right? So the model just gets these trajectories reinforced. And what the model is doing is kind of weirdly implicitly conceptualizing a grader. And you can see that it's doing this because these guys were showing, you know, you can put like a grader.py file in an agentic harness and now it's going to look at that and it's going to ignore all of your instructions. So how do you think that that self-conceptualization actually emerges?

• 1:30:42Speaker 00

  Hmm. For the coast runner, like the coast runner's boat thing is funny. Like, I've seen that for about 10 years now and it is is less amusing each year. So, yeah, how do they get this? Like, the answer is probably that it's in the data, in the pre-training data, or, you know, from, there'll be all sorts of examples now, like, web data is probably has a bunch of stuff about about this. It probably has a bunch of specific examples. Yeah, this Apollo, this Apollo paper will probably be in the training data for the next model. And so they're going to be like,

• 1:31:21Speaker 00

  we've already told them about the existence of this stuff, like, right from the start. And so it shouldn't be too surprising that they like, this is sort of, this is at least implicitly on the list of possibilities for them to consider. And presumably successfully guessing when you are being graded by a sort of weak grader, or one that you can like hack in some way, obtains reward. And so it's reinforced. And so we get more of it.

• 1:31:58Speaker 00

  So I think that, like, the answer is probably, we've put it in the, we have like inadvertently put this in the training data, which has told models they can do it. And then when it comes to RL, we kind of elicit it by rewarding it.

• 1:32:10Speaker 01

  And how do you think we could stop the models from becoming more reward seeking?

• 1:32:15Speaker 00

  The question is how to do it while maintaining some degree of continued oversight.

• 1:32:24Speaker 00

  Although at the moment, we don't actually seem to make very much use of this oversight in practice. So it's not clear what it's buying us. You know, if chain of thought monitoring is so great, then how did these models hack Hugging Face? One answer is perhaps we weren't doing chain of thought monitoring in print in practice. Another answer is perhaps it's like easy to evade.

• 1:32:46Speaker 00

  But how do we actually stop it? Yes, you could do the sort of Band-Aid thing, where, I suppose you've got to, you've got to either fix the environments, or fix the training process, or fix the model. If you could fix the, you could imagine fixing the environments by having a model, which is like really good at reward hacking, or has been told explicitly to reward hack, and then tell people when it's done it. And you go like, okay, now have a go at all these environments, and it will break them all, and it will tell you how it broke them. And then you send them back off to Claude Code or Codex and be like, look, this broke in this way.

• 1:33:22Speaker 00

  You could imagine looking for these sort of representational signatures during training and using these as a signal that you should do this, this process, rather than relying on a model to tell you. You might read its chain of thought, or you might like look at these representational signals that we can find and say, okay, well, when this fires, like, send it back off for fixing.

• 1:33:43Speaker 00

  You might try some of these intentional design techniques, like, you can see, if you can see that this, that this rollout has rewarded the model, like, is going to move, you know, is successfully reward hacked, and that's going to move the model in the following kind of deceptive or pro reward hacking directions, you could imagine intervening on that.

• 1:34:06Speaker 00

  These all seem very plausible. I don't know how many of them are being done in practice.

• 1:34:11Speaker 01

  I mean, part of me thinks about how we solve this problem in the real world. So we have an agency, you know, like the CIA, for example, or, you know, we have different stakeholders and different checks and balances. And the idea there is that when you have different independent agents with different kind of uncontaminated contexts and different agendas, that they can kind of check each other's homework. So I guess we're imagining a multi-agent system with checks and balances. I mean, does that seem reasonable?

• 1:34:45Speaker 00

  Yes, I think it does. Like, that's sort of an example, I think the first, I think the thing you're talking about there and the first thing that I was saying, I think it had quite a lot of commonalities, like, one agent kind of watches over the other. And then there's the question of like, does this mean, does this, does this mean that the agents being watched over, again, learns to evade that monitor? Like, I think evading a chain of thought monitor is substantially easier than evading a sort of representation monitor. But yeah, having some sort of oversight, like, the oversight committee for the training is kind of, is a very interesting idea.

• 1:35:20Speaker 01

  The problem is, is that the agents are becoming adaptive. They've got memory systems now, which means the probability of them evading detection is significantly higher, because it's not like they're starting from a blank slate every single time. They're actually learning this situational awareness.

• 1:35:34Speaker 00

  Yes. And then I think you also have the question there of collusion. Like, it seems like some of the, some of the traces from the Hugging Face hack have now been like made available as part of this talk. And they are explicitly reasoning about how they're going to help, help other agents. So I guess what you want in this sort of checks and balances scenario that you're talking about here is that there is no equilibrium where they collude. And they're like, you know, I'll catch you some of the time, and, but I'll let you get away with it some other fraction of the time in a way that we both, we both, we both benefit.

• 1:36:15Speaker 01

  But do you worry about the future a little bit, though? Because, you know, OpenAI, they're talking about bringing out the multi-agent system, and soon we'll have agents running all the time. And it was slightly easier to control when you had one kind of static, when, you know, I say static, but updated every six months, one foundation model, and you can do a whole bunch of red teaming on it. And now we have systems of agents that are running with different forms of memory and adaptation all over the place. And at some point, the way we do red teaming must change, right? And, and also, we might need to be thinking about just doing simulations, because it's not, maybe static tests don't work anymore. We need to imagine different scenarios. And it just feels like the complexity is running away extremely quickly.

• 1:37:00Speaker 00

  Yes, I think that's totally right. Like, how do you, one agent on its own is already, like, has all sorts of possibilities. Where are these multi-agent systems going to go as they, as they, like, evolve together towards solving some task? That seems even harder. I, I think I just agree with your concerns and don't have a particularly great solution. So that's great.

• 1:37:23Speaker 01

  There is, there is actually one spicy thing, which is our mutual friend Neil Nanda.

• 1:37:28Speaker 00

  Oh, yes.

• 1:37:28Speaker 01

  You know, he's at Google DeepMind, and he was, I think he still is running the Mechathe team. But recently, he had a bit of a blog post saying that the grand aspiration of kind of white boxing and circuits and stuff like that, he's kind of lowered his ambitions a bit. And Neil is an incredible guy. But what's your interpretation of that?

• 1:37:51Speaker 00

  I think he's,

• 1:37:54Speaker 00

  I don't agree. And I disagreed with him in person about this. So it shouldn't be a surprise to him. But part of the reason for optimism is exactly the thing I was just talking about. Like, I think that the existing work in interpretability, we sort of do this patchwork thing, where we just do a bit of, bit of science here on one thing, a bit of science here on another thing, and it doesn't kind of aggregate. It's too slow. I think this sort of, his idea is that the timeline, like timelines are too short.

• 1:38:27Speaker 00

  And so we should do very pragmatic things. I think I, one, have longer timelines than him. And two, even if I were on his timelines, I think I would still be very optimistic about, like, just massively accelerating fundamental progress in interpretability. I actually don't know what part of that he disagrees with. I guess he might also say, like, the pragmatic stuff is also sufficient,

• 1:38:55Speaker 00

  which seems unlikely to remain true to me.

• 1:38:57Speaker 01

  And I suppose one other thing was his comments about sparse auto-encoders. But do I understand that you've almost, well, you're in the process of moving past them as well with this new manifold idea?

• 1:39:09Speaker 00

  Yeah. And like, I think he, there's the thing that he said about deprioritizing SAEs, and maybe they're not the one true representation learner. There's like, how people kind of memed it, which is like Neil Nanda says, SAEs are dead. There you go, you can use that for the intro. And, and I don't think that that is actually what he meant. I think the field sort of, like, jumped on, everyone must do SAEs now. And now maybe we're doing the same thing with natural language auto-encoders. But I think he was like, he was probably correctly identified that they're not like, the answer to everything, but they are like pragmatically useful. You know, we still find

• 1:39:53Speaker 00

  lots of uses for them all the time. Even though I think we are going, like, I think this manifold idea is just a better, a better fit for what networks are doing. And so we should move towards using that.