Embryology of AI: How Training Data Shapes AI Development w/ Timaeus' Jesse Hoogland & Daniel Murfet

Jesse Hoogland and Daniel Murfet, founders of Timaeus, introduce their mathematically rigorous approach to AI safety through "developmental interpretability" based on Singular Learning Theory. They explain how neural network loss landscapes are actually complex, jagged surfaces full of "singularities" where models can change internally without affecting external behavior—potentially masking dangerous misalignment. Using their Local Learning Coefficient measure, they've demonstrated the ability to identify critical phase changes during training in models up to 7 billion parameters, offering a complementary approach to mechanistic interpretability. This work aims to move beyond trial-and-error neural network training toward a more principled engineering discipline that could catch safety issues during training rather than after deployment. Sponsors: Oracle Cloud Infrastructure: Oracle Cloud Infrastructure (OCI) is the next-generation cloud that delivers better performance, faster speeds, and significantly lower costs, including up to 50% less for compute, 70% for storage, and 80% for networking. Run any workload, from infrastructure to AI, in a high-availability environment and try OCI for free with zero commitment at https://oracle.com/cognitive The AGNTCY: The AGNTCY is an open-source collective dedicated to building the Internet of Agents, enabling AI agents to communicate and collaborate seamlessly across frameworks. Join a community of engineers focused on high-quality multi-agent software and support the initiative at ⁠https://agntcy.org NetSuite by Oracle: NetSuite by Oracle is the AI-powered business management suite trusted by over 41,000 businesses, offering a unified platform for accounting, financial management, inventory, and HR. Gain total visibility and control to make quick decisions and automate everyday tasks—download the free ebook, Navigating Global Trade: Three Insights for Leaders, at https://netsuite.com/cognitive PRODUCED BY: https://aipodcast.ing CHAPTERS: (00:00) About the Episode (04:44) Introduction and Background (06:17) Timaeus Origins and Philosophy (09:13) Mathematical Background and SLT (12:27) Developmental Interpretability Approach (Part 1) (16:09) Sponsors: Oracle Cloud Infrastructure | The AGNTCY (18:09) Developmental Interpretability Approach (Part 2) (19:24) Proto-Paradigm and SAEs (24:37) Understanding Generalization (30:15) Central Dogma Framework (Part 1) (32:13) Sponsor: NetSuite by Oracle (33:37) Central Dogma Framework (Part 2) (34:35) Loss Landscape Geometry (40:41) Degeneracies and Evidence (47:25) Structure and Data Connection (55:36) Essential Dynamics and Algorithms (01:00:53) Implicit Regularization and Complexity (01:07:19) Double Descent and Scaling (01:09:55) Big Picture Applications (01:17:17) Reward Hacking and Risks (01:25:19) Future Training Vision (01:32:01) Scaling and Next Steps (01:36:43) Outro

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome back to The Cognitive Revolution. Today, I'm excited to share my conversation with Jesse Hoogland and Daniel Murfet, founders of Timaeus, an AI safety and alignment research nonprofit that's pursuing an ambitious, mathematically rigorous, and fascinating approach to understanding the development and function of neural networks. Named after one of Plato's dialogues, Timaeus' work is based on Singular Learning Theory, or SLT, which applies algebraic geometry to statistical learning theory. Obviously, that's a mouthful, but the core premise of SLT is pretty intuitive. Training data determines the geometry of the loss landscape, which in turn determines which algorithms' models learn in training, and ultimately, how their behavior will

• 0:43Nathan Labenz?

  generalize once training is done and they're put into actual use. The driving insight of SLT is that the super-high-dimensional loss landscapes in which modern neural networks are optimized are not actually well-represented by the smooth bottom valley-shaped surfaces that we often see depicted in figures. On the contrary, Daniel, who recently left a tenured professorship in algebraic geometry to pursue this work at Timaeus full-time, calls these representations "maximally misleading", and explains that in reality, loss landscapes are highly-complex, jagged surfaces, full of singularities, also known as degeneracies, which are directions in weight space that a model can move without changing its external behavior or loss score, but which nevertheless

• 1:25Nathan Labenz?

  sometimes do involve a change to the model's internal circuitry, such that the model might behave very differently in novel situations. This, of course, has profound implications for big-picture AI safety questions. To frame it in terms that would be familiar to Eliezer Yudkowsky readers from 15-plus years ago, the difference between a model that acts nice and friendly because it is fundamentally aligned to human values, and a model that acts the same way because it's learned how to please humans while actually pursuing its own goals, could be the difference between a superintelligence-powered utopia and human extinction. And yet, today, even as AIs become more and more powerful, we don't have reliable ways to tell the difference. Anthropic and Goodfire-style mechanistic interpretability has, of course, made great progress toward identifying the concepts that trained neural networks represent, and

• 2:10Nathan Labenz?

  these days, also offering some visibility into the circuits they use. But there's a very long way to go, and I definitely believe that there's plenty of opportunity for complementary approaches to strengthen our overall understanding.

• 2:22Nathan Labenz?

  The TMAIS approach, which they call developmental interpretability, aims to understand how neural networks evolve through the training process, using a measure called the local learning coefficient to help identify what might otherwise be invisible internal phase changes that could profoundly affect downstream model behavior. This line of work, like all approaches to understanding neural networks, is still pretty early in its own developmental history. But critically, the TMAIS team has shown that it can scale beyond toy models. Their latest work applies these techniques to seven billion parameter models and is able to identify critical phase change moments that correspond to the appearance of important functional circuits. So we might say that it's roughly at the toward monosemiticity stage and hope that with more engineering and compute,

• 3:06Nathan Labenz?

  it will continue to scale to frontier models. If successful, this could perhaps prevent an episode like the one that happened in Claude 4 training, where a certain safety dataset related to harmful system prompts was mistakenly left out of the data mix, causing the model to generalize in such a way that it followed rather than refused harmful system prompts. Anthropic caught that problem with behavioral testing and patched it. But the hope for developmental interpretability is that such things might be caught by instrumentation during the training process before they ever seriously affect model behavior. In the best case scenario, this could help us move beyond the trial and error phase of neural network training and toward a more principled, engineering-like discipline, where specific datasets are used at specific times for specific purposes,

• 3:49Nathan Labenz?

  leading to predictable and reliable results. Now, this is all high-concept, mathematically sophisticated work. And this conversation, to be honest, is really just an introduction. I did my best to take my time to develop my own intuitions for what's going on inside a neural network during training, to compare and contrast some of the phenomenon that Jesse and Daniel described to other things like Grokking that we've previously covered, and to grapple with questions of how much generalization we really want from neural networks and how, in some cases, too much generalization could be very harmful. I found this conversation fascinating throughout. And while it will stretch your brain in a bit of a different way than most of our episodes, I expect that for some of you, it will immediately be among your favorite episodes that we've ever done.

• 4:32Nathan Labenz?

  Now, I hope you enjoy this introduction to developmental interpretability, a new approach to understanding neural networks, with pioneers Jesse Hoogland and Daniel Murfet, founders of TMAIS. Jesse Hoogland and Daniel Murfet, founders of AI Safety and Alignment Research nonprofit TMAIS. Welcome to the Cognitive Revolution.

• 4:51Speaker 01

  Hi Nathan, thanks for having us.

• 4:53Nathan Labenz?

  I'm excited for this conversation. I think I'm gonna learn a lot from it. And one of the things I think is really interesting about what you guys are doing is it presents as a bit of a contrast to what I sort of see as the general kind of default approach today in the safety and alignment space, which you might call like prosaic safety and alignment methods or defense in depth. I'm always on the lookout for something that feels like it, even if it's a long shot, could really work, where really work means like, give us enough of a depth of understanding and sort of a reason to believe that this thing will actually, this approach could actually work even in the surprising future that we may try to carry it into that, you

• 5:36Nathan Labenz?

  know, and when I say really work, I mean like so much so that I don't have to worry about it anymore and I can just focus on all the applications I want to build with AI and not have to do all these, but what if it gets out of control sort of episodes? So I think you guys have something that sort of fits that bill. It is at times over my head and I expect, you know, members of the audience at times will have to stretch their brains to really grok it as well, but that will be part of the fun. So I'm excited to get into it. For starters, want to give me just a little background on how you guys came together, the organization? I was intrigued by the name and I'm also very intrigued by the fact that Daniel, you recently left a tenured professorship to go all in on this work. So start off with the backstory and then we'll get into the more technical details.

• 6:19Speaker 01

  Yes, I think, I mean, the story starts what two and a half thousand years ago with Plato. So Timaeus is a dialogue by Plato in which he puts forth sort of the first theory of everything. So it's a protochemistry where the elements of earth, wind, fire are associated to platonic forms, right? The cube and the icosahedron and the dodecahedron and these kinds of things. Now, the theory is completely wrong, but it's the first instance of somebody thinking, hey, mathematics really has a chance of understanding the natural world. And so it's that spirit that drives us today. Maybe we can find ideas in mathematics and the sciences that actually help us understand this new phase of matter, these deep learning systems to help us make sure that they're safe.

• 7:05Speaker 02

  Also, in the Timaeus, the world is intelligent. So actually, the history of the universe is just a learning process and like physics is just a subset of learning theory.

• 7:19Nathan Labenz?

  That's fascinating. I've never read it, I'll confess. And it is striking that somebody had such a worldview 2500 years ago. This is the deepest in history. Usually I skip the backstory in a lot of podcasts openings because it's like, well, you know, we saw chat GPT and we knew we had to build this app. And it's like, you know, there's a lot of that. I think this is the deepest in history we've gone for the intellectual roots of a line of work. So, okay, how literally do you take the idea that

• 7:51Nathan Labenz?

  that that hierarchy that you described of like physics being a subset of learning might really be real? I mean, I know that there's a lot of differing opinions among the most, you know, much like in AI, right? There's we got Turing Award winners saying we're all going to die. We got other Turing Award winners saying the first ones are crazy. On the sort of what is the fundamental nature of the universe, you know, probably even more disparate takes. Do you come from a secular sort of tradition or perspective on like, is it a mathematical universe or are we bracketing that sort of thing for now?

• 8:20Speaker 02

  Yeah, I was mostly trying to troll your physicist listeners with that. I don't. That seems like a bit of a tangent. I don't think I want to take a position on the kind of mathematical nature of reality. But there, I mentioned that not completely in a kind of idle way, this aspect of the Timaeus. Because there are a lot of deep connections between physics and learning theory. And those are some of the connections that inspire a lot of the work we do, particularly with statistical physics and so on. So the study of learning machines is like as deep as physics in many ways. And it's not a surprise that a lot of physicists are starting to

• 9:05Speaker 02

  devote their professional careers and so on to understanding aspects of these systems. So, yeah, that's maybe the main connection.

• 9:13Nathan Labenz?

  Better than sucking all the physicists into hedge funds, I say. So.

• 9:19Nathan Labenz?

  How about just a touch more on what kind of math you were doing as a professor? And how that positioned you for this approach to AI? Because it does seem quite distinctive from almost everything else, you know, that I've seen. There's a few other little points in space that maybe are close to it, but not a lot.

• 9:41Speaker 02

  Yeah, so I'm an algebraic geometer by training. So algebraic geometry is a part of pure mathematics. We study mostly like solutions of polynomial equations, so very high dimensional geometric forms and very sophisticated techniques for trying to understand those geometric forms. Some of it is related to the real world, most of it is not. And most of what I did is, I mean, there were some connections to string theory at various points, which I've studied a bit, but mostly I was working on algebraic geometry, a little bit of mathematical logic. And at some point I was following the developments in deep learning and they were very sort of interesting, but it was completely separate from my professional life till I encountered Sumio Watanabe's work.

• 10:26Speaker 02

  So he's a Japanese mathematician, the inventor of singular learning theory. And it seems kind of hard to believe, but one of the core tenets of his work is that one of the very deep parts of algebraic geometry is sort of central to Bayesian statistics. And that was kind of curious to me and I was also aware that people were not completely satisfied with the theoretical treatments of deep learning that existed at the time. And so it sort of seemed tempting to me as an algebraic geometer to adopt this sort of language and approach to try and understand these systems. And that's sort of how I got into singular learning theory.

• 11:09Speaker 02

  And it was, that was a few years before I was drawn into thinking about AI safety. But yeah, it was sort of through the fundamental theory of Bayesian statistics. And well, all parts of mathematics are sort of ultimately connected at some level, right? And so it's not necessarily super surprising that geometry comes into learning theory. There are other ways it enters, but that's one of them.

• 11:31Nathan Labenz?

  Okay. So yeah, I mean, I guess in terms of like motivating the whole narrative, you could tell me how you think is better to motivate it. As I've studied up on the subject, one of the things that has Seemingly distinguished your approach from most others in terms of understanding AI systems generally is the approach or the focus on the developmental process. You know, how is it actually learning through the training process as opposed to run that, get a thing out and then try to, you know, make some sense out of it? So that's, that's one possible motivational line. Another could be just like, you know, it starts with this one, you know, weird thing in math

• 12:15Nathan Labenz?

  that I couldn't get over, but how, um, maybe you can kind of draw those threads together and, and tee up how you got to this, uh, you know, the current

• 12:25Speaker 01

  Yeah.

• 12:26Nathan Labenz?

  body of work.

• 12:28Speaker 01

  So all the research we're doing is based on singular learning theory, SLT. And I'd say that if I had to reduce that into one sentence, then maybe the core idea of SLT is that the geometry of the loss landscape, the surface that we walk down when we're defining some sort of optimization process, the geometry of this landscape contains important information for understanding neural networks and potentially steering them more reliably. So this has applications on the one hand for interpretability and on the other hand for alignment. And when we started looking into applications of this theory, which was at the time still very mathematical and theoretical for actually empirically understanding neural networks,

• 13:10Speaker 01

  two years ago now, one of the first applications we came up with was this agenda we called developmental interpretability. So can you use SLT to better understand how neural networks develop over the course of training? You know, the hope here is that development is an axis that can reduce the problem of interpretability to something more manageable. But at the end of the day, all interpretability techniques are hoping that we can reduce the problem of understanding some hundred billion or trillion parameter model into fewer numbers, into something that is a little more manageable. And one way you could try to do this is understand the changes that give rise to that model over the course of training.

• 13:51Speaker 01

  If there's a finite number of changes, then the number of changes is much smaller than the number of parameters, then you've got a useful simplification for the bottom. And that's what motivated developmental interpretability or Devin-Turk when we started two years ago.

• 14:07Nathan Labenz?

  When you say the number of changes are smaller than the number of parameters, that means basically like the steps, the number of steps in training? Each step would be a change?

• 14:16Speaker 01

  That is a good question because one problem is if you have to understand each individual training step, this isn't a reduction, right? Because there are still billions of training steps, many, many gradients to calculate. So somehow the unit of change needs to be larger than just one training step. So what's the right unit of change here? Well, that's where SLT actually comes in because SLT says that the right unit of change is a phase transition. So a developmental stage in biology, for example. And it's an empirical question then, whether you can actually find these phase transitions and whether they actually exist at a frequency that's useful for simplifying the problem. But in the systems we've studied so far, that does seem to be the case.

• 15:00Speaker 01

  You can meaningfully reduce the problem by looking at these stages.

• 15:05Nathan Labenz?

  Okay, that's definitely interesting. Would you make an analogy between a phase change and Grokking?

• 15:15Speaker 01

  Yeah, so Grokking would be an example of this kind of change. So typically, we identify maybe two primary kinds of changes. The first, we call type A transition, involves the model becoming more complex as it learns more information. You can imagine that a human develops a richer and richer mental model of the world that becomes more and more predictive and more accurate. But each additional step requires a more sophisticated model. And then you have the other kind of learning behavior, which is something like Grokking, where you find a simpler explanation for the same data. So you keep the same level of performance in explaining the data, but now you gain a solution that's simpler and therefore generalizes better.

• 15:58Speaker 01

  And the tension between these two things gives rise to this interesting learning process in neural networks.

• 16:04Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 16:10Nathan Labenz?

  Cool. Not too long ago, I did an episode with two of the co-founders of Goodfire. And Tom, who's the chief scientist there, gave an interesting sort of, you know, very short update on like, where are we in interpretability today? He said, "We've been pre-paradigmatic for a long time, but I'll now give us proto-paradigmatic status." And then he went on to describe the proto-paradigm as basically, you know, there seems to be a fairly broad consensus that neural networks learn interpretable things, that, you know, these things we call features, they are packed super densely into parameter space using superposition.

• 16:53Nathan Labenz?

  And that means the individual neurons are polys semantic. These, you know, features are sort of almost orthogonal to each other, but not quite. And that's sort of how they're packed in. The magnitude of their activation seems to correspond to the intensity with which they are relevant to a particular context. And these features connect up in circuits across the layers of a model. Do you guys sign on to that proto-paradigm? Or is there anything that you would add or subtract from it to put forward your own proto-paradigm?

• 17:25Speaker 02

  Yeah, maybe I'll take that one. Firstly, just to add to the discussion on the previous question. So we expect and do in fact see sort of clear phase transitions in small systems. But nobody really expects to see like hundreds of very clear step changes in the loss curve or other measurements in large models. So that's like another question, like how, if this is actually an axis of reduction, how do you find that structure inside a pretty homogeneous looking global sort of course training process? You need some sort of other means of looking more closely or sort of tuning your detection instrument to various different kinds of frequencies to figure out what's going on.

• 18:08Speaker 02

  So that's, yes, that's like additional information there. But yeah, to answer your question, I'm a big fan of the SAE's work, particularly of this recent work out of Anthropic and what Goodfire is doing. It's actually, in that episode, I think Tom is mentioning at some point also parameter-based methods. And sort of I like the analogy he gave, which is to sort of think of it as... You know, if you have a graph, you need to understand both the nodes and the edges. Right. And so it's it's very clear that ultimately the information in the model is in the parameters. Right. And there's a role for understanding what the model is doing by examining that structure.

• 18:50Speaker 02

  And one way of thinking about that is examining the geometry of the lost landscape. So I don't think of, say, SAEs and our approach to interpretability as necessarily like in the struggle. They're likely to inform each other. I think that if I was to make one critique potentially of things like SAEs, I mean, many people have concerns or doubts about SAEs. And there's also many people who are enthusiastic about them. I think I'm probably on the more enthusiastic end. But if I had to make a critique, it's that there isn't really a sort of mathematical foundation on the SAEs that connects it to things like generalization. So

• 19:33Speaker 02

  at least not that I understand, and I I don't think it exists yet. That's not to say it can't exist, and maybe we will, for example, do some work on that at some point. But if you go out and you understand what the model is doing by computing SAEs and you find circuits, it's very unclear, for example, if you were to do continual learning or you do reinforcement learning training on top of that interpretability that you did on a base model. Like what the behavior that you end up with after you continue to train, the relationship of that to the interpretability examination that you did before. Just because it really isn't clear what the relation is between the structure you found in the SAEs and the sort of the data generating process or the training process.

• 20:18Speaker 02

  So you don't know how stable it is to that. I can imagine empirically, it actually is quite stable. So when I spoke, I mean, I expressed this concern at some point to Chris Ola and his response, if he doesn't mind me passing this on, is that, well, if you understand circuits through SAEs. And then you have some further modification of the model. Probably what's happening, so you do fine tuning. Probably what's happening is that the fine tuning process is recruiting the representations and circuits that existed before and using them for a different purpose. And indeed, Anthropic has some work looking at features before and after fine tuning, and I would say it's consistent with that interpretation. So I think you can give an empirical answer to the question of like, how do SAEs relate to

• 21:03Speaker 02

  the training process and the generating process and generalization, just by kind of like empirically going out there and checking and seeing that it seems to work. I think that's reasonable. I think that isn't a path to a high level of assurance necessarily that we might hope to get. And so when people talk about generalization in connection with AI safety, it's the hope that you can ground the understanding you have of the model in something that is mathematical and something that is like a solid foundation and is not purely empirical. And so our approach to interpretability in contrast is sort of more starting from the generalization end and working towards interpretability. And so the hope is that

• 21:44Speaker 02

  and obviously we're also more about parameters rather than activations. But I mean, there are likely to be relations between these two perspectives. I mean, I can speculate on those if you want, but yeah.

• 21:58Nathan Labenz?

  It strikes me that I think everybody has an intuitive sense of what generalization means. But you may want to offer a more formal or precise meaning for that. I mean, you know, generalization at the chat GPT user level, right? It's like, I can take in anything. And it seems, as Ilya once said, like, the amazing thing is I talk to the AI and I feel I am understood. Like, that's maybe, you know, the sort of mundane utility, you know, short form story of generalization. Then there's like the Grokking generalization where you go from the sort of memorization to algorithmic. And I suspect you're going to generalize that to your, your sets of generalizations. How do you mean that?

• 22:38Speaker 02

  Yeah, I guess like maybe you're understood because you're an interpolation of the training data.

• 22:44Speaker 02

  So when people talk about generalization, they often mean multiple like sort of somewhat distinct things. So technically, generalization, I mean, usually it's a number, so a generalization error. And it represents the ability of the model to predict on samples from the data distribution, which were not seen in train. And that might be a held out test set, as it often is in machine learning. I mean, theoretically, we often suppose that there's a generating process which both generated the training data and you can continue to draw samples from. And then generalization error would be something about the gap between the predictions you make based on the model you learned from the training data.

• 23:27Speaker 02

  Evaluated on these sort of new samples from the same generating process. That is, if you like, in distribution generalization, I think, like, usually people don't add this prefix in distributions. Just understood that you are testing the trained model on samples from the same generating process that produced the training data. If you're not, then we've heard of that as out of distribution generalization. And that's kind of a very fraught, difficult thing to understand, because of course, those new samples could be absolutely anything. And why should you expect any particular good performance on that?

• 24:02Speaker 02

  Yeah, I guess like the attitude towards generalization that was existed sort of pre G P T three was, I mean, this is one of the big cognitive transformations in the last few years is people really understanding that you can pre-train a large model on very diverse data and get something like generalization to very large range of tasks. Exactly what form of generalization that is sort of not super clear, I think. And the degree to which you're actually getting capabilities which work well outside the training distribution.

• 24:39Speaker 02

  In some sense, this is of course a very controversial topic, where like R L M's really reasoning, or that they're really reasoning, they can do arbitrary things that reasoning can do. Which may be very far outside the training data. Okay, so people debate these kinds of questions all the time. It's quite unclear, but the angle, I mean, the interest we have, I mean.

• 25:01Speaker 02

  Beyond thinking about generalization as a number, right? Like a measure of this ability to predict on new samples. You can ask for kind of an understanding of generalization, which has its core mathematical like type, not a number, but like maybe something more complex, like say an algorithm, right? Like the algorithm is the thing that allows you to generalize, say. And so it's the thing behind, if you have low generalization error, maybe the reason is that you have internalized some algorithmic aspect of the generating process. And so in some sense, like the study of like interpretability is kind of like one way of phrasing it is you're just studying generalization,

• 25:45Speaker 02

  but you've changed the type signature of the object you're trying to produce rather than like being interested in a number. You're being interested in the thing behind the number. I mean, I would say that's like what the aim of in some sense, interpretability is, is to like, explain why you're able to have low generalization.

• 26:02Nathan Labenz?

  So yeah, I mean, that's intuitive, certainly at the level of like the original Grokking paper, you know, the first steps, it memorizes relatively quickly. It can get the samples that it's seen right. It still does terrible on all the examples, even from the same distribution that it hasn't seen until, and I always like to remind people of this, orders of magnitude, more steps later, then you slowly, you know, it looks like a steep rise on the log, you know, X axis on that famous graph, but it's actually the last 90% of the training time is when this sort of Grokking process slowly happens. And then you get an actual algorithm for, in that case, the modular arithmetic, and now you're like, good to go.

• 26:44Nathan Labenz?

  And I think there are some interesting things there to unpack. I mean, one that's really, I think, probably the close to the heart of why all this matters is it has been shown that there are multiple different ways to Grok that problem, right? And we know now that, I think this came out of Max Tegmark's group, if I remember correctly, there's like the sort of trigonometric way. And then there's like the pizza slice algorithm. And you know, these both like get the right answer. But they perform quite differently. Maybe that doesn't matter in that particular case. But obviously, you can imagine learning very different algorithms to solve the same problems, especially as we go into a space colonization mode or, you know, trying to put these things into

• 27:25Nathan Labenz?

  an online learning environment, you know, that could lead to potentially very different downstream behavior. And so it makes a lot of intuitive sense that we would like to understand what is going on there. Maybe you can elaborate on that or clarify anything I'm getting wrong. But maybe this is a good time to give what I'm, I kind of have started to think of as the central dogma of your approach in the same way that we've got like, you know, DNA, eventually get to proteins. You've got this story of like, we start with data and there's this sort of set of relationships that takes us from data to model. So I don't know if you like that central dogma label, but.

• 28:04Speaker 01

  I'm sure Jason's gonna use this like every week now.

• 28:09Speaker 01

  Thanks for that.

• 28:09Nathan Labenz?

  It's a good phrase. I don't know if it's a quote or bit, but yeah, give us the central dogma. And then we can kind of, when we get to the end, we'll have this, you know, still ambiguity, right? Around like, at least for now, what, it may be getting them all right now, but like what algorithms did it actually learn?

• 28:25Speaker 01

  So I think the key question is, where does neural network behavior ultimately come from? Right? All of our alignment techniques, what we're trying to do is change model behavior and make sure that they generalize robustly out of distribution. Many problems can be reduced to this understanding. Ultimately, behavior comes from training data, right? There are basically three inputs to every learning process. There's the neural network architecture, there's some choice of optimizer, and then there's the training data. And of these, training data is the most important one. The one that tells one model to learn one algorithm and another model to learn a different algorithm. So what we'd like to understand is how does training data give rise to the final behaviors

• 29:09Speaker 01

  models end up with, in particular, because this is what all of our current alignment techniques look like. RLHF, Constitutional AI, DPO, deliberative alignment, all of these techniques are basically just modifications on the same deep learning processes with different data. So in particular, data determines the geometry for loss landscape. It's that geometry which in turn tells SGD how to move around. So it tells your optimization process how to move. That learning process picks out the final weights and thus algorithms that a neural network ends up with. And it's the structures in those weights that determine how models generalize and thus whether they're aligned or not. So this central dogma, as you called it, something we called the S4 correspondence between structure

• 29:54Speaker 01

  and data, structure and geometry, structure and the learning process, and structure in the final weights of the model, is what we see as key to understanding the mapping from data to final behaviors.

• 30:08Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 30:12Nathan Labenz?

  And so what do we know in general about the loss landscape? I mean, we've all seen many one dimensional loss curves of just like loss dropping over time. You mentioned earlier, like, that may, especially in larger models, that I've always kind of understood that as there are probably lots of little gracking moments happening in there and phase transitions, but because they're also aggregated up, you know, we can't see them at that scale. And then we've also seen like many two dimensional, you know, visualizations of loss landscape. And I honestly have no idea whether, when I look at something that looks like a, you know, a well in a two dimensional space, if that is leading to-

• 30:56Speaker 02

  Can look at those pictures, yeah.

• 30:57Nathan Labenz?

  Yeah, is that so? So, yeah, I mean, I don't put too much stock in my intuition for that because I'm just like, I don't know, you know, does two is two like 200 billion? I don't know, it seems quite different or plausibly quite different. What do we really know about loss landscapes today?

• 31:12Speaker 02

  Yeah, I think it's, I mean-

• 31:16Speaker 02

  Well, to go back to algebraic geometry briefly, so in two, one or two or three dimensions, you can use pictures and use intuition from pictures. But once you go to higher dimensions, your intuitions are not like a reliable guide to how to think about these objects. And so that's where you sort of try to bake your intuitions in the lower dimensions into mathematical forms, which continue to work. Right. And that's in some sense what geometry is. Uh, I think these pictures of, I mean, if you draw a two dimensional, like a surface picture of a lost landscape, the way you do that is by choosing two directions in a very high dimensional space.

• 31:56Speaker 02

  And then plotting the, you take that little plane and you're plotting the loss like as a function of the coordinate on that plane. Right. One of the things you're with with probability one going to do, if you choose a random slice like that is not see, for example, the degeneracy that is the thing that determines the generalization according to single learning theory. So in some sense, those pictures are maximally misleading when it comes to generalization. I'm not saying they're not useful and they're certainly pretty. But the, like in many cases, the, the actual relevant geometry looks more like a bunch of planes intersecting in some kind of crazy way. That's what the level sets look like.

• 32:42Speaker 02

  So if you think about the, maybe you take a kind of valley and sort of a picture of valleys in a lost landscape that you'd usually see. The level sets where you just look at the set of points at given height above the floor, so a set of points with a given loss will be kind of, if you think about it, like the edges of the valleys and so on, will have some sort of like nice, smooth shape. Right. And then if you go to the very bottom of the valley, you'll eventually just get a single point. And this is not like the level sets of um, the lost landscapes. And there's actually two things you could mean by lost landscape. I'll have to disambiguate in a second. The level sets of those landscapes are not smooth shapes like that. They're very complex.

• 33:28Speaker 02

  Geometric forms with lots of intersecting lines, lots of high dimensional things that are kind of gnarled up in various ways. And those are the singularities that I was sort of interested in. And to be clear, the loss landscape here, I mean, the population loss. So this is a theoretical object. This is sort of like an important and subtle and difficult point. So you never actually have access to this object, the population loss, right? That would be if you could average over every sample from the true generating process, say that was images, right? Like you somehow could average the loss of every possible draw from that data distribution. Of course, you never see that. And if you're plotting an empirical loss landscape,

• 34:11Speaker 02

  you have some samples and then you're computing the loss based on that and you plot that. And so the the actual geometry, which dictates generalization mathematically, is the geometry of the population loss. And that the sort of that influences the behavior of all these empirical losses that are based on samples. But you never actually have access to that. So.

• 34:36Speaker 01

  I'd like to offer just a slightly higher level description. So usually when you ask people to imagine what the loss landscape looks like at the bottom, or you look at these slices, it'll look like a basin, right? A roundish basin, like a parabola. And what Dan calls degeneracy is the property that this is the wrong way to think about it, and there are valleys and canyons, the directions you can walk along that don't change the loss. And in a sense, SLT tells us that these are the most important directions by far. It's, in a sense, the number of valleys that you have that determines how well your model generalizes. If there are more valleys, you can perturb your weights, you can change your implementation without actually

• 35:19Speaker 01

  changing the function that you've implemented. And that means that your function is actually simpler, right? Simple functions can be implemented in more different ways. And then through an Occam's razor kind of argument, it's the simplest function was generalized as best. You can actually make this statement fairer size using SLT.

• 35:39Nathan Labenz?

  I definitely want to unpack a little bit. First of all, it is just a good reality check on all these visualizations that what you're actually doing when you're training a model is you're taking the gradient and you're just taking a step in that gradient direction and you have no actual landscape around you. It's not like you're being guided broadly by this like high, you know, resolution object that you're sort of finding your way down. You're just literally one point in time, take a step and hopefully that was a good step, but it could have even been like a bad step, right? I mean, there's you have not a lot of guarantees there. Then the next thing around the degeneracies, I have a lot of questions around that, probably some of them like fairly ignorant.

• 36:23Nathan Labenz?

  I guess one like really basic one is just how do I know that this is this even exists, right? I mean, because you mentioned the sort of challenge of like, we don't have the actual full, you know, let's say, I don't know if platonic is too loaded of a term, you know, this sort of idealized

• 36:42Speaker 02

  It's a fun term.

• 36:43Nathan Labenz?

  landscape from the, you know, idealized full data set. Since we don't have that, then I'm not exactly sure how we establish that there is this direction that we can move. Cause it's not like we can say, hey, look, you know, here's five inputs. I changed this parameter, nothing changes on the output, we're done, right? Like, how do we know that this actually exists? Or I feel like I'm sort of missing something like caveat or constraint on how this is formulated. So yeah, question. Please de-confuse me if I'm capable of being de-confused.

• 37:19Speaker 02

  Yeah, I mean, there's a few, a few ways into that. One is to study very, very simple neural networks where you can actually figure out the population loss in closed form. And that's where Waternabe started. So you can take, I mean, this is kind of decades ago. So he was studying like tan H networks rather than kind of modern neural networks. But you can take like one hidden layer tan H networks, for instance, and actually just like for a very kind of synthetic data distribution that you can understand. And then you can just write down, like do the integrals, figure out what the population loss is. And then you can actually study these degeneracies, see that they're real.

• 38:00Speaker 02

  They don't have any interesting relation to the data because the data is not interesting because for interesting data, you can't actually do that calculation. But in these kind of very synthetic settings, you can actually show what the degeneracies are and do various calculations and see that. Study the theorems which connect generalization to geometry explicitly in examples, and there's numerous examples. Including four neural networks, albeit these simple ones. You can do things that are a bit more related to current practice. So one of the papers we wrote looked at the, so Anthropic did this work on the toy model of superposition. Very interesting. And there's a, an actual, so there's a small autoencoder.

• 38:44Speaker 02

  And you can actually do the integrals there as well. So you can write down and get the actual population loss. You can study its geometry. We did that. We computed these degeneracies and you can actually see that the training process is governed by these degeneracies in the way the theory says. And so that's again, like not a super interesting data distribution. And so the degeneracies like, I mean, it's quite complicated. What's going on ultimately? But that's a case where you can actually find out what the degeneracies are and. I mean, that's, that's kind of like the interesting form of degeneracy. It's kind of a boring order degeneracy, which maybe it's worth getting out of the way, which is just kind of like scaling symmetries and so on.

• 39:25Speaker 02

  So if you have a ReLU, you can scale up the input and down the output by any real number, right? A positive real number. So that's a way of changing the weights that doesn't change the function that's computed. And so you have a lot of degeneracies like that. So anytime you have two matrices being multiplied inside a network, so like in the in attention, you have Q K, like the query and key matrix are multiplied and various other matrices are multiplied. Whenever you have structure like that in your model, you can of course like take those matrices that are being multiplied and for example, just scale the first one up and the second one down by the same factor or put any invertible matrix in between. So multiply

• 40:07Speaker 02

  the first matrix by a ma- so on the left by some matrix and the other matrix on the right by its inverse. And you haven't changed the outputs of the network, but you have changed the weights. So you have many degeneracies like that. Those are not uninteresting. For example, with the Q K matrix, this degeneracy will have like an impact on, for example, the kinds of functions that the that part of the network will tend to learn. So it's like non-trivial, even though it's a little bit like on its face a bit boring. Um, but there are other kinds of degeneracy which emerge sort of in a more overall fashion, which can't be pinned down to

• 40:50Speaker 02

  individual kind of obvious forms of degeneracy like that. Okay. So you can do it in toy cases. You can see some examples like these products or matrices where it has to arise. But then it starts at some point, the bigger you make the model and the more interesting you make the data distribution, it stops being feasible to do theoretical calculations and it becomes an empirical question. So maybe I'll come back to that in a moment. There is one more kind of theoretical thing you can do, which is you can invent toy models of what you think are the key ingredients in a more interesting model. So there are various papers which have looked at, for example, transformers doing in-context learning.

• 41:33Speaker 02

  And you sort of abstract out some of the annoying details in real transformers. And you come up with like a closed form formula, some function of the parameters which you think uh models well what transformers are trying to do.

• 41:50Speaker 02

  So there's Sax's lab, Andrew Sax has like, has made a career out of doing this very beautifully in many different settings, not specifically for transformers, across many different settings. But there's a recent paper which looks at in-context learning for transformers and actually comes up with kind of a toy model, a sort of explicit potential, some functions, a loss function, which describes that. And you can see the degeneracy in that function, for instance. So that's another way of trying to get at it. But if we're talking about like, algorithms in models, and trying to back those out of weights, so far, I don't think there's a very clear toy model of that in that form. It'll be interesting. I don't think it exists

• 42:31Speaker 02

  yet. And so it becomes an empirical question. Can you go out and extract signals of that degeneracy from information that you get from these empirical losses? The theory says you can do that. And that's kind of what, a large part of what we do.

• 42:48Speaker 01

  To come back for a moment to the link between degeneracy and generalization. So, you know, one of the core theorems of SLT says is that you can link a sensitivity analysis in weights. So this is the question, if I perturb the weights a little bit, how much does the loss increase? You can link this sensitivity analysis to a sensitivity analysis in data. And so if I change the data distribution, how much is my loss going to go up? And it's that link, which gives us, first of all, a link between structure inside of the model, because structure is reflected inside of the geometry of the loss landscape, to generalization

• 43:32Speaker 01

  and potentially out of distribution, generalization. So now you can look at, if I apply a small local change to my data distribution, I upweight some, some distribution, for example, GitHub code data versus WikiText, which parts of the model become active, which directions in the loss landscape respond most to this? And it's through this connection between structure, geometry, and data that we are developing these tools for interpretability.

• 44:03Speaker 02

  Okay, I need to dig in on that a little bit.

• 44:06Nathan Labenz?

  More, again, wouldn't maybe like ignorant question around the just the nature of these degeneracies. I certainly get what you're saying in terms of like, some of them are just in a sense trivial. You know, you can scale from one thing up, another thing down, and you get the same thing. So that's obvious.

• 44:24Nathan Labenz?

  As you get to these large scale things, it feels almost like accidental to me. It feels super contingent, but then you had made the statement that they govern the training process. And I realized like, I actually don't know really what that means. You know, what does it mean for them to be governing the training process? And if they are in fact governing the training process, then that suggests that they're like much less accidental and contingent, or maybe there's different kinds of them, some of which are like deeply built into the nature of the world and sort of reflected from that. And others maybe are sort of accidental and contingent on the way you order your batches or, you know, the exact mix of this versus that.

• 45:07Nathan Labenz?

  So, um, yeah, again, consider that your prompt and tell me what I need to know.

• 45:14Speaker 02

  Yeah, there's kind of a bit of mathematical background, which I'll maybe provide. So stochastic gradient descent is a specific kind of optimization process. But in much more generally, you know, we're interested in many different areas in just following a gradient. Right? So you have some function and then you're going to follow a gradient to maximize it or minimize it. And in cases where that gradient is the gradient of something that looks like a sum of squares. So X squared plus Y squared plus Z squared. So you're just trying to, for example, find the bottom of a bowl. That process isn't interesting, right? It's just, there's the bottom of the bowl, you're going to go there. So that's kind of convex optimization, right?

• 45:58Speaker 02

  You can still do interesting mathematics with that, but it's like fundamentally not a complex process. But many processes in nature do not look like that. And that's why there's like a field called nonlinear dynamics. So more generally, if you're trying to follow a potential, so imagine, imagine a surface and it's covered in little arrows, right? That tell you which direction you're going to go. So that's the gradient of some fixed potential. So potential is just a general term for a function whose gradient you're going to follow.

• 46:30Speaker 02

  So you have this kind of vector field, all these arrows pointing in various directions. It's a fact about dynamical processes that the places where the gradient vanishes organize the trajectories. So in two dimensions, maybe that seems a little counterintuitive, but you could think about it like this. So if instead I go back to one dimension, so let's just take a curve. And now the places where the gradient vanishes are the maxima and the minima, right? Or the saddle points. And I mean, I taught calculus for many, many years. And a large part of the course is teaching people to classify the minima and the maxima. Because, you know, the other points in the curve are kind of boring, right?

• 47:13Speaker 02

  Like you want to understand some function, like a cost function. You want to know where it's maximized or where it's minimized. What it's doing like at some random point on the curve is completely irrelevant. Like you just want to know where the maximum and minimum are. So that's an instantiation of the principle that if you're trying to understand a function, you are often looking for the places where interesting stuff happens. And those are maxima and minima in one dimension. And in higher dimensions, they're singularities. So they're places where the gradient is zero in all directions. And you can never get there because no trajectory that starts from outside the singularity ever arrives. And if it's at the singularity, it never leaves. But still, if you follow a random trajectory, following a potential,

• 47:55Speaker 02

  and there's like some singularity somewhere, it's going to do something around it. And like, this will dominate the behavior. So the trajectories will tend to approach it and then escape in some particular direction. So the singularity is organizing the set of global trajectories. And that's a kind of informal statement, but there are formal statements that you can use ideas from topology and so that there's ways of making that more precise. That's a general principle and it applies for learning with SGD just as much as it applies to any of these other systems in physics to which people would apply this principle. So that's like a general rationale for caring about critical points or singularities.

• 48:37Speaker 02

  These are almost synonymous. So that's a reason for caring about those as the organizing principle for dynamical systems and learning in particular. Now, like what it looks like for an actual learning process in deep learning to be governed by singularities. Yeah, I mean, that's sort of like, you know, in really interesting cases where there's kind of like a model doing interesting stuff is, I mean, it's a very complex question. But in simple cases, you can see that, well, yeah, like this toy model of superposition work I was describing earlier, the columns of the weight matrix. So you can visualize those columns as vectors in two dimensions.

• 49:20Speaker 02

  So there's two rows of this matrix and some number of columns. You can just plot those columns as vectors. And then you get sort of shapes like a pentagon or a square or a hexagon, connect the vertices of those, those vectors. And then moving between the neighborhoods of these critical points literally corresponds to like growing a leg or contracting a leg and rearranging the other ones in some particular way. So there's, if you like, the, the sort of growth process of the structure of the network is dictated by the movement between neighborhoods of these two different singularities. So that's, that's in a very simple case.

• 50:05Speaker 02

  In a more complex case, yeah, I think it's not so clear how to think about that. And the, the singularities aren't like isolated things where there's like one singularity here and normal stuff in between. In a large model, there's kind of singularities everywhere and it's, uh, yeah, not well understood.

• 50:24Nathan Labenz?

  I'm almost envisioning a sort of fluid dynamics type of thing where the, you know, the point that we are in parameter space kind of speeds up when it gets around the drain, but then maybe spins out and goes slower as it has some like wider orbit. Um, and probably a lot is lost in that, but that's at least the visual that's coming to mind. Another

• 50:48Speaker 02

  Yeah, there's this, you can visualize the function in function space of like a small language model. And there's some parts of the trajectory that actually look like it's orbiting around some particular kind of mode of prediction. Yeah, it's like that kind of image is is.

• 51:05Nathan Labenz?

  Yeah. So do do do these singularities, I mean, in a, you know, simple cost function, as you're describing maximize it, minimize it, whatever. It's very clear what that point means and why we're, you know, trying to identify it in the first place. In the case, especially of a more complicated neural network, I guess. I'm, you know, I'm tempted to ask, is is that the sort of, you know, if I imagine like a Grokking type phenomenon? Are those singularities the place where like the algorithm is perfectly Grocked? Such that there's never anything more to learn, at least, you know, with respect to like some subset of problems? Or is that is that reading too much into the nature of the?

• 51:49Speaker 02

  Think think of the singularities as different ways of solving the problem. Like different kinds of solutions at a given level of loss. So if you're at a given point in the loss landscape, different kinds of singularities correspond to different ways of predicting on the data you've seen so far. Jesse, maybe you want to say a bit about the ED stuff?

• 52:13Speaker 01

  Yeah, I think it's an interesting one. Um, yeah, maybe I'll say one more thing about this. All right, so coming back to generalization, one thing we're worried about is your model might learn two algorithms that look identical from the training data, but one of them generalizes in a way that you don't like, and one of them generalizes in a way that you do like. So how do you distinguish them? Well, what Dan's just saying is that maybe if you look at the neighborhood and loss landscape, they actually have different geometries associated to them. And so this is the sense in which reading this this geometry gives you information about what kind of algorithm do you actually end up with? And so what Dan was referring to as ED stands for essential dynamics. This is a paper we put out earlier this year where you can train a neural network,

• 52:56Speaker 01

  a transformer, to do in-context linear regression. So you can give it X, Y, X, Y, X, Y samples and ask it to predict the Ys from the Xs. Or each Y is generated from a simple linear transformation of the Xs plus some noise. Okay, so what happens is the model, if you train it on lots and lots of different samples of this, will learn to do regression over the course of context. However, what you can do is you can vary the number of distinct tasks that the model's exposed to during training. If you expose it to many different tasks, it learns regression. If you only expose it to a few different tasks, it'll memorize those tasks and it won't learn the regression solution. It'll learn this memorization solution.

• 53:42Speaker 01

  Now, which solution the model ends up with varies as you vary these two, the complexity or the diversity of the number of tasks. And you see a phase transition where the model initially memorizes and then eventually learns this other solution. But the selection process between these is something you can actually try to understand with singular learning theory. So each solution has an associated performance and complexity. Memorization is always better at performance than this regression solution, which I call the generalization or meta-learning solution. Generalization solution, however, is simpler past a certain level of things that you have to memorize.

• 54:23Speaker 01

  And so what happens is the simplest solution roughly occupies more volume in parameter space and is therefore easier to find. And so you you notice this phenomenon where the model will first learn the generalizing solution, the simple one, before moving to the memorization solution. You can classify exactly which kinds of dynamics qualitatively you should expect to see based on this trade-off between performance and complexity. Or complexity is reflected in the degeneracy of the loss landscape. So that's kind of the opposite order to Grokking.

• 54:59Nathan Labenz?

  Yeah, I was noticing that for sure. So, yeah, I was so caught on that. Maybe let's take.

• 55:09Nathan Labenz?

  Do another version of that, because, and help me understand how it's happening in the reverse in this case versus the Grokking case.

• 55:16Speaker 02

  Grokking is actually a strange phenomenon. This is not like a typical, I think the, I think there's been, in a sense, like, overfitting to the example of Grokking. This, this, at least in our opinion, is kind of not the typical way in which memorization and generalization probably interact.

• 55:37Nathan Labenz?

  Okay. Say more.

• 55:40Speaker 01

  What's special about Grokking is that the performance on the training set is more or less identical. Because you have one solution that is more, more complex, the memorization solution is more complex, but it's the thing you learn initially. So you end up with this transition where, at a fixed level of loss in the loss landscape, you end up in a broader basin, and in a basin that has more degeneracy.

• 56:06Speaker 02

  Yes, this is a phenomenon that can only happen, I mean, this is sort of, if you get to, conceptually, the very end of training, so at the very bottom of the loss landscape, then there are some perfect solutions that are simpler and more complex, and you can, you know, you prefer the simpler one if you can find it, and eventually you will find it. But the trade-off between memorization and generalization that occurs more frequently, like, without such sort of, like, artificial conditions, is more like, you prefer the simple solution that's bad, has high loss, you eventually trade that off for a solution that has lower loss and more complexity. You pay a complexity penalty in order to get

• 56:51Speaker 02

  the lower loss. That's the more typical relation between memorization and generalization.

• 56:57Speaker 01

  Yeah, and that's the case in this in-context linear regression example. So, what SLT says is, you should expect models to minimize a sort of effective loss, where it's, it's the loss with this emergent regularization term, this implicit bias that comes from degeneracy, that comes from the number of ways you can procure weights without actually changing the function. And so typically, the model's just going to prefer initially to learn the simple, bad thing before moving on to the complex, better thing.

• 57:31Nathan Labenz?

  And so help me understand one more time how the Grokking case is different? Like, because in that case, it takes a lot longer to learn the, what I would think of as the simpler algorithm, which presumably does have, you know, in terms of some like formal complexity metric, I would assume that that does have like a lower score for complexity than like memorizing, you know, and examples, right?

• 57:55Speaker 02

  Yeah, so the, so as Jesse referred to earlier, this kind of like two typical changes in the trade off between loss and complexity that are predicted by the core mathematical result of SLT. By trade off, I mean trade off between the loss and the complexity as estimated, we call that the LLC, local learning coefficient. So I'll just say complexity. So one way you can, like a preferred trade off is to pay the penalty of increasing the complexity by lowering the loss. That will decrease what's called the free energy, which determines which solution is more preferred from a Bayesian statistics point of view.

• 58:39Speaker 02

  So that's what we sometimes refer to as a type A transition. That's like the typical, that's just like learning, right? Like you learn more stuff, you get better. Okay. But there is another way to decrease the free energy, which is to like decrease the complexity at a fixed loss. So this is what we refer to as a type B transition. It's hard to ascertain exactly like the conditions under which this is the case, but Grokking seems to be one of them. We've seen others. So those are situations where a model really isn't improving on the loss, but it simplifies the algorithm that it's using. We can see this in the linear regression setting in some sense as well. And of course, there are many other examples

• 59:21Speaker 02

  of Grokking like behavior. But yeah, so that's the, the underlying principle is you should decrease the free energy, which is a sum of two terms, one to do with the loss and one to do with the learning coefficient. And you're allowed to increase one of those terms if you decrease the other one enough, that will decrease the free energy. That's normal learning. And then there are situations where you can

• 59:46Speaker 02

  Have a similar change, a decrease by, by just decreasing one of the terms, the complexity.

• 59:51Nathan Labenz?

  So one thing that's jumping out at me is like, you know, in general, right? Obviously, people can do all sorts of variations on this, but the sort of vanilla setup is you're just minimizing the loss. And just you would use this term, I think it was implicit regularization. So if I'm sitting, you know, looking at my single, you know, loss function and trying to drive it as low as I can. I understand what like weight decay is when it comes to, you know, and that's like one form of this sort of regularization, right? But if I understand you correctly, you're saying that there is some sort of natural pull that is happening, even if I am from an external perspective, just minimizing loss that is

• 1:00:36Nathan Labenz?

  moving me around in this complexity space and prefers simpler solutions, but I don't have an intuition as of now for why? Or like how that's happening if I'm not like to, you know, taking an active step in the learning process to enforce that.

• 1:00:51Speaker 01

  You can imagine asking the question, what happens if you randomly drop yourself somewhere in the loss landscape?

• 1:00:58Speaker 01

  More likely than not, you'll end up somewhere with pretty high loss. You'll also end up somewhere that has a very broad basin, right? There are many different ways you can perturb that solution that'll leave the loss basically the same. And so generally, as you, as you move down the loss landscape, it's going to be easier to find the solutions that occupy a greater volume. And the main thing that contributes to the volume is the degeneracy. So the number of valleys, it's just easier to find canyons that have more valleys than it is to find very narrow nooks and crannies. And this is the sense in which you get this emergent implicit regularization from the structure of the loss landscape.

• 1:01:39Nathan Labenz?

  I could do this all day in terms of just sitting here trying to visualize these things and asking you to help de-confuse me. And I think we have a lot of those listeners too, though I probably need to be somewhat disciplined about how many more of those kinds of questions I ask. This sort of seems to connect very fundamentally to like double descent, right? Or kind of, you know, just these sort of, yeah, I mean, maybe just riff on double descent for a second. And then I definitely want to take a moment and also really zoom out, right? I mean, the, it's all fun and games and modular arithmetic and, you know, toy problems to motivate the entire thing. And I'll probably put this in the intro. You know, the sort of canonical safety worry is you can't necessarily tell

• 1:02:22Nathan Labenz?

  the difference between a super intelligence that has your best interests at heart and one that doesn't. And, you know, you have a hard time, you may have a real hard time in many ways if you can't make that distinction as you get to more and more powerful systems. So there's the double descent thing, but then let's do the zoom out and kind of, you know, talk about the like, you know, the biggest picture worries that we have. And then I also think, you know, maybe we should kind of start to go toward like, what is the path for you guys from here with this, this research agenda? We've seen from Anthropic, I would say, and others, of course, you know, from whatever toy models of superposition

• 1:03:07Nathan Labenz?

  two and a half years ago, like really an unbelievable amount of progress, in my view. Obviously a long way still to go, but if you'd told me then that we'd be here now, I'd be like, that's really amazing. And it has struck me that, you know, one of the big things that has gone into that has just been like a ton of compute. And I'm, I don't have an intuition yet for whether that is sort of a similar trajectory that you guys think you will follow, or if it is going to be a more, you know, sort of eureka moment to go, you know, to recall the Greeks again, driven process. So, yeah, a lot there. But double descent, if you want to say anything about that, you know, what are the biggest, if we zoom out to like, not toys, but the real things that we're really worried about?

• 1:03:48Nathan Labenz?

  And then, and how do we sort of go from these toy model understandings to, you know, starting to tackle that? Hopefully on relevant timelines.

• 1:03:58Speaker 02

  Yeah, so regarding double descent, I mean, there's various kinds of double descent. So there's double descent with respect to training samples, there's double descent with respect to model size. I think nobody has systematically attempted to study double descent with singular learning theory yet. We haven't really tried. I think partly because it just doesn't look that mysterious. When you have approached the problem from the mathematical perspective that we do, it isn't mysterious that models with many parameters can generalize well.

• 1:04:33Nathan Labenz?

  And i- in terms of an intuition, is this another one of those dimensionality things where when I see a hill, I'm like, boy, it looks like a lot to climb over that hill. But when I'm in 200 billion dimensional space, there's always kind of a way to like wind my way through it?

• 1:04:49Speaker 02

  I mean, that's an often, often presented intuition for why optimization in high dimensions is amenable. I am not so sure. I think that's really an explanation. I think that's closer to like a folk story, maybe. I think it's not understood why. I mean, just having a large model with lots of parameters, like you can of course find things that don't optimize well, that have lots of parameters that aren't neural networks, right? Or you can get the neural networks configured badly and they won't train very well. So there remain many mysteries, why large neural networks can actually find well performing,

• 1:05:34Speaker 02

  well generalizing solutions. Singular learning theory doesn't resolve all those mysteries, right? But it does at least give you a mathematical framework from which you're not surprised that there can be models with lots of parameters which do well and predict well and generalize well. So I think I don't really have a clear answer for you on double descent. Maybe one of your listeners wants to think more about it. We haven't put a lot of effort into it. There is a story in, you can think about double descent with respect to training samples and see that. You can back out a picture that looks somewhat like the double descent curve from generalization curves that are associated to one of these kind of transitions I was describing earlier.

• 1:06:18Speaker 02

  So potentially, there's an explanation there. I don't.

• 1:06:22Speaker 01

  I'm just worried that at some points, we've gotten too deep into the technical details. So if I can take a chance to like reclarify or restate some things I've said earlier.

• 1:06:32Speaker 02

  Yeah, sure.

• 1:06:32Speaker 01

  Right, always that.

• 1:06:35Speaker 01

  Okay, so the big picture, we want to end up with a friendly AI.

• 1:06:40Speaker 01

  And we don't understand enough about what's going on inside of the model to currently make any kinds of guarantees of this model is aligned, this model is not aligned. They can exhibit the same behavior in training and then generalize differently when deployed. And so for me, the basic premise of interpretability is understanding model internals to a sufficient degree that we can disambiguate these two cases and that we can understand generalization. And SLT tells us that we can look at the geometry of the law landscape. We can look at how structure is reflected in this geometry to start getting at this question. With this principled link to in-distribution generalization, that's a starting point for developing a theory of out-of-distribution generalization

• 1:07:24Speaker 01

  and understanding better what happens in the case of S G D. So on the interpretability side, what this looks like in practice is you can probe a bunch of points in the law landscape, apply small perturbations to your model, you know, hit it with a hammer and see how it responds to that probe, to that small perturbation. And this tells you something about what's going on internally. And this is the basis for starting to extract information about model internals from geometry. So we've been looking at a series of projects, papers, where we study first small language models, subject them to these kinds of perturbations, and then measure things like how much does the performance decrease? How much worse does the performance get under these kinds of perturbations? That tells you about complexity of the solution the model has learned.

• 1:08:09Speaker 01

  How much does performance increase if I apply these structured perturbations to a specific part of the model? An attention head or a component? And this, it turns out, can tell you something about this attention head is specialized to this kind of data. These attention heads are all induction heads. These ones are doing something else, like memorizing different n-grams and skip n-grams. And so you can start to actually tear apart the internals of the model using these kinds of probes. More recently, we've been looking at applications for something like circuit discovery. So in a principled way, can you associate the components of the model with patterns and data? And do this attribution from what structures the model ends up with to the data that activates that structure?

• 1:08:52Speaker 01

  And this is starting to get the kind of understanding that could lead to an understanding of, you know, why the model exhibits the behavior it does rooted in its internal structures, without necessarily having a full mechanistic understanding of what's going on. You can do the sensitivity analysis and it can tell you that these components are involved without needing to know exactly causally what the input to output was. So this is more like a top-down approach to interpretability informed through this perturbation analysis.

• 1:09:25Speaker 01

  Okay, so on the one side, we want to develop tools for interpretability. On the other side, at some point, we want to be able to steer the learning process so that we don't end up with a misaligned model in the first place. We sort of have to take a preventative approach. Well, ultimately, that's going to look like intervening in the training data. And so the hope is, to the degree that the mapping from training data through geometry philosophy landscape, through the learning process, to the final weights you end up with is invertible, to some extent, can you go backwards and come up with techniques for choosing training data that aligns your model more robustly? And so these are the kinds of applications of SLT that we're trying to develop for alignment.

• 1:10:08Speaker 01

  We're only just getting started here, but what our real hope is for this agenda in the long term is to give us better ways to make models more aligned.

• 1:10:18Nathan Labenz?

  So are those sorts of like bell ringing exercises? I'm imagining as you try to scale this up to larger and larger systems, you would maybe be running experiments like if we add some noise, like how often do we see behavior that we didn't want, or you know, sort of has some maybe connections to like AI control type ideas too? Like how often does the thing take debate? You know, if we set up a situation where there's some bad thing that it could do,

• 1:10:48Nathan Labenz?

  how often does that happen with different perturbations? And the less it happens, maybe the like more aligned we are? Is that the sort of measurement that you would imagine taking?

• 1:11:00Speaker 02

  Yeah, I don't think I'm gonna bite on that particular proposal, but if you think that you can elicit something about the structures in the model, which at the end of training, you have some understanding, produce the concerning behavior and trace the origin, to go back to the developmental perspective again, if you can trace the origin of those structures and behaviors to some kind of pattern in the data, then you can try and intervene on that, using your understanding of that pattern, intervene at some earlier point to to shape that structure differently. Now, obviously, that's, you know, for a complex

• 1:11:41Speaker 02

  behavior and very complex internal structure, an ambitious thing to attempt, right? And so that's one, that's one of the reasons why some progress on interpretability from our point of view is essential to unpacking that, right? So if you imagine some complex mechanism which is producing the undesired behavior, uh, well, you're not just gonna, I mean, I think it's naive to pretend you're just gonna find like the be good samples out there and like throw those in the training process at 12,000 steps and be like, be good, be good. I don't think that's sufficiently fine grained, but if you have some understanding

• 1:12:20Speaker 02

  of how that complex behavior came together, right? So this mechanism, however, was shaped over development and what factors went into that shaping, then it's not crazy to imagine that there's something analogous to morphogens. So morphogens, I mean, this is a sort of umbrella term that refers to factors in development of biological organisms that shape when things happen or what things happen, right? There could be particular molecules or proteins, and these dictate the way that development works in biology, right? And if you insert the right morphogens at the right time, you can change the outcome of development. And so this is like maybe one guiding analogy for why it makes sense to

• 1:13:03Speaker 02

  hypothesize that if you understand the developmental process well enough, you might be able to intervene at various points strategically to change the outcome. And indeed, that's how synthetic biology works, right? You understand the development process, you understand the morphogens, I mean, some parts of synthetic biology, and then you can program the development in a way you prefer.

• 1:13:24Nathan Labenz?

  One question on just the nature of overgeneralization. Is reward hacking an example of overgeneralization? Where the model has sort of, you know, what we wanted to specify was some richer concept than the signal we gave, and so it sort of has gone too far in understanding the true nature of that signal beyond what we meant it to be, and so that's why we see reward hacking? Fair?

• 1:13:54Speaker 02

  I think like, uh, could be. I think there's like a bucket of things that are, I mean, reward hacking has a technical definition under which the answer to your question is no. So like the technical way that people would introduce reward hacking is simply to say that it has found a way of getting more reward than the solution you intended, right? So that's, that's purely about the reward, which is somewhat analogous to the loss, right? So like, that's, that's not really about complexity. That's just about getting more of the actual explicit target of the optimization process. So under that definition, the answer to your question is no, like reward hacking is not an instance of this trade off we were referring to earlier, which you're terming overgeneralization.

• 1:14:38Speaker 02

  However, it could very well be, I think, that instances that we are currently referring to as reward hacking are perhaps partly a trade off in favor of a simpler solution rather than the one we intended. I don't claim to have evidence for that, but I would maybe hypothesize that. But yeah, I think you're sort of referring to one of the hypotheses we have for one of the ways in which dangerous behavior could arise, which is somewhat different to the usual story of how dangerous behavior could arise from sort of over optimizing a reward target, uh, which is, um, to go back to the experiments that Jesse was referring to

• 1:15:20Speaker 02

  earlier, very briefly, um, so there, there was a simpler solution, which was ridge regression. In theory, if you train long enough, you should always switch to memorizing the data because the optimization pressure wants you to get the lowest loss possible. The lowest loss possible is to memorize the data.

• 1:15:41Speaker 02

  If you can, right? Like typically like training G P T model, you just can't memorize the data. So you have to generalize and so on. But if you can memorize, that's like the best thing to do, given the training signal. But what we see actually in that experiment is that in many cases, the model sticks at the generalizing solution and never switches to the memorizing one. So that's a situation in which if you imagine the lost landscape, and to just draw a curve with my hand despite my complaints about that earlier. So if you imagine like in the very bottom of the lost landscape, there's the thing that the training data actually specifies is the optimal solution, that in this case, you can prove mathematically is to op is to memorize the tasks. But there

• 1:16:26Speaker 02

  is a higher loss, but simpler solution, which doesn't memorize, but does some more general thing. And training can get stuck there forever, even though this is the global minimum. I mean, I say forever. Obviously, we didn't train forever, but it seems like it would stay there. And this is a sort of, I think, a good heuristic to have contra some of the normal stories about A I risk, that it may be that you don't get what you ask for, you don't get what you try to specify in the training data, you get a simplification, you get a heuristic, you get an approximation, which is not what you intended or expected.

• 1:17:07Speaker 02

  That could be good, it could be bad. So I think mostly in the context of A I safety, people have an intuition that actually the simpler thing will be better, right? Because they have a, so just to communicate an often told sort of way of thinking about this, if you imagine the the model becomes scheming, becomes a kind of scheming agent.

• 1:17:37Speaker 02

  Even though you didn't optimize for this, so it's trying to do the task. It has structure in its weights to do the task. And then it needs to like tack on some additional structure to take over the world. Right. And because that would be more complex than the solution that just does the task. Then, okay, and we have a simplicity bias. That's great because we have this very strong pressure to remove this additional stuff. That is going to be dangerous. That very well may be a reasonable way of thinking about it. I think one should have not very much confidence in this. This is like quite a imprecise story, not grounded really in very much like mathematically. So that's potentially the case, but there's a, just to tell a sort of opposite story.

• 1:18:20Speaker 02

  It could be the case that the simplification is, is quite dangerous. And if you indulge me, there's actually a historical example here. That's really quite interesting. So something called the Windrush scandal. Uh, so the UK, the 1940s, I think, allowed citizens of the British Empire to come and settle in the UK without really any documentation. So many people came to the UK, I think primarily from the Caribbean. Then in the 1970s, they switched the laws. So they, they started insisting that people have documentation of their citizenship in order to access services. And these people who came as part of this program didn't have the relevant documentation.

• 1:19:04Speaker 02

  And some of them were deported and many bad things happened. And there was an investigation and scandal in 2018, I think. Um, looking into this, this is a case where you have a policy, which is like, obviously the intended, intended policy of the state is not to like deport people who it led into the country earlier, right? But when you follow that policy, the way it's actually implemented at the kind of street level is a

• 1:19:34Speaker 02

  Simplification of the original intent. It's this kind of very brute judgment call at the level of the street level bureaucrat. And that's what this kind of phenomenon is referred to, street level bureaucracy, which is a simplification of the thing that the original policy was trying to specify. And this ends up having unintended consequences, which are negative. So I think there's a case to be made that in AI safety, we should also be quite concerned about what happens if the behavior we are specifying at great expense by collecting training data ends up being like some kind of simplification in various ways we don't

• 1:20:14Speaker 02

  anticipate of the intended target behavior in ways that are undesirable.

• 1:20:22Nathan Labenz?

  Yeah, seeing like a state and seeing like a model, maybe more similar than different than certainly relative to what I had conceived of previously. I think I maybe have two more questions. One is, I've heard Jesse kind of tell the story in the past in other fora around like, maybe model training in the future could look more like refining oil does today, where it's a very sort of, you start with a big mess, but you can sort of figure out exactly what components need to be added, when you have a sense for when these phase transitions are happening, and you have just a much more precise level of control over that process.

• 1:21:06Nathan Labenz?

  I'd love to hear a little bit better version of that story than what I just offered. And then my other one is just, what do you guys make of other safety efforts? I mean, you've sort of said, like, probably can't just like get the be good examples and hope for the best, or at least that doesn't seem adequate. But I'd be interested in your sort of handicapping of, you know, all the various, or at least some of, you know, the most prominent various techniques that are out there today. Like, what do you think has the best hope of working? Or, you know, what do you think is doomed? So yeah, kind of your vision for the future of of model training and the level of control we might really be able to require or acquire over that process? Maybe you could talk about like the compute, you know, how you scale and the compute, because I don't think we ever touched on that.

• 1:21:51Nathan Labenz?

  And then this, you know, this handicapping of the landscape, there's a lot there, but I'll let you have at it.

• 1:21:56Speaker 01

  You could start with the oil refining example. So I think it's useful to compare current deep learning to alchemy. Right, we've got some huge cauldron, that's the architecture. We've got a fire, that's the optimizer. And then we've got the reagents we put in this cauldron that we start to mix together. Those reagents are the training data. And currently, we just throw the entire internet together into this cauldron, start spurring and hope that we haven't accidentally mixed bleach and Clorox. You know? Okay. So what we have in mind for the future of training looks something more like industrial chemical manufacturing, where you know exactly which reagents you're mixing together, at

• 1:22:34Speaker 01

  what concentration, at what point in the process, and with what catalyzers and ingredients. And this level of control is not something that's accessible to us right now, but it's something that we're already empirically starting to develop. So we have splintered the post-training process into a bunch of different stages. There's, you know, Chain of Thought RL1, there's a second stage of Chain of Thought RL, there's a little bit of instruction fine-tuning maybe somewhere, there's Constitutional AI1, there's the second stage of Constitutional AI just for the personality, then maybe some more refusal training. Similarly, on the pre-training side, this process is being split into multiple stages

• 1:23:16Speaker 01

  so that we're developing an empirical understanding of how to better control the learning process. What we think is possible is that you could develop the kind of scientific understanding to actually know when you should mix two samples in a batch together to get the desired behavior out the other end. And this will give you more control over the entire process and what you end up with.

• 1:23:38Nathan Labenz?

  Yo, just to interject, one example of that that struck me recently with the Claude 4 system card, there was this, I'm sure you've seen this anecdote. It hasn't been very deeply explained, although Sam Bowman talked about it a little bit online. They reported having observed that the model was following harmful system prompts in a way that surprised them. And then, you know, they're trying to figure out why. Sure enough, it turns out that they had omitted the Cis-Prompt Harmful data set, which had been developed specifically to teach the model what to do when given a harmful system prompt.

• 1:24:18Nathan Labenz?

  And so you have the, you know, sort of simpler solution of always follow the system prompt is is what it ends up learning for lack of that. You know, curb being included as it was meant to be. Interestingly, from my perspective,

• 1:24:34Nathan Labenz?

  they did not go back and retrain. I think there's a lot. I'm still kind of unpacking like, what exactly should I be inferring about the world on multiple different dimensions from the fact that, you know, they didn't just say, oh, shit, we messed that up, like revert and pick up, you know, where that thing was supposed to be introduced and and kind of continue on. Presumably that means it was happening somewhat earlier in the process than, you know, some some very late finishing stage. Presumably it also means they have constraints at Anthropic in terms of either compute budget or timeline to launch for competitive and misreasons or whatever that, you know, there's a whole interesting analysis to be done there, too. But that's at least one of these sort of, you know, very

• 1:25:14Nathan Labenz?

  recent and like concrete examples where without necessarily a full theory to drive it, you know, they that is starting to kind of emerge and you know, you better make sure you get those of data sets in the right place at the right time, or now it's it honestly seems like we have sort of a Claude. I had a little private conversation with somebody at Anthropic just to ask for clarification on this. They basically said, we're confident we got the value of that data set in the end. And I was also kind of, so I'm like, okay, that's interesting. I sort of believe you and you definitely know more than me. At the same time, how can you be confident? Like, what do you even, you know, on what basis could you have any confidence about that, other than obviously poking at the model a lot? But I mean, that's the sort of confidence, I guess, that you hope to create in the process of continuing to do this work.

• 1:25:58Speaker 02

  Yeah, I mean, I think it's it isn't an unreasonable expectation on behalf of like all of our civilizations about to be transformed by this technology to have it be somewhat more of an engineering science and somewhat less of a big like not not that it's I mean, it's not like nobody understands anything, right? Like the data mixtures that are selected and the order that they are selected is like empirically very finely tuned. Lots of effort goes into that. Selecting hyperparameters has some underlying theory to it. So, you know, lots of people and lots of money is being poured into deep learning, obviously. So it's not not completely incomprehensible. But I think most people would agree that

• 1:26:43Speaker 02

  it would be highly desirable to have it be much more understood and much more like other aspects of engineering, which we submit to safety engineering than deep learning is currently. That seems hard to disagree with, I would say. Um, that's not an easy thing to do, especially in a short period of time. Um, to come back to your question about scaling and the situation we're in vis a vis kind of where Anthropic was with SAEs a few years ago and so on. I think this analogy seems pretty apt to us. I mean, I think it's been quite a difficult path to get from sort of fairly pure abstract

• 1:27:24Speaker 02

  theory to validate it in real systems. And then, I mean, now we're just starting to do things like circuit discovery. We can find induction circuits and some simple things like that, you know, kind of unsupervised way. Does seem like there's, you know, scaling to do. And that's one of the main things that now remains at least to get up to a point where we're finding sort of complex internal structure. Um, yeah, in terms of compute, that's a little difficult to say. I mean, the the compute overhead of computing SAEs, for instance, where you train the SAEs, and that's very expensive. Our compute burdens are going to be in different places, but I imagine it's still very substantial to try and understand large models who don't really, um, we have been studying models

• 1:28:09Speaker 02

  like up to seven billion parameters and so on. Um.

• 1:28:14Speaker 01

  We can definitely use more compute. If a listener has compute to offer, please.

• 1:28:19Speaker 02

  Yeah, that's right.

• 1:28:20Speaker 01

  Maybe, maybe something down here. So in particular, if you have more compute, you can draw more samples, apply more perturbations, and that gives you a finer signal about what's going on inside of the model. And so expect that generally, yeah, we will be able to translate more compute into a finer understanding of structure.

• 1:28:39Speaker 02

  Yeah. Regarding other sort of approaches to alignment, I mean, I think in many respects, the things we're describing are just like useful foundations for many different things you might do. So I guess that's one way we think about it. So for instance, the stuff we're doing around shaping the data distribution and so on is potentially useful for some of the work the UK AI Security Institute wants to do around elicitation. So we have various ideas about how the work we're doing can, I mean, some of it is just kind of good solid foundational science of understanding models and understanding the training process and and how model behavior works and that, you know, in the same way that interpretability is broadly useful, much of that is broadly useful.

• 1:29:24Speaker 02

  I guess, yeah, it seems unclear whether the current paradigm of shaping model behavior at the end of training from like, is going to be a very robust solution to alignment as we move towards higher capability levels. I mean, this is an oft expressed concern, right? So I think that something that looks like baking in behavior and control of behavior earlier in the training process seems like something that is broadly going to be adopted. I mean, arguably, the deliberative alignment approach is, especially if you scale up the amount of compute spent on RL, right? If that is like equal to pre-training, then that's like a lot of shaping of model behavior

• 1:30:08Speaker 02

  that is taking place or a large part of the effect of compute.

• 1:30:12Speaker 01

  And this idea is not just due to us, right? There's papers like pre-training from human feedback. A lot of people realize that we need to bring alignment earlier in the, in the learning process. Maybe I can end with a bit of a look to the next few months. So currently, what a lot of our effort looks like is scaling techniques up to larger models and starting to just run lots of experiments at larger models. So we've validated these circuit discovery techniques in small three million parameter language models. Now can we do this in seven billion parameter language models? We think this is quite likely to work by the end of the year. On the on the alignment front, we're starting to do our first experiments in terms of applying these techniques to try to steer the learning

• 1:30:53Speaker 01

  process. This is still early days. We'll probably have early signs of life by the end of the year in small language models. And then we're developing these applications for things like elicitation, data attributions. So there's a singular learning theory extension to influence functions that you can study. It gives us ideas about, again, influence of samples on other behaviors and trying to use these kinds of applications to really validate that SLT and the techniques we're developing have something to offer that goes beyond existing techniques. That's where we're headed.

• 1:31:28Nathan Labenz?

  Cool. I love it. This has been fascinating. I appreciate you indulging me in so many little side questions and, you know, attempts to develop my own intuition. And I certainly think the, you know, the big picture need for a proper developmental understanding of how these AIs are forming and, you know, any theoretical basis at all for what they're going to do when they get into a truly out of distribution situation is badly needed. And especially as we, you know, get closer to 2027 and beyond is going to really help me sleep well at night. So to quote my dad, who often quotes Leslie Nielsen from Airplane,

• 1:32:12Nathan Labenz?

  "Good luck. We're all counting on you." Anything, anything we didn't touch on you want to leave people with real quick?

• 1:32:20Speaker 01

  Yeah, I guess if you want to learn more about singular learning theory and read about some of the papers we discussed, some of the papers we didn't discuss, you can go to our website. That's timaeus.co, T I M A E U S dot C O. There's also a Discord server link there. So that's in particular the place to learn more. And we have a weekly seminar and plenty of opportunity to get engaged with the research going on.

• 1:32:45Nathan Labenz?

  Cool. Yeah, I've typed it wrong more than once myself, so it's T I M A E U S dot co. And it's a it's a brain stretching exercise, at least for me and probably for most of us. But I've definitely enjoyed stretching my brain in this way. So look forward to future progress. You know, we can do a check back in maybe in a year's time or whatever and see how much we've closed the gap on all these important questions. But for now, I will say, Jesse Hoogland and Daniel Murfet, founders of TMAUS, thank you for being part of the cognitive revolution. Thank you, Nathan.

• 1:33:20Nathan Labenz?

  It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please don't hesitate to reach out via email at T C R at turpentine.co, or you can DM me on the social media platform of your choice.