Adam Marblestone — AI is missing something fundamental about the brain

Adam Marblestone https://twitter.com/AdamMarblestone is CEO of Convergent Research https://www.convergentresearch.org/. He’s had a very interesting past life: he was a research scientist at Google Deepmind on their neuroscience team and has worked on everything from brain-computer interfaces to quantum computing to nanotech and even formal mathematics. In this episode, we discuss how the brain learns so much from so little, what the AI field can learn from neuroscience, and the answer to Ilya’s question: how does the genome encode abstract reward functions? Turns out, they’re all the same question. Watch on YouTube https://youtu.be/_9V_Hbe-N1A; read the transcript https://www.dwarkesh.com/p/adam-marblestone. Sponsors * Gemini 3 Pro https://gemini.google.com recently helped me run an experiment to test multi-agent scaling: basically, if you have a fixed budget of compute, what is the optimal way to split it up across agents? Gemini was my colleague throughout the process — honestly, I couldn’t have investigated this question without it. Try Gemini 3 Pro today gemini.google.com https://gemini.google.com * Labelbox https://labelbox.com/dwarkesh helps you train agents to do economically-valuable, real-world tasks. Labelbox’s network of subject-matter experts ensures you get hyper-realistic RL environments, and their custom tooling lets you generate the highest-quality training data possible from those environments. Learn more at labelbox.com/dwarkesh https://labelbox.com/dwarkesh To sponsor a future episode, visit dwarkesh.com/advertise https://www.dwarkesh.com/advertise. Timestamps (00:00:00) – The brain’s secret sauce is the reward functions, not the architecture (00:22:20) – Amortized inference and what the genome actually stores (00:42:42) – Model-based vs model-free RL in the brain (00:50:31) – Is biological hardware a limitation or an advantage? (01:03:59) – Why a map of the human brain is important (01:23:28) – What value will automating math have? (01:38:18) – Architecture of the brain Further reading Intro to Brain-Like-AGI Safety https://www.lesswrong.com/s/HzcM2dkCq7fwXBej8 - Steven Byrnes’s theory of the learning vs steering subsystem; referenced throughout the episode. A Brief History of Intelligence https://www.abriefhistoryofintelligence.com/book - Great book by Max Bennett on connections between neuroscience and AI Adam’s blog https://longitudinal.blog/, and Convergent Research’s blog on essential technologies https://www.essentialtechnology.blog/. A Tutorial on Energy-Based Learning http://yann.lecun.com/exdb/publis/pdf/lecun-06.pdf by Yann LeCun What Does It Mean to Understand a Neural Network? https://arxiv.org/abs/1907.06374 - Kording & Lillicrap E11 Bio https://www.e11.bio/ and their brain connectomics approach Sam Gershman on what dopamine is doing in the brain https://gershmanlab.com/pubs/GershmanUchida19.pdf Gwern’s proposal https://www.reddit.com/r/reinforcementlearning/comments/9pwy2f/wbe_and_drl_a_middle_way_of_imitation_learning/ on training models on the brain’s hidden states This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.dwarkesh.com https://www.dwarkesh.com?utm_medium=podcast&utm_campaign=CTA_1

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Dwarkesh Patel?

  The big million dollar question that I have that I've been trying to get the answer to through all these interviews with AI researchers. How does the brain do it? Right? Like, we're throwing way more data at these LLMs and they still have a small fraction of the total capabilities that a human does. So what's going on?

• 0:14Speaker 01

  Yeah, I mean, this might be the quadrillion dollar question or something like that. It's arguably, you could make an argument, this is the most important question in science. I don't claim to know the answer. I also don't really think that the answer will necessarily come even from a lot of smart people thinking about it as much as they are. My overall meta-level take is that we have to empower the field of neuroscience to just make neuroscience a more powerful field technologically and otherwise to actually be able to crack a question like this. But maybe the way that we would think about this now with like modern AI, neural nets, deep learning,

• 0:56Speaker 01

  is that there are sort of these certain key components of that. There's the architecture. There's maybe hyperparameters of the architecture. How many layers do you have or sort of properties of that architecture? There is the learning algorithm itself. How do you train it? You know, backprop, gradient descent. Is it something else? There is, how is it initialized? Okay, so if we take the learning part of the system, it still may have some initialization of the weights.

• 1:24Speaker 01

  And then there are also cost functions. There's like, what is it being trained to do? What's the reward signal? What are the loss functions, supervision signals? My personal hunch within that framework is that the field has neglected the role of this very specific loss functions, very specific cost functions.

• 1:45Speaker 01

  Machine learning tends to like mathematically simple loss functions, right? Predict the next token.

• 1:52Speaker 01

  You know, cross entropy, these simple kind of computer scientists' loss functions. I think evolution may have built a lot of complexity into the loss functions. Actually, many different loss functions were different areas, turned on at different stages of development. A lot of Python code, basically, generating a specific curriculum for what different parts of the brain need to learn. Because evolution has seen many times what was successful and unsuccessful. And evolution could encode the knowledge of the learning curriculum. So in the machine learning framework, maybe we can come back and we can talk about, yeah, where do the loss functions of the brain come from? Can that, can loss, different loss functions lead to different efficiency of learning?

• 2:31Dwarkesh Patel?

  You know, people say like, the cortex has got the universal human learning algorithm, the special sauce that humans have. What's up with that?

• 2:37Speaker 01

  This is a huge question, and we don't know. I've seen models where what the cortex, you know, the cortex has typically this like six-layered structure. Layers in a slightly different sense than layers of a neural net. Because like any one location in the cortex has six physical layers of tissue. As you go in, layers of the sheet. And then those areas then connect to each other. And that's more like the layers of a network. I've seen versions of that where what you're trying to explain is actually just how does it approximate backprop? And what is the cost function for that? What is the network being asked to do? If you are sort of are trying to say it's something like backprop. Is it doing backprop on next token prediction? Is it doing backprop on classifying images? Or what is it doing?

• 3:21Speaker 01

  And no one, no one knows. But I think, I think one, one thought about it, one possibility about it is that it's just this incredibly general prediction engine. So, so any one area of cortex is just trying to predict. Any, basically, can it learn to predict any subset of all the variables it sees from any other subset? So like omnidirectional inference or omnidirectional prediction. Whereas an LLM is just you see everything in the context window. And then it, it computes a very particular conditional probability, which is, given all the last thousands of things, what is the very probabilities for all the, all the, the next token?

• 4:03Dwarkesh Patel?

  Yeah.

• 4:04Speaker 01

  But it would be weird for a large language model to say, you know, you know, the quick brown fox, blank, blank, the lazy dog, and fill in the middle.

• 4:15Dwarkesh Patel?

  Yeah.

• 4:16Speaker 01

  Versus do the next token if it's doing just forward. It can learn how to do that stuff in this emergent level of in-context learning, but natively it's just predicting the next token. What if the cortex is just natively made so that it can, you know, any area of cortex can predict any pattern in any subset of its inputs, given any other missing subset? That is a little bit more like, quote unquote, probabilistic AI. I think a lot of the things I'm saying, by the way, are extremely similar to like what Yann LeCun would say.

• 4:48Dwarkesh Patel?

  Yeah.

• 4:48Speaker 01

  He's really interested in these energy-based models and something like that. It's like the joint distribution of all the variables. What is the, what is the likelihood or unlikelihood of just any combination of variables? And if I, if I clamp some of them, I say, well, definitely these variables are in these states, then I can compute with probabilistic sampling. For example, I can compute, okay, conditioned on these being set in this state, what are, and these could be any arbitrary subset of, of, of variables in the model. Can I predict what any other subset is going to do and sample from any other subset, given clamping this subset? And I could choose a totally different subset and sample from that subset. So it's omnidirectional inference.

• 5:31Speaker 01

  And so, you know, that could be, there's some parts of cortex that might be like association areas of cortex that may, you know, predict vision from audition.

• 5:38Dwarkesh Patel?

  Yeah.

• 5:39Speaker 01

  There might be areas that predict things that the more innate part of the brain is going to do, because remember, this whole thing is basically riding on top of the sort of a lizard brain and lizard body, if you will. And that thing is a thing that's worth predicting, too. So you're not just predicting, do I see this or do I see that? But is this muscle about to tense? Am I about to have a reflex where I laugh? You know, is my heart rate about to go up? Am I about to activate this instinctive behavior?

• 6:06Dwarkesh Patel?

  Based on my higher-level understanding of, like, I can match, uh, somebody has told me there's a spider on my back to this lizard part that would activate if I was like literally seeing a spider in front of me. And you, you learn to associate the two, so that even just from somebody hearing you say, there's a spider on your back.

• 6:25Speaker 01

  Yeah, let's, well, let's come back to this. And this, this is partly having to do with, with Steve Byrne's theories, which I'm recently obsessed about. But, but on your podcast with Ilya.

• 6:36Speaker 01

  He said, "Look, I'm not aware of any good theory of how evolution encodes high-level desires or intentions." I think this is like, this is like, very connected to all of these questions about the loss functions and the cost functions

• 6:54Speaker 01

  that the brain would use. And it's a really profound question, right? Like, like, let's say that

• 7:01Speaker 01

  I am embarrassed for saying the wrong thing on your podcast because I'm imagining that Yann LeCun is listening. And he says, "That's not my theory. You described energy-based models really badly." That's going to enact, activate in me innate embarrassment and shame, and I'm going to want to go hide and whatever. And that's going to activate these innate reflexes. And that's important because I might otherwise get, get killed by Yann LeCun's, you know, marauding army of other... The French AI researchers coming for you, Adam. And so it's important that I have that instinctual response. But of course, evolution has never seen Yann LeCun or known about energy-based models or known what a important scientist or a podcast is. And so somehow, the brain has to encode this desire to, you know,

• 7:46Speaker 01

  not, not piss off really important, you know, people in the tribe or something like this

• 7:52Speaker 01

  in a very robust way, without knowing in advance, all the things that the, the learning subsystem, okay, of the brain, the part that is learning, cortex and other parts. The cortex is going to learn this world model. It's going to include things like Yann LeCun's and, and podcasts. And evolution has to make sure that that, those neurons, whatever the Yann LeCun being upset with me neurons, get properly wired up to the shame response or this part of the reward function.

• 8:22Speaker 01

  And this is important, right? Because if we're going to be able to seek status in the tribe or learn from knowledgeable people, as you said, or things like that. Exchange knowledge and skills with friends, but not with enemies. I mean, we have to learn all this stuff. So it has to be able to robustly wire these learned features of the world, learn parts of the world model up to these innate reward functions, and then actually use that to then learn more, right? Because next time, I'm not going to try to piss off Yann LeCun if he emails me that, that I got this wrong.

• 8:50Speaker 01

  And so I'm going to do further learning based on that. So in constructing the reward function, it has to use learned information. But how can evolution, evolution didn't know about Yon Le Kuhn. So how can it, how can it do that? And so the basic idea that Steve Burns is proposing is that well, part of the cortex or other areas like the amygdala that learn, what they're doing is they're modeling the steering subsystem. Steering subsystem is the part with these more innate, innately programmed responses and the innate programming of these series of reward functions, cost functions, bootstrapping functions that exist. So there are parts of the amygdala, for example, that are able to monitor what, what those parts do and predict what those parts do.

• 9:30Speaker 01

  So, so how do you find the neurons that are important for social status? Well, you have some innate heuristics of social status, for example, or you have some innate, innate heuristics of friendliness that, that the steering subsystem can use. And the steering subsystem actually has its own sensory system, which is kind of crazy. So we think of, you know, vision as being something that the cortex does. But there's also a steering subsystem, subcortical visual system called the superior calliculus with innate ability to detect faces, for example, or threats. So it, so there's a visual system that has innate heuristics and that the steering subsystem has its own responses. So there'll be part of the amygdala or part of the cortex

• 10:15Speaker 01

  that is learning to predict those responses. And so what are the neurons that are, that matter in the cortex for social status or for friendship? Or they're the ones that predicts those innate heuristics for friendship. Right? So you train a predictor in the cortex and you say, which neurons are part of the predictor? Those are the ones that are, now it's, now you've actually managed to wire it up.

• 10:35Dwarkesh Patel?

  Yeah, this is fascinating. I feel like I still don't understand. I understand how the cortex could learn how this primitive part of the brain would respond to.

• 10:49Dwarkesh Patel?

  So it can obviously, it has these labels on, here's literally a picture of a spider. And this is bad, like be scared of this.

• 10:56Speaker 01

  Right.

• 10:56Dwarkesh Patel?

  And then the cortex learns that this is bad because the innate part tells it that. But then it has to generalize to, okay, the spider's on my back.

• 11:04Speaker 01

  Yes.

• 11:05Dwarkesh Patel?

  And somebody's telling me the spider's on your back, that's also bad.

• 11:07Speaker 01

  Yes.

• 11:07Dwarkesh Patel?

  But it never got supervision on that. So how does it?

• 11:11Speaker 01

  Well, it's because the learning subsystem is a powerful learning algorithm that does have generalization, that is capable of generalization. So the steering subsystem, these are the innate responses. So you're going to have some, let's say, built into your steering subsystem, these lower brain areas, hypothalamus, brain stem, et cetera. And again, they include, they have their own primitive sensory systems. So there may be an innate response.

• 11:38Speaker 01

  If I see something that's kind of moving fast toward my body that I didn't previously see was there and is kind of small and dark and high contrast, that might be an insect kind of skittering onto my body. I am going to like flinch, right?

• 11:53Speaker 01

  And so there are these innate responses. And so there's going to be some group of neurons, let's say in the hypothalamus, that is that I am flinching.

• 11:59Dwarkesh Patel?

  Yeah.

• 11:59Speaker 01

  Or I just flinched.

• 12:01Dwarkesh Patel?

  Right.

• 12:01Speaker 01

  Right, the, the, I just flinched neurons in the hypothalamus. So when you flinch, first of all, that negative contribution of the reward function, you didn't want that to happen perhaps.

• 12:12Speaker 01

  But that's only happen, that's a reward function then that is, it doesn't have any generalization in it. So I'm going to avoid that exact situation of the thing skittering toward me. And maybe I'm going to avoid some actions that lead to the thing skittering. So that's, that's something, a generalization you can get. What Steve calls it is downstream of the reward function. So I'm going to avoid the situation where the spider was skittering toward me. But you're also going to do something else. So there's going to be like a part of your amygdala, say, that is saying, okay,

• 12:42Speaker 01

  a few, you know, a few milliseconds, you know, hundreds, hundreds of milliseconds or seconds earlier.

• 12:49Speaker 01

  Could I have predicted that flinching response? It's going to be a group of neurons that is essentially a classifier of, am I about to flinch? And I'm going to have classifiers for that for every important steering subsystem variable that evolution needs to take care of. Am I about to flinch? Am I talking to a friend? Should I laugh now? Is the friend high status? Whatever variables the hypothalamus brainstem contain. Am I about to taste salt? So it's going to have all these variables. And for each one, it's going to have a predictor. It's going to train that predictor. Now, the predictor that it trains, that can have some generalization. And the reason it can have some generalization is because it just has a totally different input. So its input data might be things like the word spider. But the word spider can activate in all sorts of situations that lead to the word spider

• 13:32Speaker 01

  activating in your world model. So if you have a complex world model with really complex features, that inherently gives you some generalization. It's not just the thing skittering toward me. It's even the word spider or the concept of spider is going to cause that to trigger. And this predictor can learn that. So whatever spider neurons are in my world model, which could even be a book about spiders or a room where there are spiders or whatever that is.

• 13:58Dwarkesh Patel?

  The amount of heebie-jeebies that this conversation is eliciting in the audience is like...

• 14:02Speaker 01

  So now I'm activating your steering subsystem. Your steering subsystem spider hypothalamus subgroup of neurons of skittering insect are activating based on these very abstract concepts in the conversation.

• 14:14Dwarkesh Patel?

  If we keep going, I'm going to have to put in a trigger warning.

• 14:16Speaker 01

  That's because you learned this. And the cortex inherently has the ability to generalize because it's just predicting based on these very abstract variables and all these integrated information that it has, whereas the steering subsystem only can use whatever the superior click list and a few other sensors can spit out.

• 14:31Dwarkesh Patel?

  By the way, it's remarkable that the person who's made this connection between different pieces of neuroscience, Stephen Burns, like former physicist, has for the last few years has been trying to synthesize.

• 14:41Speaker 01

  He's an AI safety researcher. He's just synthesizing. This comes back to the academic incentives thing. I think that this is a little bit hard to say. What is the exact next experiment? How am I going to publish a paper on this? How am I going to train my grad student to do this? It's very speculative. But there's a lot in the neuroscience literature and Stephen has been able to pull this together. And I think that Steve has an answer to Ilya's question essentially, which is how does the brain... Ultimately code for these higher level desires and link them up to the more primitive rewards. Yeah.

• 15:05Dwarkesh Patel?

  Very naive question, but why can't we achieve this omnidirectional inference by just training the model to not just map from a token to next token, but remove the masks in the training. So it maps every token to every token or come up with more labels between video and audio and text so that it is forced to map one to each one.

• 15:30Speaker 01

  I mean, that may be, that may be the way. So it's, it's not clear to me. Some people think that there's sort of a different way that it does probabilistic inference or different learning algorithm that isn't backprop. There might be like other ways of learning energy based models or other things like that that you can imagine, but that is involved in being able to do this and that the brain has that. But I think there's a version of it where, you know, what the brain does is like crappy versions of backprop to learn to predict, you know, through a few layers. And that, yeah, it's kind of like a multimodal foundation model.

• 16:04Dwarkesh Patel?

  Right.

• 16:04Speaker 01

  Yeah, so maybe the cortex is just kind of like a, certain kinds of foundation models. There, you know, LLMs are maybe just predicting the next token, but, you know, vision models maybe are trained in learning to fill in the blanks or reconstruct different pieces or combinations. But, but I think that it does it in an extremely flexible way. So it's, you know, if you train a model to just fill, to fill in this blank at the center, okay, that's great. But what if you didn't train it to, to fill in this other blank over to the left? Then it doesn't know how to do that. It's not part of its like repertoire of predictions that are like amortized into the network. Whereas with a really powerful inference system, you could choose at test time, you know, what is the, the subset of variables it needs to infer and what, which ones are clamped.

• 16:46Dwarkesh Patel?

  Okay, so two sub questions. One, it makes you wonder whether the thing that is lacking in artificial neural networks is less about the reward function and more about the encoder or the embedding.

• 17:00Dwarkesh Patel?

  Which like, maybe the issue is that you're not representing video and audio and text in the right latent abstraction such that they could intermingle and conflict. Maybe this is also related to why L L M C bad at drawing connections between different ideas. Like, it's like, are the ideas represented at a level of generality at which you could, which you could notice.

• 17:26Speaker 01

  Well, the problem is these questions are all co-mingle. So if we don't know if it's doing a backprop like learning, and we don't know if it's doing energy-based models, and we don't know how these areas are even connected in the first place, it's like very hard to like really get to the ground truth of this. But yeah, it's possible. I mean, I think that people have done some work. My friend Joel DePello actually did something some years ago where I think he put a model, I think it was a model of V1 of sort of specifically how the early visual cortex represents images and put that as like an input into like a ConvNet and that like improved some things. So it could be, it could be like differences. The retina is also doing, you know, motion detection and certain things are kind of getting filtered out. So there may be some preprocessing of the sensory data.

• 18:11Speaker 01

  There may be some clever combinations of which modalities are predicting which or so on that that lead to better representation. There may be much more clever things than that. Some people certainly do think that there's inductive biases built in the architecture that will shape the representations, you know, differently or that there are clever things that you can do. So Astera, which is the same organization that employs Steve Behrens, just launched this neuroscience project based on Doris So's work. And she has some ideas about how you can build vision systems that basically require less training. They put in, they inbuild into the assumptions of the design of the architecture that things like objects are bounded by surfaces and this, you know, surfaces have certain types

• 18:56Speaker 01

  of shapes and relationships of how they occlude each other and stuff like that. So it may be possible to build more assumptions into the network. Evolution may have also put some changes of architecture. It's just, I think that also the cost functions and so on may be a key. A key thing that it does.

• 19:14Dwarkesh Patel?

  So Andy Jones has this amazing 2021 paper where he uses AlphaZero to show that you can trade off test time compute and training compute. And while that might seem obvious now, this was three years before people were talking about inference scaling. So this got me thinking, is there an experiment you could run today, even if it's a toy experiment, which would help you anticipate the next scaling paradigm? One idea I had was to see if there was anything to multi-agent scaling. Basically, if you have a fixed budget of training compute, are you going to get the smartest agent by dumping all of it into training one single agent or by splitting that compute up amongst a bunch of models, resulting in a diversity of strategies that get to play off each other? I didn't know how to turn this question into a concrete experiment, though, so I started brainstorming with Gemini 3 Pro in the Gemini app. Gemini helped me think through a bunch of different judgment calls.

• 19:59Dwarkesh Patel?

  For example, how do you turn the training loop from self-play to this kind of co-evolutionary league training? How do you initialize and then maintain diversity amongst different AlphaZero agents? How do you even split up the compute between these agents in the first place? I found this clean implementation of AlphaGo Zero, which I then forked and opened up in AntiGravity, which is Google's agent-first IDE. The code was originally written in 2017, and it was meant to be trained on a single GPU of that time. But I needed to train multiple whole, separate populations of AlphaZero agents, so I needed to speed things up. I rented a beefcake of a GPU node, but I needed to refactor the whole implementation to take advantage of all this scale and parallelism. Gemini suggested two different ways to parallelize self-play.

• 20:44Dwarkesh Patel?

  One, which would involve higher GPU context switching, and the other would involve higher communication overhead. I wasn't sure which one to pick, so I just asked Gemini. And not only did it get both of them working in minutes, but it autonomously created and then ran a benchmark to see which one was best. It would have taken me a week to implement either one of these options. Think about how many judgment calls a software engineer working on an actually complex project has to make. If they have to spend weeks architecting some optimization or feature before they can see whether it will work out, they will just get to test out so many fewer ideas. Anyways, with all the help from Gemini, I actually ran the experiment and got some results. Now, please keep in mind that I'm running This experiment on an anemic budget of compute, and it's very possible I made some mistakes in implementation.

• 21:27Dwarkesh Patel?

  But it looks like there can be gains from splitting up a fixed budget of training compute amongst multiple agents, rather than just dumping it all into one. Just to reiterate how surprising this is, the best agent in the population of 16 is getting 1/16 the amount of training compute as the agent trained on self-play alone. And yet, it still outperforms the agent that is hogging all of the compute. The whole process of vibecoding this experiment with Gemini was really absorbing and fun. It gave me the chance to actually understand how AlphaZero works and to understand the design space around decisions about the hyperparameters and how search is done and how you do this kind of co-evolutionary training, rather than getting bogged down

• 22:12Dwarkesh Patel?

  in my very novice abilities as an engineer. Go to gemini.google.com to try it out. I want to talk about this idea that you just glanced off of, which was amortized inference.

• 22:27Dwarkesh Patel?

  And maybe I should try to explain what I think it means, because I think it's probably wrong, and this will help you correct.

• 22:36Speaker 01

  It's been a few years for me, too.

• 22:37Dwarkesh Patel?

  Okay.

• 22:39Dwarkesh Patel?

  Right now, the way the models work is you have an input. It maps it to an output. And this is amortizing a process that the real process, which we think is like what intelligence is, which is like you have some prior over how the world could be. Like, what are the causes that make the world the way that it is? And then, when you see some observation, you should be like, okay, here's all the ways the world could be.

• 23:07Dwarkesh Patel?

  This cause explains what's happening best. Now, doing this calculation over every possible cause is computationally intractable. So then, you just have to sample like, oh, here's a potential cause. Does this explain this observation? No, forget it. Let's keep sampling. And then, eventually, you get the cause. The cause, then the cause explains the observation, and then this becomes your posterior.

• 23:33Speaker 01

  That's actually pretty good, I think, of sort of, yeah, this Bayesian inference, like in general, is like of this very intractable thing.

• 23:39Dwarkesh Patel?

  Right.

• 23:40Speaker 01

  The algorithms that we have for doing that tend to require taking a lot of samples, Monte Carlo methods, taking a lot of samples.

• 23:46Dwarkesh Patel?

  Yeah.

• 23:46Speaker 01

  And taking samples takes time. I mean, this is like the original like Boltzmann machines and stuff. We're using techniques like this.

• 23:55Speaker 01

  And still it's used with probabilistic programming, other types of methods often. And so, yeah, so the Bayesian inference problem, which is like basically the problem of like perception, like given some model of the world and given some data, like, how should I update my, what are the like, the variables, you know, missing variables in my, in my internal model?

• 24:13Dwarkesh Patel?

  And I guess the idea is that neural networks are hopefully, obviously, there's mechanistically, the neural network is not starting with like, here is my model of the world and I'm going to try to explain this data. But the hope is that instead of starting with, hey, does this cause explanation? No. Did this cause explanation? Yes. What you do is just like observation.

• 24:36Speaker 01

  What's the most, what's the cause that we, the neural net thinks is the best one?

• 24:40Dwarkesh Patel?

  Observation to cause. So the feed forward like goes observation to cause.

• 24:43Speaker 01

  Observation to cause.

• 24:44Dwarkesh Patel?

  To then the output.

• 24:45Speaker 01

  Yes. You don't have to, you don't have to evaluate all these energy values or whatever and sample around to make them higher and lower. You just say approximately that process would result in this being the top one or something like that.

• 24:58Dwarkesh Patel?

  Yeah. One way to think about it might be that test time compute, inference time compute is actually doing this sampling again, because you literally read its chain of thought. It's like actually doing this toy example we're talking about where it's like, oh, can I solve this problem by doing X? Nah, I need a different approach. And this raises the question. I mean, over time, it is the case that the capabilities which were, which required inference time compute to elicit, get distilled into the model. So you're amortizing the thing which previously you needed to do these like rollouts, these like Monte Carlo rollouts to, to figure out. And so in general, there, maybe there's this principle of. Digital minds, which can be copied, have different trade offs, which are relevant than biological minds, which cannot.

• 25:42Dwarkesh Patel?

  And so in general, it should make sense to amortize more things because you can literally copy the, copy the amortization, right? Or copy the things that you have sort of like built in.

• 25:51Speaker 01

  Yeah.

• 25:52Dwarkesh Patel?

  And it maybe, this is a tangential question where it might be interesting to speculate about in the future, as these things become more intelligent. And the way we train them becomes more economically rational. What will make sense to amortize into these minds, which evolution did not think it was worth amortizing into biological minds. They do, you have to retrain every time.

• 26:13Speaker 01

  Right. I mean, first of all, I think the probabilistic AI people would be like, of course you need test time compute. Because this inference problem is really hard. And the only ways we know how to do it involve lots of test time compute. Otherwise, it's just a crappy approximation that's never gonna like, you have to do infinite data or something to like make this. So I think that some of the probabilistic people will be like, no, it's like inherently probabilistic and like amortizing it in this way like just doesn't make sense. And so, and they might then also point to the brain and say, okay, well, the brain, the neurons are kind of stochastic and they're sampling and they're doing, doing things. And so maybe the brain actually is doing more like the non amortized inference, the real inference.

• 26:48Speaker 01

  But it's also kind of strange how perception can work in just like milliseconds or whatever. It doesn't seem like it uses that much sampling. So it's also clearly also doing some kind of baking things into, into like approximate forward passes or something like that to do this. And yeah, so in the future, you know, I don't know. I mean, I think.

• 27:10Speaker 01

  Is it already a trend to some degree that things that are, people were having to use test time compute for are getting like used to train back the, the base model, right?

• 27:20Dwarkesh Patel?

  Yeah.

• 27:20Speaker 01

  Yeah, that's so now it can do it in one pass.

• 27:22Dwarkesh Patel?

  Right.

• 27:23Speaker 01

  Yeah. So I mean, I think, yeah, you know, maybe evolution did or didn't do that. I think evolution still has to pass everything through the genome, right? To build the network. So, and the environment in which humans are living is very dynamic, right? And so. Maybe that's if we believe this is true, that there's a learning subsystem per Steve Burns and a steering subsystem, that the learning subsystem doesn't have a lot of like preinitialization or pre-training. It has a certain architecture, but then within lifetime it learns. Then evolution didn't actually like immortalize that much into that network.

• 27:58Dwarkesh Patel?

  Right.

• 27:59Speaker 01

  It immortalized it instead into a set of innate behaviors and a set of these bootstrapping cost functions.

• 28:04Dwarkesh Patel?

  Yeah.

• 28:04Speaker 01

  Or ways of building up very particular reward signals.

• 28:07Dwarkesh Patel?

  Yeah.

• 28:08Speaker 01

  Yeah.

• 28:08Dwarkesh Patel?

  This framework helps explain this mystery that people have pointed out and I've asked a few guests about, which is if you want to analogize evolution to pre-training, well, how do you explain the fact that so little information is conveyed through the genome? So 3 gigabytes is the size of the total human genome. Obviously, a small fraction of that is actually relevant to coding at the brain.

• 28:30Speaker 01

  Yeah.

• 28:32Dwarkesh Patel?

  And if previously, people made this analogy that actually evolution has found the hyperparameters of the model, the numbers which tell you how many layers should there be, the architecture, basically, right? Like, how should things be wired together? But if a big part of the story that increases sample efficiency, aids learning, generally makes systems more performant, is the reward function, is the loss function.

• 28:54Speaker 01

  Yeah.

• 28:55Dwarkesh Patel?

  And if evolution found those loss functions, which aid learning, then it actually kind of makes sense how so you can like build an intelligence with so little information. Because like the reward function, you're like right in Python, right? The reward function is like literally a line.

• 29:10Speaker 01

  Yeah.

• 29:11Dwarkesh Patel?

  And so you just like have like a thousand lines like this, and that doesn't take up that much space.

• 29:15Speaker 01

  Yes, and it also gets to do this generalization thing with the thing I was describing, where we were talking with about the spider, right, of where it learns that just the word spider, you know, triggers the spider, you know, reflex or whatever.

• 29:27Speaker 01

  It gets to exploit that too, right? So it gets to build a reward function that actually has a bunch of generalization in it, just by specifying these innate spider stuff and the thought assessors, as Steve calls them, that do the learning. So that's like a potentially a really compact solution to building up these more complex reward functions too that you need. So it doesn't have to anticipate everything about the future of the reward function, just to anticipate what variables are relevant and what are here. For like finding what those variables are. And then, yeah, so then it has to have like a very compact specification for like the learning algorithm and basic architecture of the learning subsystem. And then it has to specify all this Python code of like all the stuff about the spiders and all the stuff about friends and all the stuff about your mother and all the stuff about meeting and and and social groups and joint eye contact.

• 30:12Speaker 01

  It has to specify all that stuff.

• 30:15Speaker 01

  And so is this really true? And so I think that there is some evidence for it. So so so Fei-chen and Evan McCosco and various other researchers who have been doing like these single cell atlases. So one of the things that neuroscience technology or scaling up neuroscience technology, again, this is kind of like my, one of my obsessions, has done through through

• 30:39Speaker 01

  the Brain Initiative, a big neuroscience funding program, is they've basically gone through different areas, especially of the mouse brain and mapped like where are the different cell types?

• 30:50Speaker 01

  How many different types of cells are there in different areas of cortex? Are they the same across different areas? And then you then you look at these sub cortical regions, which are more like the like steering subsystem or reward function generating regions. How many different types of cells do they have and which neurons types do they have? We don't know how they're all connected and exactly what they do or what the circuits are, what they mean. But you can just like quantify like how many different kinds of cells are there?

• 31:15Speaker 01

  With sequencing the RNA. And there are a lot more weird and diverse and bespoke cell types in the steering subsystem, basically, than there are in the learning subsystem, like the cortical cell types. There's enough to build, it seems like there's enough to build a learning algorithm up there and specifies them hyper parameters. And in the in the steering subsystem, there's like a gazillion, you know, thousands of really weird cells, which might be like the one for the spider flinch reflex and the one for I'm about to taste salt.

• 31:45Dwarkesh Patel?

  Sorry, why would each reward function need a different cell type?

• 31:49Speaker 01

  Well, so this is where you get innately wired circuits, right? So in the in the learning algorithm part, in this, in the learning, learning subsystem, you set up, specify the initial architecture, specify a learning algorithm. It's all, all the, all the, all the juices is happening through plasticity of the synapses, changes of the synapses within that big network. But it's kind of like a relatively repeating architecture. How it's initialized, it's just like the amount of Python code needed to make, you know, an eight-layer transformer is not that different from one to make a three-layer transformer, right? You're just replicating.

• 32:22Dwarkesh Patel?

  Yeah.

• 32:23Speaker 01

  Whereas all this Python code for the reward function, you know, if superior click list sees something that's skittering in the land, you know, you're feeling goosebumps on your skin or whatever, then trigger spider reflex, that's just a bunch of like bespoke, species-specific.

• 32:37Dwarkesh Patel?

  Hmm.

• 32:39Speaker 01

  Situation-specific crap. The cortex doesn't know about spiders, it just knows about layers and.

• 32:44Dwarkesh Patel?

  Right.

• 32:45Speaker 01

  And learning.

• 32:46Dwarkesh Patel?

  But you're saying that the only way to have this, like, write this reward function.

• 32:49Speaker 01

  Yeah.

• 32:49Dwarkesh Patel?

  Is to have a special cell type.

• 32:51Speaker 01

  Yeah, yeah, well, I think so. I think you either have to have a special cell types or you have to somehow otherwise get special wiring rules that evolution can say, this neuron needs to wire to this neuron without any learning. And the way that that is most likely to happen, I think, is that those cells express like different receptors and proteins that say, okay, when this one comes in contact with this one, let's form a synapse. So it's genetic wiring. Yeah. And those need cell types to do it. Yeah.

• 33:20Dwarkesh Patel?

  I'm sure this would make a lot more sense if I knew 101 neuroscience, but like, it seems like there's still a lot of complexity, or generality, rather, in the steering system. So if a steering system has its own visual system that's separate from the visual cortex.

• 33:38Speaker 01

  Yeah.

• 33:38Dwarkesh Patel?

  Different features still need to plug into that vision system in the, so like the spider thing needs to plug into it and also the, the love thing needs to plug into it, et cetera, et cetera.

• 33:54Speaker 01

  Yes.

• 33:55Dwarkesh Patel?

  So it seems complicated.

• 33:57Speaker 01

  No, it's still complicated. And that's all the more reason why a lot of the genomic. You know, real estate in the genome and in terms of these different cell types and so on would go into wiring up the steering subsystem.

• 34:09Dwarkesh Patel?

  And can we tell?

• 34:09Speaker 01

  Pre-wiring it.

• 34:10Dwarkesh Patel?

  Can we tell how much of the genome is like clearly working? So I guess you could tell how many are relevant to the producing the RNA that manifests or the epigenetics that manifest in different cell types in the brain, right?

• 34:22Speaker 01

  Yeah, this is what the cell types helps you get at it. I don't think, I don't think it's exactly like, oh, this percent of the genome is doing this, but you could say, okay, in these, all these steering subsystems, subtypes, you know, how many different genes are involved in sort of specifying which is which and how they wire?

• 34:35Speaker 01

  And how much genomic real estate do those genes take up versus the ones that specify, you know, visual cortex versus auditory cortex? You're kind of just reusing the same genes to do the same thing twice, whereas the spider reflex hooking up, yes, you're right, they have to, they have to build a vision system and they have to build some auditory systems and touch systems and navigation type systems. So, you know, even feeding into the hippocampus and stuff like that, there's head direction cells, even the fly brain, it has innate circuits that, you know, figure out its orientation and help it navigate in the world and it uses vision, figures optical flow of how it's flying and, you know, how is it, how is its flight related to the wind direction? It has all these innate stuff that I think in the mammal brain, we would all put that

• 35:20Speaker 01

  and lump that into the steering subsystem. So there's a lot of work. So all the genes basically that go into specifying all the things a fly has to do, we're going to have stuff like that too, just all in the steering subsystem.

• 35:30Dwarkesh Patel?

  But do we, do we have some estimate of like, here's how many nucleotides, here are many megabases it takes to?

• 35:37Speaker 01

  I don't know. I mean, but, but, but, I mean, I think you might be able to talk to biologists about this, you know, to some degree, because you can say, well, we just have a ton in common. I mean, we have a lot in common with yeast from a genes perspective. Yeast is still used as a model for, you know, some amount of drug development and stuff like that in biology. And so so much of the genome is just going towards, you have a cell at all, it can recycle waste, it can get energy, it can replicate.

• 36:06Speaker 01

  And then what we have in common with a mouse. And so we do know at some level that the difference is us and a chimpanzee or something. And that includes the social instincts and the more advanced, you know, differences in cortex and so on. It's a tiny number of genes that go into these additional amount of making the eight layer transformer instead of the six layer transformer or tweaking that reward function.

• 36:28Dwarkesh Patel?

  This would help explain why the hominid brain exploded inside so fast, which is presumably like, tell me this is correct, but under the story, we, social learning or some other thing increased the ability to learn from the environment, like increased our sample efficiency, right? Instead of having to go and kill the boar yourself and figure out like how to do that. You can just be like, the elder told me this, oh, you make a spear. And then now it increases the incentive to have a bigger cortex, which can like learn these things.

• 37:01Speaker 01

  Yes, and that can be done with a relatively few genes because it's really, it's really replicating what the mouse already has, is making more of it. It's maybe not exactly the same and there may be tweaks, but it's like from a perspective, you don't have to reinvent all this stuff.

• 37:16Dwarkesh Patel?

  So then how far back in the history of the evolution of the brain, does the cortex go back? It is the idea that like the cortex has always figured out this omnidirectional inference thing. That's been a solved problem for a long time. And then the big unlock with Primace is this, we got the reward function, which increased the returns to having omnidirectional inference. Or is the cortex, is the omnidirectional inference also something that took a while to unlock?

• 37:41Speaker 01

  I'm not sure that there's agreement about that. I think there might be specific questions about language, you know, are there tweaks to be, you know, whether that's through auditory and memory, some combination auditory, memory regions. There may also be like macro wiring, right? Of like, you need to wire auditory regions into memory regions or something like that. And into some of these social instincts to get

• 38:03Dwarkesh Patel?

  I see.

• 38:03Speaker 01

  Language, for example, to happen. So there might be, but that might be also a small number of gene changes.

• 38:08Dwarkesh Patel?

  Yeah.

• 38:08Speaker 01

  To be able to say, oh, I just need from my temporal lobe over here, going over to the auditory cortex, something right. And there is some evidence for the, you know, the Broca's area, Wernicke's area. They're connected with these hippocampus and so on. And so prefrontal cortex. So there's like some small number of genes, maybe for like enabling humans to really properly do language. That could be a big one. But yeah, I mean, I think that.

• 38:34Speaker 01

  Is it that something changed about the cortex? And it became possible to do these things? Whereas that was that potential was already there, but there wasn't the incentive to expand that capability and then use it, wired it to these social instincts and use it more. I mean, I would lean somewhat toward the latter. I mean, I think a mouse has a lot of similarity in terms of cortex as a human.

• 38:57Dwarkesh Patel?

  Right. Although there's that, Suzanna Herkel who's all work.

• 39:00Speaker 01

  Yeah.

• 39:02Dwarkesh Patel?

  The the the number of neurons scales better with weight with primate brains than it does with rodent brains, right? So

• 39:11Speaker 01

  Yeah.

• 39:12Dwarkesh Patel?

  Does that suggest that there actually was some improvement in the scalability of the cortex?

• 39:16Speaker 01

  Maybe, maybe. I'm not, I'm not super deep on this. There may, there may have been.

• 39:22Speaker 01

  Yeah, changes in architecture, changes in the folding, changes in neuron properties and stuff that somehow slightly tweak this, but there's still a scaling.

• 39:29Dwarkesh Patel?

  That's right.

• 39:30Speaker 01

  Right, either way, right? And so I was not saying there aren't something special about humans in the architecture of the learning subsystem at all.

• 39:40Speaker 01

  But yeah, I mean, I think it's pretty widely thought that this is expanded, but then the question is, okay, well, how does that? How does that fit in also with the steering subsystem changes and the instincts that make use of this and allow you to bootstrap using this effectively? But I mean, just to say a few other things, I mean, so even the fly brain has some amount of, for example, even, even very far back. I mean, I think you've read this, this great book, The Brief History of Intelligence, right? I think this is a really good book. Lots of AI researchers think this is a really good book, it seems like.

• 40:12Speaker 01

  Yeah, you have some amount of learning going back all the way to anything that has a brain, basically. You have something kind of like primitive reinforcement learning, at least. Going back at least to like vertebrates, like imagine like a zebra fish, just like a.

• 40:35Speaker 01

  These other branches, birds maybe kind of reinvented something kind of cortex-like, but it doesn't have the six layers, but they have something a little bit cortex-like. So that some of those things, after reptiles, in some sense, birds and mammals both kind of made a somewhat cortex-like but differently organized thing. But even a fly brain has like associative learning centers that actually do things that maybe look a little bit like this like thought assessor concept from Behrens where there's like a specific dopamine signal to train specific subgroups of neurons in the fly mushroom body to associate different sensory information with, am I going to get food now or am I going to get hurt now?

• 41:14Dwarkesh Patel?

  Yeah. Brief tangent, I remember reading in one blog post that Darren Millett wrote that the parts of the cortex which are associated with audio and vision have scaled disproportionately between other primates and humans, whereas the parts associated, say, with odor have not. And I remember him saying something like, this is explained by that kind of data having worse scaling law properties. But I think the, and maybe he meant this, but another interpretation of actually what's happening there is that these social reward functions that are built into this hearing

• 41:56Dwarkesh Patel?

  subsystem needed to make use more of being able to see your elders and see what the visual cues are and hear what they're saying.

• 42:05Speaker 01

  Yeah.

• 42:06Dwarkesh Patel?

  In order to make a sense of these cues which guide learning, you needed to activate these, activate the vision and audio more than odor.

• 42:14Speaker 01

  I mean, there's all this stuff, I feel like it's come up in your shows before, actually, but like even like the design of the human eye where you have like the pupil and the white and everything, like we are designed to be able to establish relationships based on joint eye contact and maybe this came up in the sudden episode, I can't remember. But yeah, we have to bootstrap to the point where we can detect eye contact and where we can communicate by language, right? And that's like what the first couple of years of life are trying to do.

• 42:42Dwarkesh Patel?

  Okay. I want to ask you about RL. So currently, the way these LNs are trained, you know, they are, if they solve the unit test or solve a math problem, that whole trajectory, every token in that trajectory is upweighted. And what's going on with humans? Is there, are there different types of model based versus model free that are happening in different parts of the brain?

• 43:02Speaker 01

  Yeah, I mean, this is another one of these things. I mean, again, all my answers to these questions, any specific thing I say is all just kind of like, directionally, this is we can kind of explore around this. I find this interesting. Maybe I feel like the literature points in these directions in some very broad way. What I actually want to do is like, go and map the entire mouse brain and like, figure this out comprehensively and like, make neuroscience the ground truth science. So I don't know, basically. But, but yeah, I mean, there. So first of all, I mean, I think with Ilya on the podcast, I mean, he was like, it's weird that you don't use value functions, right? You use like, the most dumbest form of RL. And of course, there are, these people are incredibly smart and they're optimizing for how to do it on GPUs. And it's really incredible what they're achieving. But like, conceptually, it's a really dumb form of RL, even compared to like, what was

• 43:45Speaker 01

  being done in like, 10 years ago, right? Like even, you know, the Atari game playing stuff, right, was using like, Q learning, which is basically like, it's a kind of temporal difference learning, right? And the temporal difference learning basically means you have some kind of a value function of like, what action I choose now doesn't just tell me literally what happens immediately after this. It tells me like, what is the long run consequence of that from my expected, you know, total reward or something like that. And so you have value functions, like, the fact that we don't have like, value functions at all is like, in the LLMs is like, it's crazy. I think, I think because Ilya said it, I can say it, I know, you know, one, one hundredth of what he does about

• 44:28Speaker 01

  AI. But like, it's kind of crazy that this is working. Yeah. But, yeah, I mean, in terms of the brain, well, so I think there are some parts of the brain that are thought to do something that's very much like model free RL. That's our parts of the basal ganglia, sort of striatum and basal ganglia. They have like, a certain finite, like, it is thought that they have a certain like, finite relatively small action space. And the types of actions they could take, first of all, might be like, tell the spinal cord, or tell the brainstem and spinal cord to do this motor action. Yes, no. Or it might be more complicated cognitive type actions like tell the thalamus to allow

• 45:12Speaker 01

  this part of the cortex to talk to this other part. Or release the memory that's in the hippocampus and start a new one or something, right? But there's some finite set of actions that kind of come out of the basal ganglia and that it's just a very simple RL. So there are probably parts of other brains in our brain that are just like doing very simple naive type RL algorithms. Layer one thing on top of that is that some of the major work in neuroscience, like Peter Dian's work and a bunch of work that is part of why I think DeepMind did the temporal difference learning stuff in the first place is they were very interested in neuroscience. And there's a lot of neuroscience evidence that the dopamine is giving this reward prediction

• 45:53Speaker 01

  error signal rather than just reward, yes, no, you know, a gazillion time steps in the future, it's a prediction error. And that's consistent with like learning these value functions. So there's that. And then there's maybe like higher order stuff. So we have these cortex making this world model. Well, one of the things the cortex world model can contain is a model of when you do and don't get rewards, right? Again, it's predicting what the steering subsystem will do, it could be predicting what the basal ganglia will do. And so you have a model in your cortex that has more generalization and more concepts and all this stuff that says, okay, these types of plans, these types of actions will lead in these types of circumstances to rewards. So I have a model of my reward.

• 46:37Speaker 01

  Some people also think that you can go the other way. And so this is part of the inference picture. There's this idea of RL as inference. You could say, well, conditional on my having a high reward, sample a plan that I would have had to get there. That's inference of the plan part from the reward part. I'm clamping the reward as high and inferring the plan, sampling from plans that could lead to that. And so if you have this very general cortical thing, it can just do, if you have this like general, very general model based system and the model, among other things, includes plans and rewards.

• 47:12Speaker 01

  Then you just get it for free, basically.

• 47:14Dwarkesh Patel?

  So like in neural network parlance, there's a value head associated to the the omnidirectional inference that's happening.

• 47:23Speaker 01

  Yes, yeah, or there's a value input. Yeah.

• 47:26Dwarkesh Patel?

  Oh, okay.

• 47:27Speaker 01

  Yeah, and it can predict one of the one of the almost sensory variables it can predict is is what rewards is going to get.

• 47:34Dwarkesh Patel?

  Yeah, but by the speaking of this thing about amortizing things. Yeah, obviously, value is like amortized rollouts of looking up reward.

• 47:46Speaker 01

  Yeah, something like that. Yeah, yeah, it's like a statistical average or prediction of it. Yeah.

• 47:52Dwarkesh Patel?

  Right. Tangential thought. You know, Joe Henrick and others have this idea that the way human societies have learned to do things is just like, how do you figure out that, you know, this kind of bean, which actually just almost always poisons you, is edible if you do this 10-step incredibly complicated process? Any one of which, if you fail at, the bean will be poisonous. How do you figure out how to hunt this seal in this particular way with this like particular weapon at this particular time of the year, etc.? There's no way but just like trying shit over generations. And it strikes me that this is actually very much like model-free RL happening at like a civilizational level. No, not exactly.

• 48:35Speaker 01

  I mean, evolution is the simplest algorithm in some sense, right? And if we believe that all of this can come revolution, like the outer loop can be like extremely not foresighted and yeah.

• 48:45Dwarkesh Patel?

  Right. That's interesting, just like hierarchies of evolution model-free, culture, evolution model-free.

• 48:52Speaker 01

  So what does that tell you? Maybe the simple algorithms can just get you anything if you do it enough first.

• 48:56Dwarkesh Patel?

  Right, right, yeah.

• 48:57Speaker 01

  Yeah, I don't know. So.

• 48:58Dwarkesh Patel?

  But yeah, so you have like maybe this yeah, evolution model-free, basal ganglia model-free, cortex model-based, culture, model-free potentially. I mean, there's a you pay attention to your elders or whatever.

• 49:13Speaker 01

  Yeah, maybe there's like group selection or whatever of these things is like more model-free.

• 49:17Dwarkesh Patel?

  Yeah.

• 49:18Speaker 01

  But now I think culture, well.

• 49:22Speaker 01

  It stores some of the model.

• 49:24Dwarkesh Patel?

  Yeah.

• 49:24Speaker 01

  Right?

• 49:25Dwarkesh Patel?

  So let's say you want to train an agent to help you with something like processing loan applications. Training an agent to do this requires more than just giving the model access to the right tools, things like browsers and PDF readers and risk models. There's this level of tacit knowledge that you can only get by actually working in an industry. For example, certain loan applications will pass every single automated check despite being super risky. Every single individual part of the application might look safe, but experienced underwriters know to compare across documents to find subtle patterns that signal risk. Labelbox has experts like this in whatever domain you're focused on, and they will set up highly realistic training environments that include whatever subtle nuances and watch outs you need to look out for. Beyond just building the environment itself, Labelbox provides all the scaffolding you

• 50:10Dwarkesh Patel?

  need to capture training data for your agent. They give you the tools to grade agent performance and capture the video of each session and to reset the entire environment to a clean state between every episode. So whatever domain you're working in, Labelbox can help you train reliable, real-world agents. Learn more at labelbox.com/thoracash. Stepping back, how is it a disadvantage or an advantage for humans that we get to use biological hardware in comparison to computers as they exist now? So what I mean by this question is like, if there's the algorithm, would the algorithm just qualitatively perform much worse or much better if inscribed in the hardware of

• 50:55Dwarkesh Patel?

  today? And the reason to think it might, like, here's what I mean. Like, you know, obviously, the brain has had to make a bunch of trade-offs which are not relevant to computing hardware. It has to be much more energetically efficient. Maybe as a result, it has to run on slower speeds so that there can be a smaller voltage gap. And so the brain runs at 200 Hz. It has to like run on 20 watts. On the other hand, you know, with like robotics, we've clearly experienced that fingers are way more nimble than we can make motors so far. And so maybe there's something in the brain that is the equivalent of like cognitive dexterity, which is like, maybe due to the fact that we can do unstructured sparsity, we can co-locate the memory. And the compute.

• 51:32Speaker 01

  Yes.

• 51:33Dwarkesh Patel?

  Where does this all land out? Are you like, fuck, we would be so much smarter if we didn't have to deal with these brains? Or are you like, oh?

• 51:38Speaker 01

  I mean, I think in the end, we will get the best of both worlds somehow, right? I think, I think an obvious downside of the brain is it cannot be copied.

• 51:45Dwarkesh Patel?

  Yeah.

• 51:45Speaker 01

  You don't have, you know, external read write access to every neuron and synapse. Whereas you do, I can just edit something in the weight matrix.

• 51:52Dwarkesh Patel?

  Right.

• 51:53Speaker 01

  You know, in Python or whatever, uh, you know, and, and load that up and copy that, um, in principle, right?

• 52:01Speaker 01

  So the fact that it can't be copied and kind of random accessed is like very annoying. But otherwise, maybe these are, it like has a lot of advantages. So, or it also tells you that you want to like somehow do the code design of the algorithm and the, it maybe did, that even doesn't change it that much from all of what we discussed, but you want to somehow do this code design. So, um, yeah, how do you do it with really slow, low voltage switches? That's going to be really important for the energy consumption. The co-locating memory and compute. So like, I, I think that probably just like hardware companies will try to co-locate memory and compute. They will try to use lower voltages, allow some stochastic stuff. There are some people that think that this, like all this probabilistic stuff that we were talking about, oh, oh, it's actually energy based models and so on.

• 52:45Speaker 01

  It's doing lots, it is doing lots of sampling. It's not just amortizing everything. That the neurons are also very natural for that because they're naturally stochastic. And so you don't have to do a random number generator and a bunch of Python code, basically to generate a sample. The neuron just generates samples and it can tune what the different probabilities are.

• 53:06Dwarkesh Patel?

  Yeah.

• 53:06Speaker 01

  And so, and like learn, learn those tunings. And so it could be that it's very code designed with like some kind of inference method or something.

• 53:15Dwarkesh Patel?

  Yeah. It'd be hilarious. I mean, the, the message I hear in this interview is like, you know, all these people that folks make fun of on Twitter, you know, Yann LeCun, Yann LeCun and Beth Jezos and whatever. They're like, no, like, well, yeah, maybe, I don't know.

• 53:27Speaker 01

  That is actually, that is actually one read of me and granted, you know, I, I haven't really worked on AI at all since L L M's, you know, took off. So I'm, I'm just like out of the loop, but, um, I, I'm surprised and I'm, I, I think it's amazing how the scaling is, is working and everything. But yeah, I think Yann LeCun and Beth Jezos are kind of on to something about the, about the probabilistic models or at least possibly. And in fact, that's what. You know, all the neuroscientists and all the AI people thought like until 2021 or something, right?

• 53:54Dwarkesh Patel?

  So there's a bunch of cellular stuff happening in the brain that is not just about neuron to neuron synaptic connections. How much of that is functionally doing more work than the synapses themselves are doing versus it's just a bunch of collage that you have to do in order to make the synaptic thing work? So the way you need to, you know, with a digital mind, you can nudge the synapse, sorry, the parameter extremely easily. But with a cell to modulate a synapse according to the gradient signal, it just takes all of this crazy machinery. So like, is it actually doing more than it takes extremely little code to do?

• 54:36Speaker 01

  So I don't know, but I'm not a believer in the like radical, like, oh, actually, memory is not synapses mostly. Or like, learning is mostly genetic changes or something like that. I think it would just make a lot of sense. I think you put it really well for it to be more like the second thing you said, like, let's say you want to do weight normalization across all the weights coming out of your neuron, right, or into your neuron. Well, you probably have to go like somehow tell the nucleus about this of the cell. And then have that kind of send everything back out to the synapses or something, right? And so there's going to be a lot of cellular changes, right? Or let's say that, you know, you just had a lot of plasticity and like you're part of this memory. And now that's got consolidated into the cortex or whatever.

• 55:19Speaker 01

  And now we want to reuse you as like a new one that can learn again. It's going to be a ton of cellular changes. So there's going to be tons of stuff happening in the cell, but algorithmically, it's not really adding something beyond these algorithms. Right? It's just implementing something that in a digital computer is very easy for us to go and just find the weights and change them.

• 55:38Speaker 01

  And it is a cell, it just literally has to do all this with molecular machines itself without any central controller, right? It's kind of incredible. There are some things that cells do, I think, that that seem like more convincing, so. In the cerebellum, so one of the things the cerebellum has to do is like predict over time, like predict what is the time delay, you know, let's say that, um, you know, I see a flash and then, uh, you know, some number of milliseconds later, I'm going to get like a puff of air in my eyelid or something, right? Uh, the cerebellum can be very good at predicting what's the timing between the flash and the air puff, so that now your eye will just like close automatically, like the cerebellum is like involved in that type of reflex, like learned reflex.

• 56:18Speaker 01

  Um, and there are some cells in the cerebellum where it seems like the cell body is playing a role in storing that time constant, changing that time constant of delay versus that all being somehow done with like, I'm going to make a longer ring of synapses to make that delay longer. It's like, no, the cell body will just like store that time delay for you. So there are some examples, but I'm not a believer like out of the box in like essentially this theory that like what's happening is changes and connections between neurons.

• 56:53Dwarkesh Patel?

  Yeah.

• 56:54Speaker 01

  And that's like the main algorithmic thing that's going on. I like, I, I think that's a very good reason to, to still believe that it's that rather than some like crazy cellular stuff.

• 57:03Dwarkesh Patel?

  Yeah. Going back to this whole perspective of like, our, our intelligence is not just this omnidirectional inference thing that builds a world model, but really this system that teaches us what to pay attention to, what are the important salient factors to learn from, et cetera. I, I want to see if there's some intuition you can drive from this, but what different kinds of intelligences it might be like. So it seems like AGI or superhuman intelligence should still have this, um, uh, uh, like ability to learn a world model that's quite general.

• 57:40Speaker 01

  Yeah.

• 57:40Dwarkesh Patel?

  But then it might, um, be incentivized to pay attention to different things that are relevant for what, you know, the, the modern post singularity environment. How different should we expect different intelligences to be basically?

• 57:56Speaker 01

  Yeah. I mean, I think one way of this question is like, is it actually possible to like make the paperclip maximizer or whatever, right? If you make, if you try to make the paperclip maximizer, does that end up like just not being smart or something like that? Because it's, it was just, the only reward function it had was like make paperclips.

• 58:11Dwarkesh Patel?

  Interesting. Yeah, yeah.

• 58:12Speaker 01

  If I channel Steve Burns more, I mean, I think he's very concerned that the, the sort of minimum viable things in the steering subsystem that you need to get something smart.

• 58:21Speaker 01

  Is way less than the minimum viable set of things you need for it to have human like social instincts and ethics and stuff like that. So a lot of what you want to know about the steering subsystem is actually the specifics of how you do alignment essentially, or what human behavior and social instincts is versus just what you need for capabilities. And we talked about it in a slightly different way because we were sort of saying, well, in order for humans to like learn socially, they need to make eye contact and learn from others. But we already know from L L Ms, right? Depending on your starting point, you can learn language without that stuff, right? And so, yeah, and so I think that it probably is possible to make like super powerful, you know, model based R L, you know, optimizing systems and stuff like that, that don't have

• 59:05Speaker 01

  most of what we have in the human brain reward functions. And as a consequence, might want to maximize paperclips and that's a concern.

• 59:11Dwarkesh Patel?

  Yeah. Right. But you're pointing out that in order to make a competent paperclip maximizer, the kind of thing that can build the spaceships and learn the physics and whatever, it needs to have some drives which elicit learning, including say curiosity and exploration.

• 59:29Speaker 01

  Yeah, curiosity and interest in others, of interest in social interactions, curiosity. Yeah, but that's pretty, that's pretty minimal, I think. And that's true for humans, but it might be less true for like something that's already pre-trained as an L L M or something, right? And so, so most of why we want to know the steering subsystem, I think, if I'm channeling Steve, is alignment reasons.

• 59:51Dwarkesh Patel?

  Yeah. Right. How confident are we that we even have the right algorithmic conceptual vocabulary to think about what the brain is doing? And what I mean by this is, you know, there was one big contribution to AI from neuroscience, which was the idea you have the neuron, which like William, you know, 1950s, just like this original contribution. But then it seems like a lot of what we've learned afterwards. About what the high level algorithm the brain is implementing from the backprop to if there's something analogous to backprop happening in the brain to always V1 doing something like CNNs to TD learning and Bellman equations, actor critic, whatever.

• 1:00:34Dwarkesh Patel?

  Seems inspired by what is like, we come up with some idea like, but maybe we can make AI neural networks work this way.

• 1:00:40Speaker 01

  Yeah.

• 1:00:40Dwarkesh Patel?

  And then we notice that's something in the brain also works that way.

• 1:00:43Speaker 01

  Yes.

• 1:00:44Dwarkesh Patel?

  So why not think there's more things like this?

• 1:00:45Speaker 01

  There may be, yeah. I think the reason that I'm not, I think that we might be on to something is that like the AIs we're making based on these ideas are working surprisingly well. There's also a bunch of like just empirical stuff, like convolutional neural nets and variants of convolutional neural nets. I'm not for sure what the absolute latest latest, but compared to other like models in computational neuroscience of like what the visual system is doing are just like more predictive, right? So you can just like score even like pre-trained on like cat pictures and stuff, CNNs. What is the representational similarity that they have on some arbitrary other image versus, you know, compared to the brain activations measured in different ways?

• 1:01:30Speaker 01

  Jim DeCarlo's lab has like brain score and like the AI models actually like there, there seems to be some relevance there in terms of like even like neurosciences don't necessarily have something better than that. So yes, I mean, that's just kind of recapitulating what you're saying is that like the best computational neuroscience theories we have seem to have been like invented largely as a result of AI models and like find things that work. And so find backprop works and then say, can we approximate backprop with cortical circuits or something? And there's, there's kind of been things like that. Now, some people totally disagree with this, right? So like Yuri Busak is a neuroscientist who has a book called The Brain From Inside Out. He basically says like all our psychology concepts, like AI concepts, all the stuff

• 1:02:14Speaker 01

  is just like made up stuff. We actually have to do is like figure out what is the actual set of primitives that like the brain actually uses and our vocabulary is not going to be adequate to that. We have to start with the brain and make new vocabulary rather than saying backprop and then try to apply that to the brain or something like that. And, you know, he studies a lot of like oscillations and stuff in the brain as opposed to individual neurons and what they do. And, you know, I don't know, I think that.

• 1:02:39Speaker 01

  There's a case to be made for that. And from a kind of research program design perspective, I think there's like, one thing we should be trying to do is just like simulate a tiny worm or a tiny zebrafish, like from almost like as biophysical or like as as bottom up as possible, like get connecto molecules activity and like just study it as a physical dynamical system and like, look what it does. But I don't know. I mean, just when I like, it just feels like the AI is really good fodder for computational neuroscience. Like, those might actually be pretty good models. We should look at that. So I'm not a person who thinks that. I think I both think that there should be a part of the research portfolio that is like totally bottom up and not

• 1:03:23Speaker 01

  trying to apply our vocabulary that we learn from AI onto these systems. And that there should be another big part of this that's kind of trying to reverse engineer it using that vocabulary or variants of that vocabulary. And that we should just be pursuing both. And, and my guess is that the reverse engineering one is actually gonna like, kind of workish or something like, we do see things like TD learning, which, you know, Sutton also invented separately, right?

• 1:03:50Dwarkesh Patel?

  That must be a crazy feeling to just like, this like, equation I wrote down is like, in the brain.

• 1:03:56Speaker 01

  Yeah, it seems like the dopamine is like, doing some of that. Yeah.

• 1:03:59Dwarkesh Patel?

  So let me ask you about this. You know, you guys are finding different groups that are trying to figure out what's up in the brain. If we had a perfect representation, however you define it, of the brain, why think it would actually let us figure out the answer to these questions? We have neural networks, which are way more interpretable, not just because we understand what's in the weight matrices, but because there are weight matrices. There are these boxes with numbers in them. And even then, we can tell Very basic things, we can kind of see circuits for very basic pattern matching of following one token with another. I feel like we don't really have an explanation of why LLMs are intelligent, just because they're

• 1:04:42Speaker 01

  Yeah, well, I would somewhat dispute it. I think we have some architectural, we have some description of what the LLM is fundamentally doing. And what that's doing is that I have an architecture and I have a learning rule and I have hyperparameters and I have initialization and I have training data.

• 1:04:57Dwarkesh Patel?

  But those are things we learned from because we built them, not because we interpreted them from seeing the way

• 1:05:02Speaker 01

  You built them.

• 1:05:02Dwarkesh Patel?

  Which is the not the last thing to connect home is like seeing the way it's

• 1:05:05Speaker 01

  What I think we should do is we should describe the brain more in that language of things like architectures, learning rules, initialization, rather than trying to find the Golden Gate Bridge circuit and saying exactly how does this neuron actually, you know, that's going to be some incredibly complicated learned pattern.

• 1:05:21Speaker 01

  Yeah, Connor Cording and Tim Lillykrap have this paper from a while ago, maybe five years ago called, "What Does It Mean To Understand A Neural Network?" Or "What Would It Mean To Understand A Neural Network?"

• 1:05:31Speaker 01

  And what they say is, yeah, basically that. Like, you could imagine you train a neural network to like, compute the digits of pi or something, or like some crazy, you know, it's like, it's like this crazy pattern. And you also train that thing to like predict the most complicated thing you find, predict stock prices, basically predict the really complex systems, right? Computationally, you know, computationally complete systems, I could predict, I could train a neural network to do cellular automata or whatever crazy thing. And it's like, we're never going to be able to fully capture that with interpretability, I think. It's just going to just be doing really complicated computations internally. But we can still say that the way it got that way is that it had an architecture and we gave it this training data and it had this loss function. And so I want to describe the brain in the same way. And I think that this framework that I've been kind of laying out is like, we need to understand the cortex and how it embodies a learning algorithm.

• 1:06:16Speaker 01

  I don't need to understand how it computes Golden Gate Bridge.

• 1:06:18Dwarkesh Patel?

  But if you can see all the neurons, if you have the connectome, why does that teach you what the learning algorithm is?

• 1:06:24Speaker 01

  Well, I guess there are a couple of different views of it. So it depends on this different parts of this portfolio. So on the totally bottom up, we have to simulate everything portfolio. It kind of just doesn't. You have to just like see what are the, you have to make a simulation of the zebrafish brain or something. And then you like see what are the like emergent dynamics in this and you come up with new names and new concepts and all that. That's like, that's like the most extreme bottom up neuroscience view.

• 1:06:48Speaker 01

  But even there, the connectome is like really important for doing that biophysical or bottom up simulation. But on the other hand, you can say, well, what if we can't actually apply some ideas from AI? We basically need to figure out, is it an energy-based model? Or is it, you know, an amortized, you know, VAE-type model? You know, is it doing backprop? Or is it doing something else? Are the learning rules local or global? I mean, if we have some repertoire of possible ideas about this, can we just think of the connectome as a huge number of additional constraints that will help to refine to ultimately have a consistent picture of that? I think about this for the, the steering subsystem stuff too, just very basic things about it. How many different types of dopamine signal?

• 1:07:33Speaker 01

  Or of steering subsystem signal or thought assessor or so on? How many different types of what broad categories are there? Like even this very basic information that there's more cell types in the hypothalamus than there are in the cortex? Like, that's new information, right? About how much structure is built there versus somewhere else? Yeah, how many different dopamine neurons are there? Is the wiring between prefrontal and auditory the same as the wiring between prefrontal and visual? You know, it's like, uh, the most basic things we don't know. And the problem is, learning even the most basic things by a series of bespoke experiments takes an incredibly long time. Whereas just learning all of that at once by getting a connectome is just like way more efficient.

• 1:08:09Dwarkesh Patel?

  What is the timeline on this? Because presumably the idea of this is to, um, well, first, inform the development of AI. You want to be able to figure out how we do the, um, how we get AIs to want to care about what other people think of its internal thought pattern. But, inter researchers are making progress on this question just by inspecting, you know, normal neural networks. There must be some feature.

• 1:08:35Speaker 01

  You can do inter on LLMs that exist.

• 1:08:38Dwarkesh Patel?

  Yeah.

• 1:08:38Speaker 01

  You can't do inter on a hypothetical model-based reinforcement algorithm like the brain that we will eventually converge to when we do AGI.

• 1:08:46Dwarkesh Patel?

  Fair, fair. But, um, yeah, you know, what, what timelines on AI do you need? For this research to be practical and relevant?

• 1:08:54Speaker 01

  I think it's fair to say it's not super practical and relevant if you're in like AI 2027 scenario.

• 1:09:00Dwarkesh Patel?

  Yeah.

• 1:09:01Speaker 01

  You know, and so like what science I'm doing now is not going to affect the science of like 10 years from now. Because what's going to affect the science of 10 years from now is the outcome of this like AI 2027 scenario. Right? It's kind of doesn't matter that much probably if I have the connectome. Maybe it slightly tweaks certain things, but, um, but I think there, there's a lot of. Reason to think maybe that we will get a lot out of this paradigm, but then the real thing, the thing that is like the like, single event that is like transformative for the entire future or something type event is still like, you know, more than five years away.

• 1:09:38Dwarkesh Patel?

  Is that because the like, we haven't captured omnidirectional inference? We haven't figured out the right ways to get a mind to pay attention to things in a way that makes it.

• 1:09:50Speaker 01

  I would take the entirety of your like collective podcast with everyone as like showing like the distribution of these things, right? I don't know, right? But what was Carpathia's timeline, right? You know, what's Demis' timeline, right? So then not everybody has a three-year timeline. And so I think.

• 1:10:08Dwarkesh Patel?

  But there's different reasons and I'm curious.

• 1:10:09Speaker 01

  There are different reasons. What are mine?

• 1:10:12Speaker 01

  I don't know. I'm just watching your podcast. I'm trying to understand the distribution. I don't have a super strong claim that LLMs can't do it.

• 1:10:19Dwarkesh Patel?

  But it is across like data efficiency or is it the?

• 1:10:21Speaker 01

  I think part of it is just it is weirdly different than all this brain stuff.

• 1:10:25Dwarkesh Patel?

  Yeah, yeah, yeah.

• 1:10:26Speaker 01

  And so intuitively, it's just weirdly different than all this brain stuff. And I'm kind of waiting for like the thing that starts to look more like brain. Like I think if AlphaZero and model-based RL and all of these other things that were being worked on 10 years ago had been giving us the GPT-5 type capabilities, then I would be like, oh, wow, we're both in the right paradigm and seeing the results.

• 1:10:44Dwarkesh Patel?

  Right.

• 1:10:45Speaker 01

  A priori, so my model, my prior and my data are agreeing.

• 1:10:49Dwarkesh Patel?

  Yeah, yeah.

• 1:10:49Speaker 01

  Right? And now is like, I don't know what exactly my data is, looks pretty good, but my prior is sort of weird. So, yeah, so I don't have a super strong opinion on it. So I think there's a possibility that essentially all other scientific research that is being done is like not, is somehow obviated, but I don't put a huge amount of probability on that. I think my timelines might be more in the like, yeah, 10-year-ish range. And if that's the case, I mean, I think there, yeah, there is probably a difference between a world where we have connectomes on hard drives and we have understanding of steering subsystem architecture. We've compared the, you know, even the most basic properties of what are the reward functions, cost function, architecture, et cetera, of, you know, mouse versus a shrew versus a small primate, et cetera.

• 1:11:32Dwarkesh Patel?

  This is practical in 10 years?

• 1:11:34Speaker 01

  I think it has to be a really big push.

• 1:11:35Dwarkesh Patel?

  Like, how much funding? How does it compare to where we are now?

• 1:11:38Speaker 01

  It's like billion, low billions of dollars scale funding in a very concerted way, I would say.

• 1:11:44Dwarkesh Patel?

  And how much is on it now?

• 1:11:46Speaker 01

  Well, so if I just talk about some of the specific things we have going so with connectomics, so E11 bio is kind of like our main thing on connectomics. They are basically trying to make the technology of connectomic brain mapping several orders of magnitude cheaper. So the Wellcome Trust put out a report a year or two ago that basically said to get one mouse brain, the first mouse brain connectome would be like several billion dollars, billions of dollars project. What E11 technology and sort of the suite of efforts in the field also are trying to get like a single mouse connectome down to like low tens of millions of dollars.

• 1:12:30Speaker 01

  Okay, so that's a mammal brain, right? Now, a human brain is about 1,000 times bigger. So if a mouse brain, you can get to 10 million or 20 million, 30 million with technology. If you just naively scale that, okay, human brain is now still billions of dollars to just one do one human brain. Can you go beyond that? So can you get a human brain for like less than a billion? But I'm not sure you need every neuron in a human brain. I think we want to, for example, do an entire mouse brain and a human steering subsystem and the entire brains of several different mammals with different social instincts. And so I think that that with a bunch of technology push and a bunch of concerted effort can be done in the real significant progress if it's focused effort can be done in the kind of hundreds of millions to low billions.

• 1:13:13Dwarkesh Patel?

  What is the definition of a connectome? Is it presumably it's not a bottom-up biophysics model? So is it just that if it can estimate the input-output of a brain, but like what is the level of abstraction?

• 1:13:26Speaker 01

  You can give different definitions. And one of the things that's cool about, so the kind of standard approach to connectomics uses the electron microscope and very, very thin slices of brain tissue. And it's basically labeling the cell membranes are going to show up. Scatter electrons a lot and everything else is going to scatter electrons less. But you don't see a lot of details of the molecules, which types of synapses, different synapses of different molecular combinations and properties. E11 and some other research in the field has switched to an optical microscope paradigm. With optical, the photons don't damage the tissue. So you can kind of wash it and look at fragile, gentle molecules. So with the E11 approach, you can get a quote unquote molecularly annotated connectome.

• 1:14:07Speaker 01

  So that's not just who is connected to who by some kind of synapse, but what are the molecules that are present at the synapse? What type of cell is that? So molecularly annotated connectome, that's not exactly the same as having synaptic weights. That's not exactly the same as being able to simulate the neurons and say, was the functional consequence of having these molecules and connections? But you can also do some amount of activity mapping and try to correlate structure to function. Yeah, so.

• 1:14:38Dwarkesh Patel?

  Interesting.

• 1:14:39Speaker 01

  Train an ML model to basically predict the activity from the connectome.

• 1:14:42Dwarkesh Patel?

  What are the lessons to be taken away from the Human Genome Project? Because one way you could look at it is that it was actually a mistake and you shouldn't have spent whatever billions of dollars getting one genome mapped. Rather, you should have just invested in technologies which have, and now allows to map genomes for hundreds of dollars.

• 1:14:58Speaker 01

  Yeah, well, yeah, so George Church was my PhD advisor and basically, yeah, I mean, what he's pointed out is that, yeah, it was three billion or something, roughly $1 per base pair for the first genome. And then the National Human Genome Research Institute basically structured the funding process right and they got a bunch of companies competing to lower the cost. And then the cost dropped like a million fold in 10 years because they changed the paradigm from kind of macroscopic kind of chemical techniques to these.

• 1:15:29Speaker 01

  Individual DNA molecules make a little cluster of DNA molecules in a microscope and you would see just a few DNA molecules at a time on each pixel of the camera would basically give you a different in parallel, looking at different fragments of DNA. So you parallelize the thing by like millions fold and that's what reduced the cost by millions fold. And, and yeah, so, so I mean, essentially with switching from electron microscopy to optical conectomics, potentially even future types of conectomics technology, we think there should be similar patterns. That's why E11 with the focused research organization started with technology development rather than starting with saying, we're going to do a human brain or something. Let's just brute force it. We said, let's get the cost down with new technology. But then you still, it's still a big thing. Even with

• 1:16:12Speaker 01

  new next generation technology, you still need to spend hundreds of millions on data collection.

• 1:16:17Dwarkesh Patel?

  Yeah. Is this going to be funded with philanthropy by governments, by investors?

• 1:16:22Speaker 01

  This is very TBD and very much evolving in some sense as we speak. I'm hearing some rumors going around of conectomics related companies potentially forming. But so, so far E11 has been philanthropy. The National Science Foundation just put out this call for it for tech labs, which is basically somewhat of it is kind of fro inspired or related. I think you could have a tech lab for actually going and mapping the mouse brain with us. And that would be sort of philanthropy plus government, still in a nonprofit kind of open source framework. But can, can companies accelerate that? Can you credibly link conectomics to AI in the context of a company and get investment for that? It's like possible?

• 1:17:05Dwarkesh Patel?

  I mean, the cost of training these AIs is increasing so much. If you could like tell some story of like, not only are we going to figure out some safety thing,

• 1:17:13Speaker 01

  Right.

• 1:17:13Dwarkesh Patel?

  but in fact, we will, once we do that, we'll also be able to tell you how AI works.

• 1:17:19Speaker 01

  I mean, all these questions.

• 1:17:20Dwarkesh Patel?

  You should like go to these AI labs and just be like, give me one one hundredth of your projected budget in 2030.

• 1:17:25Speaker 01

  I sort of tried a little bit like, like seven or eight years ago, and there was not a lot of interest. And maybe now there, there would be. But yeah, I mean, I think all the things that we've been talking about, like,

• 1:17:38Speaker 01

  I think it's really fun to talk about, but it's ultimately speculation. What is the actual reason for the energy efficiency of the of the brain? For example, right? Is it doing real inference or amortized inference or something else? Like, this is all going to be all it's all answerable by neuroscience. It's going to be hard, but it's actually answerable. And so if you can only do that for low billions of dollars or something to really comprehensively solve that, it seems to me in the grand scheme of trillions of dollars of GPUs and stuff, it actually makes sense to do that investment.

• 1:18:06Dwarkesh Patel?

  And I think investors also just there's been many labs that have been launched in the last year where they're raising on the valuation of billions.

• 1:18:14Speaker 01

  Yes.

• 1:18:14Dwarkesh Patel?

  For things which are quite credible, but are not like RARR, next quarter is going to be whatever. It's like, we're going to discover materials and dot, dot, dot, right?

• 1:18:23Speaker 01

  Yes, yes. Moonshot startups or billion dollar billionaire backed startups, moonshot startups, I see as kind of on a continuum with Fro's.

• 1:18:30Dwarkesh Patel?

  Yeah.

• 1:18:31Speaker 01

  Fro's are a way of channeling philanthropic support and ensuring that it's open source, public benefit, various other things that may be properties of a given Fro. But yes, billionaire backed startups, if they can target the right science, the exact right science, I think there's a lot of ways to do moonshot neuroscience companies that would never get you the connectome. He's like, oh, we're going to upload the brain or something, but never actually get the mouse connectome or something, these fundamental things that you need to get to ground truth to science. There are lots of ways to have a moonshot company kind of go wrong and not do the actual science, but there also may be ways to have companies or big corporate labs get involved and actually do it correctly.

• 1:19:12Dwarkesh Patel?

  Yeah. This, uh, this brings to mind an idea that you had in a lecture you gave five years ago about, yeah, do you want to explain behavior cloning on?

• 1:19:23Speaker 01

  Right. Yeah, I mean, actually, this is funny because I think that the first time I saw this idea, it was, I think it actually might have been in a blog post by Gorn.

• 1:19:30Dwarkesh Patel?

  Oh.

• 1:19:31Speaker 01

  There's always, there's always a Gorn blog post. And there are now academic research efforts and some amount of emerging company type efforts to try to do this. So, um, yeah, so normally like, let's say I'm training an image classifier or something like that. I show it pictures of cats and dogs or whatever, and they have the label cat or dog. And I have a neural net that's supposed to predict the label cat or dog or something like that.

• 1:19:57Speaker 01

  That is a limited amount of information per label that you're putting in. It's just cat or dog. What if I also had predict what is my neural activity pattern when I see a cat? Or when I see a dog and all the other things?

• 1:20:16Speaker 01

  If you add that as like an auxiliary loss function or an auxiliary prediction task, does that sculpt the network to know the information that humans know about cats and dogs? And to represent it in a way that's consistent with how the brain represents it and the kind of representational kind of dimensions or geometry of how the brain represents things? As opposed to just having these labels, does that let it generalize better? Does that let it have just richer labeling? And of course, that's like, that sounds really challenging. It's very easy to generate lots and lots of labeled cat pictures with, you know, Scale AI or whatever can do this. It is harder to generate lots and lots of brain activity patterns that correspond to things that you want to train the AI to do.

• 1:21:02Speaker 01

  But again, this is just a technological limitation of neuroscience. If we, if every iPhone was also a brain scanner, you know, you would, you would not have this problem and we would be training AI with the brain signals. And it's just the order in which technology is developed is that we got GPUs before we got portable brain scanners or whatever, right? And that kind of thing.

• 1:21:18Dwarkesh Patel?

  What is the analog, what would you be doing here? Cause when you distill models, you're still looking at the final layer, like the, the log probes across, across all tokens.

• 1:21:30Speaker 01

  If you do distillation of one model into another, that is a certain thing. You're just trying to copy one model into another.

• 1:21:37Dwarkesh Patel?

  Yeah.

• 1:21:38Speaker 01

  I think that we don't really have a

• 1:21:43Speaker 01

  perfect proposal to like distill the brain. I think to distill the brain, you need like a much more complex brain interface. Like maybe you could also do that. You could make surrogate models. Andreas Tolia and people like that are doing some amount of neural network surrogate models of brain activity data. Instead of having your visual cortex do the computation, just have the surrogate models. You're basically distilling your visual cortex into a neural network to some degree.

• 1:22:09Speaker 01

  That's the kind of distillation. This is doing something a little different. This is basically just saying, I'm adding an auxiliary. I think of as regularization, or I think of it as adding an auxiliary loss function. That's sort of smoothing out the prediction task to also always be consistent with how the brain represents it.

• 1:22:25Dwarkesh Patel?

  What exactly are you predicting?

• 1:22:27Speaker 01

  It might help you with things like adversarial examples, for example, right?

• 1:22:29Dwarkesh Patel?

  So you're predicting the internal state of the brain?

• 1:22:32Speaker 01

  Yes. So in addition to predicting the label, the vector of labels like yes, cat, not dog, yes, not boat, you know, one-shot vector or whatever of one-hot vector of yes, it's cat, instead of these gazillion other categories, let's say, in this simple example. You're also predicting a vector, which is like all these brain signal measurements.

• 1:22:51Dwarkesh Patel?

  Right.

• 1:22:51Speaker 01

  Yeah.

• 1:22:52Dwarkesh Patel?

  Interesting.

• 1:22:53Speaker 01

  And so Gourin, anyway, had this long ago blog post of like, oh, this is like an intermediate thing. It's like, we talk about whole-brain emulation, we talk about AGI, we talk about brain-computer interface, we should also be talking about this like brain augmented, brain data augmented thing that's trained on all your behavior, but is also trained on like predicting some of your neural patterns.

• 1:23:14Dwarkesh Patel?

  Right. And you're saying the learning system is already doing this for the steering system?

• 1:23:17Speaker 01

  Yeah. And our learning system also has to predict the steering subsystem as an auxiliary task.

• 1:23:21Dwarkesh Patel?

  Yeah.

• 1:23:22Speaker 01

  Yeah. And that helps the steering subsystem. Now the steering subsystem can access that predictor and build a cool reward function using it. Yes.

• 1:23:28Dwarkesh Patel?

  Okay. Separately, you're on the board for Ovlean, which is this formal math language that mathematicians use to prove theorems and so forth. And obviously, there's a bunch of conversation right now about math, AI automating math. What's your take?

• 1:23:49Speaker 01

  Yeah. Well, I think that there are parts of math that it seems like it's

• 1:23:54Speaker 01

  pretty well on track to automate. And that has to do with like

• 1:24:01Speaker 01

  So first of all, Lean had been developed for a number of years at Microsoft and other places. It has become one of the convergent focused research organizations to kind of drive more engineering and focus on to it. So Lean is like this language, programming language, where if you, instead of expressing your math proof on pen and paper, you express it in this programming language Lean. And then at the end, if you do that that way, it is a verifiable language so that you can basically click verify and Lean will tell you whether the conclusions of your proof actually follow perfectly from your assumptions of your proof. So it checks whether the proof is correct automatically. And just like by itself, this is useful for mathematicians collaborating and stuff like that.

• 1:24:46Speaker 01

  Like if I'm some amateur mathematician, I want to add to a proof, you know, Terry Tao is not going to like believe my results. But if Lean says it's correct, it's just correct. So it makes it easy for like collaboration to happen. But it also makes it easy for correctness of proofs to be an RL signal in very much the RLVR. You know, it's like a perfect math proofing is now formalized math proving. So formal means it's like expressed in something like Lean and verifiable, mechanically verifiable. That becomes a perfect RLVR, you know, task. Yeah, and I think that that is going to just, just keep working.

• 1:25:29Speaker 01

  It seems like is the couple billion dollar, at least one like billion dollar valuation company, Harmonic, based on this. Alpha proof is based on this. A couple of their emerging really interesting companies. I think that this problem of like RLVR-ing the crap out of math proving is basically going to work. And we will be able to have things that search for proofs and find them in the same way that we have AlphaGo or what have you that can search for, you know, ways of playing the game of Go and with that verifiable signal works. So it does this like solve math. There is still the part that has to do with conjecturing new interesting ideas.

• 1:26:12Speaker 01

  There's still the kind of conceptual organization of math of what is interesting. How do you come up with new theorem statements in the first place? Or even like the very high level breakdown of what strategies you use to do proofs.

• 1:26:27Speaker 01

  I think this will shift the burden of that so that humans don't have to do a lot of the mechanical parts of math, validating lemmas and proofs and checking if the statement of this in this paper is exactly the same as that paper and stuff like that. It will just, that will just work. You know, if you really think you're, we're going to get all these things, we've been talking about real AGI, it would also be able to make conjectures. And, you know, Benjio has like a paper as more like theoretical paper, there are probably a bunch of other papers emerging about this, like, is there like a loss function for like good explanations or good conjectures? That's like a pretty profound question, right? A math, a really interesting math proof or statement might be one that kind of compresses lots of information about other, you know, has lots of implications for lots of other theorems.

• 1:27:12Speaker 01

  Otherwise, you would have to prove those theorems using long, complex passive inference. Here, if you have this theorem, this theorem is correct, and you have short passive inference to all the other ones. And it's this short, compact statement, so it's like a powerful explanation that explains all the rest of math. And like, part of what math is doing is like making these compact things that explain the other things.

• 1:27:29Dwarkesh Patel?

  So the Kull-Go-Maurer complexity of this statement or something?

• 1:27:32Speaker 01

  Yeah, of generating all the other statements, given that you know this one or stuff like that, or if you add this, how does it affect the complexity of the rest of the kind of network of proofs? So can you like make a loss function that adds, oh, I want this proof to be a really highly powerful proof? I think some people are trying to work on that. So, so maybe you can automate the creativity part. If you had true AGI, it would do everything a human can do, so it would also do the things that the creative mathematicians do. But, but way barring that, I think just RLVR-ing the crap out of proofs, well, I think that's going to be, this is a really useful tool for mathematicians, it's going to accelerate math a lot and change it a lot, but not necessarily immediately change everything about it.

• 1:28:14Speaker 01

  Will we get, you know, mechanical proof of the Riemann hypothesis or something like that, or things like that, maybe? I don't know. I don't know enough details of how hard these things are to search for. I'm not sure anyone can fully predict that, just as we couldn't exactly predict when Go would be solved or something like that.

• 1:28:33Speaker 01

  And I think it's gonna have lots of really cool applied applications. So one of the things you want to do is you want to have provably stable, secure, unhackable, et cetera, software. So you can write math proofs about software and say, this code, not only does it pass these unit tests, but I can mathematically prove that there's no way to hack it in these ways, or no way to mess with the memory, or this type of things that hackers use, or it has these properties.

• 1:29:05Speaker 01

  It can use the same lean and same proof to do formally verified software. I think that's gonna be a really powerful piece of cybersecurity.

• 1:29:14Speaker 01

  That's relevant for all sorts of other AI hacking the world stuff. And that, yeah, if you can prove a Riemann hypothesis, you're also gonna be able to prove insanely complex things about very complex software. And then you'll be able to ask the LLM, synthesize me a software that is, I can prove is correct, right?

• 1:29:32Dwarkesh Patel?

  Why hasn't provable programming language taken off as a result of LLMs? You would think that this would-

• 1:29:41Speaker 01

  I think it's starting to, yeah, I think it's starting to. I think that one, one challenge, and we are actually incubating a potential focus research organization on this, is the specification problem. So mathematicians kind of know what interesting theorems they want to formalize. If I have like some code, let's say I have some code that like is involved in running the power grid or something, and it has some security properties. Well, what is the formal spec of those properties? The power grid engineers just made this thing, but they don't necessarily know how to lift the formal spec from that. And it's not necessarily easy to come up with the spec that is the spec that you want for your code. People aren't used to coming up with formal specs, and there are not a lot of tools for it. So you also have like this kind of user interface plus AI problem of like,

• 1:30:25Speaker 01

  what security spec should I be specifying? Is this the spec that I wanted? So there's a spec problem. And it's just been really complex and hard, but, but it's only just in the last very short time that, that the LLMs are able to generate, you know, verifiable proofs of, you know, things that are useful to mathematicians. Starting to be able to do some amount of that for, for software verification, hardware verification. But I think if you project the trends over the next couple of years. It's possible that it just flips the tide that formal methods, based this whole field of formal methods or formal verification, provable software,

• 1:31:05Speaker 01

  which is kind of this weird, almost like backwater of more like theoretical part of programming languages and stuff. Very academically flavored often, although there was like this DARPA program that made like a provably secure, like quadcopter, helicopter and stuff like that.

• 1:31:21Dwarkesh Patel?

  So secure against, like, what is the property that is exactly prude?

• 1:31:26Dwarkesh Patel?

  Not for that particular project, but just in general, like, what, what, because obviously the, the, the things malfunction for all kinds of reasons, like.

• 1:31:33Speaker 01

  You could say that

• 1:31:36Speaker 01

  what's going on in this part of the memory over here, which is supposed to be the part the user can access, can't in any way affect what's going on in the memory over here or something like that, or, yeah, things like that. Yeah.

• 1:31:48Dwarkesh Patel?

  Got it.

• 1:31:48Speaker 01

  Yeah.

• 1:31:50Dwarkesh Patel?

  So there's, there's two questions. One is,

• 1:31:54Dwarkesh Patel?

  how useful is this?

• 1:31:56Speaker 01

  Yeah.

• 1:31:56Dwarkesh Patel?

  And two is like, how satisfying a, a, as a, as a mathematician, would it be?

• 1:32:05Dwarkesh Patel?

  And the fact that there's this application towards proving that software has certain properties or hardware has certain properties, obvious, like if that works, that would obviously be very useful. But from a pure, like, are we going to figure out mathematics?

• 1:32:18Speaker 01

  Right.

• 1:32:20Dwarkesh Patel?

  Yeah. Is there, is your sense that there's something about finding that one construction cross-maps to another construction in a different domain? Or finding that, oh, this like lema is, if you reconfigure it, like if you redefine this, this term, it's still like, kind of satisfies what I meant by this term, but it, it no longer, a counter example that previously knocked it down, no longer applies. Like that kind of dialectical thing that happens in mathematics.

• 1:32:48Speaker 01

  Will this software like replace that?

• 1:32:50Dwarkesh Patel?

  Yeah. And like, how, how much of the value of this sort of pure mathematics just comes from actually just coming up with entirely new ways of thinking about a problem?

• 1:32:58Speaker 01

  Yeah.

• 1:32:59Dwarkesh Patel?

  Like mapping it to a totally different representation and, yeah, do we have examples of?

• 1:33:03Speaker 01

  I don't know. I think of it as, I think of it maybe a little bit like the, if whenever you have, everybody had to write assembly code or something like that. Just like the amount of fun, like cool startups that got created was like a lot less or something, right? And so it was just like.

• 1:33:18Speaker 01

  Less people could do it. Progress was more grinding and slow and lonely and so on. You had more false failures because you didn't get something about the assembly code right, rather than the essential thing of like, was your concept right?

• 1:33:31Speaker 01

  Harder to collaborate and stuff like that. And so I think it will like be really good. There is some worry that by not learning to do the mechanical parts of the proofs, that you fail to generate the intuitions that inform the more conceptual part, creative part, right?

• 1:33:46Dwarkesh Patel?

  Yeah, it's the same with assembly and.

• 1:33:48Speaker 01

  Right. And so at what point is that applying? Is vibe coding? Are people not learning computer science, right? Or actually, are they like vibe coding and they're also simultaneously looking at the LLM? It's like explaining to them these abstract computer science concepts and it's all just like, all happening faster, their feedback loop is faster and they're learning way more abstract computer science and algorithm stuff because they're vibe coding. You know, I don't know. It's not obvious. That might be something the user interface and the human infrastructure around it. But I guess there's some worry that people don't learn the mechanics and therefore don't build like the grounded intuitions or something. But my hunch is it's like super positive. Exactly on net, how useful that will be or how much overall math like breakthroughs or like math breakthroughs even that we care about will happen?

• 1:34:33Speaker 01

  I don't know. I mean, one other thing that I think is cool is actually the accessibility question. It's like, okay, that sounds a little bit corny. Okay, yeah, more people can do math, but who cares? But I think there's actually lots of people that like could have interesting ideas, like maybe the quantum theory of gravity or something. Like, yeah, one of us will come up with the quantum theory of gravity instead of like a card carrying physicist. In the same way that Steve Burns is like reading the neuroscience literature and he's like, hasn't been in a neuroscience lab that much. But he's like able to synthesize across the neuroscience literature, be like, oh, learning subsystems, steering subsystems. Does this all make sense? He's, you know, it's kind of like he's an outsider neuroscientist in some ways. Can you have outsider, you know, string theorists or something because the math is just done for them by the computer?

• 1:35:18Speaker 01

  And does that lead to more innovation in the string theory? Right? Maybe yes.

• 1:35:22Dwarkesh Patel?

  Interesting.

• 1:35:22Speaker 01

  So.

• 1:35:23Dwarkesh Patel?

  Okay, so if this approach works and you're right that LLMs are not the final paradigm, and suppose it takes at least 10 years to get the final paradigm.

• 1:35:32Speaker 01

  Yeah.

• 1:35:32Dwarkesh Patel?

  In that world, there's this fun sci-fi premise where. You have, Terrence Tao today had a tweet where he's like, these, these models are like automated cleverness, but not automated intelligence. And you can quibble with the definitions there. But yeah, if you have automated cleverness and you have some way of filtering, which if you can formalize and prove things that the LLMs are saying, you could do.

• 1:36:02Speaker 01

  Yes.

• 1:36:02Dwarkesh Patel?

  Then you could have this situation where quantity has a quality all of its own.

• 1:36:06Speaker 01

  Yes.

• 1:36:06Dwarkesh Patel?

  And so what are the domains of the world which could be put in this provable symbolic representation?

• 1:36:14Speaker 01

  Yeah.

• 1:36:15Dwarkesh Patel?

  Furthermore, okay, so in the world where AGI is super far away, maybe it makes sense to like literally turn everything the LLMs ever do, or almost everything they do, into like super provable statements. And so LLMs can actually build on top of each other because everything they do is like super provable.

• 1:36:29Speaker 01

  Yeah.

• 1:36:30Dwarkesh Patel?

  Maybe, maybe this is like just necessary because you have billions of intelligences running around, even if they are super intelligent. The only way the future AGI civilizations can collaborate with each other is if they, they can prove each step.

• 1:36:41Speaker 01

  Yeah.

• 1:36:42Dwarkesh Patel?

  And they're just like brute force churning out, this is what the Jupiter brains are doing.

• 1:36:46Speaker 01

  It's a universe, it's a universal language, it's provable. And it's also provable from like, are you trying to exploit me? Or are you sending me some?

• 1:36:52Dwarkesh Patel?

  Yeah.

• 1:36:52Speaker 01

  Some message that's actually trying to like sort of hack into my, my brain effectively? Are you trying to socially influence me? Are you actually just like sending me just the information that I need and no more?

• 1:37:01Dwarkesh Patel?

  Right.

• 1:37:02Speaker 01

  For this. And yeah, so David Dodd, who's like this program director at ARIA now in the UK. I mean, he has this whole design of a, of a kind of ARPA style program of sort of safeguarded AI that very heavily leverages like provable safety properties. And can you apply proofs to like, can you have a world model? But that world model is actually not specified just in neuron activations, but it's specified in, you know, equations. Those might be very complex equations, but if you can just get insanely good at just auto proving these things with cleverness, auto cleverness, can you have explicitly interpretable world models, you know,

• 1:37:40Speaker 01

  as opposed to neural net world models? And like move back basically to symbolic methods just because you can, you can just have insane amount of ability to prove things. Yeah, I mean, that's an interesting vision. I don't know how, you know, in the next 10 years, like whether that will be the vision that plays out. But I think it's really interesting.

• 1:37:57Speaker 01

  To think about, yeah, and even for math, I mean, I think Terry Tao is like doing some amount of stuff where it's like, it's not about whether you can prove the individual theorems. It's like, let's prove all the theorems en masse. And then let's like, study the properties of like, the aggregate set of proved theorems. Right? Which are the ones that got proved and which are the ones that didn't? Okay, well, that's like, the landscape of all the theorems instead of one theorem at a time, right?

• 1:38:18Dwarkesh Patel?

  Speaking of symbolic representations, one question I was meaning to ask you is, how does the brain represent the world model? Like, obviously, that's out in neurons, but I don't mean sort of extremely functionally. I mean, sort of, conceptually, is it in something that's analogous to the hidden state of a neural network? Or is it something that's closer to a symbolic language?

• 1:38:39Speaker 01

  We don't know. I mean, I think there's, there's some amount of study of this. I mean, there's, there's these things like, you know, face patch neurons that represent certain parts of the face that geometrically combine in interesting ways. That's sort of with geometry and vision. Is that true for like, other more abstract things? There's like, this idea of cognitive maps, like, a lot of the stuff that a rodent hippocampus has to learn is like, place cells and like, where is the rodent gonna go next? And is it gonna get a reward there? Is like, very geometric and like, do we organize concepts with like, a space, abstract version of a spatial map?

• 1:39:13Speaker 01

  There's some questions of can we do like, true symbolic operations? Like, can I have like, a register in my brain that copies a variable to the another register regardless of what the content of that, that variable is? That's like, this variable binding problem. And basically, I just don't, I don't know if we have that, like, machinery, or if we, or if it's like, more like, cost functions and architectures that like, make some of that approximately emerge, but maybe would also emerge in a neural net. There's a bunch of interesting neuroscience research trying to study this, what, what the representations look like.

• 1:39:41Dwarkesh Patel?

  But what was your hunch?

• 1:39:42Speaker 01

  Yeah. My hunch is it's gonna be a huge mess and we should look at the architectures, the loss functions and the learning rules and we shouldn't really, I don't expect it to be pretty in there. Yeah.

• 1:39:51Dwarkesh Patel?

  We should say this is not a symbolic language.

• 1:39:53Speaker 01

  Yeah, probably, probably it's not that symbolic. Yeah, but, but, but other people think very differently, you know, yeah.

• 1:40:00Dwarkesh Patel?

  I have other random questions. Speaking of binding. Yeah, but what is up with feeling like there's an experience? It's like both all the parts of your brain, which are modeling very different things, have different drives. Feel like, at least presumably, feel like there's an experience happening right now. And also that across time, you feel like, what is?

• 1:40:20Speaker 01

  Yeah, I'm pretty much at a loss on this one.

• 1:40:25Speaker 01

  I don't know. I mean, Max Hodak has been giving talks about this recently. He's another really hardcore neuroscience person, neurotechnology person. And the thing I mentioned with Dorsso is maybe also it sounds like it might have some touching on this question. But yeah, I think this, I don't think anybody has any idea. It might even involve new physics. It's like, you know, yeah.

• 1:40:54Dwarkesh Patel?

  Another question, which might not have an answer yet. What, so continual learning?

• 1:41:01Dwarkesh Patel?

  Is that the product of something extremely fundamental, the level of even the learning algorithm? Where you could say, look, at least the way we do backprop in neural networks is that you freeze the way, there's a training period and you freeze the weights.

• 1:41:16Dwarkesh Patel?

  And so you just need this active inference or some other learning rule in order to do continual learning? Or do you think it's more a matter of architecture? And how is memory exactly stored? And is it like, what kind of associative memory you have, basically?

• 1:41:29Speaker 01

  Yeah, so continual learning. I don't know. I think that there's probably things that, there's probably some, at the architectural level, there's probably something interesting stuff that the hippocampus is doing.

• 1:41:44Speaker 01

  And people have long thought this.

• 1:41:49Speaker 01

  What kinds of sequences is it storing? How is it organizing, representing that? How is it replaying it back? What is it replaying back?

• 1:41:57Speaker 01

  How is it exactly how that memory consolidation works? In sort of training the cortex using replays or memories from the hippocampus or something like that? There's probably some of that stuff. There might be multiple time scales of plasticity or sort of clever learning rules. That can kind of, I don't know, can sort of simultaneously kind of be storing sort of short-term information and also doing backprop with it. I mean, neurons might be doing a couple of things, you know, some fast weight plasticity and some slower plasticity at the same time, or synapses that have many states. I mean, I don't know. I mean, I think that from a neuroscience perspective, I'm not sure that I've seen something that's super clear on what continual learning, what causes it, except maybe to say that this, this systems consolidation idea of sort of hippocampus consolidating cortex,

• 1:42:42Speaker 01

  like some people think is a big piece of this and we don't still fully understand the details. Yeah.

• 1:42:47Dwarkesh Patel?

  Speaking of fast weights, is there something in the brain which is the equivalent of this distinction between parameters and activations that we see in neural networks? And specifically, like in transformers, we have this idea, like some of the activations are the key and value vectors of previous tokens that you build up over time. And there's like the so-called the fast weights that you, whenever you have a new token, you query them against these, you query against these activations, but you also obviously query them against all the other parameters in the network, which are part of the actual built-in weights. Is there some such distinction that's analogous?

• 1:43:27Speaker 01

  I don't know. I mean, we definitely have weights and activations. Whether you can use the activations in these clever ways, different forms of like actual attention, like attention in the brain, is that based on, I'm trying to pay attention. I think there's several, probably several different kinds of like actual attention in the brain. I want to pay attention to this area of visual cortex. I want to pay attention to this, the content in other areas that is triggered by the content in this area, right? Attention that's just based on kind of reflexes and stuff like that. So I don't know. I mean, I think that there's not just the cortex, there's also the thalamus. The thalamus is also involved in kind of somehow relaying or gating informations. There's cortical-cortical connections. There's also some amount of connection between cortical areas that goes

• 1:44:12Speaker 01

  through the thalamus. Is it possible that this is doing some sort of matching or kind of constraint satisfaction or matching across, you know, keys and, you know, keys over here and values over there? Is it possible that it can do stuff like that? Maybe. I don't know. This is all part of what's the architecture of this cortical-thalamic, yeah, system. I don't know how transformer-like it is, or if there's anything analogous to like that attention. Be interesting to find out.

• 1:44:42Dwarkesh Patel?

  We gotta give you a billion dollars so we can come on the podcast again and then tell me how exactly the

• 1:44:47Speaker 01

  Yeah, mostly I just do data collection, just like really, really unbiased data collection. So all the other people can figure out these questions. Yeah.

• 1:44:54Dwarkesh Patel?

  Maybe the final question to go off on is, what was the most interesting thing you learned from the gap map? And maybe you want to explain what the gap map is?

• 1:45:03Speaker 01

  So the gap map, so in the process of incubating and coming up with these focused research organizations, these sort of nonprofit startup like moonshots that we've been getting philanthropists and now government agencies to fund, we talked to a lot of scientists and some of the scientists were just like, here's the next thing my graduate student will do. Here's what I find interesting, exploring these really interesting hypothesis spaces, like all the types of things we've been talking about. And some of them are like, here's this gap. I need this piece of infrastructure, which like, there's no combination of a grad students in my lab or me loosely collaborating with other labs with traditional grants that could ever get me that. I need to have like an organized engineering team that like builds, you know, the miniature equivalent of the Hubble Space Telescope.

• 1:45:48Speaker 01

  And if I can build that Hubble Space Telescope, then like, I will unblock all the other researchers in my field or some like path of technological progress in the way that the Hubble Space Telescope made lifted the boats, improved the life of every astronomer, but wasn't really an astronomy discovery in itself. It was just like, you had to put this giant mirror in space with a CCD camera and like organize all the people in engineering and stuff to do that. So some of the things we talked to scientists about look like that. And so the gap map is basically just like a list of a lot of those things. And it's like, we call it a gap map. I think it's actually more like a fundamental capabilities map, like what are all these things like mini Hubble Space Telescopes?

• 1:46:26Speaker 01

  And then we kind of organize that into gaps for like, helping people understand that or like, search that.

• 1:46:32Dwarkesh Patel?

  And what was the most surprising thing you found?

• 1:46:34Speaker 01

  So, I mean, I think I've talked about this before, but I think it, one thing is just like,

• 1:46:40Speaker 01

  kind of like the overall size or shape of it or something like that is like, it's like a few hundred fundamental capabilities.

• 1:46:47Speaker 01

  So if each of these was like a deep tech startup sized project, that's like only a few billion dollars or something like, you know, each one of those was a series A, that's only like not, you know, it's not like a trillion dollars to solve these gaps. It's like lower than that. And so that's, that's like one, maybe we assumed that and we also came to, that's what we got. It's not really comprehensive. It's really just a way of summarizing a lot of conversations we've had with scientists.

• 1:47:11Speaker 01

  I do think that in the aggregate process, like things like lean are actually like surprising because I did start from sort of neuroscience and biology. And it was like very obvious that there's sort of like these omics. We need genomics, but we also need connectomics and, you know, we can engineer E. coli, but we also need to engineer the other cells. And like, there's like somewhat obvious parts of biological infrastructure. I did not realize that like math proving infrastructure like was a thing. And so, and that was kind of like immersion from trying to do this. So I'm looking forward to seeing other, other things where it's like, not actually this like hard intellectual problem to solve it. It's maybe the kind of slightly the equivalent of AI researchers just needed GPUs or something like that and focus and, and, and really good PyTorch code to like, start doing this, like,

• 1:47:54Speaker 01

  what is the full diversity of fields in which that exists? We've even now found,

• 1:48:03Speaker 01

  and which are the fields that do or don't need that? So fields that have had gazillions of dollars of investment, do they still need some of those? Do they still have some of those gaps? Or is it only more like neglected fields? We're even finding some interesting ones in actual astronomy, actual telescopes that have not been explored because maybe because of that kind of,

• 1:48:25Speaker 01

  if you're getting above a critical mass size project, then you have to have like a really big project. And that's a more bureaucratic process with the federal agencies. Yeah.

• 1:48:32Dwarkesh Patel?

  I guess, I guess you just kind of need scale in every single domain of science these days.

• 1:48:36Speaker 01

  Yeah, I think you need scale in many of the domains of science, and that does not mean that the low scale work is not important. Does not mean that kind of creativity, serendipity, et cetera. Each student pursuing a totally different direction or thesis that you see in universities is not like also really key. But yeah, I think we need some amount of scalable infrastructure is missing in essentially every area of science, even math, which is crazy because mathematicians, I thought, just needed whiteboards. Right, yeah. Right, but they actually need Lean. They actually need verifiable programming languages and stuff. Like, I didn't know that. Yeah. Cool, Alan, this is super fun. Thanks for coming on. Thank you so much. Where can people find your stuff? It's a pleasure. The easiest way now, my AdamMarblestone.org website is currently down, I guess.

• 1:49:18Speaker 01

  But you can find convergentresearch.org can link to a lot of the stuff we've been doing, yeah. And then you have a great blog, Longshored National Science. Yes, Longshored National Science, yes, on WordPress, yeah. Cool. Thank you so much. Thanks. Yeah.

• 1:49:29Dwarkesh Patel?

  Hey, everybody. I hope you enjoyed that episode. If you did, the most helpful thing you can do is just share it with other people who you think might enjoy it. It's also helpful if you leave a rating or a comment on whatever platform you're listening on. If you're interested in sponsoring the podcast, you can reach out at dwarkesh.com/advertise. Otherwise, I'll see you on the next one.