#106 – Matt Botvinick: Neuroscience, Psychology, and AI at DeepMind

Matt Botvinick is the Director of Neuroscience Research at DeepMind. He is a brilliant cross-disciplinary mind navigating effortlessly between cognitive psychology, computational neuroscience, and artificial intelligence. Support this podcast by supporting these sponsors: – The Jordan Harbinger Show: https://www.jordanharbinger.com/lex – Magic Spoon: https://magicspoon.com/lex and use code LEX at checkout If you would like to get more information about this podcast go to https://lexfridman.com/ai or connect with @lexfridman on Twitter https://twitter.com/lexfridman, LinkedIn https://www.linkedin.com/in/lexfridman/, Facebook https://www.facebook.com/lexfridman, Medium https://medium.com/@lexfridman, or YouTube https://www.youtube.com/lexfridman where you can watch the video versions of these conversations. If you enjoy the podcast, please rate it 5 stars on Apple Podcasts https://podcasts.apple.com/us/podcast/artificial-intelligence/id1434243584, follow on Spotify https://open.spotify.com/show/2MAi0BvDc6GTFvKFPXnkCL, or support it on Patreon https://www.patreon.com/lexfridman. Here’s the outline of the episode. On some podcast players you should be able to click the timestamp to jump to that time. OUTLINE: 00:00 – Introduction 03:29 – How much of the brain do we understand? 14:26 – Psychology 22:53 – The paradox of the human brain 32:23 – Cognition is a function of the environment 39:34 – Prefrontal cortex 53:27 – Information processing in the brain 1:00:11 – Meta-reinforcement learning 1:15:18 – Dopamine 1:19:01 – Neuroscience and AI research 1:23:37 – Human side of AI 1:39:56 – Dopamine and reinforcement learning 1:53:07 – Can we create an AI that a human can love?

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Lex Fridman?

  The following is a conversation with Matt Botmanek, Director of Neuroscience Research at DeepMind. He's a brilliant, cross-disciplinary mind navigating effortlessly between cognitive psychology, computational neuroscience, and artificial intelligence. Quick summary of the ads: two sponsors, The Jordan Harbinger Show and Magic Spoon Cereal. Please consider supporting the podcast by going to jordanharbinger.com/lex and also going to magic spoon.com/lex and using code LEX at checkout after you buy all of their cereal. Click the links, buy the stuff. It's the best way to support this podcast and the journey I'm on.

• 0:44Lex Fridman?

  If you enjoy this podcast, subscribe on YouTube, review it with five stars on Apple Podcasts, follow on Spotify, support on Patreon, or connect with me on Twitter at LexFriedman, spelled surprisingly without the E, just F-R-I-D-M-A-N. As usual, I'll do a few minutes of ads now and never any ads in the middle that can break the flow of the conversation. This episode is supported by The Jordan Harbinger Show. Go to jordanharbinger.com/lex. It's how he knows I sent you. On that page, subscribe to his podcast on Apple Podcasts, Spotify, and you know where to look. I've been binging on his podcast.

• 1:26Lex Fridman?

  Jordan is a great interviewer and even a better human being. I recently listened to his conversation with Jack Barsky, former sleeper agent for the KGB in the 80s and author of Deep Undercover, which is a memoir that paints yet another interesting perspective on the Cold War era. I've been reading a lot about the Stalin and then Gorbachev and Putin eras of Russia, but this conversation made me realize that I need to do a deep dive into the Cold War era to get a complete picture of Russia's recent history. Again, go to jordanharbinger.com/lex, subscribe to his podcast. It's how he knows I sent you. It's awesome. You won't regret it.

• 2:06Lex Fridman?

  This episode is also supported by Magic Spoon, low-carb, keto-friendly, super amazingly delicious cereal. I've been on a keto or very low carb diet for a long time now. It helps with my mental performance. It helps with my physical performance, even during this crazy push up, pull up challenge I'm doing, including the running. It just feels great. I used to love cereal. Obviously, I can't have it now because most cereals have a crazy amount of sugar, which is terrible for you. So I quit it years ago. But Magic Spoon amazingly, somehow, is a totally different thing. Zero sugar, 11 grams of protein, and only three net grams of carbs.

• 2:50Lex Fridman?

  It tastes delicious. It has a lot of flavors, two new ones, including peanut butter. But if you know what's good for you, you'll go with cocoa. My favorite flavor and the flavor of champions. Click the magic spoon.com/lex link in the description and use code Lex at checkout for free shipping and to let them know I sent you. They have agreed to sponsor this podcast for a long time. They're an amazing sponsor and an even better cereal. I highly recommend it. It's delicious. It's good for you. You won't regret it. And now here's my conversation with Matt Botvnik.

• 3:46Lex Fridman?

  How much of the human brain do you think we understand?

• 3:50Speaker 01

  I think we're at a weird moment in the history of neuroscience in the sense that there's a, there, I feel like we understand a lot about the brain at a very high level.

• 4:06Speaker 01

  But a very, very coarse level.

• 4:09Lex Fridman?

  When you say high level, what are you thinking? Are you thinking functional?

• 4:12Speaker 01

  Yeah.

• 4:12Lex Fridman?

  Structurally?

• 4:14Speaker 01

  So, in other words, what is, what is the brain for? You know, what, what, what kinds of computation does the brain do?

• 4:22Speaker 01

  What kinds of behaviors would we have to, would we have to explain? If we were going to look down at the mechanistic level?

• 4:33Speaker 01

  And at that level, I feel like we understand much, much more about the brain than we did when I was in high school. But what, but it's, it's at a very, it's almost like we're seeing it through a fog. It's only at a very coarse level. We don't really understand what the, the neuronal mechanisms are that underlie these computations. We've gotten better at saying, you know, what are the functions that the brain is computing? That we would have to understand, you know, if we were going to get down to the neuronal level. And at the other end of the spectrum, we, you know, in the last few years, incredible progress has been made in terms of technologies that allow us to see, you know, actually literally see in some cases, what's going on at the, at the single unit level,

• 5:18Speaker 01

  even the dendritic level. And then there's this yawning gap in between.

• 5:23Lex Fridman?

  Well, that's interesting. So at the high level, so that's almost a cognitive science level?

• 5:26Speaker 01

  Yeah.

• 5:26Lex Fridman?

  Yeah. And then at the neuronal level, that's neurobiology and neuroscience.

• 5:31Speaker 01

  Yeah.

• 5:31Lex Fridman?

  Just studying single neurons, the, the, the synaptic connections and all the dopamine, all the kind of neurotransmitters.

• 5:38Speaker 01

  One blanket statement I should probably make is that,

• 5:43Speaker 01

  as I've gotten older, I have become more and more reluctant to make a distinction between psychology and neuroscience. To me, the point of neuroscience is to study.

• 5:56Speaker 01

  What the brain is for. If you, if you, if you're, if you're a nephrologist and you want to learn about the kidney, you start by saying, what is this thing for? Well, it seems to be for taking blood on one side that has metabolites in it that are, that shouldn't be there, sucking them out of the blood while leaving the good stuff behind, and then excreting that in the form of urine. That's what the kidney is for. It's like obvious. So the rest of the work is deciding how it does that. And this, it seems to me, is the right approach to take to the brain. You say, well, what is the brain for? The brain, as far as I can tell, is for producing

• 6:38Speaker 01

  behavior. It's from going, it's foregoing from perceptual inputs to behavioral outputs. And the behavioral outputs should be adaptive. So that's what psychology is about. It's about understanding the structure of that function. And then the rest of neuroscience is about figuring out how those operations are actually carried out at a mechanistic level.

• 7:01Lex Fridman?

  That's really interesting. But so unlike the kidney, the, the brain, the, the gap between the electrical signal and behavior. So you truly see neuroscience as the science of that, that touches behavior, how the brain generates behavior, or how the brain converts raw visual information into understanding? Like, like, you, you basically see cognitive science, psychology, and neuroscience as all one science?

• 7:33Speaker 01

  Yeah.

• 7:34Lex Fridman?

  Is that?

• 7:35Speaker 01

  It's a personal statement.

• 7:36Lex Fridman?

  Is that a hopeful, is that, is that a hopeful or realistic statement? So certainly, you will be correct in your feeling in some number of years, but that number of years could be 200, 300 years from now.

• 7:48Speaker 01

  Oh, well, well, there's a.

• 7:50Lex Fridman?

  Is that aspirational, or is that pragmatic engineering feeling that you have?

• 7:56Speaker 01

  It's, it's both, in the sense that.

• 8:00Speaker 01

  This is what I hope and expect will bear fruit over the coming decades. But it's also pragmatic in the sense that I'm not sure what we're doing in either in either psychology or neuroscience, if that's not the framing. I don't, I don't, I don't know what it means to understand the brain if there's no, if, if part of the enterprise is not about understanding the behavior that's being produced.

• 8:37Lex Fridman?

  I mean, yeah, but I would, I would compare it to maybe astronomers looking at the movement of the, the planets and the stars and without any interest of the underlying physics. Right. And I would argue that at least in the early days, there is some value to just tracing the movement of the planets and the stars without thinking about the physics too much because it's such a big leap to start thinking about the physics before you even understand even the basic structural elements of.

• 9:06Speaker 01

  Oh, I agree with that. I agree.

• 9:08Lex Fridman?

  But you're saying in the end, the goal should be.

• 9:10Speaker 01

  Yeah.

• 9:11Lex Fridman?

  Deeply understand.

• 9:11Speaker 01

  Well, right. And I, I think, so I thought about this a lot when I was in grad school because a lot of what I studied in grad school was psychology. And I found myself a little bit confused about what it meant to, it seems like what, what we were talking about a lot of the time were virtual causal mechanisms. Like, oh, well, you know, attentional selection then selects some object in the environment and that is then passed on to the motor, you know, information about that is passed on to the motor system. But these are, these are virtual mechanisms. These are, you know, they're metaphors. They're, you know, that there's no, they're not, there's no reduction to, there's no

• 9:52Speaker 01

  reduction going on in that conversation to some physical mechanism that, you know, which is really what it would take to fully understand, you know, how, how behavior is arising. But the causal mechanisms are definitely neurons interacting. I'm, I'm willing to say that at this point in history. So in psychology, at least for me personally, there was this strange insecurity about trafficking in these metaphors, you know, which were supposed to explain the, the function of the mind.

• 10:24Speaker 01

  If you can't ground them in physical mechanisms, then what is the explanatory validity of these explanations? And I managed to soothe my own nerves by thinking about the history of genetics research. So I'm very far from being an expert on the history of this field, but I know enough to say that Mendelian genetics preceded Watson and Crick. And so there was a significant period of time during which people were productively

• 11:07Speaker 01

  investigating the structure of inheritance using what was essentially a metaphor, the notion of a gene. And, "Oh, genes do this and genes do that." But where are the genes? They're sort of an explanatory thing that we made up. And we ascribed to them these causal properties. Oh, there's a dominant, there's a recessive, and then they recombine it. And then later, there was a kind of blank there that was filled in with a physical mechanism. That connection was made. But it was worth having that metaphor, because that gave us a good sense of what kind of causal mechanism we were looking for.

• 11:51Lex Fridman?

  And the fundamental metaphor of cognition, you said, is the interaction of neurons. What is the metaphor?

• 11:59Speaker 01

  No, no. The metaphors we use in cognitive psychology are things like attention, the way that memory works. I retrieve something from memory. A memory retrieval occurs. What is that? That's not a physical mechanism that I can examine in its own right. But it's still worth having, that metaphorical level.

• 12:31Lex Fridman?

  Yeah, I misunderstood, actually. So the higher level abstractions is the metaphor that's most useful?

• 12:36Speaker 01

  Yes.

• 12:37Lex Fridman?

  But what about... So how does that connect to the...

• 12:45Lex Fridman?

  The idea that that arises from interaction of neurons. Well, even i- is the interaction of neurons also not a metaphor to you? Is or is it literally like that's no longer a metaphor? That's that's already that's already the lowest level of abstraction that could actually be directly studied.

• 13:07Speaker 01

  Well, I'm hesitating because I think what I want to say could end up being controversial. So what I want to say is, yes, the interaction of the interactions of neurons, that's not metaphorical. That's a physical fact. That's that's where that's where the causal interactions actually occur. Now, I suppose you could say, well, you know, even that is metaphorical relative to the quantum events that underlie. You know, I don't want to go down that rabbit hole.

• 13:34Lex Fridman?

  It's always turtles on top of turtles.

• 13:36Speaker 01

  But but but there is a there is a there's a reduction that you can do. You can say these psychological phenomena are can be explained through a very different kind of causal mechanism, which has to do with neurotransmitter release. And and so what we're really trying to do in neuroscience writ large, you know, as I say, which for me includes psychology, is to take these psychological phenomena and map them onto neural events.

• 14:07Speaker 01

  I think remaining forever at the level of

• 14:13Speaker 01

  description that is natural for psychology, for me personally, would be disappointing. I want to understand how mental activity arises from neural neural activity. But the converse is also true. Studying neural activity without any sense of what you're trying to explain

• 14:37Speaker 01

  to me feels like at best groping around, you know, at random.

• 14:44Lex Fridman?

  Now, you've kind of talked about this bridging of the gap between psychology and neuroscience. But do you think it's possible like my love is like I fell in love with psychology and psychiatry in general, with Freud and when I was really young and I hoped to understand the mind. And for me, understanding the mind, at least at that young age, before I discovered AI and even neuroscience was to. Is psychology and do you think it's possible to understand the mind without getting into all the messy details of neuroscience? Like you kind of mentioned, to you, it's appealing to try to understand the mechanisms at the lowest level, but do you think that's needed? That's required to understand how the mind works?

• 15:28Speaker 01

  That's an important part of the whole picture, but I would be the last person on earth to suggest that that reality renders psychology in its own right unproductive. I trained as a psychologist. I am fond of saying that I have learned much more from psychology than I have from neuroscience. To me, psychology is a hugely important discipline. And one thing that warms my heart is that ways of investigating behavior

• 16:11Speaker 01

  that have been native to cognitive psychology since it's dawn in the 60s are starting to become, they're starting to become interesting to AI researchers for a variety of reasons. And that's been exciting for me to see.

• 16:28Lex Fridman?

  Can you maybe talk a little bit about what you see as beautiful aspects of psychology, maybe limiting aspects of psychology? I mean, maybe just start it off as a science, as a field. To me, was when I understood what psychology is analytical psychology, like with the way it's actually carried out, it's really disappointing to see two aspects. One is how few, how small the N is, how many, how small the number of subjects is in the studies. And two was disappointing to see how controlled the entire, how much it was in the lab. It wasn't studying humans in the wild. There was no mechanism for studying

• 17:11Lex Fridman?

  humans in the wild. So that's where I became a little bit disillusioned to psychology. And then the modern world of the internet is so exciting to me, the Twitter data or YouTube data, data of human behavior on the internet becomes exciting because the N grows and then in the wild grows. But that's just my narrow sense. Do you have a optimistic or pessimistic cynical view of psychology? How do you see the field broadly?

• 17:38Speaker 01

  When I was in graduate school, it was early enough that there was still a thrill in seeing that there were ways of doing, there were ways of doing experimental science that provided insight to the structure of the mind. One thing that impressed me most when I was at that stage in my education was neuropsychology, looking at, looking at analyzing the behavior of populations who had brain damage of different kinds and trying to understand what, what the, what the specific deficits were that arose from a lesion in a particular part of the brain.

• 18:23Speaker 01

  And the, the kind of experimentation that was done and that's still being done to get answers in that context was so creative. And it was so deliberate, you know, it was good science. An experiment answered one question, but raised another. And somebody would do an experiment that answered that question. And you really felt like you were narrowing in on some kind of approximate understanding of what this part of the brain was for.

• 18:52Lex Fridman?

  Do you have an example from memory of what kind of aspects of the mind could be studied in this kind of way?

• 18:58Speaker 01

  Oh, sure. I mean, the very detailed neuropsychological studies of language,

• 19:05Speaker 01

  language function, looking at production and reception and the relationship between, you know, visual function, you know, reading and auditory and semantic. And there were these beaut, and still are these beautiful models that came out of that kind of research that really made you feel like you understood something that you hadn't understood before about how, you know, language processing is organized in the brain. But having said all that,

• 19:36Speaker 01

  You know, I, I think, I think you are, I mean, I agree with you that the cost of doing highly controlled experiments is that you, by construction, miss out on the richness and complexity of the real world. One thing that, so I, I was drawn into science by what in those days was called connectionism, which is of course the, you know, what we now call deep learning. And at that point in history, neural networks were primarily being used in order to model human cognition. They weren't yet really useful for industrial applications.

• 20:17Lex Fridman?

  So you always found neural networks in biological form beautiful?

• 20:21Speaker 01

  Oh, neural networks were very concretely the thing that drew me into science. I was handed, are you familiar with the, the PDP books from, from the eighties? Some, when I was, I went to medical school before I went into science and

• 20:36Lex Fridman?

  Really? Interesting. Wow.

• 20:38Speaker 01

  I also, I also did a graduate degree in art history. So I'm, I kind of explored.

• 20:43Lex Fridman?

  Well, art history, I understand. That's, that's just a curious, creative mind. But medical school with the dream of what? If we take that slight tangent, what, did you, did you want to be a surgeon?

• 20:56Speaker 01

  I actually was quite interested in surgery. I was, I was interested in surgery and psychiatry. And I thought that must be, I must be the only person on the planet who had, who was torn between those two fields. And I, I, I said exactly that to my advisor in medical school, who, who turned out, I found out later to be a famous psychoanalyst. And, and he said to me, no, no, it's actually not so uncommon to be interested in surgery and psychiatry. And he, he conjectured that the reason that people develop these, these two interests is that both fields are about going beneath the surface. And kind of getting into the kind of secret.

• 21:35Lex Fridman?

  Yeah.

• 21:36Speaker 01

  I mean, maybe you understand this as someone who was interested in psychoanalysis.

• 21:39Lex Fridman?

  Yeah.

• 21:39Speaker 01

  At an early stage, there's sort of a, you know, there's a cliche phrase that people use now on, you know, like NPR, the secret life of blank, it is blank, right?

• 21:48Lex Fridman?

  Yeah.

• 21:48Speaker 01

  You know, and that was part of the thrill of surgery was seeing, you know, the secret, you know, the secret activity that's inside everybody's abdomen and thorax.

• 21:57Lex Fridman?

  That's a very poetic way to connect the two disciplines that are very practically speaking, different from each other.

• 22:03Speaker 01

  That's for sure. That's for sure. Yes.

• 22:05Lex Fridman?

  So, uh, so how did we get onto medical school?

• 22:09Speaker 01

  So, so I was in medical school and I, I was doing a psychiatry rotation and my kind of advisor in that rotation, uh, asked me what I was interested in. And, um, I said, well, maybe psychiatry. He said, why? And I said, well, I've always been interested in how the brain works. I'm pretty sure that nobody's doing scientific research that addresses my interests, which are, I didn't have a word for it then, but I would have said about cognition. And he said, well, you know, I'm not sure that's true. You might, you might be interested in these books. And he pulled down the, the PDB books from his shelf and they were still shrink wrapped.

• 22:51Speaker 01

  He hadn't read them, but he handed to me, handed them to me and said, you, you feel free to borrow these. And that was, uh, you know, I went back to my dorm room and I just, you know, read them cover to cover.

• 23:00Lex Fridman?

  And what's PDP?

• 23:02Speaker 01

  Parallel distributed processing, which was the, one of the original names for deep learning.

• 23:08Lex Fridman?

  And so I apologize for the romanticized question, but what, uh, what idea in the space of neuroscience and the space of the human brain is to you the most beautiful, mysterious, surprising?

• 23:21Speaker 01

  What, what had always fascinated me, um, even when I was a pretty young kid, I think, uh, was the,

• 23:33Speaker 01

  The paradox that lies in the fact that the brain is so mysterious and so it seems so distant. But at the same time, it's responsible for the full transparency of everyday life. It's the brain is literally what makes everything obvious and familiar. And and and there's always one in the room with you.

• 24:04Lex Fridman?

  Yeah.

• 24:05Speaker 01

  I used to teach when I taught at Princeton, I used to teach a cognitive neuroscience course. And the very last thing I would say to the students was, you know, people often, when people think of scientific inspiration, the metaphor is often, well, look to the stars, you know, the stars will inspire you to wonder at the universe and, you know, think about your place in it and how things work. And I'm all for looking at the stars, but I've always been much more inspired and my sense of wonder comes from the, not from the distant mysterious stars, but from the

• 24:46Speaker 01

  extremely intimately close brain.

• 24:51Lex Fridman?

  Yeah.

• 24:52Speaker 01

  There's something just endlessly fascinating to me about that.

• 24:56Lex Fridman?

  That like Jessica said, the one is close and yet distant in terms of our understanding of it. Do you, are you also captivated by the fact that this very conversation is happening because two brains are communicating?

• 25:14Speaker 01

  Yes, exactly.

• 25:17Lex Fridman?

  The, I guess what I mean is the subjective nature of the experience, if you can take a small tangent into the mystical of it, the consciousness, or, or when you're saying you're captivated by the idea of the brain, you're, are you talking about specifically the mechanism of cognition? Or are you also just, like, at least for me, it's almost like paralyzing the beauty and the mystery of the fact that it creates the entirety of the experience, not just the reasoning capability, but the experience?

• 25:50Speaker 01

  Well.

• 25:52Speaker 01

  I definitely resonate with that that latter thought. And I I often find discussions of artificial intelligence to be disappointingly narrow. You know, speaking of someone who has always had an interest in in in art.

• 26:13Lex Fridman?

  Right, I was just gonna go there because it sounds like somebody who has an interest in art.

• 26:17Speaker 01

  Yeah, I mean, I there there there are many layers to, you know, to full bore human experience and in some ways it's not enough to say, oh, well, don't worry. You know, we we're talking about cognition, but we'll add emotion, you know?

• 26:33Lex Fridman?

  Yeah.

• 26:34Speaker 01

  There's there's there's an incredible scope to what humans go through in in every moment. And

• 26:47Speaker 01

  and yes, so it's that's part of what fascinates me is that is that our brains are producing that.

• 26:57Speaker 01

  But at the same time, it's so mysterious to us. How, you know, like, we literally, our brains are literally in our heads, producing this experience.

• 27:07Lex Fridman?

  Producing the experience.

• 27:09Speaker 01

  And yet there's there it's so mysterious to us. And so and the scientific challenge of getting at the the the actual explanation for that is so overwhelming. That's just, I don't know, that certain people have fixations on particular questions. And that's always that's just always been mine.

• 27:28Lex Fridman?

  Yeah, I would say the poetry that is fascinating and I'm I'm really interested in natural language as well. And when you look at artificial intelligence community, it always saddens me how much when you try to create a benchmark for the community together around how much of the magic of language is lost when you create that benchmark? That there's something we talk about experience, the the music of the language, the wit, the something that makes a rich experience, something that would be required to pass the spirit of the Turing test is lost in these benchmarks. And I wonder how to get it back in because it's very difficult. The moment you try to do like real good rigorous science, you lose some of that magic.

• 28:14Lex Fridman?

  When you try to Study cognition in a rigorous scientific way, it feels like you're losing some of the magic.

• 28:20Speaker 01

  Mm hmm.

• 28:21Lex Fridman?

  The seeing cognition in a mechanistic way that AI folk at this stage in our history.

• 28:27Speaker 01

  Well, I I agree with you, but at the same time, one, one thing that I found really exciting about that first wave of of deep learning models in cognition was

• 28:42Speaker 01

  there was the, the fact that the people who were building these models were focused on the richness and complexity of human cognition. So an early debate in cognitive science, which I sort of witnessed as a grad student, was about something that sounds very dry, which is the formation of the past tense. But there were these two camps. One said, well, the, the mind encodes certain rules and it also has a list of exceptions because of course, you know, the rule is add E D, but that's not always what you do. So you have to have a list of exceptions. And, and then there were the connectionists who, you know,

• 29:24Speaker 01

  evolved into the deep learning people who said, well, you know, if you look carefully at the data, if you look at actually look at corpora, like language corpora, it's, it turns out to be very rich because yes, there are, there are, there's a, you know, the, there are most verbs that, and you know, you just tack on E D and then there are exceptions, but there are also, there's also, there are, there are rules that, you know, there's, the exceptions aren't just random. They, they, there are certain clues to which, which, which verbs should be exceptional. And then there are exceptions to the exceptions. And there was a word that was kind of deployed in order to capture this, which was quasi

• 30:07Speaker 01

  regular. In other words, there are rules, but it's, it's messy and there, there's, there are structure even among the exceptions. And, and it would be, yeah, you could try to write down, we could try to write down the structure in some sort of closed form, but really the right way to understand how the brain is handling all this, and by the way, producing all of this, is to build a deep neural network and train it on this data and see how it ends up representing all of this richness. So. The way that deep learning was deployed in cognitive psychology was that was the spirit of it. It was about that richness. And that's something that I always found very, very compelling. Still do.

• 30:50Lex Fridman?

  Is there something especially interesting and profound to you in terms of our current deep learning neural network, artificial neural network approaches and the whatever we do understand about the biological neural networks in our brain? Is there, there's some, there's quite a few differences. Are some of them to you either interesting or perhaps profound in terms of, in terms of the gap we might want to try to close in trying to create a human level intelligence?

• 31:24Speaker 01

  What I would say here is something that a lot of people are saying, which is that one seeming limitation of the systems that we're building now is that they lack the kind of flexibility, the readiness to sort of turn on a dime when the, when the context calls for it. That is so characteristic of human behavior. So.

• 31:50Lex Fridman?

  Is that connected to you to the, like which aspect of the neural networks in our brain is that connected to? Is that closer to the cognitive science level of, now again, see like my natural inclination is to separate into three disciplines of neuroscience, cognitive science and psychology. And you've already kind of shut that down by saying you're kind of see them as separate, but just to look at those layers, I guess where, is there something about the lowest layer of the way the neuro, neurons interact that is profound to you in terms of its difference to the artificial neural networks? Or is all the

• 32:33Lex Fridman?

  diff, the key differences at a higher level of abstraction?

• 32:37Speaker 01

  One thing I often think about is that, you know, if you take an introductory computer science course and they are introducing you to the notion of Turing machines, one way of articulating what the significance of a Turing machine is, is that it's a machine emulator. It can emulate any other machine.

• 33:04Speaker 01

  And that, to me, you know, that, that, and that way of looking at a Turing machine, really sticks with me. I think of humans as maybe sharing in some of that character. We're capacity limited, we're not Turing machines, obviously, but we have the ability to adapt behaviors that are very much unlike anything we've done before, but there's some basic mechanism that's implemented in our brain that allows us to run, run software.

• 33:39Lex Fridman?

  But you're just on that point, you mentioned Turing machine, but nevertheless, it's fundamentally, our brains are just computational devices in your view? Is that what you're getting at? Like, is, I was a little bit unclear to this line you drew. Is, is there any magic in there or is it just basic computation?

• 33:57Speaker 01

  I'm happy to think of it as just basic computation. But mind you, I won't be satisfied until somebody explains to me how, what the basic computations are that are leading to the full richness of human cognition.

• 34:11Lex Fridman?

  Yes.

• 34:11Speaker 01

  I mean, it's not going to be enough for, for me to, you know, understand what the computations are that allow people to, you know, do arithmetic or play chess. I want, I want the whole, the whole, you know, the whole thing.

• 34:23Lex Fridman?

  And a small tangent, because you kind of mentioned coronavirus, there's group behavior.

• 34:29Speaker 01

  Oh, sure.

• 34:30Lex Fridman?

  Is that, is there something interesting to your search of understanding the human mind? Where behavior of large groups, so just behavior of groups is interesting. You know, seeing that as a collective mind, as a collective intelligence, perhaps seeing the groups of people as a single intelligent organisms, especially looking at the reinforcement learning work you've done recently.

• 34:52Speaker 01

  Well, yeah, I can't, I can't, I mean, I, I have the, I have the, the honor of working with a lot of incredibly smart people and I wouldn't want to take any credit for, for leading the way on the, the multi-agent work that's come out of, out of my group or DeepMind lately, but I do find it fascinating. And, I mean, I think they're, you know, I think it, it can't be debated, you know, that human behavior arises within communities. That just seems to me self-evident.

• 35:26Lex Fridman?

  But to me, it is self-evident, but that seems to be a profound aspects of something that created, that was like, if you look at like 2001 Space Odyssey, when the, when the monkeys touched the, like, that's the magical moment, I think Yuval Harari argues that the ability of our large numbers of humans to hold an idea, to converge towards idea together, like you said, shaking hands versus bumping elbows, somehow converge, like, without even, like, like without, you know, without being in a room all together, just kind of this like distributed convergence towards an idea over a particular period of time, seems to be fundamental to, to just every aspect of our cognition, of our intelligence.

• 36:10Lex Fridman?

  Because humans, I will talk about reward, but it seems like we don't really have a clear objective function under which we operate, but we all kind of converge towards one somehow. And that, that to me has always been a mystery, that I think is somehow productive for also understanding. AI systems, but I guess I guess that's the next step. The first step is try to understand the mind.

• 36:36Speaker 01

  Well, I don't know. I mean, I think there's something to the argument that that kind of bottom, like strictly bottom up approach is wrong headed. In other words, you know, there are, there are basic phenomena that, you know, you know, basic aspects of human intelligence that, you know, can only be understood in, in the context of groups. I, I'm perfectly open to that. I've never been particularly convinced by the notion that we should be, we should consider intelligence to adhere at the level of communities. I, I, I don't know why. I just, I'm sort of stuck on the notion that the basic unit that we want to understand is individual humans.

• 37:19Speaker 01

  And if, if we have to understand that in the context of other humans, fine. But for me, intelligence is just, I'm stubbornly, I stubbornly define it as something that is, you know, an aspect of an individual human. That's just my, I don't know.

• 37:37Lex Fridman?

  But I'm with you, but that's, that could be the reductionist dream of a scientist because you can understand a single human. It also is very possible that intelligence can only arise when there's multiple intelligences. When there's multiple sort of, it's a sad thing if that's true because it's very difficult to study. But if, if it's just one human, that one human would not be homo sapien, would not become that intelligent. That's a real, that's a possibility.

• 38:05Speaker 01

  I, I, I'm with you. One, one thing I will say along these lines is that I think,

• 38:15Speaker 01

  I think a serious effort to understand human intelligence.

• 38:22Speaker 01

  And maybe to build a human-like intelligence needs to pay just as much attention to the structure of the environment as to the structure of the, you know, the cognizing system, whether it's a brain or an AI system. That's one thing I took away, actually, from my early studies with the pioneers of neural network research, people like Jay McClelland and John Cohen. You know, the, the structure of cognition is really, it's only, only partly a function of the, the, you know, the, the architecture of the brain and the learning algorithms that it implements.

• 39:04Speaker 01

  What it's really a function, what it, what, what really shapes it is the interaction of those things with the structure of the world in which those things are embedded, right?

• 39:13Lex Fridman?

  And that's especially important for, that's made most clear in reinforcement learning, where the simulated environment is, you can only learn as much as you can simulate. And that's what made, with DeepMind, made very clear with the other aspect of the environment, which is the self-play mechanism of the other agent, of the competitive behavior, which the other agent becomes the environment, essentially.

• 39:36Speaker 01

  Yeah.

• 39:37Lex Fridman?

  And that's, I mean, one of the most exciting ideas in AI is the self-play mechanism that's able to learn successfully. So there you go. There's a, there's a thing where competition is essential for learning.

• 39:50Speaker 01

  Yeah.

• 39:50Lex Fridman?

  At least in that context. So if we can step back into another sort of beautiful world, which is the actual mechanics, the dirty mess of it, of the human brain. Is there something for people who might not know? Is there something you can comment on or describe? The key parts of the brain that are important for intelligence? Or just in general, what are the different parts of the brain that you're curious about, that you've studied and that are just good to know about when you're thinking about cognition?

• 40:23Speaker 01

  Well, my, my area of expertise, if I have one, is prefrontal cortex. So.

• 40:32Lex Fridman?

  What's that? Or do we?

• 40:35Speaker 01

  It depends on who you ask. The, the, the, the, the, the technical definition is, has, is anatomical. There are, there are, um, uh, parts of your brain that are responsible for motor behavior, and they're very easy to identify. And, um, the region of your cerebral cortex, the out, the sort of outer crust of your brain, that lies in front of those is defined as the prefrontal cortex.

• 41:06Lex Fridman?

  And when you say anatomical, sorry to interrupt. So that's referring to, uh, sort of the geographic region?

• 41:14Speaker 01

  Yeah.

• 41:14Lex Fridman?

  As opposed to some kind of functional definition?

• 41:17Speaker 01

  Exactly. So that, this is kind of the coward's way out. And I'm telling you what the prefrontal cortex is, just in terms of like, what part of the real estate it occupies.

• 41:26Lex Fridman?

  It's the thing in the front of the brain.

• 41:28Speaker 01

  Yeah, exactly. And, and in fact, um, the early history of, uh, you know, the, of neuroscientific, um, investigation of what this like, front part of the brain does is sort of funny to read because, uh, you know, it was really, it was really World War I that started people down this road of trying to figure out what different parts of the brain, the human brain do, in the sense that, uh, there were a lot of people with brain damage who came back from the war with brain damage. And it, that provided, as tragic as that was, it provided an opportunity for scientists to try to identify the functions of different brain regions. And it, that was actually incredibly productive.

• 42:13Speaker 01

  But one of the frustrations that neuropsychologists faced was, they couldn't really identify exactly what the deficit was, um, that arose from damage to this, these most, you know, kind of frontal parts of the brain. It was just a very difficult thing to, you know, to, um, you know, to pin down. There were a couple of neuropsychologists who identified, uh, through, through a large amount of clinical experience and close observation, they started to, um, put their finger on a syndrome that was associated with frontal damage. Actually, one of them was a Russian, uh, neuropsychologist named Luria. Who, you know, students of cognitive psychology still read.

• 42:53Speaker 01

  And, and what he started to figure out was that the frontal cortex was somehow involved in flexibility, the, in, in, in guiding behaviors that required someone to override a habit, or to do something unusual, or to change what they were doing in a very flexible way from one moment to another.

• 43:19Lex Fridman?

  So focused on like new experiences. And so, so the way your brain processes and acts in new experiences.

• 43:28Speaker 01

  Yeah. What later helped bring this function into better focus was a distinction between controlled and automatic behavior, or, or to, in, in, in other literatures, this is referred to as habitual behavior versus goal-directed behavior. So it's very, very clear that the human brain has pathways that are dedicated to habits, to things that you do all the time, and they need to be automatized so that they don't require you to concentrate too much, so that that leaves your cognitive capacity free to do other things. Just think about the difference between driving when you're learning to drive versus driving

• 44:13Speaker 01

  after you're fairly expert. There are brain pathways that slowly absorb those frequently performed behaviors so that they can be habits, so that they can be automatic.

• 44:29Lex Fridman?

  So that, that's kind of like the purest form of learning, I guess, is happening there, which is why, I mean, this is kind of jumping ahead, which is why that perhaps is the most useful for us to focusing on and trying to see how artificial intelligence systems can learn. Is that the way?

• 44:45Speaker 01

  It's interesting. I, I do think about this distinction between controlled and automatic, or goal-directed and habitual behavior, a lot in thinking about where we are in AI research.

• 44:58Speaker 01

  But but just to finish to finish the the kind of dissertation here, the the role of the front of the prefrontal cortex is generally understood these days, sort of in in contradistinction to that habitual domain. In other words, the prefrontal cortex is what helps you override those habits. It's what allows you to say, whoa, whoa, what I usually do in this situation is X, but given the context, I probably should do Y. I mean, the elbow bump is a great example, right? If you know, reaching out and shaking hands is probably a habitual behavior. And it's the prefrontal cortex

• 45:42Speaker 01

  that allows us to bear in mind that there's something unusual going on right now. And in this situation, I need to not do the usual thing. The kind of behaviors that Luria reported, and he built tests for, you know, detecting these kinds of things, were exactly like this. So in other words, when I stuck out my hand, I want you instead to present your elbow. A patient with frontal damage would have a great deal of trouble with that. You know, somebody proffering their hand would elicit, you know, a handshake. The prefrontal cortex is what allows us to say, hold on, hold on, that's the usual thing. But I'm I have the ability to bear in mind even very unusual contexts, and

• 46:26Speaker 01

  to reason about what behavior is appropriate there.

• 46:29Lex Fridman?

  Just to get a sense, is are us humans special in the presence of the prefrontal cortex? Do mice have a prefrontal cortex? Do other mammals that we can study? If if no, then how do they integrate new experiences?

• 46:46Speaker 01

  Yeah, that's a that's a really tricky question and a very timely question, because we have

• 46:57Speaker 01

  Revolutionary new technologies for monitoring, measuring, and also causally influencing neural behavior in mice and fruit flies. And these techniques are not fully available even for studying brain function in monkeys, let alone humans. And so it's a very, sort of, for me at least, a very urgent question, whether the kinds of things that we want to understand about human intelligence can be pursued in these other organisms.

• 47:39Speaker 01

  And, you know, to put it briefly, there's disagreement. You know, people who study fruit flies will often tell you, "Hey, fruit flies are smarter than you think." And they'll point to experiments where fruit flies were able to learn new behaviors, were able to generalize from one stimulus to another in a way that suggests that they have abstractions that guide their generalization. I've had many conversations in which I will start by observing, you know, recounting some some observation about mouse behavior,

• 48:22Speaker 01

  where it seemed like mice were taking an awfully long time to learn a task that for a human would be profoundly trivial. And I will conclude from that that mice really don't have the cognitive flexibility that we want to explain. And then a mouse researcher will say to me, "Well, you know, hold on. That experiment may not have worked because you asked a mouse to deal with stimuli and behaviors that were very unnatural for the mouse. If instead you kept the logic of the experiment the same, but put, you know, kind of put it in a, you know, presented it, the information in a way that aligns with what mice are used to dealing with in their natural habitats, you might find..."

• 49:06Speaker 01

  That a mouse actually has more intelligence than you think. And then they'll go on to show you videos of mice doing things in their natural habitat, which seem strikingly intelligent, you know, dealing with, you know, physical problems, you know, I have to drag this piece of food back to my, you know, back to my lair, but there's something in my way and how do I get rid of that thing? So I think, I think these are open questions to put it, you know, to sum that up.

• 49:32Lex Fridman?

  And then taking a small step back, so related to that is you kind of mentioned, we're taking a little shortcut by saying it's a geographic, geographic part of the, the prefrontal cortex is a region of the brain. But if we, what's your sense in a bigger philosophical view? Prefrontal cortex and the brain in general. Do you have a sense that it's a set of subsystems in the way we've kind of implied? That are, they're pretty distinct? Or to what degree is it that? Or to what degree is it a giant interconnected mess where everything kind of does everything? And it's impossible to disentangle them?

• 50:12Speaker 01

  I think there's overwhelming evidence that there's functional differentiation, that it's clearly not the case that all parts of the brain are doing the same thing. This follows immediately from the kinds of studies of brain damage that we were chatting about before. It, it's obvious from what you see if you stick an electrode in the brain and measure what's going on at the level of neural activity.

• 50:43Speaker 01

  Having said that, there are two other things to add, which kind of, I don't know, maybe tug in the other direction. One is that it's, when you look carefully at functional differentiation in the brain, what you usually end up concluding, at least this is my observation of the literature, is that the, the differences between regions are graded rather than being discrete. So it doesn't seem like it's easy to divide the brain up into true modules, where, you know, that are, you know, that have clear boundaries and that have, you know, like,

• 51:29Speaker 01

  Like clear channels of communication between them.

• 51:33Lex Fridman?

  And this applies to the prefrontal cortex?

• 51:35Speaker 01

  Yeah, oh yeah, yeah. The prefrontal cortex is made up of a bunch of different subregions. The, you know, the, the functions of which are not clearly defined and which, you know, the borders of which seem to be quite vague.

• 51:49Speaker 01

  And then, then there's another thing that's popping up in very recent research, which, you know, which

• 51:57Speaker 01

  involves application of these new techniques.

• 52:02Speaker 01

  Which, there are a number of studies that suggest that

• 52:06Speaker 01

  parts of the brain that we would have previously thought were quite

• 52:11Speaker 01

  focused in their function are actually carrying signals that we wouldn't have thought would be there. For example, looking in the primary visual cortex, which is classically thought of as basically the first cortical waystation for processing visual information. Basically, what it should care about is, you know, where are the edges in this scene that I'm viewing?

• 52:34Speaker 01

  It turns out that if you have enough data, you can recover information from primary visual cortex about all sorts of things, like, you know, what, what behavior the animal is engaged in right now? And what, what, how much reward is on offer in the, in the task that it's pursuing? So it's clear that even, even regions whose function is pretty well defined at a coarse grain are nonetheless carrying some information about information from very different domains. So, you know, the, the history of neuroscience is sort of this oscillation between the two views that you articulated, you know, the kind of modular view and then the big, you know, mush view. And, uh, you know, I think, I, I, I guess we're going to end up somewhere in the middle,

• 53:20Speaker 01

  which is, which is unfortunate for our understanding because the module, there's something about our, you know, conceptual system that finds it's easy to think about a modularized system and easy to think about a completely undifferentiated system. But something that kind of lies in between is confusing. But we're going to have to get used to it, I think.

• 53:38Lex Fridman?

  Unless we can understand deeply the lower level mechanism of neuronal communication.

• 53:43Speaker 01

  Yeah, yeah.

• 53:43Lex Fridman?

  So on that, on that topic, you kind of mentioned information. Just to get a sense, I imagine something that there's still mystery and disagreement on is how does the brain carry information and signal, like what in your sense? Is the basic mechanism of communication in the brain?

• 54:03Speaker 01

  Well, I guess I'm old fashioned in that I consider the networks that we use in deep learning research to be a reasonable approximation to, you know, the mechanisms that carry information in the brain. So the usual way of articulating that is to say, what really matters is a rate code. What matters is how quickly is an individual neuron spiking? You know, what's the frequency at which it's spiking?

• 54:33Lex Fridman?

  So the timing of the spike?

• 54:35Speaker 01

  Yeah. Is it firing fast or slow? Let's, you know, let's put a number on that. And that number is enough to capture what neurons are doing. There's, you know, there's still uncertainty about whether that's an adequate description of how information is transmitted within the brain. There, you know, there are studies that suggest that the precise timing of spikes matters. There are studies that suggest that there are computations that go on within the dendritic tree, within a neuron, that are quite rich and structured and that really don't equate to anything that we're doing in our artificial neural

• 55:18Speaker 01

  networks. Having said that, I feel like we can get, I feel like, I feel like we're getting somewhere by sticking to this high level of abstraction.

• 55:28Lex Fridman?

  Just the rate, and by the way, we're talking about the electrical signal. I remember reading some vague papers somewhere recently where the mechanical signal, like the vibrations or something, of the neurons also communicates information.

• 55:46Speaker 01

  I haven't seen that, but.

• 55:47Lex Fridman?

  Somebody was arguing that the electrical signal.

• 55:52Lex Fridman?

  This is in a nature paper, something like that, where the electrical signal is actually a side effect of the mechanical signal. But I don't think that changes the story. But it's almost the interesting idea that there could be a deeper. It's like, it's always like in physics with quantum mechanics, there's always a deeper story that could be underlying the whole thing. But you think it's basically the rate of spiking that gets us, that's like the lowest hanging fruit that can get us really far.

• 56:21Speaker 01

  This is a, this is a classical view. I mean, this is, this is, this is not, the only way in which this stance would be controversial is, you know, in the sense that there are, there are members of the neuroscience community who are interested in alternatives. But this is really a very mainstream view. The way that neurons communicate is that neurotransmitters arrive, you know, at, at a, you know, they, they wash up on a neuron. The neuron has receptors for those transmitters. The, the, the, the meeting of the transmitter with these receptors changes the voltage of the neuron. And if enough voltage change occurs, then a spike occurs, right? One of these like discrete events.

• 57:05Speaker 01

  And it's that spike that is conducted down the axon and leads to neurotransmitter release. This is just, this is just like neuroscience 101. This is like the way the brain is supposed to work. Now, what we do when we build artificial neural networks of the kind that are now popular in the AI community is that we don't worry about those individual spikes. We just worry about the frequency at which those spikes are being generated. And the, you know, we consider, you know, people talk about that as the activity of a neuron. And, you know, so the, the activity of units in a deep learning system is, you know, broadly analogous to the spike rate of a neuron.

• 57:50Speaker 01

  There, there are people who, who believe that there are other forms of communication in the brain. In fact, I've been involved in some research recently that suggests that the, the voltage, the voltage fluctuations that occur in populations of neurons that aren't, you know, that, that are sort of below the level of, of spike production may be important for, for communication. But I'm still pretty old school in the sense that I think that the things that we're building in AI research constitute reasonable models of how a brain would work.

• 58:28Lex Fridman?

  Let me ask, just for fun, a crazy question, because I can. Do you think it's possible we're completely wrong about the way this basic mechanism of neuronal communication, that the information is stored in some very different kind of way in the brain?

• 58:43Speaker 01

  Oh, heck yes. I mean, I wouldn't be a scientist if I didn't think there was any chance we were wrong. But, but, I mean, if you look, if you look at the history of deep learning research as it's been applied to neuroscience, of course, the vast majority of deep learning research these days isn't about neuroscience, but, you know, if you go back to the 1980s, there's a, you know, sort of an unbroken chain of research in which a particular strategy is taken, which is, hey, let's train a deep, a deep learning system, let's train a multi-layered neural network on this task that we trained our, you know, rat on, or our monkey on, or

• 59:27Speaker 01

  this human being on. And then let's look at what the units deep in the system are doing. And let's ask whether what they're doing resembles what we know about what neurons deep in the brain are doing. And over and over and over and over, that strategy works in the sense that the learning algorithms that we have access to, which typically center on back propagation, they give rise to, you know, patterns of activity, patterns of response, patterns of like neuronal behavior in these, in these artificial models that look hauntingly, hauntingly similar to what you see in the brain.

• 1:00:10Speaker 01

  And, you know, is that a, I mean, is that a coincidence?

• 1:00:14Lex Fridman?

  At a certain point, it starts looking like such coincidence is unlikely to not be deeply meaningful. Yeah.

• 1:00:20Speaker 01

  Yeah. That's, yeah, the circumstantial evidence is overwhelming, but it could.

• 1:00:25Lex Fridman?

  But you're always open to a total flipping of the table.

• 1:00:27Speaker 01

  Hey, of course.

• 1:00:28Lex Fridman?

  So you have co-authored several recent papers that sort of weave beautifully between the world of neuroscience and artificial intelligence. And This maybe if we could, can we just try to dance around and talk about some of them? Maybe try to pick out interesting ideas that jump to your mind from memory. So maybe looking at, we're talking about the prefrontal cortex, the 2018, I believe, paper called the prefrontal cortex is a matter of reinforcement learning system. What is there a key idea that you can speak to from that paper?

• 1:01:04Speaker 01

  Yeah, the, I mean, the key idea is about meta learning. So

• 1:01:11Lex Fridman?

  What is meta learning?

• 1:01:12Speaker 01

  Meta learning is by definition

• 1:01:18Speaker 01

  a situation in which you have a learning algorithm. And the learning algorithm operates in such a way that it gives rise to another learning algorithm. In the, in the earliest applications of this idea, you had one learning algorithm sort of adjusting the parameters on another learning algorithm. But the case that we're interested in this paper is one where you start with just one learning algorithm and then another learning algorithm kind of emerges out of, out of thin air. I can say more about what I mean by that. I don't mean to be, you know, scurrantous. But that's the idea of meta learning.

• 1:02:00Speaker 01

  You, it, it relates to the old idea in psychology of learning to learn.

• 1:02:06Speaker 01

  Situations where you, you, you have experiences that make you better at learning something new. Like a familiar example would be learning a foreign language. The first time you learn a foreign language, it may be, you know, quite laborious and disorienting and, and novel. But if you, let's say you've learned two, two foreign languages. The third foreign language obviously is going to be much easier to pick up. And why? Because you've learned how to learn. You know how this goes. You know, okay, I'm going to have to learn how to conjugate. I'm going to have to.

• 1:02:41Speaker 01

  That's a, that's a simple form of meta learning, right? In the sense that there's some slow learning mechanism that's giving, that's helping you kind of update your fast learning mechanism. Does that, does that?

• 1:02:52Lex Fridman?

  That makes so much sense. Yeah. So how, from, from our understanding, from the psychology world, from, in neuroscience, our, our understanding how meta learning works.

• 1:03:02Lex Fridman?

  Might work in the human brain. What lessons can we draw from that that we can bring into the artificial intelligence world?

• 1:03:10Speaker 01

  Well, yeah, so we, the origin of that paper was in AI work that we were doing in my group. We were, we were looking at what happens when you train a recurrent neural network using standard reinforcement learning algorithms. But you train that network not just in one task, but you train it in a bunch of interrelated tasks. And then you ask what happens when you give it yet another task in that sort of line of interrelated tasks. And, and what we started to realize is that

• 1:03:46Speaker 01

  a form of meta learning spontaneously happens in recurrent neural networks. And, and the simplest way to explain it is to say,

• 1:03:56Speaker 01

  a recurrent, a recurrent neural network has a kind of memory in its activation patterns. It's recurrent by definition in the sense that you have units that connect to other units that connect to other units. So you have sort of loops of connectivity, which allows activity to stick around and be updated over time. In psychology, we call, in neuroscience, we call this working memory. It's like actively holding something in mind. And, and, and so that memory gives the recurrent neural network a dynamics, right? The way that the activity pattern evolves over time is inherent to the connectivity of the recurrent neural network. Okay, so that's, that's idea number one.

• 1:04:40Speaker 01

  Now, the dynamics of that network are shaped by the connectivity, by the synaptic weights. And those synaptic weights are being shaped by this reinforcement learning algorithm that you're, you know, training the network with. So the punchline is if you train a recurrent neural network with a reinforcement learning algorithm that's adjusting its weights and you do that for long enough, the activation dynamics will become very interesting. Right, so imagine, imagine I give you a task where you have to press one button or another, left button or right button. And sometime, and there's some probability that I'm going to give you an M and M. If you press the left button and there's some probability I'll give you an M and M if you press the other button.

• 1:05:24Speaker 01

  And you have to figure out what those probabilities are just by trying things out.

• 1:05:29Speaker 01

  But as I said before, instead of just giving you one of these tasks, I give you a whole sequence. You know, I give you two buttons and you figure out which one's best and I go, good job, here's, here's a new box, two new buttons, you have to figure out which one's best. Good job, here's a new box and every box has its own probabilities and you have to figure it. So if you train a neural, a recurrent neural network on that kind of sequence of tasks, the, what, what happens, it seemed almost magical to, to us when we first started kind of realizing what was going on. The slow learning algorithm adjusting the, the synaptic weights, tho- those slow synaptic changes give rise to a network dynamics that themselves, that, you know, the dynamics themselves turn into a learning algorithm.

• 1:06:14Speaker 01

  So in other words, you can, you can tell this is happening by just freezing the synaptic weights, saying, okay, no more learning, you're done. Here's a new box. Figure out which button is best. And the recurrent neural network will do this just fine. There's no, like, it, it, it figures out which, which button is best. It tran- it kind of transitions from exploring the two buttons to just pressing the one that it likes best in a very rational way. How is that happening? It's happening because the activity of the, the activity dynamics of the network have been shaped by this slow learning process that's occurred over many, many boxes. And so what's happened is that this slow learning algorithm that's slowly adjusting the weights is changing the dynamics of the network, the activity dynamics into its

• 1:06:58Speaker 01

  own learning algorithm. And as we were, as we were kind of realizing that this is a thing.

• 1:07:08Speaker 01

  It just so happened that the group that was working on this included a bunch of neuroscientists. And it started kind of ringing a bell for us, which is to say that we thought this sounds a lot like the distinction between synaptic learning and activity, synaptic memory and activity-based memory in the brain. And it also reminded us of recurrent connectivity that's very characteristic of prefrontal function. So this is kind of why it's good to have people working on AI that know a little bit about neuroscience and vice versa, because we started thinking about whether we could apply this principle to neuroscience. And that's where the paper came from.

• 1:07:50Lex Fridman?

  So the kind of principle of the recurrence they can see in the prefrontal cortex, then you start to realize that it's possible to, for something like an idea of a learning to learn, emerging from this learning process, as long as you keep varying the environment sufficiently.

• 1:08:12Speaker 01

  Exactly. So the kind of metaphorical transition we made to neuroscience was to think, okay, well, we know that the prefrontal cortex is highly recurrent. We know that it's an important locus for working memory, for activation-based memory. So maybe the prefrontal cortex supports reinforcement learning. In other words, you, what is reinforcement learning? You take an action, you see how much reward you got, you update your policy of behavior. Maybe the prefrontal cortex is doing that sort of thing strictly in its activation patterns. It's keeping around a memory in its activity patterns of what you did, how much reward

• 1:08:51Speaker 01

  you got, and it's using that activity-based memory as a basis for updating behavior. But then the question is, well, how did the prefrontal cortex get so smart? In other words, how did it, where did these activity dynamics come from? How did that program that's implemented in the recurrent dynamics of the prefrontal cortex arise? And one answer that became evident in this work was, well, maybe the mechanisms that operate on the synaptic level, which we believe are mediated by dopamine, are responsible for shaping those dynamics.

• 1:09:27Lex Fridman?

  So this may be a silly question, but because this kind of several temporal sort of classes of learning are happening and the learning to learn is emerges. Can it just, can you keep building stacks of learning to learn to learn, learning to learn to learn to learn to learn? Because it keeps, I mean, basically abstractions of more powerful abilities to generalize of learning complex rules.

• 1:09:58Speaker 01

  Yeah.

• 1:09:58Lex Fridman?

  Or is this, that's overstretching the, this kind of mechanism?

• 1:10:03Speaker 01

  Well, one of the, one of the people in AI who started thinking about meta learning from very early on, Jürgen and Schmidt Huber, sort of cheekily suggested, I think it is, it may have been in his, his PhD thesis, that we should think about meta, meta, meta, meta, meta, meta learning, you know, that, that's really, that's really what's going to get us to true intelligence.

• 1:10:30Lex Fridman?

  Certainly, there's a poetic aspect to it, and it seems interesting and correct that that kind of levels abstraction would be powerful. But is that something you see in the brain? This kind of, is it useful to think of learning in these meta, meta, meta way? Or is it just meta learning?

• 1:10:49Speaker 01

  Well, one thing that really fascinated me about this mechanism that we were starting to look at, and you know, other groups started talking about very similar things at the same time. And then a kind of explosion of interest in meta learning happened in the AI community shortly after that. I don't know if we had anything to do with that, but, but I was gratified to see that a lot of people started talking about meta learning.

• 1:11:15Speaker 01

  One of the things that I liked about the kind of flavor of meta learning that we were studying was that it didn't require anything special. It was just if you took a system that had some form of memory.

• 1:11:28Speaker 01

  That the function of which could be shaped by pick your RL algorithm, then this would just happen.

• 1:11:35Lex Fridman?

  Yes.

• 1:11:36Speaker 01

  Right? I mean, there, there, there are a lot of forms of, there are a lot of meta learning algorithms that have been proposed since then that are fascinating and effective in, in their, you know, in their domains of application. But they're, you know, they're engineered, there, there are things that somebody had to say, well, gee, if we wanted meta learning to happen, how would we do that? Here's an algorithm that would, but there's something about the kind of meta learning that we were studying that seemed to me special in the sense that it wasn't an algorithm. It was just something that automatically happened if you had a system that had memory and it was trained with a reinforcement learning algorithm. And, and in that sense, it can be as meta as it wants to be, right?

• 1:12:16Speaker 01

  There's no limit on how abstract the, the meta learning can get because it's not reliant on a human engineering, a particular meta learning algorithm to get there. And, and that's, I, I also, I don't know, I guess I hope that that's relevant in the brain. I think there's a kind of beauty in the, in, in the ability of this emergent.

• 1:12:40Lex Fridman?

  The emergent aspect of it.

• 1:12:41Speaker 01

  Yeah, it's something.

• 1:12:42Lex Fridman?

  As opposed to engineered.

• 1:12:43Speaker 01

  Exactly. It's something that just, it just happens in, in a sense, in a sense, you can't avoid this happening. If you have a system that has memory and the function of that memory is shaped by reinforcement learning and this system is trained in a series of interrelated tasks, this is going to happen. You can't stop it.

• 1:13:05Lex Fridman?

  As long as you have certain properties, maybe like a recurrent structure to.

• 1:13:09Speaker 01

  You have to have memory. It actually doesn't have to be a recurrent neural network. One of, a paper that I was honored to be involved with even earlier, used a kind of slot based memory.

• 1:13:19Lex Fridman?

  Do you remember the title? Just.

• 1:13:22Speaker 01

  It was memory augmented neural networks. I think it, I think the title was meta learning in memory augmented neural networks. And, and, you know, it was the same exact story, you know? If you have a system with memory, here it was a different kind of memory, but the function of that memory is. Shaped by reinforcement learning. Here it was the reads and writes that occurred on this slot-based memory. This will just happen. And so, but this brings us back to something I was saying earlier about the importance of the environment.

• 1:14:03Speaker 01

  This will happen if the system is being trained in a setting where there's like a sequence of tasks that all share some abstract structure. Sometimes we talk about task distributions. And that's something that's very obviously true of the world that humans inhabit. We're constant, like if you just kind of think about what you do every day, you never do exactly the same thing that you did the day before. But everything that you do is sort of, has a family resemblance. It shares a structure with something that you did before.

• 1:14:40Speaker 01

  And so, you know, the real world is sort of, you know, saturated with this kind of, this property. It's endless variety with endless redundancy. And that's the setting in which this kind of meta-learning happens.

• 1:14:57Lex Fridman?

  It does seem like we're just so good at finding, just like in this emergent phenomenon you described, we're really good at finding that redundancy, finding those similarities, the family resemblance. Some people call it sort of, what is it, Melanie Mitchell was talking about analogies. So we're able to connect concepts together in this kind of way, in this same kind of automated emergent way, which there's so many echoes here of psychology and neuroscience and obviously now with reinforcement learning with recurrent neural networks at the core.

• 1:15:35Lex Fridman?

  If we could talk a little bit about dopamine, you have really, you're a part of co-authoring really exciting recent paper, very recent in terms of release on dopamine and temporal difference learning. Can you describe the key ideas of that paper?

• 1:15:52Speaker 01

  Sure. Yeah. I mean, one thing I want to pause to do is acknowledge my co-authors on actually both of the papers we're talking about. So this dopamine

• 1:15:59Lex Fridman?

  I'll just, I'll certainly post all their names.

• 1:16:02Speaker 01

  Okay, wonderful. Yeah. Cause I, you know, I'm, I'm sort of abashed to be the spokesperson for these papers when I had such amazing collaborators on both. So it's, it's a comfort to me to know that you all, you all acknowledge them.

• 1:16:15Lex Fridman?

  Yeah, this is an incredible team there, but yeah.

• 1:16:17Speaker 01

  Oh, yeah. It's such a, it's so much fun. And, and in the case of the, the dopamine paper, we also collaborated with Nowichi at Harvard, who, you know, obviously a paper simply wouldn't have happened without him. But, um, so, so you were asking for like a thumbnail sketch of

• 1:16:34Lex Fridman?

  Yeah, thumbnail sketch or key ideas or, you know, things, the insights that, you know, continue on our kind of discussion here between neuroscience and AI.

• 1:16:44Speaker 01

  Yeah. I mean, this was another, a lot of the work that we've done so far is taking ideas that have bubbled up in AI and, you know, asking the question of whether the brain might be doing something related, which I think on the surface, sounds like something that's really mainly of use to neuroscience. We see it also as a way of validating what we're doing on the AI side. If we can gain some evidence that the brain is using some technique that we've been trying out in our AI work, that gives us confidence that, you know, it may be a good idea, that it'll, you know, scale to rich, complex tasks,

• 1:17:28Speaker 01

  that it'll interface well with other mechanisms. So

• 1:17:32Lex Fridman?

  You see as a two-way road.

• 1:17:34Speaker 01

  Yeah, for sure.

• 1:17:34Lex Fridman?

  Just because a particular paper is a little bit focused on from one to the, from AI, from neural networks to neuroscience. Ultimately, the discussion, the thinking, the productive, long-term aspect of it is the, the two-way road nature of the whole

• 1:17:50Speaker 01

  Yeah. I mean, we, we've talked about the notion of a virtuous circle between AI and neuroscience. And, you know, the way I see it, That's always been there since the two fields jointly existed. There have been some phases in that history when AI was sort of ahead. There are some phases when neuroscience was sort of ahead. I feel like, given the burst of innovation that's happened recently on the AI side, AI is kind of ahead in the sense that there are all of these ideas that we, you know, for which it's exciting to consider that there might be neural analogues.

• 1:18:33Speaker 01

  And neuroscience, in a sense, has been focusing on approaches to studying behavior that come from, you know, that are kind of derived from this earlier era of cognitive psychology. And, you know, so in some ways fail to connect with some of the issues that we're grappling with in AI, like how do we deal with, you know, large, you know, complex environments? But, you know, I think it's inevitable that this circle will keep turning and there will be a moment in the not-too-distant future when neuroscience is pelting AI researchers with insights that may change the direction of our work.

• 1:19:15Lex Fridman?

  Just a quick human question. Is it, you have parts of your brain, this is very meta, but they're able to both think about neuroscience and AI. You know, I don't often meet people like that. So do you think, let me ask a meta-plasticity question. Do you think a human being can be both good at AI and neuroscience? Like, what, on the team at DeepMind, what kind of human can occupy these two realms? And is that something you see everybody should be doing, can be doing? Or is that a very special few can kind of jump? Just like we talk about art history, I would think it's a special person that can major

• 1:20:00Lex Fridman?

  in art history and also consider being a surgeon.

• 1:20:03Speaker 01

  Otherwise known as a dilettante?

• 1:20:05Lex Fridman?

  A dilettante, yeah. Easily distracted. No.

• 1:20:13Speaker 01

  I think it does take a special kind of person to be truly world-class at both AI and neuroscience, and I am not on that list. I happen to be someone who whose interest in neuroscience and psychology involved using the kinds of modeling techniques that are now very central in AI. And that sort of, I guess, bought me a ticket to be involved in all of the amazing things that are going on in AI research right now. I do know a few people who I would consider pretty expert on both fronts, and I won't embarrass them by naming them, but there are exceptional people out there who are

• 1:20:58Speaker 01

  like this. The one thing that I find is a barrier to being truly world-class on both fronts is just the complexity of the technology that's involved in both disciplines now. So the engineering expertise that it takes to do truly frontline, hands-on AI research is really, really considerable.

• 1:21:27Lex Fridman?

  The learning curve of the tools, just like the specifics of just whether it's programming or the kind of tools necessary to collect the data, to manage the data, to distribute, to compute, all that kind of stuff. And on the neuroscience, I guess, side, there'll be all different sets of tools.

• 1:21:41Speaker 01

  Exactly, especially with the recent explosion in neuroscience methods. But having said all that, I think the best scenario for both neuroscience and AI is to have people interacting, who live at every point on this spectrum from exclusively focused on neuroscience to exclusively focused on the engineering side of AI. But to have those people

• 1:22:17Speaker 01

  You know, inhabiting a community where they're talking to people who live elsewhere on the on the spectrum. And I may be someone who's very close to the center in the sense that I have one foot in the neuroscience world and one foot in the AI world. And that central position, I will admit, prevents me, at least someone with my limited cognitive capacity, from being a truly, you know, having true technical expertise in either domain. But at the same time, I at least hope that it's worthwhile having people around who can kind of, you know, see the connections.

• 1:22:53Lex Fridman?

  Yeah, the community, the, yeah, the emergent intelligence of the community.

• 1:22:57Speaker 01

  Yeah, yeah.

• 1:22:58Lex Fridman?

  When it's nicely distributed is useful. Okay.

• 1:23:01Speaker 01

  Exactly. Yeah. So hopefully, I mean, I've seen that work. I've seen that work out well at DeepMind. There are, there are people who, I mean, even if you just focus on the AI work that happens at DeepMind, it's been a good thing to have some people around doing that kind of work, whose PhDs are in neuroscience or psychology. Every academic discipline has its kind of blind spots and kind of unfortunate obsessions and its metaphors and its reference points. And having some intellectual diversity is really healthy.

• 1:23:41Speaker 01

  People get each other unstuck, I think. I see it all the time at DeepMind. And, you know, I like to think that the people who bring some neuroscience background to the table are helping with that.

• 1:23:54Lex Fridman?

  So one of the, one of my, like, probably the deepest passion for me, what I would say, maybe we kind of spoke off Mike a little bit about it, but that I think is a blind spot for at least robotics and AI folks is human robot interaction, human agent interaction. Maybe.

• 1:24:14Lex Fridman?

  Do you have thoughts about how we reduce the size of that blind spot? Do you also share the feeling that not enough folks are studying this aspect of interaction?

• 1:24:27Speaker 01

  Well, I'm actually pretty intensively interested in this issue now. And there are people in my group who've actually pivoted pretty hard over the last few years from doing more traditional cognitive psychology and cognitive neuroscience to doing experimental work on human agent interaction. And there are a couple of reasons that I'm pretty passionately interested in this. One is it's kind of the outcome of having thought for a few years now about what we're up to. Like, what are we doing? Like, what is this AI research for?

• 1:25:10Speaker 01

  So what does it mean to make the world a better place? I think, I'm pretty sure that means making life better for humans.

• 1:25:17Lex Fridman?

  Yeah.

• 1:25:19Speaker 01

  And so how do you make life better for humans? That's a proposition that, when you look at it carefully and honestly, is rather horrendously complicated, especially when the AI systems that you're building are learning systems. They're not, you're not, you know, programming something that you then introduce to the world and it just works as programmed, like Google Maps or something. We're building systems that learn from experience. So that typically leads to AI safety questions. How do we keep these things from getting out of control?

• 1:26:02Speaker 01

  How do we keep them from doing things that harm humans? And I mean, I hasten to say, I consider those hugely important issues. And there are large sectors of the research community at DeepMind, and of course elsewhere, who are dedicated to thinking hard all day, every day about that. But there's, I guess I would say, a positive side to this too, which is to say, well, What would it mean to make human life better? And how can we imagine learning systems doing that? And in talking to my colleagues about that, we reached the initial conclusion that it's not sufficient to philosophize about that.

• 1:26:47Speaker 01

  You actually have to take into account how humans actually work and what humans want and the difficulties of knowing what humans want. And the difficulties that arise when humans want different things. And so, human agent interaction has become, you know, a quite intensive focus of my group lately. If for no other reason that, in order to really address that issue in an adequate way, you have to, I mean, psychology becomes part of the picture.

• 1:27:24Lex Fridman?

  Yep. And so, there's a few elements there. So if you focus on solving, like, if you focus on the robotics problem, let's say AGI, without humans in the picture, you're missing fundamentally the final step. When you do want to help human civilization, you eventually have to interact with humans. And when you create a learning system, just as you said, that will eventually have to interact with humans, the interaction itself has to become part of the learning process.

• 1:27:57Speaker 01

  Right.

• 1:27:58Lex Fridman?

  So you can't just watch, well, my sense is, it sounds like your sense is, you can't just watch humans to learn about humans.

• 1:28:05Speaker 01

  Yeah.

• 1:28:06Lex Fridman?

  You have to also be part of the human world. You have to interact with humans.

• 1:28:08Speaker 01

  Yeah, exactly. And, I mean, then questions arise that start imperceptibly, but inevitably to slip beyond the realm of engineering. So, questions like, if you have an agent that can do something that you can't do...

• 1:28:28Speaker 01

  Under what conditions do you want that agent to do it? So, you know, if, you know, if I, if I have a, if I have a robot that can play Beethoven sonatas better than any human in the sense that the, you know, the, the sensitivity, the, the expression, the expression is just beyond what any human, do I, do I want to listen to that? Do I want to go to a concert and hear a robot play? These are, these are, these aren't engineering questions. These are questions about human preference and human culture. Psychology bordering on philosophy. Yeah. And then, and then you start asking, well, well, even if we knew the answer to that,

• 1:29:11Speaker 01

  is it our place as AI engineers to build that into these agents? Probably the agents should interact with humans beyond the population of AI engineers and figure out what those humans want.

• 1:29:26Speaker 01

  And then, you know, when you start, I referred this a moment ago, but even that becomes complicated because what if, what if two, what if two humans want different things? And, and you have only one agent that's able to interact with them and try to satisfy their preferences, then you're into the realm of, of, of like economics and, and social choice theory and, and even politics. So there's a sense in which if you, if you kind of follow what we're doing to its logical conclusion, then it goes beyond questions of engineering and technology and, you know, starts to shade imperceptibly into questions about what kind of society do you want?

• 1:30:08Speaker 01

  And actually that, once, once that dawned on me, I actually felt, I don't know what the right word is, quite refreshed in my, in my involvement in AI research. It's almost like this, building this kind of stuff is going to lead us back to asking really fundamental questions about what's, you know, what is this, like what, what's the good life and, and who gets to decide? And, and, you know, you know, bringing in viewpoints from multiple sub communities to help us, you know, shape the way that we live. This, it's, it's, there's something, it, it started making me feel like doing AI research in, in a fully responsible way.

• 1:30:54Speaker 01

  Would, you know, could potentially lead to a kind of like cultural renewal.

• 1:31:00Lex Fridman?

  Yeah, it's, it's the way to, it's the, it's the way to understand human beings at the individual, the societal level. It may become a way to answer all the silly human questions of the meaning of life and all the, all those kinds of things.

• 1:31:13Speaker 01

  Even if it doesn't, even if it doesn't give us a way of answering those questions, it may force us back to thinking about them.

• 1:31:20Lex Fridman?

  Thinking about.

• 1:31:20Speaker 01

  You know, and it might bring, it might bring, it might restore a certain, I don't know, a certain depth to, or even dare I say spirituality to the way that, you know, to, to, to the world. I don't know. Maybe that's too grandiose, but.

• 1:31:36Lex Fridman?

  Well, I don't, I, I'm with you. I think it's a, it's AI will be the, the philosophy of the 21st century, the, the way which will open the door. I think a lot of AI researchers are afraid to open that door of exploring the beautiful richness of the human agent interaction, human AI interaction. I'm really happy that somebody like you have opened that door and.

• 1:32:01Speaker 01

  One thing, one thing I often think about is, you know, the, the usual, the usual schema for thinking about, um, uh, human, human agent interaction is this kind of dystopian, um, you know, oh, you know, we're, our robot overlords. And, and again, I hasten to say AI safety is hugely important. And I, you know, I'm not saying we shouldn't be thinking about those risks. Totally on board for that. But there's, what, having said that, there's, there's a, I, what often follows for me is the thought that, you know, there's another, there's another kind of narrative that might be relevant, which is when we think of, when

• 1:32:43Speaker 01

  we think of, um, humans gaining more and more information about, you know, like human life. The narrative there is usually that they've gained more and more wisdom and more, you know, they get closer to enlightenment and, you know, and they become more benevolent. And, you know, like the Buddha is like the, like, that's the, that's a totally different narrative. And why isn't it the case that we imagine that the AI systems that we're creating are just gonna like, they're gonna figure out more and more about the way the world works and the way that humans interact and they'll, they'll become beneficent. I'm not saying that will happen. I'm not, I'm not, you know, I, I'm, I don't honestly expect that to happen without some careful, you're setting things up very carefully, but it's another way things could go, right?

• 1:33:28Lex Fridman?

  Yeah, and I would even push back on that. I personally believe that the, the most trajectories, natural human trajectories will lead us towards progress. So for me, there is a kind of sense that most trajectories in AI development will lead us into trouble. To me, and we over-focus on the worst case, it's like in computer science, theoretical computer science, there's been this focus on worst-case analysis. There's something appealing to our human mind at some lowest level to be, I mean, we don't want to be eaten by the tiger, I guess. So we want to do the worst-case analysis, but the reality is that shouldn't stop us

• 1:34:12Lex Fridman?

  from actually building out all the other trajectories which are potentially leading to all the positive worlds, all the, all the enlightenment. There's a book, Enlightenment Now, with Stephen Pinker and so on, this is looking generally at human progress. And there's so many ways that human progress can happen with AIs. And I think you have to do that research. You have to do that work. You have to do the, not just the AI safety work of the one worst-case analysis, how do we prevent that? But the, the actual tools and the glue and the mechanisms of human.

• 1:34:47Lex Fridman?

  AI interaction that would lead to all the positive directions and go.

• 1:34:51Speaker 01

  Yeah, right.

• 1:34:52Lex Fridman?

  Super exciting area, right?

• 1:34:53Speaker 01

  Yeah, you know, we should be spending, we should be spending a lot of our time saying, what can go wrong? I think it's harder to see that there's work to be done to bring into focus the question of what, what it would look like for things to go right.

• 1:35:10Lex Fridman?

  Yeah.

• 1:35:10Speaker 01

  It's, you know, that's not obvious. There, there's, and we wouldn't be doing this if we didn't have the sense there was huge potential, right? We're not doing this, you know, you know, for no reason. We, we, we have a sense that AGI would be a major boom to humanity. But I think, I think it's worth starting now, even when our technology is quite primitive, asking, well, exactly what would that mean? We can start now with applications that are already going to make the world a better place, like, you know, solving protein folding, you know? I, I, I think this, Deep, DeepMind has gotten heavy into science applications lately, which I think is, you know, you know, a wonderful, wonderful move for us to, to be making.

• 1:35:53Speaker 01

  But when we think about AGI, when we think about building, you know, fully intelligent agents that are going to be able to, in a sense, do whatever they want, uh, you know, we should start thinking about what do we want them to want, right? What, what, what kind of world do we want to live in? That's not an easy question, uh, and I think we just need to start working on it.

• 1:36:14Lex Fridman?

  And even on the path to sort of AGI, it doesn't have to be AGI, but just intelligent agents that interact with us and help us enrich our own existence on social networks, for example, on recommender systems and various intelligence. There's so much interesting interaction that's yet to be understood and studied and.

• 1:36:30Speaker 01

  Yeah.

• 1:36:30Lex Fridman?

  How, how do you create, uh, I mean, Twitter is, uh, is struggling with this very idea, how do you create AI systems that increase the quality and the health of a conversation?

• 1:36:41Speaker 01

  For sure.

• 1:36:42Lex Fridman?

  That's a beautiful, beautiful human psychology question.

• 1:36:45Speaker 01

  And how do you do that without, um, without deception being involved, without manipulation being involved, uh, you know, maximizing human autonomy? And how do you, how do you make these choices in a democratic way? How do you, how, how do we? How do we face the, how do we, again, I'm speaking for myself here. How do we face the fact that it's a small group of people who have the skillset to build these kinds of systems? But the, you know, what it means to make the world a better place is something that we all have to be talking about.

• 1:37:28Lex Fridman?

  Yeah, the world that we're trying to make a better place includes a huge variety of different kinds of people.

• 1:37:37Speaker 01

  Yeah, how do we cope with that? This is, this is a problem that has been discussed, you know, in, in gory extensive detail in social choice theory. You know, there, one thing I'm really enjoying about the recent direction work has taken in some parts of my team is that, yeah, we're reading the AI literature, we're reading the neuroscience literature, but we've also started reading like economics and, as I mentioned, social choice theory, even some political theory, because it turns out that it, you know, it all becomes relevant. It all becomes relevant and, but, you know, at the same time, we've been trying not to write philosophy papers, right? We've been trying not to write position papers.

• 1:38:19Speaker 01

  We're trying to figure out ways of doing actual empirical research that kind of take the first small steps to thinking about what it really means for humans with all of their complexity and contradiction and, you know, paradox, you know, to be brought, to be brought into contact with these AI systems in a way that, that really makes the world a better place.

• 1:38:42Lex Fridman?

  And often reinforcement learning frameworks actually kind of allow you to, to do that, machine learning. And so that's the exciting thing about AI is it allows you to reduce the unsolvable problem, philosophical problem to something more concrete that you can get a hold of.

• 1:38:58Speaker 01

  Yeah, and it allows you to kind of define the problem in some way that allows for

• 1:39:06Speaker 01

  Growth in the system that's sort of, you know, you're not responsible for the details, right? You say, this is generally what I want you to do, and then learning takes care of the rest. Of course, the safety issues are, you know, arise in that context. But I think also some of these positive issues arise in that context. What would it mean for an AI system to really come to understand what humans want? And, you know, in, with all of the subtleties of that, right? You know, humans, humans want help with certain things, but they don't want everything done for them, right? There is part of, part of the satisfaction that humans get from life is in accomplishing things. So if there were devices around that did

• 1:39:51Speaker 01

  everything for, you know, I often think of the movie Wally, right? That's like dystopian in a totally different way. It's like, the machines are doing everything for us. That's, that's not what we wanted. You know, anyway, I just, I find this, you know, this kind of, this opens up a whole landscape of, of research that feels affirmative and exciting.

• 1:40:10Lex Fridman?

  To me, it's one of the most exciting and it's wide open.

• 1:40:12Speaker 01

  Yeah.

• 1:40:13Lex Fridman?

  We have to, because it's a cool paper. Talk about dopamine.

• 1:40:16Speaker 01

  Oh, yeah. Okay. So I can, we were gonna, we were gonna, I was gonna give you a quick summary?

• 1:40:22Lex Fridman?

  Yeah, a quick summary of, what's the title of the paper?

• 1:40:27Speaker 01

  I, I think we called it a distributional, a distributional code for value in dopamine.

• 1:40:34Lex Fridman?

  Based reinforcement learning. Yes.

• 1:40:36Speaker 01

  So that's another project that grew out of pure AI research. A number of people at DeepMind and a few other places had started working on a new version of reinforcement learning, which was defined by taking something in traditional reinforcement learning and just tweaking it. So the thing that they took from traditional reinforcement learning was a value signal. So the, at the, at the center of reinforcement learning, at least most algorithms, is some representation of how well things are going. Your expected cumulative future reward. And that's usually represented as a single number.

• 1:41:19Speaker 01

  So if you imagine a gambler in a casino and the gambler is thinking, well, I have this probability of winning such and such an amount of money and I have this probability of losing such and such an amount of money, that situation would be represented as a single number, which is like the expected, the weighted average of all those outcomes. And this new form of reinforcement learning said, well, what if we, what if we generalize that to a distributional representation? So now we think of the gambler as literally thinking, well, there's this probability that I'll win this amount of money and there's this probability that I'll lose that amount of money and we don't reduce that to a single number. And it had been observed through experiments, through, you know, just trying

• 1:42:00Speaker 01

  this out, that that, that kind of distributional representation really accelerated reinforcement learning and led to better policies.

• 1:42:11Lex Fridman?

  What's your intuition about, so we're talking about rewards?

• 1:42:14Speaker 01

  Yeah.

• 1:42:14Lex Fridman?

  So what's your intuition why that is? Why, why does it?

• 1:42:17Speaker 01

  Well, it's, it's kind of a surprising historical note, at least surprised me when I learned it, that this had been tried out in a kind of heuristic way. People thought, well, gee, what would happen if we tried? And then it had this empirically, it had this striking effect. And it was only then that people started thinking, well, gee, why, wait, why, why, wait, why, why is this working? And, and that's led to a series of studies just trying to figure out why it works, which is ongoing. But one thing that's already clear from that research is that one reason that it helps is that it drives richer representation learning. So if you imagine, imagine two situations that have the same expected value,

• 1:43:02Speaker 01

  that the same kind of weighted average value, standard deep reinforcement learning algorithms are going to take those two situations and kind of, in terms of the way they're represented internally, they're going to like squeeze them together. Because the, the thing that you're trying to

• 1:43:19Speaker 01

  Represent, which is their expected value is the same. So all the way through the system, things are going to be mushed together. But what if, what if, what if those two situations actually have different value distributions? They have the same average value, but they have different distributions of value. In that situation, distributional learning will, will maintain the distinction between these two things. So to make a long story short, distributional learning can keep things separate in the internal representation that might otherwise be conflated or squished together. And maintaining those distinctions can be useful in, in when the system is now faced with some other task where the distinction is important.

• 1:44:00Lex Fridman?

  If we look at the optimistic and pessimistic dopamine neurons. So first of all, what is dopamine?

• 1:44:08Speaker 01

  Oh, God.

• 1:44:08Lex Fridman?

  Why is this, why is this at all useful to, to, to, to think about in the artificial intelligence sense? But what do we know about dopamine in the human brain? What is, what is it? Why is it useful? Why is it interesting? What does it have to do with the prefrontal cortex and learning in general?

• 1:44:27Speaker 01

  Yeah. So, well, this, this, this is also some, a case where there's a huge amount of detail and debate. But one, one, one currently prevailing idea is that the function of this neurotransmitter dopamine resembles a particular component of standard reinforcement learning algorithms, which is called the reward prediction error. So I was talking a moment ago about these value representations. How do you learn them? How do you update them based on experience? Well, if you, if you made some prediction about a future reward, and then you get more

• 1:45:09Speaker 01

  reward than you were expecting, then probably retrospectively, you want to go back and increase the, the, the, the value representation that you attached to the, to that earlier situation. If you got less reward than you were expecting, you should probably decrement that estimate.

• 1:45:25Lex Fridman?

  And that's the process of temporal difference.

• 1:45:27Speaker 01

  Exactly. This is the central mechanism of temporal difference learning, which is one of several kind of, you know, kind of back, the sort of the backbone of our, our mementarium in, in RL. And it was, this connection between

• 1:45:41Speaker 01

  The reward prediction error and dopamine was was made, you know, in the in the 1990s. And there's been a huge amount of research that, you know, seems to back it up. Dopamine may be doing other things, but this is clearly, at least roughly, one of the things that it's doing. But the usual idea was that dopamine was representing these reward prediction errors, again, in this like, kind of, single number way, that representing your surprise with a single number. And in distributional reinforcement learning, this this kind of new elaboration of the standard approach, it's not only the value, the value function that's represented as a single number,

• 1:46:25Speaker 01

  it's also the reward prediction error. And so what happened was that Will Dabney, one of my collaborators, who was one of the first people to work on distributional temporal difference learning, talked to a guy in my group, Will, Zeb Christ Nelson, who's a computational neuroscientist, and said, gee, you know, is it possible that dopamine might be doing something like this distributional coding thing? And they started looking at what was in the literature, and then they brought me in, and we started talking to Nowo Czita, and we came up with some specific predictions about, you know, if the brain is using this kind of distributional coding, then in the tasks that now has studied, you should see this, this, this, and this. And that's where the paper came from.

• 1:47:07Speaker 01

  We kind of enumerated a set of predictions, all of which ended up being fairly clearly confirmed, and all of which leads to at least some initial indication that the brain might be doing something like this distributional coding, that dopamine might be representing surprise signals in a way that is not just collapsing everything to a single number, but instead is kind of respecting the the variety of future outcomes, if that makes sense.

• 1:47:33Lex Fridman?

  So, yeah, so that's showing, suggesting possibly that dopamine has a really interesting representation scheme for for in the human brain, for its reward signal.

• 1:47:44Speaker 01

  Exactly.

• 1:47:45Lex Fridman?

  That's fascinating. It's just, that's another beautiful example of AI revealing something that's about neuroscience, potentially suggesting possibilities.

• 1:47:53Speaker 01

  Well, you never know. So the minute you publish a paper like that, the next thing you think is? I hope that replicates. I hope we see that same thing in other data sets. But of course, several labs now are doing the follow-up experiments. So we'll know soon. But it has been, it has been a lot of fun for us to, you know, to take these ideas from AI and kind of bring them into neuroscience and, you know, see how far we can get.

• 1:48:16Lex Fridman?

  So we kind of talked about it a little bit, but where do you see the field of neuroscience and artificial intelligence heading broadly? Like, what are the possible exciting areas that you can see breakthroughs in the next, let's get crazy, not just three or five years, but the next 10, 20, 30 years?

• 1:48:39Lex Fridman?

  That would make you excited and perhaps you'd be part of?

• 1:48:46Speaker 01

  On the neuroscience side,

• 1:48:50Speaker 01

  there's a great deal of interest now in what's going on in AI. And, um, and at the same time,

• 1:49:00Speaker 01

  I feel like, so neuroscience, especially, um,

• 1:49:06Speaker 01

  the part of neuroscience that's focused on circuits and, uh, and systems, you know, kind of like really mechanism focused.

• 1:49:15Speaker 01

  There's been this explosion in new technology. And up until recently, the, the experiments that have exploited this technology have,

• 1:49:28Speaker 01

  have not involved a lot of interesting behavior. And this is for a variety of reasons, uh, you know, one of which is in order to employ some of these technologies, you actually have to, if you're, if you're studying a mouse, you have to head fix the mouse. In other words, you know, you have to like immobilize the mouse. And so it's been, it's been tricky to come up with ways of eliciting interesting behavior from a mouse that's, that's restrained in this way. But people have begun to, you know, create very, uh, um, interesting solutions to this, like virtual reality environments where the animal can kind of move a track ball. And, and, um, and as people have kind of, uh, begun to explore what you can do with these technologies, I feel like more and more people are asking, well, let's try to bring behavior into the picture.

• 1:50:13Speaker 01

  Let's try to like reintroduce behavior, which was supposed to be what this whole thing was about. And I'm hoping that those two trends, the kind of growing interest in behavior and the widespread interest in what's going on in AI will come together to kind of open a new chapter in neuroscience research where there's a rebirth of interest in the structure of behavior and its underlying substrates, but that that research is being informed by computational mechanisms that we're coming to understand in AI.

• 1:50:55Speaker 01

  If we can do that, then we might be taking a step closer to this utopian future that we were talking about earlier, where there's really no distinction between psychology and neuroscience. Neuroscience is about studying the mechanisms that underlie whatever it is the brain is for. And what is the brain for? It's for behavior. I feel like we could maybe take a step toward that now if people are motivated in the right way. You also asked about AI. So that was the neuroscience question.

• 1:51:27Lex Fridman?

  You said neuroscience, that's right. And especially places like DeepMind are interested in both branches. So what about the engineering of intelligent systems?

• 1:51:36Speaker 01

  I think one of the key challenges that a lot of people are seeing now in AI is to build systems that have the kind of flexibility and the kind of flexibility that humans have in two senses. One is that humans can be good at many things. They're not just expert at one thing. And they're also flexible in the sense that they can switch between things very easily and they can pick up new things very quickly because they very ably see what a new task has in common with other things that they've done.

• 1:52:19Speaker 01

  And that's something that our AI systems just blatantly do not have. There are some people who like to argue that deep learning and deep RL are simply wrong for getting that kind of flexibility. I don't share that belief, but the simpler fact of the matter is we're not building things yet that do have that kind of flexibility. And I think the attention of a large part of the AI community is starting to pivot to that question. How do we get that? That's going to lead to a focus on abstraction. It's going to lead to a focus on what in psychology we call cognitive control, which

• 1:53:00Speaker 01

  is the ability to switch between tasks, the ability to quickly put together a program of behavior that you've never executed before, but you know makes sense for a particular set of demands. It's very closely related to what the prefrontal cortex does on the neuroscience side. So I think it's going to be an interesting, an interesting new chapter.

• 1:53:21Lex Fridman?

  So that's the reasoning side and cognition side, but let me ask the over romanticized question. Do you think we'll ever engineer an AGI system that we humans would be able to love and that would love us back? So have that level and depth of connection?

• 1:53:43Speaker 01

  I love that question. And it relates closely to things that I've been thinking about a lot lately, you know, in the context of this human AI research. There's social psychology research, in particular by Susan Fisk at Princeton in the department I used to, where I used to work, where she, she dissects human attitudes toward other humans into a sort of two dimensional, you know, a two dimensional, two dimensional scheme. And one dimension is about ability, you know, how able, how capable is this other person? And the, but the other dimension is warmth.

• 1:54:29Speaker 01

  So you can imagine another person who's very skilled and capable, but is very cold. Right. And you wouldn't, you wouldn't really like highly, you might have some reservations about that other person, right. But there's also a kind of reservation that we might have about another person who, who elicits in us or displays a lot of human warmth, but is, you know, not good at getting things done. Right. That, that like the greatest esteem that we, we reserve our greatest esteem really for people who are both highly capable and also quite warm. Right. That's, that's like the best of the best. This, I mean, I'm just,

• 1:55:12Speaker 01

  this isn't a normative statement I'm making. This is just an empirical, it's an empirical statement. This is what humans seem, this, these are the two dimensions that people seem to kind of like, along which people size other people up. And, and in AI research, we really focus on this capability thing. Like, we want our agents to be able to do stuff, you know, this thing can play Go at a superhuman level. That's awesome. And, but that's only one dimension. What's the, what about the other dimension? What would it mean for an AI system to be warm? And, you know, I don't know, maybe there are easy solutions here. Like, we can put a, put a face on our AI systems. It's cute. It has big ears. I mean, that's probably part of it. But I think it also has to do with a pattern of behavior,

• 1:55:53Speaker 01

  a pattern of, you know, what would it mean for an AI system to display caring, compassionate behavior in a way that actually made us feel like it was for real? That we didn't feel like it was simulated. We didn't feel like we were being duped. To me, that, you know, people talk about the Turing test or some, some descendant of it. I feel like that's the ultimate Turing test. You know, is there a, is there an AI system that can not only convince us that it knows how to reason, and it knows how to interpret language, but We're comfortable saying, yeah, that AI system's a good guy. You know, like, I mean,

• 1:56:33Lex Fridman?

  On the warmth scale, whatever warmth is, we kind of intuitively understand it. But we also want to be able to, to, yeah, we don't understand it explicitly enough yet to be able to engineer it.

• 1:56:48Speaker 01

  Exactly.

• 1:56:48Lex Fridman?

  And that's, and that's an open scientific question. You kind of alluded it several times in the human AI interaction. That's the question that should be studied and probably one of the most important questions.

• 1:56:59Speaker 01

  And humans, and humans, we humans are, are so good at it.

• 1:57:03Lex Fridman?

  Yeah.

• 1:57:03Speaker 01

  You know, it's not just,

• 1:57:04Lex Fridman?

  weird.

• 1:57:04Speaker 01

  It's not just that we're born warm, you know? Like, I suppose some people are, are warmer than others, given, you know, whatever genes they managed to inherit. But there's also, there's also, there are also learned skills involved, right? I mean, there are ways of communicating to other people that you care, that they matter to you, that you're enjoying interacting with them, right? And we learn these skills from one another. And it's not out of the question that we could build engineered systems. I think it's hopeless, as you say, that we could somehow hand design these sorts of, these sorts of behaviors. But it's not out of the question that we could build systems that kind of,

• 1:57:47Speaker 01

  we, we, we, we instill in them something that sets them out in the right direction. So that they, they end up learning what it is to interact with humans in a way that's gratifying to humans. I mean, honestly, if that's not where we're headed,

• 1:58:06Speaker 01

  I want out.

• 1:58:09Lex Fridman?

  I think it's exciting as a scientific problem, just as you described. I honestly don't see a better way to end it than talking about warmth and love. And Matt, I don't think I've ever had such a wonderful conversation where my questions were so bad and your answers were so beautiful. So I deeply appreciate it. I really enjoyed it.

• 1:58:28Speaker 01

  Well, it's been very fun.

• 1:58:29Lex Fridman?

  Thanks for talking today.

• 1:58:30Speaker 01

  I, as you can probably tell, I, I really, you know, I, there's something I like about kind of thinking outside the box.

• 1:58:37Lex Fridman?

  Yeah.

• 1:58:37Speaker 01

  And like, so it's good.

• 1:58:38Lex Fridman?

  Having fun with it.

• 1:58:39Speaker 01

  To do that.

• 1:58:40Lex Fridman?

  Awesome. Thanks so much for doing it. Thanks for listening to this conversation with Matt Boppenick, and thank you to our sponsors, The Jordan Harbinger Show and Magic Spoon Low Carb Keto Cereal. Please consider supporting this podcast by going to jordanharbinger.com/lex and also going to magic spoon.com/lex and using code LEX at checkout. Click the links, buy all the stuff. It's the best way to support this podcast and the journey I'm on in my research and the startup. If you enjoy this thing, subscribe on YouTube, review it with five stars on Apple Podcasts, support it on Patreon, follow on Spotify, or connect with me on Twitter at Lex Fridman,

• 1:59:25Lex Fridman?

  again, spelled miraculously without the E, just F R I D M A N. And now let me leave you with some words from neurologist V S Samachandra. How can a three-pound mass of jelly that you can hold in your palm, imagine angels, contemplate the meaning of infinity, even question its own place in the cosmos? Especially awe-inspiring is the fact that any single brain, including yours, is made up of atoms that were forged in the hearts of countless far-flung stars billions of years ago. These particles drifted for eons and light years until gravity and change brought them together here now.

• 2:00:10Lex Fridman?

  These atoms now form a conglomerate, your brain, that can not only ponder the very stars they gave it birth, but can also think about its own ability to think and wonder about its own ability to wander. With the arrival of humans, it has been said, the universe has suddenly become conscious of itself. This truly is the greatest mystery of all.

• 2:00:37Lex Fridman?

  Thank you for listening and hope to see you next time.