Your Brain is Running a Simulation Right Now [Max Bennett]

Tim sits down with Max Bennett to explore how our brains evolved over 600 million years—and what that means for understanding both human intelligence and AI. Max isn't a neuroscientist by training. He's a tech entrepreneur who got curious, started reading, and ended up weaving together three fields that rarely talk to each other: comparative psychology (what different animals can actually do), evolutionary neuroscience (how brains changed over time), and AI (what actually works in practice). *Your Brain Is a Guessing Machine* You don't actually "see" the world. Your brain builds a simulation of what it *thinks* is out there and just uses your eyes to check if it's right. That's why optical illusions work—your brain is filling in a triangle that isn't there, or can't decide if it's looking at a duck or a rabbit. *Rats Have Regrets* *Chimps Are Machiavellian* *Language Is the Human Superpower* *Does ChatGPT Think?* (truncated description, more on rescript) Understanding how the brain evolved isn't just about the past. It gives us clues about: - What's actually different between human intelligence and AI - Why we're so easily fooled by status games and tribal thinking - What features we might want to build into—or leave out of—future AI systems Get Max's book: https://www.amazon.com/Brief-History-Intelligence-Humans-Breakthroughs/dp/0063286343 Rescript: https://app.rescript.info/public/share/R234b7AXyDXZusqQ_43KMGsUSvJ2TpSz2I3emnI6j9A --- TIMESTAMPS: 00:00:00 Introduction: Outsider's Advantage & Neocortex Theories 00:11:34 Perception as Inference: The Filling-In Machine 00:19:11 Understanding, Recognition & Generative Models 00:36:39 How Mice Plan: Vicarious Trial & Error 00:46:15 Evolution of Self: The Layer 4 Mystery 00:58:31 Ancient Minds & The Social Brain: Machiavellian Apes 01:19:36 AI Alignment, Instrumental Convergence & Status Games 01:33:07 Metacognition & The IQ Paradox 01:48:40 Does GPT Have Theory of Mind? 02:00:40 Memes, Language Singularity & Brain Size Myths 02:16:44 Communication, Language & The Cyborg Future 02:44:25 Shared Fictions, World Models & The Reality Gap --- REFERENCES:Person: [00:00:05] Karl Friston (UCL) https://www.youtube.com/watch?v=PNYWi996Beg [00:00:06] Jeff Hawkins https://www.youtube.com/watch?v=6VQILbDqaI4 [00:12:19] Hermann von Helmholtz https://plato.stanford.edu/entries/hermann-helmholtz/ [00:38:34] David Redish (U. Minnesota) https://redishlab.umn.edu/ [01:10:19] Robin Dunbar https://www.psy.ox.ac.uk/people/robin-dunbar [01:15:04] Emil Menzel https://www.sciencedirect.com/bookseries/behavior-of-nonhuman-primates/vol/5/suppl/C [01:19:49] Nick Bostrom https://nickbostrom.com/ [02:28:25] Noam Chomsky https://linguistics.mit.edu/user/chomsky/ [03:01:22] Judea Pearl https://samueli.ucla.edu/people/judea-pearl/ Concept/Framework: [00:05:04] Active Inference https://www.youtube.com/watch?v=KkR24ieh5Ow Paper: [00:35:59] Predictions not commands [Rick A Adams] https://pubmed.ncbi.nlm.nih.gov/23129312/ Book: [01:25:42] The Elephant in the Brain https://www.amazon.com/Elephant-Brain-Hidden-Motives-Everyday/dp/0190495995 [01:28:27] The Status Game https://www.goodreads.com/book/show/58642436-the-status-game [02:00:40] The Selfish Gene https://amazon.com/dp/0198788606 [02:14:25] The Language Game https://www.amazon.com/Language-Game-Improvisation-Created-Changed/dp/1541674987 [02:54:40] The Evolution of Language https://www.amazon.com/Evolution-Language-Approaches/dp/052167736X [03:09:37] The Three-Body Problem https://amazon.com/dp/0765377063

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Tim Scarfe?

  What's really interesting about about this book, Max, is, you know, obviously, I've read loads and loads of books in the space and there's, you know, people like Hinton and Hawkins and Damasio and Friston and, I mean, God, you know, even like Sutton. And what's interesting is it's a bit like The Blind Men and The Elephant. So they've all got a completely different story to tell. And I think the magic that you have pulled off with this book is somehow you've woven it together into a coherent story. Like, what do you think about that?

• 0:29Speaker 01

  Well, first, I'm very appreciative of the kind words. Yeah, I think I came from a very unique perspective, just because I was a complete outsider. And I think, you know, and I didn't come to it with the objective of writing an academic book at all. I came to it from the objective of I was just learning on my own. And I just started building this corpus of notes because I was so independently curious. And I kind of stumbled on this idea, really for myself, of how do I make sense of all of these disparate opinions? And really, this complete lack of information about how the brain actually works. And I had my own set of, I think, biases coming from sort of the technology entrepreneurial

• 1:13Speaker 01

  world, where we tend to think about things as ordered modifications. When you think about product strategies or how to roll things out, we like to think about things as what's step one, then what's step two, what's step three. So I think I did have sort of a cognitive bias to, when presented with an incredible amount of complexity, to try and make sense of it in a similar type of way. But yeah, I think as an outsider, I felt very free to sort of explore and cross the boundaries between fields. I mean, I look at the book as a merging of three fields. One is comparative psychology, so trying to understand what are the different intellectual capacities of different species. Evolutionary neuroscience, so what do we know about the past brains of humans and

• 1:56Speaker 01

  the ordered set of modifications of how brains came to be? And then AI, which is, how do we ground sort of the highfalutin conceptual discussions about how the brain works in what works in practice? I think is a really important grounding principle because it helps really hold us accountable towards the principles that we think work. If we can't implement them in AI systems, it makes, it should make us question if we actually have the ideas right.

• 2:23Speaker 01

  So yeah, I think being an outsider in some ways comes with disadvantages, but there are some advantages also, which is you're kind of free to borrow from a variety of different fields and think freshly about things.

• 2:34Tim Scarfe?

  Yes, I mean, if you can point to any particular ideas that you found really, really difficult to reconcile, what would those be?

• 2:41Speaker 01

  One thing that's really challenging is if we were to actually lay out what's the data richness of comparative psychology studies across species, if you put that on a whiteboard and looked at it, you would realize we have so little data on what intellectual capacities different animals in fact have. For example, the lamprey fish, which is sort of the canonical animal that's used as a model organism for the first vertebrates, because it's the, of all vertebrates alive today, it's one of our most distant vertebrate cousins. To my knowledge, there are absolutely no studies examining the map-based navigation of the lamprey fish. So we have no idea if it's in fact capable of recognizing things in 3D space.

• 3:26Speaker 01

  Now, when we look at other vertebrates like teleost fish, it seems like they're eminently capable of doing that. We look at lizards, they're eminently capable of doing that. So we sort of infer, it seems likely that the first vertebrates were able to do this. We know the brain structures from which it emerges and reptiles and teleost fish are present in the lamprey. So we sort of back into an inference that, okay, probably the lamprey fish can do that. But this is all sort of, in some sense, guessing and trying to put the pieces together from very little information. So I think that's one challenging aspect to reconcile.

• 4:02Speaker 01

  The other one that's really hard is in neuroscience, there's a lot of really interesting ideas about how the brain might work that have not really been tested in the wild from the AI perspective. And then there's a lot of AI systems that work really well that have diverged substantially from at least the evidence, what the evidence suggests is how brains work. So how do you bridge the gap between these two things, I think is a really fascinating space to operate in, which is like, what can we learn about the brain, if anything, from the success of transformers, as an example? What can we learn, if anything, from the success of generative models in general?

• 4:44Speaker 01

  What can we learn from the success of and failures of modern reinforcement learning? I mean, in some ways reinforcement learning has been a success, in other ways, it's really fallen short of what a lot of people hoped it would be. So I think the gap between neuroscience and AI is still a challenging one to bridge in a lot of ways. For example, Carl Friston has all these incredible ideas in active inference. In 100 years, will we look back on this and be like Carl Friston was on to something? If you look at the AI systems today, there's very little, you know, usage of active inference principles working in practice. So that could mean that the ideas don't have legs, or it could mean that there's a breakthrough behind the corner where we're actually missing some of the key principles that he's devising. And these are, you know, questions we don't have the answers to.

• 5:27Tim Scarfe?

  I think there might possibly be some breakthroughs around the corner. I don't know if you know, but I'm Carl Friston's personal publicist.

• 5:34Speaker 01

  All right.

• 5:35Tim Scarfe?

  I do all of his stuff. I probably interviewed him more than anyone else. I love, I love the guy.

• 5:40Speaker 01

  He's a great friend. He's an amazing guy. Yeah, he's an amazing man.

• 5:43Tim Scarfe?

  Honestly, he is the man. He is the man.

• 5:45Speaker 01

  So kind.

• 5:46Tim Scarfe?

  I know. But for me as well, he has so much time to explain things. But yeah, then you could cynically argue that, you know, the effective active inference agent is just a reinforcement learning agent of a particular variety. I think it's equivalent to a, like an inverse reinforcement learning maximum entropy agent or something. But there's so much more than that. There's so much richness and explanatory power of kind of modeling this thing as a generative model that can generate policies and plans of actions and so on. And also, we want to have agents that, you know, we understand what they're doing and like with steerability and being able to do the simulations that you talk to so eloquently in your, in your paper. But just, I love, for instance, we'll do, we'll do him properly later. But to come back to what you were saying though, so you were saying, oh, you know, there might

• 6:30Tim Scarfe?

  be a parallel between transformers and AI models. In your book, actually, on this page, you sort of like, you kind of analogize model-based reinforcement learning and the neocortex. And of course, I interviewed Hawkins back in the day, and the main criticism of his book is that, you know, the tree and brain type argument. So he's kind of giving. Giving the explanation that the brain developed a bit like geological strata with one layer and then another layer, rather than kind of co-evolving together. And it's so hard not to think like that, because you give so many beautiful examples in your book, not only just morphologically, but in terms of capability, how the neocortex,

• 7:14Tim Scarfe?

  I mean, you said like with stroke victims, for example, it's not like the brain recovers those dead cells. It learns to kind of repurpose those functions in other parts of the brain. So it seems like Mount Castle was correct in the neocortex, is this just magic, general purpose learning system. I mean, what do you think?

• 7:31Speaker 01

  So I think, well, there's two different, there's two different ways to look at the neocortex enabling things like mental simulation and model-free or model-based reinforcement learning. One is that that function and algorithm is being implemented in the neocortex. But another, which is a slightly less strong claim, which is the one I would make, is the addition of the neocortex enables the overall system to engage in this process. So it is not saying that the entire process is implemented in the neocortex. I think it seems very clear that the thalamus and basal ganglia are essential aspects of enabling the pausing, the mental simulation, the modeling of one's own intentions, the evaluation of the results, et cetera.

• 8:14Speaker 01

  But it is possibly to say, which is what I'm arguing in the book, is in the absence of the neocortex, that process does not happen. And so I think where my ideas would synergize with what Hawkins is saying is the neocortex builds a very rich model of the world and a model of sufficient richness that you can explore it in the absence of sensory input. And that's a really essential aspect of model-based reinforcement learning, because if I have a model of the world that has sufficient richness, that I can mentally simulate actions that I'm not actually taking, and it at least somewhat accurately predicts the real consequences of those actions, then that model is really useful. Because I can now imagine outcomes before having them, I can flexibly adjust to new situations.

• 8:59Speaker 01

  And of course, there's so many really deep, interesting questions that are yet to be answered about that. For example, Just because you can render a simulation of the world doesn't answer the question, "Okay, what do you simulate?" I mean, this is one of the hardest problems of model-based reinforcement learning is, fine, you can have a model of the world, but how do you prune the search space of which aspects of that model you explore before evaluating outcomes? And so that's another really hard challenge. And I think there's a lot of good evidence is actually a partnership between the neocortex and the basal ganglia, which is a much older structure. So yeah, I'm not really of the, you know, I think the triune brain has been, you know, amongst evolutionary neuroscientists largely discredited. And I

• 9:41Speaker 01

  think in part somewhat unfairly, if you actually read McLean's writings, he was, he actually is very open about the fact that, you know, this is an approximation, it's not exactly accurate. Like, he couches his claims much more than the popular culture than just converted into a dogma. But I think the popular interpretation of the triune brain is not accurate, which is it is clearly not the case that the brain evolved in three key layers. It's not the case that a reptile brain doesn't have anything limbic-like. If you, a reptile brain absolutely has a cortex that does a lot of what our limbic structures do, et cetera, et cetera. So yeah, those would be my thoughts on that.

• 10:17Tim Scarfe?

  Yeah, it is fascinating because we as humans need to have models to explain and understand the thing itself, just like active inference, for example. I mean, I'll get to planning and agency and goals in a little while, but a lot of these things are instrumental fictions. I'm not saying that our brains don't plan, but you know, like the way, the abstract mathematical way that we understand planning is probably not how the brain works. It's much more complicated than that. But why don't we just rewind to the beginning? So, you know, we're going to be talking about this chapter on, you know, simulation, if you like. And you kind of lead by saying that what the neocortex does is it does learning by imagining. And Hawkins spoke about this as well. He said, "We've got the matrix inside

• 11:01Tim Scarfe?

  our brains," right? "We're always just doing all of these simulations of future things." And we're using that to kind of help us understand the world. And you get this really interesting example of some of the features of the brain that kind of lead you to believe that we are basically living in a simulation. And it's almost like rather than perceiving things, we're testing if our simulation is correct. But that means that we can only simulate one thing at a time. So we can't, we can't see two things, we can only see one thing. So can you talk through that?

• 11:34Speaker 01

  Sure, so one of the first sort of introspections and explorations into how perception works in the human mind happened in the late 19th century with all of these explorations of visual illusions that you see in pretty much every neuroscience textbook or book that you open. So listeners will be familiar with, you've probably seen examples of triangles where you actually perceive a triangle in a picture when there is in fact no triangle, or, yes, that isn't, yeah, you can find that picture.

• 12:05Tim Scarfe?

  Yeah, sorry, I hope I'm not distracting you.

• 12:08Speaker 01

  No, no, no. So that's, you know, a standard finding that was observed in the 19th century, which is this idea that clearly the brain observes the presence of things even though they're not actually there. So we perceive a triangle there, we perceive a sphere, we perceive sort of a bar, we perceive the word editor, when in fact, if you actually examine that, the word E is not there. There's evidence that suggests E is there by virtue of showing the shadows, but we did not actually write the letter E there. But the brain regularly observes that. So that finding led this scientist Herman von Helmholtz to sort of come up with this concept that what the brain, what you actually consciously perceive

• 12:51Speaker 01

  is not your sensory stimuli. You are not receiving sensory input and experiencing the sensory input. What's happening is your brain is making an inference as to what is true in the world, what's actually there. And then the evidence is just, the sensory input's giving evidence to your brain as to what's there. And so you start from this prior, and then that prior maintains itself until you get sufficient evidence to the contrary, and then you change your mind. And so it's not hard to imagine why this would be extremely useful in any sort of environment that an animal might evolve in. You know, suppose you have a mouse running across a tree branch at night. First, I see the tree branch in the moonlight, so I build a mental model of the tree branch. As I move forward, I

• 13:34Speaker 01

  lose moonlight, I no longer see the tree branch. As long as, as I'm stepping forward, the evidence is consistent with my prior of the tree branch, it makes way more sense for me to maintain the mental model of that tree branch, as opposed to all of a sudden the tree branch disappears because I no longer see the sensory stimuli of it. So because sensory stimuli is very noisy, it makes a lot of sense that we integrate it over time, build a prior, and then until something gives us evidence to the contrary, we maintain our prior about the world. So that was sort of the first idea that there's some form of inference. There's some difference between sensory input and some model of the world that we infer and then thus perceive. What's interesting that is not as discussed, but is also present in the discussions among

• 14:18Speaker 01

  scientists in the late 20th century about this, the late 19th century about this, is this idea that you can't actually render a simulation of two things at once. So there's lots of really interesting sort of visual illusions around this where you can see something. The famous one is it's either a duck or a rabbit. Yep, exactly. And it's interesting, yeah, and then so you can see that staircase is either moving up to the left or you're under the staircase looking upwards and it's actually a ceiling that's jagged. And why can't the brain perceive both of those things at the same time? Well, it would make sense if you have a model that there are such things as ducks, there are such things as rabbits, there are such things as 3D shapes that operate under certain assumptions.

• 15:02Speaker 01

  And if that's true, then you cannot see a duck and a rabbit at the same time because there's no such thing. It cannot be the case that the staircase is looking from above and below at the same time. So what your brain is not doing is just perceiving the sensory stimuli. It's trying to infer what is a real 3D thing in the world that I am aware about that this sensory stimuli is suggesting is true. And that is the thing that I'm going to render in your mind. And I think one way this parallels nicely to some of Hawkins's ideas, actually, is if you hold the thousand brains theory to be true, which is the neocortex has sort of all of these redundant overlapping models of objects, then it would make a lot of sense that we want to synergize these models to render one thing at a time. You don't want to have 15 different things rendered because then it's really hard to

• 15:47Speaker 01

  evaluate them and vote between these different columns. So it makes sense that the brain does is say, let me integrate all the input across sensory stimuli and render one sort of symphony of models in my mind so I can see one thing at a time. So that's this idea of perception by inference. At the time, no one really connected that to the idea of planning. So this was just this idea that what we perceive is different from the sensory stimuli we get. But later on, sort of as the world of AI started thinking about things from the perspective of perception by inference, what we end up realizing is this idea of perception by inference, if you're going to train a model to do that, it comes with this notion of generation.

• 16:30Speaker 01

  So because the way it self-supervises is it takes the prior and it tries to make predictions and it compares the predictions in the world to the sensory stimuli. And as long as those predictions are below a threshold, I maintain my prior. So a famous version of this is the Helmholtz machine, which Hinton devised, and I think that was in the 80s. It could be later, I forget. Yep. And so this is this basic idea that you can build a model, he called, you know, a lot of people use the term latent representation. Some people don't like that term for variety of philosophical reasons. But it builds a representation of things by virtue of building a model, in other words, perception by inference. And the way you build that model is you're constantly comparing, generating predictions from that model to what actually occurs.

• 17:14Speaker 01

  And this also has synergies to a lot of Hawkins's ideas where we think about intelligence as prediction. And so what that means is if you build a model of perception by inference by virtue of generation, then it's relatively easy to say, okay, well, what happens if I just turn off sensory stimuli and I start exploring the latent representation? Well, now we're exploring a simulated world. I'm able to cut off sensory stimuli, close my eyes and imagine a chair, rotate the chair, change the color of the chair. And because this model is a relatively good, has relatively rich features about how the world actually works, I can model things without ever having experienced them, without

• 17:55Speaker 01

  ever having done them, and relatively reasonably predict what actually happened if I were to do those things. So what I think is interesting and perhaps somewhat of a novel.

• 18:08Speaker 01

  Proposal in the book is, I think, a lot of people think about the neocortex as having adaptive value because of how good it is at recognizing things in the world. Right? So a lot of, you know, if you read a standard textbook, a lot of what people will talk about the neocortex is how good it is perceiving things, object recognition. Some of the best studied parts of the brain are this visual neocortex. So we understand reasonably well how we're building models of visual objects, et cetera. But from an evolutionary perspective, this is a little bit hard to find convincing, because if you actually examine the object recognition of vertebrates, it's incredibly accurate. I mean, a fish can recognize human faces. A fish can one-shot recognize an object

• 18:53Speaker 01

  when rotated in 3D space. So it's hard to find a dividing line between object recognition in animals with a neocortex and object recognition in objects without a neocortex and with with brain structures that seem more similar to early vertebrates.

• 19:09Speaker 02

  Now, more than ever, we need someone on our side. But Rob Whitman? Whitman's votes drove up costs, cut Medicaid, and risked 350,000 Virginians' healthcare to pay for tax breaks for himself and his donors. He's not on our side. Shannon Taylor will fight corruption, cut taxes, and bring down costs. This November, vote for Shannon Taylor. She's for us. Paid for by HMP, the HouseMajorityPac.com. Not authorized by any candidate or candidate's committee.

• 19:40Speaker 03

  You sound horrible.

• 19:41Speaker 04

  Oh, I feel worse. This flu is not playing around.

• 19:45Speaker 03

  Dad, you promised you'd be in my band, and it's showtime. Then it's Dayquil Intense Flu time.

• 19:51Speaker 05

  For fast and powerful relief of your flu symptoms, try Dayquil Intense Flu. Hit it.

• 19:56Speaker 01

  Yeah.

• 19:58Speaker 05

  Dayquil Intense Flu, the fast-acting daytime coughing, aching, aching, aching fever power through your flu medicine. Use is directed, keep out of reach of children.

• 20:08Speaker 01

  Did you have a question?

• 20:09Tim Scarfe?

  Yeah. Well, yeah, I just wanted to touch on a couple of things there.

• 20:12Speaker 01

  Go, go.

• 20:12Tim Scarfe?

  So Hawkins said that the reason, I mean, he said first of all, that we overcome the binding problem by having this kind of profusion of individual sensory motor models rather than having this kind of, you know, feed forward enrichment of representations. And that was really interesting, but he also said, the reason why having, let's say we've got 150,000 mini cortical columns that are just wired to different sensory motor signals. And he said, it's the, it's the kind of diversity and sparsity that gave the robustness of recognition. And then you can kind of think, okay, well, well, what do the representations look like? So if our brain builds some kind of a model of the world, some kind of topological model, it must be a representation, not necessarily, it's not like a homunculus, or it's not like

• 20:55Tim Scarfe?

  a stapler inside the brain, if I'm modeling a stapler. It's actually some weird structure of the stapler, but as seen by every way I can touch, feel, hear and, you know, lick a stapler or whatever. So it's difficult for us to imagine what that is. But the reason, the reason I'm going down, down this road though, because that's a bit weird, isn't it? So we have this very weird representation of things. And then I come to Hinton's Helmholtzian generative model. And you can get it to generate, let's say, a number eight, if it's trained to do numbers, right? And Hinton would argue, and I would disagree with him, that, that the model understands what an eight is. Now, this is weird, isn't it?

• 21:35Tim Scarfe?

  Because we understand what a mouse is, but intuitively, we feel that a neural network doesn't understand what an eight is. And I would argue that we're getting into semantics here. So I think the reason we understand things is because there's a relational component to understanding. So semantics is about the ontology, you know, our, the way we feel about things, where the thing came from, what was the intention guiding the creation of the thing, what was the provenance of the thing? It's almost like the interconnectedness of the thing tells you more about the meaning of the thing than the actual thing itself in a weird way. What, what do you think?

• 22:14Speaker 01

  Well, I do think, I think this is where the word "understanding," I think, can mean different things to different people. People can understand the word "understanding" in different ways. I think there is absolutely something to the idea that just because you can recognize something, so that would be a feed-forward network that can observe a stapler alone, is insufficient to what most people would mean when they use the word "understanding." And just to talk about interrelatedness, if I have a feed-forward network that, you know, let's say it's just a binary classifier, is this thing a stapler or not? Now, I can't ask lots of things of that feed-forward network that I would

• 22:56Speaker 01

  expect of an agent or a model that understands what a stapler is. I couldn't ask it, for example, what would happen if I burned the stapler? What would happen if I opened it? What would you see inside of it? I can't ask, what does a stapler do? I can't say, show a human holding a stapler and a set of objects in front of them, and what do you think the person's going to do next with this stapler? And so, clearly, understanding, most of our intuitive understanding of the word "understanding," contains some richness that's not included in just classifying or recognizing the presence of objects. So I think that's absolutely the case. My intuitions sort of fall in the direction of what we

• 23:39Speaker 01

  mean typically when we use the word "understanding," it comes to having something that can be mentally explored. And so, I think that requires what you're describing, which is the interrelatedness of things, because when I see someone holding a stapler and you ask me the question, what are the things they would likely do? Well, I start imagining what they might do with that stapler, and then I can evaluate which ones seem plausible to me. And so, in the imagination and evaluation of which things seem plausible, there's interconnectedness between the thing and the world around it. But yes, I absolutely agree with you that just recognizing objects clearly lacks something that we mean when we say "understanding."

• 24:17Tim Scarfe?

  Yeah. Do you take into consideration the memosphere? So, you know, we've got this kind of, we've got ideas that are quite collective as well. I guess I was going to explore this with you later, but...

• 24:29Tim Scarfe?

  Maybe, maybe let me frame it like this. I, I have this intuition that a lot of our intelligence is outside of the brain. So it's almost like if, if I was in the wilderness and, you know, disconnected from society, I would be much, you know, a lesser human being. In a weird way, I might have more agency, but, you know, I, I wouldn't have access to all of these rich cultural tools and knowledge and patterns and so on. So it, it's almost like that's where a lot of our intelligence comes from, rather than, and, and also meaning as well, rather than, than just being able to plan in the brain and so on. And there must be some kind of interplay. So culture must shape the development of our brain and, and vice versa. But culture seems

• 25:11Tim Scarfe?

  to be more dynamic. But how, how do you wrestle with that?

• 25:15Speaker 01

  Well, I, I mean, there, it's, that's undeniably true. I mean, the, the first example that comes to mind is just writing. I mean, what would humans be if you removed the technology of writing? I mean, we would, we would all realize we're not that smart. I mean, writing is a technology that externalizes a feature of the brain, which is memory, which brains are not that great at. We do a good job of like condensing aspects of a memory. So we, for episodic like things and procedural memory, we're relatively good at. But for semantic memories, we're terrible. And so externalizing that with writing is one of the key technologies that enables us to be

• 25:58Speaker 01

  way smarter than we are in fact, because we now have this external device that enables us to store largely infinite numbers of memories and translate them across generations. So that alone, I think, proves your point, which is what humans are capable of is clearly some relationship between brains and external things that can be writing tools, that can be other brains, you know, sharing ideas and getting challenged. So yeah, you're absolutely, I agree with you.

• 26:25Tim Scarfe?

  Intelligence is like the, it's the dynamics, isn't it? It's, it's all of the low level, you know, dynamics of things interacting with each other. And we can take a snapshot of language and, and we might say, oh, well, yeah, you know, language isn't, isn't the intelligence, but, but language itself is a form of intelligence. So I'm not just talking about the words and language models and so on. We're talking about the actual language in, in our culture. And I guess I, I think of that as, as almost a distinct form of intelligence.

• 26:56Speaker 01

  Yeah, I mean, I think it almost gets into a philosophical territory as where do you draw the bounding boxes over the things that are imbued with intelligence and the things that are supportive mechanisms that allow those things to have intelligence. So through one lens, you could you could think about brains as the physical entity in which intelligence is instantiated, and then language is a supportive tool. You could take a very perhaps odd view, which is language is the thing that's evolving, and it's just instantiated in these brains that sort of produce the language, consume the language, but it's language that's that's evolving. The same way that, you know, we don't think about intelligence on the level of an individual neuron.

• 27:40Speaker 01

  We don't imbue a neuron with intelligence, but we think on the scale of the 86 billion neurons, something has emergently appeared that we do deem intelligent. You know, there's some great sci fi books where, you know, intelligence gets instantiated in colonies of ants, where each individual ant is not intelligent, but somehow the colony itself is capable of doing incredible abstractions and whatever. So, so, yes, I think there's, there's very interesting ways to, and it's not obvious how one divides the lines between what are the physical entities in which these are instantiated. That said, I have a particular interest in brains because I think if we're looking for, for what is the physical manifestation that we can learn from and thus try to, one, to

• 28:24Speaker 01

  understand ourselves, just I think all species have an interest in understanding ourselves. But then also, if we want to try and borrow some ideas from how biological intelligence works into AI, I think the, of all the physical things to examine, the brain, to me, seems clearly the one that is probably the most rich with insight. But I, no, I agree, I, I think your point is well taken.

• 28:49Tim Scarfe?

  Interesting, okay. I want to close the loop on what you said about the brain being an imagination-filling-in machine. So you said that it does filling in, it's one at a time, it can't unsee visual illusions, and evidence is seen in the wiring of the neocortex itself, you say, so it's shown to have many properties consistent with a generative model. The evidence is seen in the surprising symmetry, the ironclad inseparability between perception and imagination that is found in generative models in the neocortex. And you give examples like illusions and how humans succumb to hallucinations, why we dream and sleep, and even the inner workings of imagination itself. So it really seems plausible when you kind of think of it in that way.

• 29:32Speaker 01

  Yeah, I think there's a reason why so much of the neuroscience community, I mean, what I'm saying there is not really novel, so much of the neuroscience community has sort of rallied around this idea of predictive coding, which is very related to active inference and generative models. Because there's just so much evidence that what's going on in the neocortex, the imagination of things, episodic memories, I mean, there's been some good evidence that episodic memory, in other words, thinking about past and imagining the future are in fact the same underlying process happening in the neocortex, which is again consistent with this idea that there's a generative model. If we look at the connectivity

• 30:17Speaker 01

  patterns, I didn't talk about this too deeply in the book, it's a little technical, but what you would expect from a generative model is backwards connections would be much richer than forward connections, because you're modulating downstream. Of course, the neocortex is not perfectly hierarchical, but things that are in general lower in the hierarchy would have lots of inputs from parts of the neocortex that are higher in the hierarchy, it's absolutely what we see. So yeah, there's a lot of evidence that these are two sides of the same coin, which is there's some form of generative model being implemented. I do think in AI, one way in which this manifests is the very clear success of self-supervision. I mean, this idea, although the actual predictive coding algorithms that people have

• 31:00Speaker 01

  sort of devised as the new as neocortex implementing, when we've actually modeled them, they haven't outperformed any of the stuff going on in AI world, the principle of self-supervision, which is, can a system end up having really interesting emergent properties and generalize well, when you only train it on predicting sensory input that it receives? And that clearly has become the case. I mean, the transformer, I think, is a great example of, if you just give it a bunch of data, and you train it through self-supervision, i.e. masking, so you hide certain data inputs, it becomes remarkably accurate and good at generalizing across data that it hasn't seen before, which is, in principle, all what people are predicting or claiming that the neocortex is doing as a generative model.

• 31:44Tim Scarfe?

  Yeah, it feels to me that there is a bright difference, though, between, let's say, a transformer and the neocortex. And I think the difference is, maybe agency is not the right word, but you can think of the neurons, I think, as having some kind of autonomy. So they're sending messages to each other, and then the other, you know, it's eventually consistent. So the other neuron will get the message, and it will decide itself what it's going to do. And in a transformer, just because of the way they're connected and the backprop algorithm and so on, they all, they all kind of ride a Mexican wave together, to use an analogy. So it feels like a difference in kind, to me.

• 32:19Speaker 01

  Well, I mean, clearly, as I argue in the book, I don't think that the brain is just one big transformer. So I would agree with you. In the human brain, unless you think there's something that's non-deterministic and sort of magical happening, you know, I think you would still say that there is either, you know, base firing rates of neurons, and then there's sensory input that flows up and then goes through the brain until eventually there's sort of, it's affecting muscles, and so you're responding. So there is, you know, there is sort of a deterministic flow happening. It might not be as feed-forward as what's happening in transformer, which is definitely the case. But they might both be sort of

• 33:02Speaker 01

  You know deterministic in a similar fashion. I think a lot of people I've had this exact sort of argument with a lot of people and one counter argument that people have towards this idea. I don't know if I fully agree with it, but it's interesting is that attention heads really are doing something more magical than we give them credit for, which is they are kind of dynamically rerouting and effectively resetting the network based on the context that the prompt is getting. And so although technically it's just a series of, you know, matrix multiplications, et cetera. If in principle, what's happening is these attention heads are doing something really clever, where they're looking at the context of a prompt and then effectively dynamically reweighting the network to decide what it cares about or what it doesn't. So there are people that think,

• 33:45Speaker 01

  you know, there is something really interesting happening in a transformer that might be analogous to certain things that are happening in the brain. But yeah, clearly, you know, these feed forward networks are not capturing everything that's going on in the brain.

• 33:58Tim Scarfe?

  Yeah, it's really interesting what you said because the way I read that is things like chat GPT and language models, they are entropy smuggling or agency smuggling. So what that means is they kind of just do what you tell them to do and all of the kind of the agency. So my directedness comes from me. So I give it a prompt, it does the thing that I wanted it to do. And then the kind of the mapping that you were talking about, I interpret that a bit like a database query. So, you know, depending on the prompt you give it, it'll activate a certain part of the representation space and it will give you a certain result back. But the brain has this thing where all of the neurons have their own directedness. And the weird thing is at the cosmic scale, agency is, it seems to emerge. So even transformer models that were

• 34:42Tim Scarfe?

  acting autonomously could presumably in large enough scale, give rise to something that we think of as directedness or goals or purpose or whatever. But it's almost like in the natural world, because there are so many levels, scales and scales of independent autonomous things just kind of mingling with each other independently and then like downstream mixing their information together and rinsing and repeating over many, many different scales. That seems to be the thing that gives rise to all of these amazing things like agency and creativity and et cetera.

• 35:16Speaker 01

  Yeah, yeah, I mean, I think the the notion of agency is an interesting one where I really am amenable. I mean, there is sort of a, I don't know if I would call it a schism in the field, but there are, there are, there is a debate where between sort of the reinforcement learning world and the active inference worlds where how much of intelligence can be conceived as optimizing a reward function and sort of the hardcore reinforcement learning world is like, everything is just a reward. And then the active inference world is, you know, would argue that not all behavior is driven just by optimizing a single reward function. There is some uncertainty minimization. There is trying to satisfy your own model

• 35:58Speaker 01

  of yourself, fulfill your own predictions, these sort of things that seem very well aligned to behavior we see, but, you know, it's unclear which of these is right. It's probably some balance of the two. But to me, agency, people would conceive of agency differently in these two worlds, right? So I think some people in the RL world would say, agency is just, you give something a reward function, and then it just learns over time, try and optimize that reward. In the more active inference world, which I do think has legs, and I'm obviously amenable to, the idea of agency is a little bit more. It's building a model of yourself and trying to infer what your goals are, based on observing

• 36:40Speaker 01

  yourself, and then trying to make predictions to fulfill those end goals. In other words, it's constructed, goals are constructed. And this is sort of one of my favorite Fristin papers is Predictions Not Commands. I don't know if you've read that paper, but I think it's a brilliant paper about how you could reconceive motor cortex, not as sending motor commands to your body, but actually as building a model of yourself and predicting what will happen. And the way the spinal cord is wired is it just fulfills those predictions. And I think that's a really interesting sort of reframe of how you could get agency and really interesting smart behavior in the absence of just a strict reward function,

• 37:21Speaker 01

  right? So how that would learn is it's trying to model the behaviors it observes, and then it's trying to sort of predict those and fulfill them. So, but yeah, I, yeah, I think agency is a really interesting concept because it sort of manifests itself in these different paradigms in different ways.

• 37:38Tim Scarfe?

  Yeah, I find it fascinating. I mean, because the way I read it in active inference literature, it's a very principled definition of an agent. And there's still a bit of a gap because I think Friston would argue, well, in the natural world, because of the laws of physics and particles and whatnot, you get the emergence of things and things become agents when they have a certain, you know, depth of planning, shall we say.

• 38:04Speaker 01

  Yep.

• 38:04Tim Scarfe?

  But yeah, it's really, really interesting. And I guess he would argue that you get all of these phenomena that give rise to agency, like biotic self-organization and so on. But yeah, maybe we should do, we'll slowly go in that direction. So you give the example of mice doing planning. Can you sketch that out?

• 38:26Speaker 01

  Yeah. So this is another like area of neuroscience research that I just absolutely love. So it was the case, I think it was the 40s or 50s, I forget the exact decade, where Tolman observed that worms, when they, mice, when they would reach choice points in mazes, where he was training them to navigate around mazes, would pause. And then they would sniff back and forth, and then they would choose an action. And so he hypothesized this idea that they must be engaging in vicarious trial and error. They must be imagining possible outcomes before deciding. And of course, this was hugely controversial because there's no evidence, he had no evidence that they were in fact imagining anything. And of course, most people

• 39:08Speaker 01

  like to, in the absence of evidence, assume animals are as dumb as possible. Only when there's irrefutable evidence will we imbue them with any intellectual capacities, which I think is an interesting human bias, but that's fine. And then David Reddish, who is also a close friend and mentor of mine, he did some amazing research with, as one of his PhD students. Where they were recording hippocampal place cells. So it's very quick background for viewers. You can go into the hippocampus of really any mammal, but this is best well studied in rats. And part of hippocampus, a region called CA1, has these things called place cells. And so if you look at, if you record it as a mouse is moving around a maze,

• 39:50Speaker 01

  what you find is this incredible thing where there are neurons that activate only in specific locations in that two dimensional plane. And it's not based on how they got there. It is egocentric, is independent of their egocentric path. It is allocentric, meaning in this, in the plane of external space. So they can come back to the same place from any route and that same place cell will activate. And as an aside, from the evolutionary story, we find similar types of cells. It's not exactly as accurate, but we find similar types of cells in fish, in the homologous region of the hippocampus, of their cortex, where they have place-like cells. It's not as accurate, but cells that activate in certain locations in a maze. What he found is when mice engage in this act of vicarious trial

• 40:35Speaker 01

  and error, meaning when they pause and look back and forth, the place cells in the hippocampus cease to only activate in the location they are. It actually starts activating down the paths of each route it might take. In other words, you can literally watch rats imagining the future. One of the most, I think, one of the most incredible neuroscience findings. And so he then took this and then did a bunch of other experiments, which I think reveal even further the power of sort of imagination in rats. And one of my favorites is his counterfactual learning studies, where he puts rats in this thing called restaurant row, where it's a sort of a square like, yeah, exactly. It's just like a square like maze. And as the rat

• 41:16Speaker 01

  is going counterclockwise, at each sort of door, a sound is released or made. And that sound signals to the rat whether or not they can go right through the door and get food in like, I think it's like three seconds, or they're going to have to wait 45 seconds before they get the food. And they're given a bunch of time to try and get as much food as they want. Rats have clear preferences, so some rats will really prefer bananas and they don't really like the bland food. So what happens? Well, this presents a set of irreversible choices to a rat. So let's say they come up and they can either get, they can either get a treat right now, they can, they can get a treat right now that they don't really like that much. Let's say it's the bland treat, or they can go to the next one and hope that

• 42:01Speaker 01

  they're going to get the banana really fast. If they go to the next one and the banana sound is long, meaning forty five seconds, then they regret their choice because it would have been better if they just went in and quickly got the food. How do we know they're regretting the choice? We can literally watch them imagining eating the foregone choice. We can go on a part of their brain called the orbital frontal cortex, which activates for certain types of tastings. And we know that we can see them imagining the foregone choice, and they end up making different choices the next time around. They end up being less likely to forego that, that choice in the future. So I think this is just such an incredible finding of what we mean when we say model-based reinforcement learning is clearly happening in these brains of very, very simple mammals.

• 42:43Tim Scarfe?

  There's a real challenge knowing which simulations to run because if you think about it, we've got a search problem, right? There's an intractable number of simulations to run. So how do we, how do we fix that in AI? And how do humans fix that?

• 42:54Speaker 01

  So, so this is one of the, I think, of the many, but this is one of the big outstanding questions in AI, which is how do you effectively prune the search space? We do not know how mammal brains do this so well. So I can, I'll give you some like high level ideas or theories, but we, we just don't know. And this is one of the big possible breakthroughs in figuring out, you know, how, how do mammal brains do such a good job of this? So the thing that AlphaGo does, which I think perhaps is a clue and is clever, is the selection of the search space actually is bootstrapped on the temporal difference learning model under the hood. So this is actually very clever, which is that say you train something to learn without a model of the world. All it's doing is it gets sensory stimuli, it gets

• 43:39Speaker 01

  a model of a go board, and then it just predicts the right next action. It just has a policy value function that bootstraps on each other, et cetera. So no planning. If you want to add planning to that, what they did, which is quite brilliant, is you say, "Okay, well, you know what? Instead of building some other system to try and choose good trajectories, why don't we just use the policy network and we just don't only pick the first one. We pick its favorite move, but then we also look, what's your second favorite move, your third favorite move, and maybe your fourth favorite move. And then let's literally play the games out. And let's just see, let's play a bunch of games against ourselves and then see the ratios in which we win one of them. And then what we might learn is your second best guess was actually better than your first best guess. But we're

• 44:21Speaker 01

  not starting from every possibility. We're saying, let's bootstrap on our best guesses of good moves, but then check them by playing out the possible futures. So if we were to analogize that to the brain, what that would suggest is perhaps it's the basal ganglia, which a lot of evidence suggests is engaging in this type of sort of model free reinforcement learning, actually is the thing that chooses the moves, but there's some other system that lets us choose the second best move, the third best move, et cetera. One way this might happen, there's some evidence for this, far from conclusive, that there's some notion of uncertainty that frontal cortex or basal ganglia is measuring. And when the level

• 45:01Speaker 01

  of uncertainty between the next actions passes a threshold, pausing occurs. Because when we see animals do this vicarious trial and error, it almost always occurs in moments of high uncertainty, when contingencies have changed, when the right answer is not obvious. And so you could conceive of this as a policy network where you're evaluating its best choice, second best choice, third best choice. And when there's uncertainty about it, in other words, they're close together, or there's some other measure of uncertainty, perhaps you have parallel policy models that, and you're comparing the similarity between them, a lot of different ways to do this, that triggers a process of playing forward. This is another key thing that mammal brains do that AlphaGo does not do. AlphaGo engage in planning on every move.

• 45:45Speaker 01

  So there was never the question of when do we pause to plan? In a game of Go, it doesn't matter. Just engage in planning in every move because we can do it so fast. In the real world, there's so much uncertainty and noise. And we need to be so energy efficient with human brains. We can't engage in planning every instant. So we need some mechanism that tells us when to stop and think about what I'm going to do next and when I can just continuously go with model-free choice. This is also something we don't know how mammal brains do. But I think, you know, a reasonable sort of speculation is that there's some uncertainty measurement that's occurring. One last point I'll make that I think parallels, an interesting way to some Hawkins' ideas

• 46:24Speaker 01

  and Person's ideas, is if you take the thousand brains model and you apply it to frontal cortex, in other words, we have multiple parallel models of ourselves. You could imagine that there's an uncertainty measurement, the same way we do uncertainty measurement in a lot of deep learning models, where you create parallel models and you just measure how similar are the predictions of the model. And if multiple parallel models predict similar things, we just measure it as low uncertainty. When they diverge substantially, all of a sudden, we measure high uncertainty. So again, speculation, but you could imagine if it is the case we have redundant models in the neocortex, then might it be the case that somewhere, perhaps the thalamus, the basal ganglia, the similarity or differences between these predictions are a measure of

• 47:08Speaker 01

  uncertainty that triggers pausing. Steven Grossberg has similar ideas. He calls this like matching and non-matching.

• 47:14Tim Scarfe?

  Yeah, yeah. Yeah, like almost because our ability to do abduction is something that fascinates me and there is some kind of a model selection or matching step that goes on there. But anyway, so we've got the neocortex. It's an absolute beast when it comes to predicting sensory signals. And then we see the emergence of planning and also smart planning, as we've just spoken about. So now we're actually talking about traversing these sensory networks over space and time. And then something else really interesting happened. So the next two moves you make. Are bringing in selfhood, so like, you know, bringing yourself as an explicit actor and including that in the planning.

• 47:59Tim Scarfe?

  And then that naturally leads to this idea of, I would call it teleology, but you know, why or intentionality. So let's go on that journey. So maybe the self modeling first. How does that come into the picture?

• 48:14Speaker 01

  So I think there's two notions of self modeling. One notion of self modeling is the kind that I think we see in early mammals. And this is an idea where the frontal cortex of mammals, a region in general called a granular prefrontal cortex, which is present in all mammals and largely believed to have existed in the very first mammal brains, gets sensory input from an animal's own introspective signals, so interceptive signals. So it gets input from the hypothalamus, which measures things like hunger. It gets input from amygdala, which measures things like valence in the world, fear, danger, et cetera.

• 48:57Speaker 01

  And when does the a granular prefrontal cortex get most excited? It's in these moments of uncertainty and in these moments where they're engaging in planning and episodic memory. And if you damage a granular prefrontal cortex in rats, they seem to dramatically have impaired, if not completely lose the ability to engage in mental simulation, episodic memory, et cetera. Okay. So I think what one might speculate is happening here, which is not a novel suggestion by me, is it's modeling the self. In other words, modeling what is the activations of the amygdala and the hypothalamus that are happening and why am I doing what I'm doing? In other words, if I wake up and I see I have these hypothalamic activations and then

• 49:41Speaker 01

  I go down to go get water, then it builds a model of in the presence of these hypothalamic activations, the next action is I'm going to go over and get water, which, and it constructs an explanation. And so as odd and philosophical as that sounds, that is in principle computationally exactly the same thing as when we showed a picture of that triangle and your posterior sensory cortex construct an explanation of what it saw. I perceive the triangle. So this idea of constructing an explanation of one's own behavior is this idea of the first idea of self, which is constructing intent. And there's lots of evidence that even in rats, if you record neurons in their a granular prefrontal cortex, they seem to be very sensitive

• 50:24Speaker 01

  to tasks that they're in and measuring progress towards goals. There's lots of evidence that it's doing something, something akin to that. So that's one notion of self. When you get to primates, you see a whole new region of frontal cortex emerge, what's called granular prefrontal cortex, which is only seen in the primate lineage. There's no other mammals that have this region of prefrontal cortex. As a quick aside for anyone who's interested, the reason it's called granular versus a granular is most neocortex has six layers. The fourth layer is called the granular layer because it contains granular cells, just a certain type of neuron. And for a variety of really unknown reasons, but there's some interesting speculations, a granular prefrontal cortex is missing layer four. That's why it's called

• 51:09Speaker 01

  a granular. Same thing with motor cortex is missing layer four. So most mammals, their front, whole frontal cortex is missing layer, a fourth layer. It only has five layers. But in primates, you get this granular prefrontal cortex, this huge region of neocortex that does contain a layer four. And the best explanation for this, I've actually seen Fristin talk about, I'm happy to go into it if you think your folks would be interested in it. But the point is, there's a new, do you want me to go into that?

• 51:34Tim Scarfe?

  Oh, yeah. I mean, so we love Fristin. And I mean, because active inference is actually about preferences. You know, because an agent expresses agency by kind of like adapting the environment to suit its preferences, or to kind of make the environment like its preferences. So basically, this is a theory of volition, right? And what Carl Fristin's talking about is where do these goals come from? Where does the volition come from? And you spoke with Carl about that, I think, over several years, didn't you?

• 52:03Speaker 01

  Yep. Yep. Yeah. Carl's been a wonderful mentor of mine. I honestly, like, it's a funny story. I like one, I didn't know this was a term in academia, but there's the reviewer lottery, where you just get lucky and you get a reviewer. So the first paper I submitted, Carl Fristin was a reviewer on, which was just lucky. And then he became a mentor and reviewed my book and gave me lots of good feedback. So yeah, he's an amazing person, been a wonderful mentor of mine. So yes. So, okay, so let's talk about granular versus a granular, because I think the best theory I've seen is Fristin's theory on this. So what does layer four do in neocortex? Across the entire neocortex, layer four is where sensory input is received. The primary sensory input is received into the neocortical column. This comes from the thalamus.

• 52:46Speaker 01

  So the canonical model is sensory input from sensors, eyes, ears, skin flows up through the brainstem to the thalamus, and then from the thalamus propagates to layer four. And then from layer four, it goes within the variety of other layers of the cortex. So, and then other layers of the cortex project back to the thalamus and the rest of the brain. So why would it be the case that regions of neocortex would not have a layer four? Well, if you actually watch an animal's development, what's interesting is mammals with a granular prefrontal cortex is not always a granular. It actually starts having a layer four, and the layer four atrophies over development.

• 53:26Speaker 01

  And so I think this is very, mirrors well, first since idea of active inference, where what's happening is the neocortical column can kind of be in two phases. It can either be trying to match its model of the world to its sensory input. In other words, I see sensory input and I'm trying to infer what's there. And I'm going to construct the idea of a triangle. But there's another state of a neocortical column, which is generation, which is I'm going to start from the latent representation of a triangle. I'm going to imagine and explore it. And so one idea is that what frontal cortex does is primarily try and fit the world to its model. In other words, it spends the vast majority of its time constructing intents and not

• 54:11Speaker 01

  trying to modify that intent to fit what it observes, but in fact, try to change what an animal does to satisfy its intent. So it's so layer four atrophies, doesn't actually go all the way away. If you go deep into a brain, you see some basic layer four. So it's not completely gone, but it atrophies because frontal cortex spends very little time trying to change what it perceives its intent to be to map what the animal's doing. But in fact, what it does, it tries to change what an animal does to map it matched to its intent. And what I think is so interesting, brilliant about this idea is it explains exactly why layer four doesn't start not existing, because at first, an animal needs to build a model of itself, thus layer four is present. But over time, it shifts towards, once I have a model of what I want and who I am

• 54:56Speaker 01

  and the things I would do, I don't need to spend as much time changing my model of self. I'm going to spend most of my time trying to change my behavior. So this is a very speculative idea, but it makes a lot of sense in the context of active inference, and it's the best to personally, in all of my reading of explanations of why a granularity exists, the best explanation I've seen.

• 55:19Tim Scarfe?

  Yeah, so this is absolutely fascinating, and in my mind, it bridges the gap between internalism and externalism, because it is describing this kind of didactic exchange, to use that, you know, there are certain high entropy words that only Friston uses. So if I say didactic exchange, you know, and by the way, if you, just for the folks at home, if you read Friston's papers, there are certain words, so he says, this licenses something. So you know, if you see the word licenses, then Friston wrote the paper. But anyway, but yeah, so but you have this kind of like, you know, like the agents have models of the world, but they're kind of exchanging information with the other agents.

• 56:01Tim Scarfe?

  And then you get, what an agent does is it has this generative model of policies, you know, which is just a sequence of actions. And here's where I want to get into the nitty gritty a tiny bit. So you could think of those, those, those plans as being goals. So you could think of a goal as just being an end state in one of your plans. But that doesn't really satisfy me, because I think of goals like eating food as being like a kind of a category, not, not like a kind of a pointillistic traversal of, you know, a specific state in the future. Because it kind of feels like, well, if there's an infinite number of goals, then in some sense, there are no goals at all. So what is a goal to you?

• 56:46Speaker 01

  So, okay, so great question. So I think this is where semantics matters a lot. I think we can think about goals in several ways. So I think if we think strict RL lands, they would just think about goals as you're just optimizing a reward function. The goal, the goal is simple. There's only one goal, which is maximize reward. And in a changing, complex world, your reward function might fluctuate over time, et cetera, et cetera. But the goal is singular. In the active inference role, what I find compelling is it introduces a different

• 57:21Speaker 01

  Components of what we mean by a goal, which has not is not just sort of a intellectual, you know, of intellectual interest is because it's cool, but has like very real AI implications, actually, because it contains the notion of explainability. So for example, if I am imagining, if I'm if I wake up and I'm hungry and I start imagining ways to satiate my hunger, and then I decide I'm going to get in a car and I'm going to go to this restaurant and then I'm going to go eat this specific food. When I get into the car and someone calls me and go, why did you get in the car? The reason I can explain that so easily is because there was

• 58:03Speaker 01

  a rendered simulation of a plan that terminated in an end state that I deemed that I wanted, I selected. So it's very easy to explain why I did that. In the absence of that, it's actually very hard to explain why you're doing things. If you're walking down the street, if I asked you to explain any one of your model free behaviors, why did you move your foot there instead of there? You have no explanation. And so I think you can think about, you can assign the word goals to multiple levels here. I think you could say it's terminating in the end state. You could say the goal is some more abstract representation of the satiation of thirst in general. And you could think about that as distinct from just the reward function. Some might challenge that and just say, well, all that you're talking about then

• 58:48Speaker 01

  is just optimize reward function. But I think you could make an argument that there is a distinction happening there. But what I what I think is so critical, that's unique that happens within mammal branch, which is why I was sort of really honing in on that, is the ability to plan a series of actions that terminates in the end result and then execute those that that plan. And I think that has very clear implications for explainability, which sort of model free actions do not. But but yeah, yeah, that would be how I would think about goals. But I think it is a little bit of a semantic where we can think about the concept of goals in multiple ways.

• 59:22Tim Scarfe?

  How do we actually know what the cognitive abilities were of early animals? And why should we care?

• 59:30Speaker 01

  Great question. So I think there's two reasons why we should care about the evolution of our brains and intelligence. The first is to understand who we are. So the the scope of what it means to be a human is not constrained to what it means to be a Homo sapiens. So so much ink has been spilled on the last 70,000 years of us being Homo sapiens. But if aliens were to come down and engage with us and analyze us as a species, most of the things they would observe about us don't come from our legacy as Homo sapiens. They come from our legacy of being a primate, and our legacy of being a mammal, and our legacy of being a vertebrate, and our legacy of being an

• 1:00:14Speaker 01

  animal in general. And so I think if we want to understand what it means to be a human being, I don't think we can skip the full 600 million year story of how we came to be. And I think in there is so much rich history and insight about what it means to be us. So I think that's one really key reason. It's our it's our legacy, it's our history, how we came to be. The second, perhaps more practical reason, is I think understanding the evolution of the human brain and the evolution of human intelligence is a key tool in our toolbox to understanding how the brain works, and how human intelligence works. It's by no means the only method. It might not even be the main method, but it's a very useful method to add to the toolbox. So the problem with going into the

• 1:00:58Speaker 01

  human brain and trying to directly reverse engineer it, is that evolution doesn't work in clean ways. It doesn't work the way a human designer would. It doesn't work from first principles. It tinkers. And so when we go into the brain, we see all of this messiness. There's redundant systems, there's vestigial systems, you know, new things evolve that make old things redundant, but they're still there. Lots of processing is duplicated in different regions. And so, um, one way to understand the brain is to continuously probe it as the human brain is, which is fine. But another method that's also useful and can impose constraints for us is to actually track the history of how it came to be. And that

• 1:01:38Speaker 01

  can provide insights as to, uh, when this brain modification occurred, such as when the neocortex evolved, um, or, uh, when the basal ganglia evolved. Um, what were the new abilities that this enabled? Um, and how did it affect the prior brain regions that were already there? And so this can give us insight into how the brain works today. So I think in the toolbox that we have of ways to reverse engineer the brain, I think this is just an underappreciated one, um, that is worthy of being included. Um, and of course, understanding how the human brain works has so many different applications. Helps us with mental health, helps us with understanding why people do what we do. Um, it helps us with, um, building AI systems. Um, I think there's lots of insights to garner

• 1:02:22Speaker 01

  for the brain. So that's, that's why to do it. Now, how to reverse engineer what behavioral abilities exist in our ancestors is a really interesting question. Um, of course, we can't go back in time. Um, so, so what we can do though, um, is there are, there are mechanisms to reverse engineer what their brains looked like, and there are mechanisms to reverse engineer what abilities they had. So what their brains looked like, we can do just by looking at other animals in the animal kingdom. So for example, we can look at all of the existing primates and all of the existing non-primate mammals. And we can see what are the common brain structures that exist between them. We can look at genetic analysis, meaning what things seem to derive from similar roots. And we can back into what do we, what

• 1:03:07Speaker 01

  seems to be common and shared amongst them. And thus, what do we think was actually existing in the brains of the first mammals? We do the same thing with fish and reptiles to do that, uh, with early vertebrates, and we can do that with invertebrates to try and infer what was existing in the first bilaterians. In other words, the first animal with brains. So we can compare different brains to try and back into what the brains looked like. For behavioral abilities, there are sort of three ways you do this. And you need, this is like my sort of approach to trying to infer behavioral abilities. I call them the in-group condition, the out-group condition, and the stem-group condition. So in order to make the argument that a behavioral ability emerged at a certain location in our evolutionary history.

• 1:03:48Speaker 01

  So for example, a behavioral ability like episodic memory evolving with the first mammals. You need to satisfy these three criteria. The in-group condition stipulates that most ancestors, or sorry, most descendants of this species, in other words, most mammals, should show this ability. Doesn't mean all of them, abilities get lost all the time, but most of them should show this ability. And the neural mechanisms by which the ability emerges should come from homologous regions. What that means is a shared neural underpinning. So if, for example, mammals show episodic memory, but they come from neurological regions that independently evolved along different mammal lineages, and that suggests it wasn't present in the first mammals. But if they all emerge from regions that emerged with early mammals,

• 1:04:33Speaker 01

  that's good evidence that this ability also emerged in mammals. The out-group condition says most, doesn't have to be all, but at least many non-mammal, so out-group, non-mammal vertebrates, so the group right above, should not show this ability. And if they do show the ability, it should emerge from non-homologous regions. In other words, parts of their brain that evolved independently. So for example, birds definitely show episodic memory. But when we look into the brain regions from which episodic memory emerges, it's clearly non-homologous. It's a part of the brain that mammals, that early vertebrates did not have. And the stem-group condition is, in the ecological sort of dynamics that early mammals existed,

• 1:05:16Speaker 01

  or this ancestor existed, we should be able to devise an argument for why this ability would have been adaptive. So why would episodic memory have evolved? So with these three things, we can start to infer the story of when behavioral abilities emerged. Is this perfect? Absolutely not, because we do not have enough, we do not have a ton of data on behavioral abilities across species. So as new evidence emerges, the story might change. But with these three conditions, we can do a reasonable job inferring what abilities emerged. And the main finding of the book, which, or the research which led me to be so excited about the book, is when you do this, what's kind of crazy is you find a, as a first approximation,

• 1:05:59Speaker 01

  a really coherent story, which is a lot of the behavioral abilities that emerge at each milestone in brain evolution don't seem to be sort of a haphazard array of different skills, but often emerge from really one, what I call breakthrough, but one underlying intellectual capacity applied in different ways. So that's sort of the idea of the five breakthroughs. One thing to note, actually, about the basal ganglia is, I think, this is a little bit of a sidebar, but I think, fun. The basal ganglia, I think, is one of the most underappreciated parts of the brain, in the sense that so much work has gone into understanding how the neocortex functions, so much work has gone into the neocortical column, and that's all wonderful work to be done.

• 1:06:43Speaker 01

  But the basal ganglia not only is evolutionarily much older, if you look in a lamprey fish, as we talked about, I think, last time, a lamprey fish has a common ancestor with us 500 million years ago, it's one of the most distant vertebrate cousins that still exists today, and they have a basal ganglia that looks exactly like our basal ganglia, same inner structure. And the basal ganglia has perhaps one of the most beautiful internal structures that can be computationally reverse engineered. There's not good consensus on the actual inner wiring and the computations performed by a neocortical column, but there is much broader consensus as to what's being executed by the basal ganglia.

• 1:07:25Speaker 01

  And without getting overly technical and perhaps boring people, I would encourage anyone who's computationally interested in this to dive into the literature here, because it is, it is almost beautiful that evolution came up with this. For example, the input structure of the basal ganglia has this mosaic of neurons that each express two different types of dopamine receptors, this will be my own little technical diatribe. One are called D1 receptors, and then there are D2 receptors. D1 receptors, when it receives dopamine strengthens connections. D2 receptors, when they lose dopamine strengthen connections. And now you track these different neurons and they actually splits their paths.

• 1:08:05Speaker 01

  The D1 receptors go to a nuclei that when activated disinhibits behaviors. And D2 neurons go through a different set of nuclei that when activated inhibits behaviors. And so we can literally watch how dopamine signals drive repeated behavior and inhibit. And dopamine drops inhibits behavior. We can literally look at the mosaic of connectivity here and say, oh, when you spike dopamine, it weakens the stop pathway, which through D2 neurons and it disinhibits the go pathway through D1 neurons and makes you more likely to repeat the behavior. And vice versa, if something bad happens and you lose dopamine. And I think that is just so beautiful that evolution stumbled on something that that

• 1:08:50Speaker 01

  clean in its macrostructure. So just diatribe on, I think the basal ganglia is cool.

• 1:08:58Tim Scarfe?

  Yeah, it's quite interesting as well, you know, people get addicted to drugs. A lot of that is about wireheading in the basal ganglia. And of course habitual learning is something that, you know, when when something becomes so habituated, it kind of, you know, just moves down the stack into the basal ganglia. And, you know, so drug taking would be an example of that. And you actually cited, I think, an experiment in China where they removed parts of the basal ganglia and there was a 40% recidivism from addiction.

• 1:09:30Speaker 01

  Yeah, very controversial study that probably violates many ethical codes in the US, but yes, they did this study for intractable heroin addiction. And they lesioned a part of the basal ganglia called the nucleus accumbens, which is sort of where goals are habitually selected. And it showed a dramatic reduction in heroin addiction. Also had other sort of side effects that maybe doctors would deem unreasonable, but it worked and it absolutely reduced sort of the addict. of cravings that are triggered by stimuli.

• 1:10:04Tim Scarfe?

  Yeah, the the the story of this chapter, incredible chapter. I've just I've just been studying it today in great detail. Is the story of mentalizing, but I I would I would call it, you know, social complexification. Actually, that's a hypothesis of why our brains expanded so dramatically. So there was this extinction event. I think it was at the Devonian extinction event and only birds and, you know, not many things survived. And then we got the chance to kind of evolve after that and and our brains rapidly exploded and there are different theories about why that happened. You know, maybe it's because we had access to loads of calories in the form of fruit, preferential access, and it gave us

• 1:10:45Tim Scarfe?

  an incredible amount of time and energy, the excess of which might have led to social complexification. But can can you just sort of give us a little bit of a background about that first piece first?

• 1:10:56Speaker 01

  Absolutely. So we don't know, there's lots of speculations, but we don't know why this occurred with primates, but we do have some really good evidence that at least part of what drove the sort of explosion in primate brains was social dynamics. And so Robin Dunbar did sort of the seminal work here, where what he showed is in primates, the encephalization quotient, which is just the ratio, like brain size, especially the neocortical ratio. So the ratio of the size of the neocortex to the rest of the brain is extremely correlated with social group size in primates.

• 1:11:37Speaker 01

  So the bigger a social group size you find in a group of primates, the bigger their neocortex seems to be relative to their their body size. And what's so interesting is you don't see this in most other mammals. So this is not a standard correlation that applies across the animal kingdom. It seems to be this correlation that's very specific to primates. There might be other mammals, but for most mammals, you don't see this correlation. And so Robin Dunbar's famous social brain hypothesis is what drove the explosion in primate brains is some form of social dynamic between people. And the more social relationships you're managing, the bigger brain has to be. Now, it doesn't explain

• 1:12:19Speaker 01

  What specifically happened in the brain, which is what we can get to with mentalizing and why that applies, but it does suggest that whatever drove this explosion in brain size seems to be something correlated to social grouping. And what's interesting about primate social groups relative to not all mammals, but many other mammals, is they're very, very political. So many mammals live in sort of solitary sort of social lives where the males mostly live alone and females will rear a child and then usually they'll go off on their own. There are animals that live in sort of herds where they socially group together, but there's not really very rigorous sort of hierarchies amongst them.

• 1:13:02Speaker 01

  But primates, especially apes, have these really complicated social structures with rigid hierarchies. So there truly is someone at the top of the hierarchy and we can measure this through, you know, primatologists have gone to painstaking links to verify this. For example, there's transitivity. So if you show that one primate tends to show a submissive signal to another primate and that other primate shows a submissive signal to another one, then it's almost definitely the case that that first one will show a submissive signal to the last one. In other words, these are real rigid hierarchies, they're not like random interactions of submission and dominance. And what we see is one of the main ways you survive as a primate is you successfully climb this hierarchy.

• 1:13:47Speaker 01

  And so what's so interesting is in many mammals, what makes someone the top dog or the sort of person who's the top of a hierarchy is sort of bronze, it's just strength. They're just show, they're trying to flaunt who would win in a physical altercation, which evolutionarily is beneficial because if you can prevent actually fighting each other and just say this is what who would win the fight, then we both save energy just having these sort of fake battles and then I can, if whoever wins gets to eat the food, et cetera. But with primates, it's not always the strongest one that reaches the top. It's the most socially savvy one. And socially savviness comes into these alliances that are built within primate. So you'll see that people at the top of the hierarchy frequently will befriend and groom

• 1:14:32Speaker 01

  and come to the aid of certain other non-family members and those folks will thus reciprocate and come to their aid. And there's these really interesting dynamics that play out. You even see wars. So there's mutinies that take place in primate societies. So, so in this sort of soup of the way to survive and get evolutionary advantage as a primate is not perhaps only to make sure you get access to food, but to climb a social hierarchy. All of a sudden, there are these huge social pressures to be able to do things like infer what someone else would do in a certain circumstance, or what someone knows, or what you can get away with, or how to change someone's opinion of you.

• 1:15:16Speaker 01

  And so, and so what that, you know, lines nicely to is what we see of the new brain regions that emerged in primates. Most notably is brain region called the granular prefrontal cortex and these areas in the back of the brain called the superior temporal sulcus, temporal parietal junction. These brain regions across primates are highly implicated in what I call mentalizing, which is thinking about thinking. But the, the standard literature would call this just theory of mind. So being able to infer the intent or knowledge of someone else. And so it's very, it's easy to understand why this would be so adaptively valuable in a sort of politicking arms race, where you're trying to sort of deceive each other. There's a great study

• 1:15:59Speaker 01

  that really revealed this with primates by M. L. Menzel in the 1970s, and I love this, this story. So he was trying to, he was, he had this like one acre forests. Do you want to ask a question?

• 1:16:11Tim Scarfe?

  Are you going to do the Machiavellian apes?

• 1:16:14Speaker 01

  Yes.

• 1:16:14Tim Scarfe?

  Yeah, yeah, go on.

• 1:16:16Speaker 01

  Okay, okay.

• 1:16:18Speaker 02

  Now, more than ever, we need someone on our side. But Rob Whitman? Whitman's votes drove up costs, cut Medicaid, and risked 350,000 Virginians' healthcare to pay for tax breaks for himself and his donors. He's not on our side. Shannon Taylor will fight corruption, cut taxes, and bring down costs. This November, vote for Shannon Taylor. She's for us. Paid for by HMP, the House with Jody Pack.com, not authorized by any candidate or candidate's committee.

• 1:16:47Speaker 01

  So, M. L. Menzel had this one acre forest. And his main objective was not to study ape social behaviors in the sense of how they would sort of climb social hierarchies. His only objective was to measure spatial reasoning and chimpanzees. And so he had this group of chimps. There was one chimp named Bell, another one named Rock, and there was a few others. And he would show Belle the location of food. So he would hide food under a bush. And then he would see, would Belle go back to that same location looking for food? In other words, could she remember locations in 3D space or two dimensional map like space? And what he found is readily, yes, they do that.

• 1:17:30Speaker 01

  And you know, we now know that lots of mammals are capable of that. In fact, even fish can do things like that.

• 1:17:36Speaker 01

  But in this study, he started finding something that was odd. When Belle would find the food, what she would frequently do is share it with her fellow chimpanzee group members. And that was great until Rock, who was a high sort of ranking aggressive male, when she would share would take the food from her.

• 1:17:55Speaker 01

  And so what she started doing is hiding the food when she found it. So instead of sharing with Rock, she would just sit on the food. And so Rock realized she was doing this and not sharing. So Rock would come over and push her to try and get the food under her. So then what she started doing was when she knew the location of the food, because on some recurring cycle or experiment or a signal that the food was now available, what she would do is not go to it until Rock was not looking.

• 1:18:24Speaker 01

  So then what did Rock start doing? Rock started pretending not to look. Rock would look away while Belle would go towards the food. And once he noticed she was doing that, he would turn around and run to try and grab the food before her. So then Belle started trying to lead him in the wrong directions. And so this, this cycle of deception, counter deception kept playing out. And what that becomes this beautiful anecdote of is sort of a key study in what happens when you have a bunch of sort of hierarchically interacting animals in an arms race for things like this. And what you get is deception and counter deception. And that's really only conceivable with some notion of theory of mind. Because in order for Belle to trick or try to trick Rocks,

• 1:19:07Speaker 01

  you need to be able to say that in order for me to change the knowledge in Rock's head of where the food is, what I need to do is walk in this other direction. And what that will do is make Rock think the food is in this direction, when in fact I know it's in this other location. Or also to reason about someone's intentions. I know Rock intends to trick me. So when he's looking away, I don't believe that he in fact is not paying attention. And so this was sort of one of the first early anecdotes that some form of theory of mind is is occurring in primates. There has since been lots of studies that show this. And so, for example, just to give some some case studies here. You can take a chimpanzee and you can teach them that when there's two boxes.

• 1:19:49Speaker 01

  The box with the red mark on it is the one with food in it. So they easily will learn that. Then what you do is you have an experimenter come in with two boxes and they bend over and they mark one. And then they accidentally, they pretend to accidentally drop the marker on the other. And then they leave. So the marking is identical in both cases. The chimpanzees always go for the one that was intentionally marked. They can infer the difference between of the same stimuli of someone meaning to do something, intending, and something being an accident. There's other studies of chimpanzees playing with different goggles. And so one goggle, you can't see through another goggle, you can. And if you put those goggles on human experimenters, the chimpanzees always go,

• 1:20:32Speaker 01

  who have food, they always go to the experimenter with the see-through goggles, asking for food. So they're somehow inferring that the other person can't see them, so why would they ask? So there's lots of studies that show this ability. And evidence outside of primates and other mammals is very loose. So it's inconclusive, controversial. But the loose evidence shows up in only the smartest mammals, which possibly suggests some independent convergence. So yeah, there's lots of rich evidence that this theory of mind exists within primates and it emerges from these uniquely primate regions. And I'm happy to go into the evidence of brains, but I'll stop there for a second.

• 1:21:04Tim Scarfe?

  So with the Machiavellian apes, the X-risk people, you know, there are people who talk about A O I killing everyone, and they make the argument, it's called instrumental convergence from Nick Bostrom, which is basically that, you know, things like power-seeking and deceptive behavior would be instrumental to any end goal. And this is a great example from the animal kingdom of deceptive Machiavellian behavior. So I guess it does seem plausible, at least on the surface, that a level of, you know, let's say sophisticated agents following their own intentions and inferring the intentions of others would seek to deceive each other. That's like a natural phenomenon.

• 1:21:45Speaker 01

  I absolutely think so. I absolutely think it is the case that if the more autonomy you give an intelligent agent, and the more ability you give it to define its own subgoals, the more risk emerges. Because you absolutely get what Nick Bostrom is talking about, which is a sub-goal to trying to help cure cancer might be dominate all of Earth and control the labor supply and allocation of resources amongst all of Earth. So, but I don't think that's necessarily inevitable. I think it is a risk. Evolution is a constrained search algorithm for intelligent entities. It is not, it does not give moral weight

• 1:22:26Speaker 01

  to what emerges. This is, I think, an important distinction, which is just because something is a natural consequence of evolutionary systems, does not mean that we should deem it morally superior.

• 1:22:37Tim Scarfe?

  Yeah, that's the naturalistic fallacy.

• 1:22:40Speaker 01

  Right. So it might be the case that it is very likely, if you, that species will eventually enter a politicking arms race, and certain forms of deception will emerge, and power-seeking will emerge. That doesn't mean when we produce our own sort of intelligent entities in AI, that we should imbue them with that, with those features. One of the, I think, optimistic outcomes of this new AI world we're going to enter in the next 100 years, is that we actually now, as designers, can do our best to try and remove some of the evolutionary baggage that we don't like, that's evolved in humans, in these new entities. And so, there's of course, risk. But I think there's also a really great opportunity that

• 1:23:25Speaker 01

  we could have benevolent beings that do not seek to dominate. You know, Yann LeCun talks a lot, a lot about this. And are less selfish. And so, so yeah, I think there's a great opportunity, but there's definitely risk. Because the second you give an autonomous agent the opportunity to produce its own subgoals, you need to have really rich either constraints, or a really well-defined reward function, or, you know, one ability that I think comes from mentalizing, actually, is, and this is an idea in alignment research, which is, if you can convince an AI agent to try and do what it thinks the human wants it to do. What you're actually doing is you're requiring it to engage in some form of

• 1:24:10Speaker 01

  mentalizing to infer the preferences of the requester and then try and do what is best for that for that individual because you can't just have them take requests at at face value because then there's all these opportunities for misinterpretations. The fan, you know, Nick Bostrom's famous paperclip factory maximized production of paperclips. Earth is turned into paperclips. We obviously don't want that. But with mentalizing with the ability to model the internal simulation of another mind and be able to play out, how would this person feel about possible futures? You could imagine optimistically an outcome where an AI agent could easily infer, if I turn all of Earth into paperclips, that's not what the person giving me this request would have in fact wanted. They would regret

• 1:24:50Speaker 01

  that outcome. So of course, it isn't fully de-risk things, but it is one methodology and one learning from sort of evolutionary neuroscience, I think we can garner, that mentalizing is a tool that can be used to try and stabilize sort of requests that we give each other in a more grounded way. So there's not these types of misinterpretations. Of course, caveat, humans misinterpret each other all the time. It's by no means perfect, but it is a tool.

• 1:25:16Tim Scarfe?

  And and I think that's a very natural phenomenon. I think any intelligence system is naturally incoherent. I think it's impossible to have a single monolithic intelligence, which is monomaniacally, you know, focused in a particular direction. But but anyway, I want to just slightly rewind a little bit to what we were saying. So the the first animals, they they had quite simplistic social games that they were playing. So they were interested in strength and submission, and it was a fairly fixed interface. And what was really interesting is that deer, for example, they lock horns, don't they? So it's predictive. They don't actually have to have a fight because that would be, you know, evolutionary, evolutionarily, not a smart thing to do. So so the social game

• 1:26:01Tim Scarfe?

  they play, even though the game is fixed, it's predictive, which is fascinating. And then you were telling the story of, I think, monkeys and macaques, how they have this really interesting virtual social game where strength and social

• 1:26:16Tim Scarfe?

  Status diverged. So your social status actually became this virtual thing that was based on grooming and pruning and lots of completely unrelated things. And it was entirely possible for a very weak macaque to have a significantly higher social status than than a big strong one. So that's really fascinating. But then we get into, I mean, maybe a broader question is, we are still very social creatures ourselves. We have Facebook, for example, and could you just arguably, you know, could you cynically argue that Facebook or, you know, all socializing is just a kind of arms race to improve our social status? So when we're kind of posting on Facebook, in a way, it's like the deer locking horns. It's us playing these status games without having to have a fight with each other.

• 1:27:03Speaker 01

  Hmm. I think there is an aspect of human behavior that can absolutely be explained by this. There's a great book called The Elephant in the Brain.

• 1:27:15Tim Scarfe?

  Oh, yes.

• 1:27:15Speaker 01

  It's a great book called The Elephant in the Brain that talks about how much of human behavior can be explained by this sort of status-seeking behavior. And the reason why it's so hard to study is because it's what they call, I think, a cognitive taboo or an intellectual taboo, where we don't want to admit it to ourselves. So we self-delude ourselves into believing we're doing things for virtuous reasons, because it makes it easy, it's more convincing. If I know that I'm doing something to deceive someone, it's easier to tell that I'm doing it. If I genuinely believe that I'm doing these things to help the world, but subconsciously, of course, they're actually just benefiting me,

• 1:27:57Speaker 01

  it's more convincing to other people. So their argument is primates evolved this sort of self-deceit to make them more convincing. And this really accelerated with language and humans and all that. So I think it's definitely, it's very likely to be the case that the core thesis of their. A book is right, that a lot of human behavior is this sort of subtle status seeking, which I do think has, not to go on a sort of tangent here, but I do think has sort of social, political theory implications here, where, you know, how do we make sure that society doesn't devolve into just a hedonic treadmill, where, you know, the interesting

• 1:28:38Speaker 01

  thing about social status is it's always definitionally a scarce resource, because it's just a ranking game. So unlike physical resources, where it is possible for all of us to live better than kings did a thousand years ago, kings or queens did a thousand years ago, we can all have better access to medical care, all have better access to information, all have better access to food, social status is always a zero-sum game, unless someone can conceive of a better way to do it. But I think in most cases, it's a zero-sum game. And this is problematic, because if, over time, most of our actions become pursuing social status, then we're going to forever be in this sort of game. Now, I don't think personally that we're doomed to this. I think there are absolutely

• 1:29:23Speaker 01

  better virtues in the human psychology, where not everything we do is based on pursuing social status. And I think you can conceive of dynamics where humans are doing things for other reasons, not just to gain status. But I think it's absolutely fair to say that a surprising amount of human behavior is status-seeking, and maybe a depressing amount.

• 1:29:48Tim Scarfe?

  Yeah, yeah. Well, I agree with you, and I don't necessarily want to get too philosophical on that. But yeah, that book was Will Stor's Status Games, where he said that there were three meta-status games that we played. I think he gave, yeah, the virtue game, the dominance game, the success game. So, you know, I might be playing the success game, I want to have the best podcast or whatever. But, no, the reason I bring this up is the difference between humans and animals is they're just playing one game. So it's really interesting that they have this memetic social score, but the the game is the same everywhere. Whereas for us, we go one level of abstraction up. So the success game to us can be manifest in in a myriad of different ways.

• 1:30:28Tim Scarfe?

  It could be successful at playing computer games. It could be, you know, writing books or or making podcasts or whatever. So it's almost like we we fractionate our our social ranking into a myriad of different games. And I think that's a little bit of a testament to the difference us humans have in general with our metacognition, which is our ability to kind of like create the memetics, you know, in a novel way.

• 1:30:54Speaker 01

  One thing that that's sort of related to this, at least in early human societies, is one way to sort of, I'm taking a little bit from the business world here. Where one way to reduce sort of status infighting is to make it such that members of a team have distinct roles. And I don't think this lesson only comes from management theory and, you know, entrepreneurship. I think this probably derives from either early human or maybe even early primate societies where it's much more stable and you can introspect that this feels much more comforting. Being a part of a troop of a hundred humans where pretty much everyone is pulling their weight and everyone matters because they're doing their own distinct thing.

• 1:31:38Speaker 01

  And that is a that's a state that's a very stable state where we're not infighting as much because we're all sort of doing, we all matter to some degree. But when now there's, when there's infighting for, you know, there's only there's five blacksmiths or there's five podcasts or there's five books about the evolution of the brain. Now, all of a sudden, these other types of things start emerging because we're no longer all fulfilling a role that matters. It feels like there's only it feels like there's a ranking and only one of these is going to matter. So I think. As an example of ways to reduce this sort of status seeking, I see this in sort of business all the time, which is the more you can create an environment where it's not zero sum,

• 1:32:21Speaker 01

  where everyone's pulling their weight and together we all win, the more sort of the best version of humans emerge, the more zero sum it becomes and the less divided the sort of roles and what people are doing is, the more of these sort of, I would argue, sort of primitive primate behaviors start emerging. You know, an early-stage company, so a company of 30 people, has such different dynamics than, you know, my last company where, you know, when I was, when I left that we were 400 people. The social dynamics is so different. And I do think one could speculatively correlate that to sort of our evolutionary history here, where in a 30-person company, you don't need that much structure.

• 1:33:05Speaker 01

  I mean, if you have, if you have like, people that work well together, that are aligned on a mission, that, you know, you get rid of people that are, in general, mean-spirited or have bad intent, you don't need a lot of structure and process to get people to work well together, to support each other, to move in the common direction. I think what that demonstrates, when, when one observes that, is that what's playing out is an evolutionary program that got groups of 30 humans to work really well together. When you're at 400 people, what very quickly happens and takes a lot of work to fight this, is you start getting sort of internal factions emerge. Because what splinters out is these subgroups of, you know, 30 to 100 people that then have their own points of view, and then it's very easy to us versus them, other groups,

• 1:33:50Speaker 01

  and you start seeing things break down. And that's where, you know, one mechanism for solving this is very rigid hierarchies, that's what the military does. Another mechanism is embrace the chaos, which is sort of a little bit more what Google does. Another mechanism is to have, is to effectively make it a satellite of different startups, which is what Amazon does, where each group is a kind of autonomous and has very clean interfaces to other groups, so it can kind of, so there's many different sort of management approaches to this. But the breakdown, I do think... Emerges from humans and not evolved to interact with, you know, a hundred thousand people. We did evolve in an environment where we interacted naturally with about a hundred people. And that's why that comes very, we don't need as much sort of process to make that work, but we do and we scale it up.

• 1:34:35Tim Scarfe?

  Yeah, yeah, it's fascinating. I mean, as you say, you could argue that Amazon has one overarching goal to make money. But as soon as you increase the autonomy in the organization, it's a very human trait, isn't it? You were talking about the Machiavellian behaviors and the deceptive behaviors. And you just wonder how much energy is just wasted within fighting. And even I've made the comment that in the military, you know, they might be doing quite simplistic jobs compared to Google. But even at Google, there's an obsession with job level. That's the, I mean, if you go on Quora, if you go on Blind, that's the only thing people talk about is their total conversation and job level. So, you know, I mean, maybe we should save the cynicism. But so coming, coming back to the, to the chapter.

• 1:35:18Tim Scarfe?

  So we're telling the story basically of how this metacognition and this predictive apparatus, you know, gave rise to an entire suite of complex social behaviors that we see in primates, which is fascinating. And maybe we should just talk a little bit about, I call it why bootstrapping. So it's like, why, why, why? I think there was a, one guy at Toyota Research was quite famous because he would get people to ask why five times. So you say, why, why, why? And it's almost as if there's some magic number, you know, like everyone is only a certain number of degrees of separation. And it's a similar thing that you only need to ask why a few times and you'll always get to some kind of base reason.

• 1:36:00Tim Scarfe?

  And maybe that's why evolutionarily we have, you know, two levels of causation, metacognition in our brain. You know, we have the, the a granular prefrontal cortex and we have the granular prefrontal cortex. So I guess like one potential question there is why is there not a third level of asking why? And what would that look like? And can you just kind of sketch out that metacognition picture in general?

• 1:36:22Speaker 01

  When we think about what, when we speculate what a granular prefrontal cortex does. A reasonable framework for it is it generates explanations of an animal's behavior. It models an animal's own behavior. So one sort of cognitive tool to reason about that is if it observes a rat wake up and have certain hypothalamic activations and run to in a certain direction to drink water, it produces a representation that could be interpreted as I am explaining this behavior by I am hungry as an animal. And so that can be useful in a variety of ways. It can trigger simulations to find alternative solutions to satiate the same need, right?

• 1:37:05Speaker 01

  So if I, if you think about this, if you put a rat in a novel situation, but the aggricular prefrontal cortex infers that right now I am hungry. Now we can start triggering a bunch of simulations to try and satiate the same desire to fulfill, this is a little bit an active inference land, to fulfill sort of what I believe about myself through alternative means and enables an animal to be flexible. Okay. So this is sort of the, this is an explanation of an animal itself. Why would it be the case? So what I sort of argue in the book is the granular prefrontal cortex builds a model of that model. So instead of a simulation, it's a simulation of the simulation.

• 1:37:47Speaker 01

  And so what that would mean is if you, if you could, as a thought experiment, if you could ask, let's, let's go one step further or one step back. If we could ask the basal ganglia, which, you know, is the sort of reinforcement learning system, why did you turn left to go in this direction to drink water? It would just say, because that turning left maximizes reward. The answer would always be the same. If you ask the aggricular prefrontal cortex, why did you turn left? You would say, oh, because I'm thirsty. There's a specific thing that me as an entity, this animal I'm modeling, wants to achieve. But if you ask the granular prefrontal cortex, it would say, well, I turned left because I am thirsty and that made me think about ways to satiate my thirst.

• 1:38:30Speaker 01

  And I simulated going to the left and I remembered water being there because last time I was there, there was water and so I went to the left. And so, in other words, it enables you to simulate different types of simulations and reason about what you would think in a new setting, which of course enables you to think about what someone else might be thinking. And we do this all the time. I mean, someone doesn't respond to a text message, someone makes an odd facial expression in social interaction, and you know, we're immediately trying to figure out what is this person thinking, why would they do this, et cetera. So, the question first is, why do we even need this new level at all? And sort of, I think, one of the main adaptive values is it enables survival in the politicking arms race, because now if I can simulate

• 1:39:13Speaker 01

  a simulation, I can infer why you might do a certain behavior, how to manipulate someone's knowledge, your intentions behind things. So this is why you would have one layer. To go a level above, I mean, you could make an argument that theoretically, there should be an infinite scaling up of whys. I think, this is maybe a cop out, but I think there is huge energetic costs to any sort of scale up. So what that means is, the question is not, would there be benefits to a third level of hierarchy? But the question is, would the benefits of a third level of hierarchy outweigh the massive energetic costs of producing it? And so I think that would be my first blushed explanation as to why we might only

• 1:39:55Speaker 01

  have two levels instead of three or four, because the second level added a clear adaptive value relative to the cost to survive in the politicking arms race, and the third one perhaps was superfluous and unnecessary relative to the energetic cost.

• 1:40:07Tim Scarfe?

  I think having that second level of metacognition, it does a lot of work, right? And I'm going to talk a little bit about that now. But one of the things is, you can infer the intents and knowledge of others through the same process of doing simulations yourself. So you can kind of imagine, let's say, yourself doing something, but you can kind of swap out the pointer to be someone else, and swap out the knowledge to be someone else. And that's incredibly valuable. But the knowledge thing is really, really, really interesting. So- I asked the question last time, and this is something that I've been quite confused about, and I feel that reading this chapter is actually really cleared it up for me, which is about goals, because when you look one level down at the a granular prefrontal

• 1:40:49Tim Scarfe?

  cortex, it's modeling intense, and then this granular prefrontal cortex, which is trying to seek explanations about the level below, which is the a granular prefrontal cortex, it's going a level of abstraction up and it's modeling goals, not intense, but it's actually modeling knowledge. What it's doing is it's categorizing. So when you have a simulation of simulations, what it's doing is it's creating a category. So to the example you just gave before, thirsty becomes a category. So rather than it being a pointillistic intent, you know, it's a little bit like saying, I can go and have a sandwich, or I can go and have McDonald's, or my abstract simulator could kind of draw a boundary around those

• 1:41:32Tim Scarfe?

  things, and now I'm getting food. So as well as being able to categorize intents in yourself and other people, you're also categorizing knowledge, and then it can be shared memetically. So it's almost like just going to that second level of metacognition just gives you so much that you didn't have before.

• 1:41:51Speaker 01

  Hundred percent. Yeah, I think the thinking about the level of granular prefrontal cortex, the new primate regions, as enabling something akin to knowledge, I think is a really wonderful way to look at it. Especially, one, from the connectivity analysis, and then two, just from what we mean by knowledge. So connectivity analysis, if you look at superior temporal sulcus and the temporal parietal junction, these are regions of the posterior cortex that, you know, in simple terms are at the very top of the hierarchy. I mean, they get multimodal input from all the other regions of sensory cortex. So a very sort of simple

• 1:42:36Speaker 01

  Rule of thumb for understanding this is just this models the rest of sensory cortex. I understand the full rendering of the simulation of the external world that is happening. And this is where I build a representation of that. And it is perhaps no coincidence that's also where we see brain regions light up when you're engaging in things like theory of mind and solving false belief tests. In other words, trying to infer the knowledge of someone else, these same regions light up. And so, and what do we mean by knowledge? I would argue that knowledge can mean a few things. One is procedural knowledge, where I just know how to do certain motor behaviors. I don't think that's what we mean. I think we mean more semantic knowledge or episodic knowledge, which would be, I know that water is over there.

• 1:43:20Speaker 01

  And I know that if I do this behavior, this will be the causal outcome. That type of knowledge, I think, is absolutely rendered in the mental simulation. When I imagine certain things, when I imagine a case of lightning hitting the ground, what do I see afterwards? I see fire. And that, that, that's the source of my knowledge about the causal relationship between these two things. So having a layer that models the simulation enables me to reason about my own knowledge and to see what the effect of changing knowledge is on behavior. And this, of course, enables us to flexibly adapt to other people's behavior and predict what they would do under cases of different knowledge and different intents.

• 1:43:58Tim Scarfe?

  Yeah, that's fascinating, but you did say that the, there was a bit of a riddle about the granular prefrontal cortex, because there was one study where it could be damaged and the person would still score really highly in IQ. But you said it's about being able to project yourself in simulations, you know, this kind of like abstract modeling your own mind. So in this particular case, how could the person still score the same IQ without that part of the brain?

• 1:44:23Speaker 01

  So this is such a cool story in the history of neuroscience, where you would think that if you look at a human brain, I mean, the granular prefrontal cortex is this huge region in the front of the brain. I mean, it takes up a gargantuan amount of space. You would think that taking a chunk out of that part of the brain would have a gross effect on a human being. Just like if you took a part of even a Relatively small region of the back of your brain, which is where your visual cortex are. You become hugely visually impaired. You take a region of your motor cortex out and humans become largely paralyzed for months until they recover from that. You take a region of auditory cortex out and they can lose the ability to recognize even words. So, so there's relatively small regions of neocortex and if there's damage to it,

• 1:45:06Speaker 01

  there's gross, obvious effects to the hu- human intelligence of behavior. And after World War II, there were so many patients with uh, brain damage that, you know, there's all these studies and people could not figure out, it was a puzzle. What does this huge region of prefrontal cortex do? When people don't have, something seems off about them, but it's not obvious what is wrong with them. People would note personality changes, they don't seem to be themselves. But on sort of logic tests, on IQ tests, it wasn't obvious they were dramatically impaired. In many cases, and there was one famous case where they could test someone before and after, because for surgical reasons, they were going to remove parts of the granular prefrontal cortex. And this patient actually improved in IQ tests. So render this huge puzzle, what does this part of the brain do?

• 1:45:52Speaker 01

  And so then if you track sort of the studies from that point forward, we start learning that what granular prefrontal cortex does in large part, isn't related to these types of logic puzzles. It's related to thinking about thinking and modeling ourselves. So for example, if you look at someone who has damage to granular prefrontal cortex, someone who has damage to hippocampus, and then someone who has a normal brain, and you ask them something very simple, you give them a random word and you say, just tell me a story, just imagine a story of, of you with this word, you could be, the word could be restaurant. And you compare these stories, you immediately see something very different. The people with hippocampal damage give a very, very rich story about themselves, but

• 1:46:37Speaker 01

  this, the external world misses details. So there's not a lot of rich details about the external world. This is consistent with the idea we talked about with early mammals where hippocampus helps render a simulated external world.

• 1:46:49Speaker 01

  The people with granular prefrontal damage could render a very rich external world. They could tell you the details of the leaves, the smell of food, exactly what a restaurant looked like, but they themselves were woefully missing from the stories. They could not just, they could not project themselves into this imagined world. And so then if we go back and look at all these other things that light up granular prefrontal cortex, if you ask someone to think about how they're feeling, granular prefrontal cortex lights up, self-reference. But if you ask another question, such as what does it look like outside, the granular prefrontal cortex does not activate. APFC will activate in both cases. So we start to see that it's in cases of thinking about yourself and thinking about

• 1:47:33Speaker 01

  others that this granular region gets very activated. And now if you go back and study these people more deeply, you notice that they become hugely impaired at false belief tests. They can't recognize faux pas, so they don't understand what's like not really appropriate, which of course makes sense because how do I know what's appropriate? I'm going to infer how you feel about the things that I'm saying. And so you see all of these sort of mentalizing impairments that emerge, but it's not related directly to these logic puzzles that are typically in things like IQ tests.

• 1:48:02Tim Scarfe?

  Yep.

• 1:48:02Speaker 01

  Yeah.

• 1:48:04Tim Scarfe?

  You mentioned the false belief test. Can you just briefly sketch out what that is?

• 1:48:08Speaker 01

  Yes. So there's a, there's a good picture. Do you want to hold it up?

• 1:48:13Tim Scarfe?

  Oh, yeah.

• 1:48:13Speaker 01

  Or show on the podcast? Okay. So the way the test works is you have Sally on the left, who has a basket, and then you have Anne on the right, who has a box. So Sally puts a marble in the basket. And then she walks away. Then Anne goes over and moves the marble from Sally's basket and puts it into her box and then leaves. When Sally comes back, where does she look for the marble? So it's so simple, but in order to figure out that Sally will look into the box, you have to understand that it's possible for another mind to have incorrect knowledge, to have false belief about something. And so, young children don't understand this.

• 1:48:57Speaker 01

  They assume that knowledge is omnipotent. Everyone has the same knowledge about the world. But at a certain point, they start learning that it's possible for people to have false beliefs. And so, we actually know that non-human primates can do this. They've done studies on macaques where you do exactly the same saliante test and you just look where their eyes look when the person comes back into the room to look between the two boxes. And they always look or tend to look in the direction of where that person thinks the marble is or the piece of food is, not where it actually is. If you inhibit their granular prefrontal cortex through an injection or another mechanism, this bias goes away. They no longer look in the right direction. There's lots of really good evidence that this sort of false belief mechanism is occurring in these primate regions.

• 1:49:41Tim Scarfe?

  Yeah, and what really hit home to me is that, in a way, it's not even knowledge. It's all simulations. You know, it's just simulations of other agents. And we've always kind of spoken about knowledge in some weird, platonistic, abstract sense. And I quite like the idea that the primitive form of communication between humans is just simulations, even when we're speaking to each other.

• 1:50:06Speaker 01

  Yeah, exactly. So how do we solve this saliante problem? It probably happens so quickly, but we just simulate, what would we think if we were in Sally's shoes? And then I realized, well, I would look in this place. And this helps us reason about other people. And this begs a really almost profound question about, you know, how unique is theory of mind? This brings me to a question that I've been asked multiple times, which is, does ChatGPT have theory of mind? If by theory of mind, and so the evidence, I should stipulate for anyone curious, is if you ask GPT-3 these sort of theory of mind puzzles, it does terribly. So that's an easy one to sort of discard out of hand. But if you ask GPT-4 these theory of mind puzzles, it performs remarkably accurate,

• 1:50:50Speaker 01

  like as human level at these theory of mind puzzles. And there's been people that have explored, is it just in the training data? And there's good evidence that it's not just because they're, you know, they're just like regurgitating what was in the training data. So does this mean that ChatGPT has theory of mind? So I think there's a few ways to reason about this. One is, what do we mean by theory of mind? If by theory of mind, we just mean the ability to solve these sort of false belief puzzles, then I think you have to accept the fact that yes, it can solve those tasks. The problem is the way in which it renders this model of other minds is not through having a similar mind itself.

• 1:51:31Speaker 01

  And so what this means is we should be concerned, doesn't mean it won't work well, but we should be concerned about how well this will generalize to real tasks where we might care about this much more deeply. So for example, with a human, because part of, there's good evidence to suggest that part of my ability to reason about your mind is because I have a mind that works quite similarly. We are almost bound together by some common mechanistic synergy between the way in which our brains work, because our brains are quite similar, which, which enables a lot of data efficiency, which is I'm pretty good at predicting what people do, not perfect, but pretty good at predicting what people do, because we're all people. And there's similarities between how we act.

• 1:52:12Speaker 01

  And so that makes us quite data efficient and decent at generalizing to new situations where we put people in new places that we've never seen before. I can kind of guess, well, if I were in that situation, this is what I would do. GPT-4 has learned to build a theory of mind simply by reading text of these puzzles. And so clearly it has some mechanism to build a model of predicting what people will do in certain circumstances and differentiating knowledge and intent, et cetera. But the concern is twofold. One, what will happen if we take those types of models and put them in very new situations that are not based on just these puzzles, but for example, we're asking them to optimize a paperclip factory. Now, that's a situation where we should be concerned, how well will it do at actually inferring what we mean by what we say?

• 1:52:57Speaker 01

  And the second is data efficiency, which is how much data did it have to see to build this model? If it was a ton of data, then it's going to be problematic if we have these new situations where we want to teach them to model people's behaviors in this new place. If it requires a ridiculous amount of data, then it's always going to sort of be slow to learn these things and always be at risk of not generalizing well when we put them in these new situations. So. My answer here is nuanced, which is, I think if by we theory of mine, we mean solving puzzle questions, I think it's very hard to say that ChatGPT does not have some model of human behavior. But I do think the human and primate mechanism for doing so has a data efficiency gain advantage

• 1:53:40Speaker 01

  and a mechanistic synergy advantage. In other words, we can use ourselves to reason about things that is relevant. And if we want to have these systems do a good job listening to human requests, we should be, we shouldn't translate performance on false belief tests to believing that they'll do a good job correctly inferring our intent and knowledge in new situations.

• 1:54:04Tim Scarfe?

  Yeah, I would agree with that. I think ChatGPT, it's in the world of text and it's learned all of this structured narratology and things on Reddit and things on Twitter. And as we were saying last week, you know, language has evolved to be very simple as a, it's learnable by children, it has a small subspace. But it is, it is a real kind of generalization over human behaviors and it's, and it's in this very low resolution substrate. Whereas in the Machiavellian apes example that we were talking about before, these are agents performing real time sensing and inferencing and making like, you know, in the moment judgments. And they're, they're in this continuous sensor domain where they have many, many different types of signals, you know, visual signals, sound signals, and also, you know, memory

• 1:54:48Tim Scarfe?

  of what happened in those dynamics just before. So it feels like a difference in kind to me between those two situations. But it is remarkable that in the GPT domain, any kind of theory of mine could, could work.

• 1:55:01Speaker 01

  One good example of this, I think, is, is there a difference in our human ability to predict behavior between a car and a person? So the brain is always able to model things it observes and simulate it and predict what it will do. So I can look at a car and I can imagine different colors of it and I can imagine what will happen if I drop it and it rolls down a hill and I can, we build models of things all the time. We build models of computers and bottles of, you know, I'm just looking around the room of books. So the brain produces models of things. Is the way that the brain produces models of other human behaviors exactly the same? Or is there some unique advantage?

• 1:55:43Speaker 01

  And my my argument is that there's something unique happening when I'm building a model of another person, which is I'm leveraging my own inner simulation of things as a as useful prior to try and predict what other people will do. And so ChatGPT models human behaviors in to draw a crude analogy, the way we would model a a random object, which is I'm only modeling it based on seeing its behaviors in certain situations with the data I I received. On the other hand, when we model someone else's behavior, we're doing some form of projection and and using the prior of how we would behave. And and we probably bootstrap part of our model of human knowledge and intent based on our own introspection. And I think in that way, it is a difference in kind.

• 1:56:28Tim Scarfe?

  Fascinating. I completely agree with that. The selfish gene is kind of saying, it doesn't matter what what you folks do. The gene is kind of directing your behavior and you don't really have as much agency as you as you think you do. And it's a similar thing with language. If you think of language as being a super organism or a virus and we are the hosts and information is being shared memetically and and it's kind of, you know, it's it's shaping our evolution, but it's also shaping our behavior. So it's almost like when we become infected by certain memes, it might be like religion, for example, it completely, you know, it's almost like it parasitically affects our behavior. But I think there is a difference between social memes and physical memes. So tool use, for example, that doesn't seem to have the same parasitic effect.

• 1:57:12Tim Scarfe?

  So, you know, the if you look at the behavioral complexity of of apes, because they don't have these, you know, novel virtual memes, you know, in in their in their culture, their behavior seems quite monolithic compared to ours. But I just wondered if you could kind of contrast that next level of of mimesis.

• 1:57:31Speaker 01

  So there's been lots of great sort of writing about the distinction in, you know, the literature is typically called cultural transmission between sort of non-human primates and humans. And a lot of sort of the general consensus here is that although there is transmission amongst non-human primates, which we see in particular with tool use, it doesn't sort of accumulate in the same way that it does with humans. In other words, you know, humans can pass a piece of information to another generation, which that next generation will reliably copy and then can merge with other new information, which they can then reliably

• 1:58:13Speaker 01

  copy. And you do this over a thousand years and you go from, you know, I know how to use a whittle a bone into a needle for sewing to all of a sudden, now I've built a loom, right? And these ideas keep accumulating on top of each other. Whereas in non-human primates, you don't see the same type of accumulation. And that's what I almost, in like a pithy way, in the book, call the singularity that already happened, which is, once you enable these memes or ideas to accumulate across generations, you get what you're describing as this sort of, sort of memetic organism that, that we are the substrate for, for sure. Now, what I think is interesting here is one lens through which I like to think

• 1:58:54Speaker 01

  about this is sources of learning. So if you think about how non-human primates learn, there's sort of three sources. One is they learn from direct experience. So their own actual actions. This is reinforcement learning writ large. I do something, it succeeds, it fails, fine. Another is their own imagined actions. This is the part that evolved in early mammals. So I can imagine doing five different strategies to try and get to the sort of food over there. And I find the one that worked, and that's a source of learning. My model of the world became a source of figuring out the right path. So my own imagined actions. What mentalizing enabled with primates is this third mechanism, which is learning

• 1:59:37Speaker 01

  from other people's actual actions. So I can see my, you know, if I'm a young chimpanzee, I can see my mother using a stick to put into this termite mound and pull it out and this complicated behavior and eat food. And I don't have to do my own behaviors to do that. I don't even have to simulate doing it. By watching her do that skill, I will adopt and learn. But what non-human primates don't have, which is what's very uniquely human, is learning from other people's imagined actions. This is sort of the key breakthrough that happens with language, which is the bandwidth through which non-human primates can communicate, what we're sort of calling knowledge here, is only through actions themselves. I can't describe, if I were a non-human primate,

• 2:00:21Speaker 01

  what I saw when I imagined five different ways to try and hunt the boar over there. I can just do it and you can learn from what you saw me do. But language enables us to share the outcomes of our imaginations. And so that is a way higher bandwidth mechanism for translating information. So for example, and so that enables accumulation across generations. So for example, I mean, it's so easy to think about ways in which this would be adaptive. You know, two would be sharing semantics. I go into the forest, there's two snakes there. One bites me and I'm fine, the other bites me and I get really sick. I come back and I say, green snakes are okay, red snakes don't go near. And that semantic knowledge now exists amongst the whole troop. In the old world, before there was language,

• 2:01:03Speaker 01

  you would have, only the people surrounding who saw this happen would have the knowledge. Now I can translate it just through, I simulate it in my mind, the episodic memory, and I translate it to everyone. The other is coordinated planning. So before language, it would not be possible for, you know, five humans to jump in trees and say, okay, here's how we're going to hunt these boar. We're going to stay silent and then I'm going to whistle three times and then we're all going to jump down and surround the one in the back. That type of planning is only possible because one person can simulate something and then translate and say, hey, when I imagine this happened, we succeed. And other people, of course, can edit that simulation and say, when I imagine that happened, I don't see us succeed for this reason. You can start refining. So the, this ability to have a source of learning

• 2:01:47Speaker 01

  from other people's mental simulations is sort of what I would argue is the source of this like very unique human superpower that emerges from language. And of course, now with such a high bandwidth transference of mental simulations, you do get this sort of quasi evolutionary process, which is what Richard Dawkins is talking about. Where you do have a process by which the memes, the ideas that do a good job propagating are the ones that will propagate and the ones that for whatever reasons are not viral and either don't do a good job of maintaining the host, so the ideas are bad and I end up dying, or I just don't have an incentive to share them, they're not viral, those ideas die. And so then you get this sort of meme evolution. But to me, the source

• 2:02:32Speaker 01

  is the fact that language enables us to share inner simulations, becomes a much higher bandwidth communication mechanism.

• 2:02:37Tim Scarfe?

  I'm fascinated by this idea of the meme itself being an agent, being a virtual agent. And in expressing its agency, it needs to manipulate us. So you might argue, as you do in your book, that there has to be some kind of traceable chain down to the basal ganglia. So we have many, many levels of bootstrapping. And at some point, the thing exists because the basal ganglia says, oh, that's good. I like that. So then we have one level, and then we have another level, and then we have like the memesphere. And it's almost as if that thing is manipulating us, because when you have weakly emergent macroscopic phenomena, part of the definition of emergence is surprise. So it's macroscopically surprising. So it does something which is completely unexpected and unlike the thing

• 2:03:21Tim Scarfe?

  that went below. And it's just kind of weird that it might be manipulating us down here, but doing something completely different up there.

• 2:03:27Speaker 01

  Hmm. Yeah, so I think there's probably, this is mostly fun speculation, but I think there's probably two ways, if I'm going to draw analogies to brain regions and intellectual features of the human brain. I mean, there's probably two lightweight ways we could think about why memes become sort of attractive. One would be sort of the older vertebrate-like structures. So this would be basal ganglia plus amygdala. And these are things, a meme that makes me feel fear, or makes me think that unless I take an action, something bad is going to happen to me, and one of those actions has to be sharing it. Then then that's going to be highly viral.

• 2:04:12Speaker 01

  You know, if you make me afraid for my family's well being, you're going to activate my my amygdala. And even if there's only a 2% chance that this is true, I might still share it. Right. So you get these sort of we're not good. Humans are not good at dealing with low probability, high magnitude events, which is another brain constraint. The other key thing that also exists at the level of sort of vertebrate, early vertebrate like structures is a preference for surprise. So in order for reinforcement learning to work well, it's very effective. And we see this in AI systems to make people pursue actions that are novel, because that's one way in which we can explore new areas and learn new actions that sort of explore the space of possible choices to make. This is one intrinsic way to get trial and error to work.

• 2:04:54Speaker 01

  So the way casinos make money from you is they hack into this sort of preference for surprise. So they have something, you know, if there's a 0% chance of winning, you would never play. But if there's a 48, a net 48% chance, meaning in the long run, you'll lose money. But every once in a while, there's some surprising thing that actually is over the threshold of being worth it to the basal ganglia. Because the surprise is so exciting is one way to think about it. So I think you get something that creates sort of innate fear or some great outcome or surprise. You get these older structures. I think with mammalian structures, I think there is sort of a active inference play here. I mean, you could even correlate it to granular prefrontal cortex with things like identity.

• 2:05:39Speaker 01

  Where if you give me some information that is consistent with the my model of myself. And I'm highly motivated to maintain my model of myself for a variety of reasons that, you know, we can talk about. Then I'm more likely to maintain this belief versus if you give me information that's inconsistent with my model of who I am. People are highly likely to reject these beliefs. So so this is another speculative way to think about why memes sort of persist within their little echo chambers. And it can quickly become sort of identity wars is if my if. One's identity is consistent with a certain set of beliefs, then that almost creates a gated

• 2:06:24Speaker 01

  wall for certain types of memes to enter. It becomes much harder for certain ideas to be entered into my mind, and it creates a very porous film for other types of ideas that are consistent with my identity, become very easy for me to adopt. And I think that's another, if we're going to frame memes as having agency, one strategy by which a meme would seek to survive is you find a way to be consistent with certain people's view of themselves and the world, so that what you're doing is you're reinforcing it as opposed to challenging it. Now, there is a very clear difference, though, which is the human brain is analog and these machine brains are digital. And there's pros and cons of each of these. So a digital brain, Jeff Hinton talks about this to make sure I'm citing these

• 2:07:09Speaker 01

  cool ideas. But he has a great talk, where he very well describes that the benefit of a digital brain is it's immortal, where all the weights are stored in sort of binary, so I can very easily transfer it to different brains. But it's hugely energy inefficient, because I need to model everything exactly to be copyable. But the human brain is way more efficient, but it's not copyable, because the information exists in the physical representation of the analog connection between all these neurons, the actual protein receptors that exist in them, the gene expressions, all this crazy stuff that makes it non-copyable. And so

• 2:07:50Speaker 01

  it'll be interesting to see what is the energy efficiency, for example, of an AI, digital AI system that actually does attempts to recapitulate a human brain, that might be very

• 2:08:04Speaker 01

  energy inefficient. And it might open the door for a whole new area of research, I think would be really fascinating, which is building analog brains. Can we have these systems that actually work in a more analog way? And the way they pass information to each other, this is also a Jeffrey Hinton idea. Is by teaching each other. And because they're AI systems, they can teach each other with better fidelity than a human, because they can actually share probabilistic outcomes opposed to just the words we say. I mean, they can generate way more samples for each other than a human could because they can live much longer, etc. But there's a whole emerging world of, you know, the distinction between these digital machines that are immortal, but very energy inefficient, and then analog machines that are much more energy efficient, but less good at translating information.

• 2:08:51Speaker 01

  And I, to one of your points that I think is really key is one of the main things missing that I don't think is talked about enough. And maybe there'll be a breakthrough soon. That would be great, but I don't see very clear ideas over the horizon that will solve this is the continual learning problem. I would say this is one of the essential lines that differs, that differentiates biological brains from modern AI systems, which is the way in which AI systems are trained are such that we cannot let them continuously learn from new experiences because it disrupts the old information they have. So whether that is an architectural constraint or something that needs to change in the underlying learning algorithm itself,

• 2:09:33Speaker 01

  lots of open research and debate about that, but the fact remains that if you allowed ChatGPT to learn from every chat that happens to it, it would get rapidly dumber. And that is not the case with humans. We can continuously update our information. Our representations are robust. And I think for many of the applications for AI systems that are going to be most impactful, continual learning is going to be an essential component because we're going to want to bring an AI agent in, show it new information and immediately have it incorporate that without forgetting old things. So I think that is very clearly aligned that there's a lot of really interesting research happening and a lot of research.

• 2:10:07Speaker 02

  Now, more than ever, we need someone on our side, but Rob Whitman? Whitman's votes drove up costs, cut Medicaid and risked 350,000 Virginians healthcare to pay for tax breaks for himself and his donors. He's not on our side. Shannon Taylor will fight corruption, cut taxes and bring down costs. This November, vote for Shannon Taylor. She's for us. Paid for by HMP, the HouseMajorityPac.com, not authorized by any candidate or candidate's committee. What's left to be done?

• 2:10:37Tim Scarfe?

  Why are we superior to animals?

• 2:10:40Speaker 01

  So there's been such a long history of

• 2:10:44Speaker 01

  Us pontificating on the various chasms or attempting to create a chasm intellectually between us and other animals. The most famous form of this, which I think still shows threads in modernity, is from Aristotle, where he sort of took the same kind of ideas that you see in a clean triune brain, where, you know, other animals might have these basic instincts, they might have some form of emotions, but what they all lack, which humans uniquely have, is this notion of reason. We can uniquely reason about things in the world. And I think, as I try to argue in the book, and I think most comparative psychology demonstrates

• 2:11:23Speaker 01

  quite clearly, is there's clearly forms of reasoning that we see in other animals. And so, you know, of all of the different abilities and capacities that seem unique to humans, the one that stands out as most salient is undeniably language. Because we, despite many painstaking attempts, have not even been able to teach chimpanzees, bonobos, or gorillas to speak with the same degree of fidelity of human-like language. Now, there is some controversy as to the extent to which did Kanzi and Koko and Washu pass the threshold that we define as language? And so that can be debated, where do we draw the line?

• 2:12:08Speaker 01

  But undeniably, most people would agree that these, you know, non-human primates do not learn language naturally without painstaking attempts to teach them, and when they do learn language, it does not show the same sort of flexibility as humans. Now, what makes language unique is two things. One is declarative labeling. So there is a distinction between imperative labels and declarative labels. An imperative label is learning that a phrase, a cue, leads, if you take an action in response to a cue, you get a reward. So when a dog responds to a specific cue with a response, and then you give them a treat, that's not what we define as language, it's an imperative label.

• 2:12:49Speaker 01

  A declarative label is when I say "dog," and in your head, you know that reference is a concept or a thing. Now we have a label for a concept or a thing. And so it's not at all clear that other species perform these types of declarative labeling, and if they do, it surely independently evolved. Like, we're quite confident that early primates didn't have this. Ability. The second thing that makes language unique is grammar. So we can take these declarative labels that reference things or actions and then we can weave them together in certain structure and the structure itself has meaning. So that basic example is just the ordering of phrases. So if I say Ben hugged James, that means something different than James hugged Ben,

• 2:13:33Speaker 01

  despite the fact that it's the same phrases or the same declarative labels. The order presents meaning. So there's a whole interesting world of why language? If it is the case that language is the fundamental difference, why is it language that has allowed humans to sort of take over the world? And that's another interesting topic we should discuss. But I would argue that primarily what makes humans different is language. And Aristotle's idea of reason, we see at least in smaller forms of other animals.

• 2:14:03Tim Scarfe?

  Yeah, I mean, it's quite interesting because you said right at the very beginning that, you know, yeah, Aristotle, he spoke about the rational soul that we have.

• 2:14:10Speaker 01

  Yep.

• 2:14:11Tim Scarfe?

  And even in the 20th century, you know, we spoke about things like mental time travel and our sense of self and tool use. And it's really interesting because we look in the animal kingdom and one by one, all of these things that we thought placed a bright line between us and animals faded away. Some people think that language is a continuum, that there's just a gradation. That, you know, if you scale up the brain of the ape, that you will get human language. Is that the case?

• 2:14:36Speaker 01

  So I'm, the reason I'm very skeptical of that claim is we don't see variance in language abilities based on brain size. Children who learn language at the age of four still have small brains relative. I mean, I actually, I'm not sure the exact brain size comparison, but I'd be curious the brain size of a four-year-old child relative to an adult chimpanzee, just based on, you know, volume. The other, the other interesting case is, yeah, homo flores, floresiensis. Yeah, homo floresiensis.

• 2:15:11Tim Scarfe?

  Oh, yeah, from Indonesia, the ones with the small brain.

• 2:15:13Speaker 01

  Yeah, yeah, yeah.

• 2:15:14Tim Scarfe?

  Yeah.

• 2:15:14Speaker 01

  Yes. So homo floresiensis is a great case study here. Because they, we found fossils of ancestral humans on Indonesia, who were effectively miniature humans. I mean, they had shrunk in size to, I think they were like three and a half feet to four feet tall. And their brain capacity, we can look at their fossilized brains, had shrunk actually from our ancestral humans. So they were marginally larger than the size of a modern chimpanzee brain. And yet, they showed a lot of signs of superior human intelligence despite having smaller brains. They showed tool use that was akin to ancestral humans.

• 2:15:54Speaker 01

  They had Aldwin tools, which is supposedly a sign of uniquely human intelligent tool making. And so that is suggestive of the idea that whatever unique intellectual capacities humans earned around 2 million to 1 million years ago, was present despite their shrinking brain. So either one has to argue that language evolved much earlier, which some people do, or whatever sort of proto-language emerged back then was present even when these brains started shrinking. Which suggests to me, which is actually aligned with the ideas in The Language Game, in my understanding of them, which is a great book, is that fundamentally, what's unique is we have an instinct to learn language.

• 2:16:39Speaker 01

  It's not that we have some unique capacity for language. And I think that is a key difference that we can talk about. Because when you look at children who learn language, there are two very unique features of how they go through language learning. By the young age of around two, they're already engaging in proto-conversations. So even actually a younger infant will pause, will match the pausing of their mother. So even if they're just babbling, they will engage in the synchrony of babbling time intervals. And so that is clearly a demonstration of some initial instincts,

• 2:17:21Speaker 01

  which demonstrates the ability for me to want to engage in some turn-taking action with you. The other unique thing that emerges a little bit later is joint attention. And where human children will uniquely attempt to get their parents to engage in attention towards the same object. And scientists have gone to painstaking efforts to demonstrate that this attempt to get a parent to engage in attention of an object is not the attempt to get the object. So a child or an infant will be dissatisfied if the parent doesn't look at the object, but they get the object. So a third party comes in and hands it to them, they're dissatisfied. If the parent looks at them and is excited when they're pointing at an object, the kid will also not be satisfied.

• 2:18:05Speaker 01

  But only when the parent looks at the object and then looks back at the kid and smiles is the child satisfied. So there's this instinct to engage in conversation and to jointly attend to things, which gives us the sort of instinctual foundation on which you can start adding declarative labels. Because when you have joint attention to something and you're paying attention to this turn-taking, it enables you to label things and say, well, this means run, or this means book. So I think, I think that all of that makes it hard to argue that it's just a consequence of a scaled-up brain.

• 2:18:41Tim Scarfe?

  One of the things that's really fascinating is that animal communication seems extremely superficial. And when I say superficial, I mean that when you take different populations of the same species or different species, the kind of the expression, the complexity, it's very, very simple. We don't see this incredible fractionation and divergence that we see in human language. And as you articulated just a minute ago, a big part of that is this declarative labeling, which is that, you know, one of the reasons presumably for language is the ability to do variable binding on symbols. So to say, you know, this thing is a dog, that thing's a bear, and to be able to dynamically manipulate that. It seems to me that you can think of language as a form of agentic communication.

• 2:19:24Tim Scarfe?

  So the difference between humans as languages is that we are agents, and agents is about being able to kind of like, you know, have your own directedness and plan many steps ahead and take control of your environment and so on. So the difference in communication with animals is the information content is more in the environment around them, whereas, you know, human languaging, a lot of it comes from the agent itself. So I just wondered whether you could think of any weird way to distinguish human languaging from animal communication?

• 2:19:53Speaker 01

  One line that I think there's some good evidence to suggest exists between human communication and non-human primate communication is humans have much more of a desire to share what's going on in our own mind. There is a unique pleasure we have from sharing our thoughts. And when we look at the communication styles that happens in non-human primates, there's much less of a desire, even for these, when we go through these language learning experiments where they have forms of communication, there's much less of a desire to share thoughts that are going on in one's mind.

• 2:20:34Speaker 01

  And one line, there's some controversy around this, is humans from a very young age will ask questions, they'll inquire as to what's going on in someone else's head. And with the exception of maybe Kanzi, who there is some argument maybe asked questions, you did not see non-human primates probe the mind of other individuals, even though we know they have theory of mind. We know when they're trying to deceive others, or they're trying to learn actions by observation, they clearly engage in theory of mind, but when it comes to language, they weren't interested in inquiring as to what someone is thinking about. And so I think in that sense, there's an agency to a language being a tool for inquiring as to what's going on in someone else's mind and sharing what's going on in your mind.

• 2:21:17Speaker 01

  And this is where I think language is in part of why language is a superpower, because it provides a completely unique source of learning and data, or data for learning. So, so non-human primates can engage in learning through observation, because I can see someone take actions, as you said, with imitation learning, I can see you open a puzzle box to get food, and I can learn from observing your actual actions. And so, and the way I do that is because I can infer the intent of what you're trying to do, and then I can figure out which of the actions are relevant and which are irrelevant. So a monkey and a chimpanzee, or an ape, will ignore irrelevant actions when they observe you do a task. So they've done these experiments with humans and chimpanzees where they do all these actions

• 2:22:01Speaker 01

  to open a puzzle box, and they do some random actions, and chimpanzees will ignore the random actions, which suggests they can infer the intent of it. Which is great, but chimpanzees don't learn from what's going on in your head. And so the ability to learn from from other mental simulations is what's so powerful about language, because I can say, you know, and I just went over to that forest over there, and I saw a red and blue snake, and I saw that the red snake is really dangerous, but the blue snake is not, because the blue snakes bit me and nothing happened. And so I share that that episodic memory, and now everyone has that knowledge, even though that was just in my mental simulation. Or when planning a hunt, you know, a group of five humans, I can imagine a strategy

• 2:22:46Speaker 01

  of how all five of us are going to coordinate, see it succeed in my mind, and then share the results and the plan with everyone. So language enables us to tether our mental simulations to each other. And I think there there is a sense of agency in the idea that there's a purpose to that. There's a volitional purpose to the communication. The neurological underpinnings of communication that occurs in non-human primates is more analogous to our emotional expressions than it is to language, and we see this also in the brain. So like, you know, monkeys and then and non-human apes have these innate expressions that they do, which are this genetically hardcoded, and we know that because it's the same in

• 2:23:30Speaker 01

  even across species often who have never interacted with each other. And it comes from neurological structures similar to our laughing and crying. And so this and so it's clearly a hardcoded emotional expression, and in that sense, it doesn't have the same volition, because I'm not doing this action to communicate a concept to you. I'm doing this action as an innate response to a cue or a feeling I have. So yeah, I think there's there's meat to that idea.

• 2:23:57Tim Scarfe?

  Well, a few few things to explore there. I mean, first of all, we should just talk about how we became a collective intelligence after the fact. So I'm not sure whether that's unique to humans. If you look at other forms of collective intelligence, there's always a kind of juxtaposition between the intelligence of the individual versus the collective. And actually, usually you find that having very intelligent individuals is not good for the intelligence of the collective. But what's interesting about humans is like, we clearly didn't evolve as a collective intelligence. So we had this kind of bootstrapping process where we were very, very useful, you know, independent agents. And then this collective intelligence just emerged out of nowhere. So is that is that an interesting observation?

• 2:24:35Speaker 01

  There's different degrees of collectiveness. And so I think we can draw distinctions between different flavors of collectiveness, but I don't think humans are uniquely collective. So for example, the imitation learning of non-human primates is a form of collective intelligence, because you can teach one member of a chimpanzee troop how to use a tool and then over time, the rest of the troop will learn just through observation. So that's a sense of collective intelligence. Many vertebrates, and likely the first vertebrate, you can even see fish, will learn through observation. In other words, when a fish swims in a certain direction to get food, other fish can see that fish do that and follow them.

• 2:25:17Speaker 01

  There can be an instinct to follow others around you.

• 2:25:24Speaker 01

  I think there is flavors of collectiveness that exists across many different species. But what's unique about the collectiveness in humans is the fidelity with which we transfer our mental simulations enables it to accumulate across generations. And in that sense, it has almost its own, you know, you could argue it becomes, it has its own agency or is its own thing, because it can actually go through its own process of evolution as ideas propagate through generations of people, which is not the same thing that you see in other animals.

• 2:25:59Tim Scarfe?

  Yeah, I mean, a couple of things on that. I mean, first of all, I would quite like to distinguish knowledge and intelligence. So collective intelligence and intelligence in general is a process of discovering models. And when I, just to sort of get the language down here, I'll use models and skills and knowledge pretty much interchangeably. So I think of like an intelligent process as epistemic foraging. So just finding interesting models and then they can be discovered and shared by other people. So it's a little bit like when you distribute a GPU workload, you can do like model parallelism and you can do data parallelism. So you can either split up the CPU, like, you know, the processing, or you can split up the actual representation.

• 2:26:44Tim Scarfe?

  So I think the kind of collective intelligence that you've just been speaking about is, okay, we've got all of these independent agents and they are finding models and sharing models and, you know, the models get refined over time and it adapts and so on. But I also think a big important element is sharing the computation. So even though there's some kind of redundant work going on, but you know, epistemic areas over here are being explored, but also in many cases, the same problems are being explored, but in slightly different variations. So we're kind of sharing the workload with other humans.

• 2:27:16Speaker 01

  Yes, I think that totally makes sense. There's some interesting ideas in AI here, actually, where there's this concept of knowledge distillation in AI, where one way in which you can have a model A teach model B the things that model A knows. One way is you can wholesale copy the parameters of model A. Of course, that's totally biologically implausible. There are aspects of parameter copying, which is the components of our brain that are genetically hardcoded. That is a version of parameter copying. But for other applications, it's not feasible to just copy parameters, or it's maybe not desirable. So knowledge distillation

• 2:27:56Speaker 01

  is saying, okay, well, we can have a set of data that we give model A, and we either look at the outputs of model A or the layer before the outputs. So we can see sort of more richness in its representation of the input you give it, and then take that data, that almost labeled data, to model B, and then train model B on it. So that's distilling some of the knowledge through almost training model B to try and act similarly to model A. And so that type of information transfer, I think, does occur in non-human primates,

• 2:28:34Speaker 01

  and that's imitation learning. However, it is not nearly as rich, because what happens in non-human primates is it's primarily grounded in just the actions that I'm taking, which is much less rich than I can share not only the data of what you see me actually do, but I can also include in the data I'm transferring to you, things that happen only in my mind. And that opens the door for much more transference of, and the word you used, the computations that I'm performing.

• 2:29:06Tim Scarfe?

  Yeah.

• 2:29:07Speaker 01

  So I think it's absolutely, absolutely true.

• 2:29:09Tim Scarfe?

  Before, we were Learning in the physical world, so we were learning from physical things that we were directly observing. And now we are learning from imagined actions. But there's a bit of a latent component to language as well. So for example, someone might come up to me and say, oh, the blue swirly thing is over there. And I'll say, well, I don't know what you mean about the blue swirly thing because I've I've never seen one before. So there's this kind of inference process. And this is where it starts to get really interesting because there's a diffusion, right? There's a kind of, there's a message passing that happens between all of the different agents and it's filling in missing information. So even though many of the agents wouldn't have seen anything like what we're talking about, sometimes it can be filled in with subsequent interactions with people.

• 2:29:52Tim Scarfe?

  And sometimes it can just become a kind of latent category which can be filled in later. So there's this real diffusion process going on, which I think is quite difficult to articulate.

• 2:30:02Speaker 01

  Part of what's so interesting about language is it's still an area of such controversy amongst cognitive psychologists, linguists and even AI people. And so much is still, you know, unsettled about it. There are still, you know, debates today. I mean, there's debates today about whether language is primarily a tool for thinking or communication. And and most people were, you know, Chomsky is the most famous proponent of the idea of language for thinking. So, you know, and he has evolutionary arguments that language initially evolved not as a tool for communication, but for our own process of thinking. And then later was accepted or used for communication. That's a minority view.

• 2:30:47Speaker 01

  And then other people argue, which I I'm more amenable to, that language was primarily used as a tool for communicating. And these ideas actually are reemerging with language models. Because the way language models learn about the world in some sense is language becomes the reasoning tool itself, which is more Chomsky like, even though, you know, there's a lot of.

• 2:31:13Speaker 01

  You know, I think the successive language models, I think, in a lot of ways, discredits a lot of Chomsky's ideas, and we can talk about that. But interestingly, the fact that we're using language as the fundamental mechanism for reasoning and thinking is actually somewhat Chomsky-like. Versus "language is communication," the idea is, "language is a condensed set of tokens that I'm passing between minds." But the goal, the real communication I'm trying to share with you, is what's going on in my mind. In other words, the mental simulation, the more mammalian component here, the rendered 3D world is what I'm trying to transfer to you, and I condense it into this code that then you, you know, reverse engineer back into a mental simulation.

• 2:31:57Speaker 01

  And a theory of mind, one reason why language might be so rare in the animal kingdom, is mentalizing theory of mind, which is relatively rare in the animal kingdom, is a prerequisite. Because in order for me to reverse engineer the language code you've provided me, I need to be able to infer what you might have meant by what you're saying, and reason about why you would have said this, and what knowledge you have, and et cetera, et cetera. So, yeah, so I think language is intended to cue to another person to render something in their mind. This is also where, you know, teaching is so important, is such a key aspect of language learning, because we can infer, you know, what declarative labels does this person aware of?

• 2:32:39Speaker 01

  And when they're confused, then you can start, you have to start trying to iterate to understand what are they confused about that I'm saying, that I can disambiguate for them. So there's also a disambiguation process where you ask follow-up questions when you feel like you don't fully understand what's going on in someone else's head.

• 2:32:56Tim Scarfe?

  Yeah, I mean, the guardrails thing is interesting, because they are not necessarily thinking guardrails, they're also pragmatic guardrails. And there's a really interesting figure in the book, actually, yeah, here it is. And it talks about how language is sharing information over generations. So without language, you know, we learn a little bit inside a generation, then it goes to, you know, pretty much back to zero again. But now we have the ability to pass on these memetic bits of information over several Generations. But the thing is, there's a real structure to it. I think of it as a bit like, you know, um, a directed acyclic graph. So it's a tree structure. And every single bit of knowledge that we discover kind of stands on the shoulders of giants. So it needs all of the things that we

• 2:33:37Tim Scarfe?

  discovered beforehand. So in a sense, you know, we're all of these little agents and we're doing this epistemic foraging. So we, you know, we're finding new skill programs. We're sharing them and so on. But it's almost like we shouldn't think of the mass as being like an entire convex whole. It's only on the boundary where all of the creativity and and all of the information sharing happens, you know, like on the surface of this object that that's being created. And what I mean by that is like now in modern cities, for example, you can't live without a driver's license. You can't live without the internet. You need to do things a certain way. And even though it's not technically constraining our brains and how we think, like, we live in a very, very constrained and weird world now.

• 2:34:17Speaker 01

  Yeah, the great, totally great point. There's um, we are, there's a biological constraint as to how much knowledge a given human brain can contain. And so one lens through which to see the last, you know, hundred thousand years, especially the last, you know, hundred years is us finding solutions to getting past the biological constraint of human brains. Language was one tool because it used to be the case that uh all of the information that a given entity learned needed to be learned by my brain um, within my lifetime. And language enables us to, our group, to have shared knowledge, but not

• 2:34:59Speaker 01

  every brain contains all of the knowledge. So if you think about a troop of a hundred, a hundred people, it's possible for that hundred people and all their descendants for a thousand years to have tons of skills, despite the fact that any one brain never had all of the skills. So someone becomes really good at, you know, uh, hunting. Someone becomes really good at weaving uh, animal skins into clothing and all of these types of skills. And actually, there are cases um, in anthropology. Of groups of humans that get separated from each other of their technology degrading. Because there is a limit, there is a minimum number of brains needed to contain and store a certain amount of information in the absence of writing. And so writing, language was maybe innovation one here, writing was another innovation, which

• 2:35:44Speaker 01

  is great. Now we can more reliably transfer these ideas across generations, even if there are gaps. In other words, even if there's a period of time for maybe two generations, no brains contain it, a third generation can go back to the writing and pick up that knowledge. And then, of course, now with, you know, with the internet, we have just scaled up writing even more. But you're absolutely right. You know, sometimes I think about this as like, you know, if me and a group of 20 friends ended up on an island, how much of human, we were the only 20 humans left, not that I think about this all the time, but it is, it is crazy how little of human knowledge would be contained in our 20 brains. How dramatically we would degrade.

• 2:36:27Tim Scarfe?

  Essentially, like, we've got this thing where we've got all of these different brains and individual people can, you know, have about 150 friends or something, you know, it's the social Dunbar limit. But as you say, because we have like this ability to share simulations and we have common myths and so on, we can actually address a much larger carrying capacity of people and, and knowledge. And actually, you said something really interesting in the book, which is four things, right? So bigger brains, specialization, more brains, bigger population size, and writing and sharing simulations and the internet and all of these things. So, so we've increased our carrying capacity and like now something very interesting and arbitrary has emerged. So we've got all of these different specializations of skills.

• 2:37:09Tim Scarfe?

  And I guess the question is, where does it end? Has it converged? Like, could, could we, could we carry much more knowledge than we already have? Or would we have to wait for a top-down kind of genetic pressure for our brains to get a bit bigger again?

• 2:37:26Speaker 01

  I mean, I think we are, um, we are about to go through this. The Google and the Internet has become, has turned us all into epistemic hybrids. I mean, our, Google has become a shared knowledge store that we all use. And of course, there's problems because now there's sub areas on the Internet where we can use different knowledge stores. And now we live in, you know, uh, these different epistemic bubbles and that creates political problems as well. But we have already become hybrids where we use technology to be, to, to overcome limitations in our own brains. Writing is a tool to overcome challenges in memory and at times thinking.

• 2:38:08Speaker 01

  The Internet has become, uh, you know, a tool to answer any question at a whim. And, and some people have concerns with this because it can also atrophy parts of our brain that maybe we want. So for example, um, through mere introspection, I will say, once I started using Google Maps as a kid, um, my, the part of my brain that was learning how to navigate a city through, you know, actually remembering the grid and map of a city just started atrophying. Like now I have no capacity to do that. Whereas my dad, you know, he'd take him to any new city and he's like, you can see him rendering a map of the city in his mind and he won't use Google Maps. Now, now one could argue that it doesn't matter because I'll always have Google Maps. So why do I need this skill?

• 2:38:51Speaker 01

  And then another argument would be, um, that atrophying may have other consequences in my life and it would be important for me to go through the cognitive exercise, even though technology enables me to do it. And, and we do make these trade-offs at different times. Why do we teach kids arithmetic? They can always just use a calculator, but we deem it important for us to go through the process of understanding arithmetic, even though technology can already do a better job for us. And so this new frontier with, you know, using large language models and there's some really cool things with education happening, like, uh, you know, in Khan Academy, they're working on building language models to help children go through reasoning steps, which is a really cool application because instead of just asking the question, it'll probe the student to go through a process so they can come to the conclusion themselves.

• 2:39:37Speaker 01

  And so, so there's a, there's a sort of pessimistic and optimistic world here. An optimistic world is these new AI systems are actually gonna be a new step forward in sort of cyborgizing ourselves, but it's not necessarily gonna be as atrophying as something like Google, because these systems won't only just get us so the dopamine hit of a factual answer, but will also guide us towards better understanding how it came to this conclusion to ensure that we understand when we're probing and asking questions. That's an optimistic state of the world. The pessimistic state of the world would be that we offload more and more of our own

• 2:40:18Speaker 01

  sort of cognitive reasoning to these systems and we become even more atrophied in these abilities and that might be, you know, not a good world we want to live in if we keep offloading more and more of reasoning to systems that, and we lose the ability to do it well ourselves.

• 2:40:34Tim Scarfe?

  Yeah, so I've been thinking about this a lot recently. So, I mean, I was involved in a startup that did transcription and large language models and augmented reality glasses. So the idea was, you know, you can be in a lecture, and by the way, I still think this is very useful for people with, you know, accessibility concerns like hard of hearing or something like that, but, you know, but, you know, we were kind of thinking of it as something which can augment your cognition. So you're in a lecture and now you don't need to pay attention to the lecturer because, you know, you're transcribing and GPT is making notes for you and so on. And I think this is really wrong, but you give a counter example of Sat Nav, so we don't need to read the maps anymore because we can now externalize that cognition. But I feel like this is different.

• 2:41:15Tim Scarfe?

  So you're in a, you're in a lecture or something like that, and now all of these kind of AI language tools, they are a form of understanding procrastination, right? So, understanding or intelligence is, is the process of creating a model. So you're creating a simulation and in order to create a simulation, you actually have to think and you, you know, you normally think you externalize the thinking a bit, you do some writing and you pay attention. Now, here's the thing, in the situation, there are so many more cues, right? Because it's in 4D, you can hear things, you can see things, it's a social activity, it's a physical activity, even the The sort of the the dance, the the performativity of the of the lecturer, it's all information. It helps you understand. So now I'm transcribing the thing and people say, oh, it's okay. I can I can just read the transcription later and I can understand it.

• 2:41:59Tim Scarfe?

  Well, yeah, maybe, but you're already at a disadvantage and you probably won't because this procrastination, you're just like, you're paying it down the line. You're saying, I might do it later, I might do it later, and you never will. And that's going to create a society of automatons that just don't think for themselves.

• 2:42:15Speaker 01

  Yeah, well, I think that's a very, uh, I'm sort of torn between the optimistic and pessimistic uh state uh of the future, but I think there's a very good argument behind what you're saying. So I'm not, I don't, uh, I definitely don't reject that out of hand. I think the, I actually really liked the um uh analogy to sort of model based versus model free that you were suggesting there, because that is, actually applies very well towards Google Maps. Because when my dad navigates a city, he has a model of the world and he's engaging in this model based planning of how to get somewhere. When I use Google Maps, I've externalized the model and all I do is respond to the queue of when do I turn right or left. And so, and so I think that is absolutely a good way to to think about this, which is,

• 2:42:59Speaker 01

  we use technology to externalize building models, which can sometimes make things more efficient, because then we can just be model free actors. But there are places, like in the example you're suggesting, where we really want people to engage in the more painful, hard process of building models of things. And in those cases, uh, you know, obviously it's dangerous to make it so easy to externalize these models.

• 2:43:22Tim Scarfe?

  Yeah, I mean, it's it's hard to articulate. I I think part of it is, um, it's a kind of acquiescence. So I think you're sequestering your agency when you externalize too much of your cognition, particularly if it's parts of your cognition that are useful, in the sense that it has core knowledge which will generalize and help you acquire new knowledge, or um it's just the the proto ability of, you know, um discovering knowledge. It's just your intelligence and you're not exercising that muscle. So you become acquiescent and then you become less of of an agent. And from a collective intelligence point of view, I think this, you know, we're just saying like language and intelligence is a bad. Discovering knowledge and if we are all sequestering our agency and becoming less intelligent as individuals as a collective, maybe we will suffer. But it's one of those things that it's so easy for us now just to make grand statements about this.

• 2:44:07Tim Scarfe?

  And people in 200 years will look back on this and just laugh and say, oh, you know, it's a little bit like when when they introduced bicycles. Um, they were saying, oh, bicycles are there. There's a moral panic, apparently, because they said, oh, women will start cheating on their husbands and using the bicycles to go to the next town.

• 2:44:24Speaker 01

  You know, that is an interesting fact. Yeah, well, I think um, I I do. I think history is such a good. I love thinking about history as a tool when trying to reason about how people in the future will think about us, because I think, you know, we are the people in the future to the past, which is obvious, but um, a useful tool. So, for example, in some sense, we already live in this dystopian world when it comes to physical exercise. I mean, roll back the clock 500 years. And most people, most people uh, didn't have to think about physical exercise as much because most work required physical exercise. So we exercise with our work. And so much of jobs, uh, at least in the developed world, are information related jobs where we don't exercise. And so we go to the gym. I mean,

• 2:45:07Speaker 01

  the gym is a is a weird, if the like, if aliens came down and observed gyms, it would be an anthropologically very bizarre behavior. Because we just go into a room and we run on treadmills. And we do it because obviously we've evolved to require exercise. And uh, modernity has removed exercise as a prerequisite to most of the things that we need in life. But now there's this gaping hole. And so what we do is we just go to the gym and we run in place uh, to satiate this physical need. And so you could imagine, and one might interpret this as dystopian or utopian, but you could imagine a world where we've offloaded so much cognition. But because humans need to think about things, um, or as a society, we value it the same way we value physical fitness.

• 2:45:49Speaker 01

  That there is now social pressures to go to these intellectual gyms. Um, just to make sure, even though, you know, you don't need to do it for work, or it's not necessary for the world to to function. But we feel like there's just value in a human who knows how to reason about things. So we just, you know, we go to intellectual gyms for that. And so we might, I don't know if that's utopian or dystopian future, but however we feel about it, I would venture to guess people 500 years ago who looked at a treadmill. Would probably feel similarly.

• 2:46:18Tim Scarfe?

  Hundred percent. Well, MLST is my intellectual gym, by the way.

• 2:46:22Tim Scarfe?

  You spoke about DNA, so Dawkins, of course, wrote this book, The Selfish Gene. And you said, actually, that the value of DNA was not what it creates. So, you know, it creates hearts and lungs and so on, but what it enables, which is this evolution process. But then it gets to this concept of what we mean by a meme in general. So you said that it's an idea or behavior which spreads contagiously. I mean, how do you think about memes?

• 2:46:48Speaker 01

  Well, I think Dawkins did a wonderful job articulating this idea in a way that's really understandable. Where a meme is a concept or a behavior. So a meme can be just like the idea that individuals should have rights. Or the idea of equality, or something sillier as the idea that we shake hands before we sit down for a meeting. And these things, because humans can share simulations through language and we engage in imitation learning. These ideas or behaviors propagate throughout societies.

• 2:47:27Speaker 01

  And because these things are propagating, a different form, not evolution in the sense of, you know, genetic evolution, but a form of evolution emerges because some ideas will propagate better than other ideas. So by nature of that process unfolding, memes, these concepts or behaviors, actually go through an evolutionary process. So ideas that either are viral because people want to share them with each other, or ideas that somehow support the survival of the individuals that hold them. Those are going to be ideas that propagate correctly. Ideas that negatively affect the survival of the individuals that hold them,

• 2:48:10Speaker 01

  or for whatever reason, people do not desire to share, or ideas that are going to do a worse job propagating. And so it's a really, almost brilliant lens to look at human culture when you reframe sort of cultural ideas and concepts as memes, a different take on a gene, that goes through its own sort of process of iteration. Which is not my idea. This is Richard Dawkins's.

• 2:48:37Tim Scarfe?

  Oh, yeah. Well, we can we can thank Richard very much for this. But no, I'm I'm fascinated with with memes. And I kind of think of languages as being a collection of memes. But now we're in this very, very interesting space, right? So, um, you know, before before language, we learned by, you know, observing physical skills as performed by other people. And we could kind of imitate them and so on. Now we are sharing, um, you know, kind of simulations, basically, without actually needing to see the thing. And that means that we are kind of one step removed from reality. So all sorts of memes have cropped up, and some of them are better described, as you say in your book, as shared delusions. And but they have some utility as well, you know, so they they have this, um,

• 2:49:20Tim Scarfe?

  you know, when we have a common myth, for example, it might be a religion, it might be a nation state. It allows us to kind of cooperate with each other in in a way that we we wouldn't be able to do before. And you actually cited, um, uh, some some ideas by John Searl and and Yuval Harari in his book Sapiens on that.

• 2:49:37Speaker 01

  Yeah, they both have great, they both, uh, you've all you've all famously like popularized this idea. Um, but Searl was one of the original ideators of it. Um, but what's so powerful about these sort of shared fictions is they can propagate much more easily than a human can talk to everyone in a group. And so because they propagate much more easily and with very high fidelity, it enables me to meet someone who is a a New Yorker, who I never have met before, and immediately have shared views. We we probably both believe in individual rights, we probably both believe in, uh, you know, that money can be used for transacting things.

• 2:50:21Speaker 01

  So I if I give them a dollar, they'll believe that the dollar will be used elsewhere, or they can give me a dollar. Um, and you can, and of course, today it's hard to reason about these things because there's so many rules in place that you don't realize it's all a shared fiction. So, you know, the reason we think we believe in money is because we're like, well, I know that all the other stores I go to will take this money. So that's the reason it works. But why do they all take the money? It's this all this this just shared belief that we all trust that this thing will will be used, uh, for transacting. And so because of that, it enables really large groups of people to coordinate. Um, and and and that is a very powerful aspect of language.

• 2:51:02Speaker 01

  But the sort of the argument I make in the book, which is is is that similar to how genes are powerful, not because of the structures it creates, but because it enables a process of evolution by which good structures will emerge. Language is similar in that sense. It's not what's powerful about language per se is not that we can engage in these shared simulations for coordination. It's that language enables the propagation of ideas and concepts across generations, which will thereby go under its own evolutionary process. So of course, these good ideas that enable survival are going to emerge. And that's really what's so powerful about about language.

• 2:51:42Tim Scarfe?

  I guess like the arbitrariness is quite interesting. So some of them on the surface don't seem like good ideas. They just seem like really bad ideas. And there's this kind of, I mean, I guess you can think about it in terms of creativity as well, which is like, you know, for a meme to be established in the sphere of possible memes. Does it need to have intrinsic value? And possibly not, you know, because we're getting into creativity. Is it novelty? Does it have intrinsic value? Is it just social proof? You know, is the meme, does it only exist because lots of people have been fooled into thinking it has value? So it's kind of like extrinsic value via social proof. And then there's like almost a double entendre with the meme or a deeper meme meaning because you talk about altruism. So the meme itself might actually be quite a stupid meme. But

• 2:52:26Tim Scarfe?

  if it causes altruism, so there's actually a group selection advantage to it, then it's almost like that's the lens of analysis to understand how good the meme is.

• 2:52:34Speaker 01

  Yeah, the so there's so much like, it's a really fun area of literature to read through because there's still like zero consensus as to how language evolved. And the one reason why it's so controversial is the way in which we disambiguate, and I'll get to your question. The way we disambiguate evolutionary arguments is typically by observing

• 2:52:59Speaker 01

  Uh, gradation in extant or currently present animals. And so that enables us to observe these intermediary steps between, you know, morphological or uh, sort of aspect of, of the body A and morphological aspect of a body B. And the problem with language is we have non-human primates that for the most part, don't have any language. And then we have humans that have very complex language. And all of the intermediary humans that existed between uh, our divergence with chimpanzees about six million years ago, and our divergence with all, all other modern humans between 50,000, 100,000 years ago, um, we don't have, they're all dead.

• 2:53:39Speaker 01

  All those lineages are lost. And so that means that there's this broad spectrum of, of arguments that could be made between, uh, you know, Chomsky argues, I find this a very strong claim and thus hard to, to defend, but there are people that argue that it happened all at once, or like very rapidly. There was like no language, then all of a sudden there's language. Prometheus. And then there's, right. And there's other arguments that it was a sort of gradual, gradual process. But the, one of the most controversial aspects of language evolution goes to what you're talking about, which is evolutionary arguments for why language evolved has a, a, almost a harder burden of proof than other adaptations.

• 2:54:17Speaker 01

  So when we argue um, uh, about the evolutionary benefit of something like theory of mind, um, there's no complex evolutionary machinations one needs to conceive of to defend it because you can see why it would be beneficial for an individual um, uh, chimpanzee to be born with the ability to infer what's going on other people's heads, because they can better defend themselves when someone is going to be mean. They can better figure out who to trust. They can better climb the social hierarchy, et cetera. But language, unless you may take the Chomsky view that its primary adaptation is for thinking. Um, but if you hold the argument that uh, uh, language evolved for communication, that's more challenging because it's, it's not valuable for an individual human to be born with a

• 2:55:02Speaker 01

  little bit of language skill unless other humans are also engaging in language as well. And so this then means that the only benefit is if we're both sharing true useful information with each other. And although it seems intuitive that like, well, the way this would function is that a group of humans that are sharing knowledge with each other is going to survive better than another group of human that's not, and that's how evolution will ensue. This is actually quite controversial in evolutionary biology because that's invoking something called group selection, which now some people, the modern incarnation of this, something called multi-level selection, where there's some consensus that yes, there are group

• 2:55:40Speaker 01

  level effects where that kind of can impact things, but it's, but most people think that group level effects are not nearly as strong as we would intuit. And the issue, the issue is the following. If you have a group of 100 humans that use language with each other, and then you have one human that's born that actually is just going to try and trick all the others. So all they're going to do is use language just to be disingenuous. It's not at all clear that that human would be at a disadvantage. And so if people then start, in fact, they might be at an advantage relative to everyone else. And so if you play that forward over time, language will be lost because the someone

• 2:56:22Speaker 01

  born that's not going to be tricked by the individual that's trying to lie to them with language is actually going to survive better than the people that have language skills. And so, and so there's been so much debate throughout linguistics, evolutionary linguistics about these arguments as to how language evolved. I like the sort of, there's a great book called The Evolution of Language by Fitch, and I think he makes a really great argument around how, you know, you could think about this occurring. But, but a lot of people argue that it probably started with something called reciprocal altruism. And so the way altruism exists in the animal kingdom, there's two forms of accepted altruism. One is something called kin selection, which is quite straightforward.

• 2:57:05Speaker 01

  I'm going to, I'm willing to sacrifice something, other words, share something with an individual if I share genes with them. So, you know, that's easy. Reciprocal altruism, we do see in the animal kingdom, which is I'll scratch your back if you scratch my back. But if you start not scratching my back, then I'm going to stop scratching your back. And so what this suggests is in order for language to be stable, in other words, it be beneficial for me to truthfully share information, there needs to be costs to me lying. And so this is one argument that people speculate this is one reason why humans have such strong sort of moral preferences towards punishing of liars and outgroups

• 2:57:50Speaker 01

  and in-groups, because what we do is we really try to identify individuals that are lying. Robin Dunbar has a beautiful argument that this is why gossip evolved. One way that evolution can stabilize using language is by virtue of us having a preference to share moral violations. And so gossip being a tool of language where if you see someone lie or cheat and you share it with a bunch of other individuals, that becomes a huge cost to someone lying and cheating. Because if one person catches them, then the whole group is aware of it. And so there's this sort of special feedback loop that happens where language skills require

• 2:58:33Speaker 01

  more punishment, a violation, to be a stable strategy. And then the one way you get that is by having more gossip and making sure there's higher costs to defecting. So this is not by any means like the only story of language evolution, but it's one that there's, you know, a lot of interesting evidence behind. There are some people that argue that the feedback loop, one emerging idea, which I don't talk about in the book, which I do think is interesting, is that the feedback loop of language evolution is actually one in which we try to detect lying in others.

• 2:59:13Speaker 01

  And they make the counter-arguments to me saying that the effect of lying is loss of language. There is an argument that you get the reverse, where you get really good theory of mind in humans, because we're so sensitive to trying to detect people who are actually giving us false information. So still a lot of controversy around it, but the main takeaway is that the whole, the blanket group-level selection arguments, that language is obviously beneficial, because once a group has language, they're all going to survive better, is Not a sufficient argument for a language evolution. You need a more nuanced evolutionary argument as to why it's a stable strategy for an individual to be born with superior language skills. Or you have to argue that language did not evolve primarily for communication.

• 2:59:58Tim Scarfe?

  You know, that was quite interesting, first of all, that you were writing this book actually a couple of years ago. So this was before GPT four, although you did put a note in about GPT four.

• 3:00:05Speaker 01

  Yeah.

• 3:00:05Tim Scarfe?

  And you were speaking about Blake Lemayon. And he was this Google engineer and he famously came out and, you know, he was convinced that these things had developed sentience. I think much of this actually hinges on this concept of a world model. So one view of language models is that, you know, they're just modeling a statistical distribution of tokens. And that seems quite low resolution, like another take is that they're learning a world model. And what that means is that rather than just kind of capturing the the state of language, they are actually, they're simulators, they're generating the underlying processes of language. They're capturing the dynamics of language. How do you say that something is or is not sentience? Especially given that the models could potentially be so high resolution that they are generating the same thing for all intents and purposes.

• 3:00:50Speaker 01

  Yeah, this is where I don't see myself as a philosopher, but this is where I do think scientists need to include philosophers because when questions become non-scientific, I think the scientific instinct is to argue that we don't draw distinctions between things that scientists, that the scientific method can't draw a distinction between. But the problem is there might be moral differences between them. So for example, it might be scientifically impossible for us to have, we have no methodology for differentiating whether two systems that look indistinguishable in their outputs and inputs, which one is sentient? And so scientifically, we might say, well, for for the, because we can't differentiate the two, we're going to say they're the same. But that doesn't mean they're the same.

• 3:01:33Speaker 01

  That just means that because we have no methodology for for drawing a distinction between them, we're from a scientific perspective, we're not going to draw a distinction because we're entering philosophical territory. But if you if you take that and then you start talking about policy implications and the actual values we attribute to them and how we introduce these things to society, I think we need to include a sort of a philosophy lens here. Because it might not actually be the case that they're the same, just because we can't distinguish them. So that's just one thought. Other thought on world models.

• 3:02:07Speaker 01

  So, so, one distinction I want to draw, because I've seen a lot of confusion on the internet about, about the world model dilemma. So, there's a difference between a world model and a model. It is undeniable that language models have a model. And all that means is clearly, in order for GPT-4 to correctly predict the next token in these really complicated language questions, it clearly has some model of something. And because we can ask common sense questions about the world to it, and it answers many of them correct, you can say this is a model of aspects of our world, without question. I think that would be very hard to argue that's not the case, if you look at the performance

• 3:02:50Speaker 01

  of GPT-4 on many of these, these questions. But what most people mean when they say "a world model" is they mean a specific process of stimulating and ordered, simulating and ordered states of consequences of different actions, and identifying the, the end results of these actions in your head. Another way to think about what we mean by world model is the ability to reason about interventions and causality. And so, so, uh, this is the Judea Pearl sort of argument here, which is, um, with our world model, we can hypothesis test. I can say, I imagine that if I do this thing in the world, I think this will be the consequence of it, because that's what I see in my head.

• 3:03:35Speaker 01

  Now I have a hypothesis, now I'm going to actually do that thing in the world, and see if my hypothesis is correct. And so, that's very different than what's happening in a language model, where its understanding of the world derives solely from its input data. Versus in a world model, my understanding of the world comes from the delta, the difference between what I hypothesize is going to happen in the world, and then my actual experience of it. And this distinction really matters, the more we are going to start offloading our cognition to these systems, because, for example... Everything that ChatGPT knows is on the basis of its input data. And so that means if false information or wrong information is in the input data, ChatGPT is going to know that information.

• 3:04:19Speaker 01

  There's absolutely no hypothesis testing embedded into ChatGPT. Versus our true AGI agents that one will be invented. What it would do is it's going to hypothesize aspects of the world. And it's going to test its own hypotheses. And so if you give it false information, if it reads articles about how the earth is flat, it's not going to just start talking about how the earth is flat. It's going to say, okay, well, that is incongruent with my model of the world. I'm going to now render some tests where I could differentiate them. And I'm going to perform those tests and then I'm going to conclude that the world is not flat. So when one says that ChatGPT does not have a world model, I think some people misinterpret that as suggesting that it's just dumbly looking at the statistics.

• 3:05:03Speaker 01

  And I think that's not at all what we're saying. In order to correctly look at the statistics of language, clearly it's built up a very rich and complex model of the text that it's seeing. And that's how it's able to predict the next word so well. But it's not what most people mean when we say world model.

• 3:05:18Tim Scarfe?

  Yeah, I mean, a couple of things on that, yeah. I think people conflate the machinations of language models with how we represent them statistically or abstractly. Because if you look at a lot of papers, they actually represent it like a probability, you know, like a joint probability distribution. And of course, you know, the way that language models work is completely different to that. But you were bringing in some very interesting things. So first of all, we are agents in the world. So the agential lens is quite interesting. We interact with the world, so we're not just learning from observational data. It's quite interesting, actually, I was talking with Nick Chater. You know, we said, oh, why is it that in our everyday experience, we see, we experience the world in 4D color? And he said it's because it's interactive. So in your experience plane, you can actually seek new information, right?

• 3:06:02Tim Scarfe?

  You can circade your eyes, you can get new information in, you can touch things. And when you're doing future or past simulations, you don't have that interactivity. So there's something about interactivity which is really important. But even then, right, you know? We could, you know, how far, how far could you go? So a complete one-to-one simulacrum of the world wouldn't be a particularly good model. And in, in physics, there is no causality, right? So it's just dynamics. So causality is actually something which emerges very, very far up. So we're talking about a model which is an approximation of the real world, which may or may not include causality. It probably would because it's an interactive model and it has this kind of agential map. But I guess we're just kind of drawing the line somewhere and we're saying, well, that is a world model. And yeah.

• 3:06:46Speaker 01

  Well, I think the, um, it's, I'm actually, uh, I'm not sure if, even if we rendered, let's think about it. If we rendered a perfect three-dimensional map of every particle in the universe and, and that was the input data to some infinitely large model, I still would argue that it is learning something different than a model that is given some form of agency where it can hypothesize rules and then test its own rules. Now, given infinite time, it is possible that those will converge because given infinite time, every possible hypothesis I could conceive of will end up showing up in the training data. So eventually I'll see the training data of every possible experiment I could run.

• 3:07:32Speaker 01

  If, yeah, if time is infinite, I guess you could suppose that happens. But what's so different is there's this dramatic dimensionality reduction that happens when you show me something uncertain and then I can conceive of specifically the tests I want to run to map the uncertain thing to my mental model of the world. And so that's a very different way of learning about things. It's not just input data and then, and then self-supervising on predicting one's own input data. It's building a model that can, I can simulate possible outcomes and then hypothesis test those outcomes. And so I think, you know, when we look at even, this is not uniquely human at all. If you look at, uh, you know, the way a rat would deal with something novel in its environment,

• 3:08:13Speaker 01

  it's drawn to the novel thing and it explores the novel thing until it feels like it understands it. And then it will move away. And so when you show a child an object that is perplexing, they will touch it, turn it around, try and understand it until they feel like they've built a model of it. That simple act is doing something very different than the self supervision we see in most AI models today. Because I see something I'm uncertain about and I'm volitionally going to create new training data for me. I'm I know the training data I want now. I want to see what happens when I pick it up and I turn it to the left and I turn it to the top. A convolutional neural network doesn't do that. So the way we teach CNNs to understand rotations in 3D objects is we manipulate the training data ourselves.

• 3:08:55Speaker 01

  We we we audit, we take data, we take imagery and then we rotate a bunch of different ways. So that we are the ones sort of curating the data set to teach it these things.

• 3:09:04Tim Scarfe?

  Yeah.

• 3:09:05Speaker 01

  But that's different than the way we learn about things. So I think this is a key aspect that's missing from AI systems today that we're that folks are working on, but that's something we're going to have to add in.

• 3:09:14Tim Scarfe?

  Yeah, completely agree. And it feels like, I mean, you're saying basically what I think, which is that there is, you use the word volition, I use the word agency, there's a creativity and an agency gap. And a lot of that is because we are agents, as you say, we create our own training data. And we do this active inferencing and sense making and we build we build these models in real time. And as a collective intelligence, it creates a kind of divergent search process for knowledge. It's this epistemic foraging that we spoke about. GPT is a monolithic model and, you know, it does have models, but the models are only learned at train time. The inference actually happens at train time. And then when you put a prompt into GPT, you're just retrieving one of the models that are already learned a long time ago. It's not creating a new model in the moment.

• 3:09:58Tim Scarfe?

  So it creates this kind of sclerotic system rather than this divergent creative system, you know, which is what we experience in biomimetic intelligence.

• 3:10:07Speaker 01

  The the one sort of mental model I have of this, because there's so much debate around. So I'd be curious, I haven't put this to the test of the gauntlet of what other people think about. So maybe in the comments, people will either agree or disagree with this. But I think an interesting alternative experiments or eval of an AI model, which I haven't heard before, is if you give it knowingly false information. In the training data, not at inference time, in the training data, will it reject it wholesale? And to me, this is, this is the distinction, which is a agent that can hypothesis test and intervene in the world will reject false information.

• 3:10:50Speaker 01

  You can, if you tell it, if you say the world is flat, it will know that the world is not flat. And so versus GPT, any data you give it is given equal weight to every other data. So the only reason it would reject that is if there's other data in the training set that it would, that it's going to ignore. And so almost, it's almost cheating because by definition, we know a language model is going to fail at this task. The only way you can fix it is if you give it other data in the training set. There's no notion of hypothesis testing. Versus an agent, the only way you could get it to be wrong is if you manipulate its sensors on the actual hypothesis testing of it that it does. You can, of course, manipulate it by, you know, when it does these tests, you change the actual test, which happens in the book,

• 3:11:33Speaker 01

  Three Body Problem, which is an amazing book, where aliens manipulate our experiments. But anyway, I think that's another way to eval these, these systems, which is, does it, can it figure out that you're giving it false information and reject it?

• 3:11:47Tim Scarfe?

  Yeah, a hundred percent. A couple of things though. So there's something magic about the, having a gentle density in the system, right? So when you have something like GPT, just to make it statistically tractable, it's generally doing a kind of low entropy search. And what, you know, what I mean by that is it's just looking for the baseline patterns. It's not doing a lot of exploration. It's not searching outside of the, the main sources of statistical regularity. Whereas when you have a divergence, you know, in the search process, you have all of these individual agents doing their own things. As a system, it's much more of a high entropy search, which means you're actually bringing in lots of new information to solve problems in creative and interesting ways. But in the physical world though, it's quite interesting, right? Because the problems come from the physical world. So the trees get big.

• 3:12:31Tim Scarfe?

  Now the giraffes need to have a, a long neck in order to eat the leaves from the trees. And this whole thing just wrenches and repeats. So, you know, the, the environment produces novel solutions. And then like we see this divergence and we find novel, you know, creative solutions to. The problems that get generated. But in the memetic sphere, it's so much more difficult than that, right? Because the the problems and the guardrails aren't constrained in in the way that they are in the physical world. So, for example, we have capitalism, or we have the nation states. And again, there's there's all kinds of interesting divergence going in different directions. But like, it doesn't seem like there are the same pressures that ground the thing to reality.

• 3:13:06Speaker 01

  Well, yeah, I I think it's definitely not grounded in truth. It's tethering to truth is it's tethering towards knowingly false information that leads me to take actions that will hurt my survival will fade. But false information that helps me survive better will propagate freely or is at least neutral. And, you know, another way this shows up is in, this is where I'll go into maybe some pontificating.

• 3:13:38Tim Scarfe?

  Please.

• 3:13:38Speaker 01

  But but where where I think there is sort of memetic evolution that can drive us away even from things like happiness. So so if we think about what are systems of coordination that survive, there are systems of domination, of of militarism. If you take two groups, one had, let's say, two, you take two groups of individuals. Let's say one is really happy and calm and they see no desire for domination and they do not attempt to innovate and build more technology. Another, they're unhappy, but they're super aggressive and they want power and they want to expand. These ideas will die out.

• 3:14:17Speaker 01

  And so so what what this suggests, what's important, and I think I talked about this in one of our previous conversations, is delineating, in my view, the Darwinian component of what does survive from the moral component of what is right or wrong. Because it is definitely not the case that what is right, what survives is definitionally right. It is absolutely possible that the things that survive and do well evolutionarily are not the things that we feel like are aligned morally. And that, of course, is not to propose a correct or incorrect system, but it is an important distinction to draw when we're trying to decide what do we deem to be morally right or wrong. So I think that's just one example of what you're saying, where the ideas that propagate successfully might not be the ones that are truth.

• 3:15:02Speaker 01

  They might also not be the ones that we deem to be moral or not even be the ones that lead to human happiness. They're just ones that do a good job keeping humans alive and reproducing of the idea.

• 3:15:12Tim Scarfe?

  Yeah, so on that then, I mean, people say that language models, you know, they confabulate and they don't preserve epistemic factfulness. But you could also argue the same thing about us, right? So we actually confabulate everything. We don't really have goals. We just kind of generate these post hoc confabulations and then we explain our behavior. And we kind of pretend that that was what we wanted to do, that we had beliefs and so on. But we just kind of make it up as as we go along using this kind of active inference. So I guess the question is, like, we do think of ourselves as being like, even though we are emotional and subjective and like, you know, we believe in religion and lots of things that we presumably made up, but but we have Wikipedia, we have objectivity. Even though it's an illusion, right? There's there's no such thing, even like general relativity.

• 3:15:57Tim Scarfe?

  It's not as objective as as we think, like if you keep asking why and why and why it just kind of disintegrates into incoherence. But there seems to be some objective structure which is preserved. And how is that explained, given that our brain simulations are, you know, they don't seem to select for truthfulness?

• 3:16:14Speaker 01

  I think the desire to say is, are humans better or worse than ChatGPT is almost like a red herring. Where I look at ChatGPT as like an alien, it's like an alien brain. And there are certain things it does that are clearly better than us. I mean, information retrieval in ChatGPT blows a human away without a question. So like, in many ways, it's way better than humans.

• 3:16:36Speaker 01

  And but but there are certain things that human brains do that ChatGPT does not have. And if we're trying to build human-like intelligence, there's certain inspiration we can garner from from human brains. So I think, you know, there's a component of our model-based sort of, you know, rendering a plan and then executing that plan that has a level of explainability. That is unique relative to a system that is sort of just iteratively predicting the next token. But we also do the same thing that ChatGPT does. When we make model-free choices, and then you say, why did you do that thing? What we engage in is exactly as you're describing, a post hoc explanation. Because I didn't render a plan, I was just walking down the street, and you say, why did you move your foot there as opposed

• 3:17:21Speaker 01

  to two inches to the right? What I'm going to do is render a post hoc explanation of why I did that, but I didn't really think about it. I'm just explaining it after the fact. So it's definitely the case that humans have that component. But it means there's also another component, which I would argue is unique and important, which is our ability to pause, render a plan, and then execute against that plan. But the key thing that I think is the dividing line between these models and us is the ability to render hypotheses and make interventions in the world. That's the key thing. And so it's not the case that our brain has the true objective state of the world in our head. I don't think

• 3:18:01Speaker 01

  that's... There might be components of objective truth in ChatGPT that it contains that we don't have. And I think in its information retrieval, it has probably, in some ways, more aspects of reality than I do, in terms of having read all of Wikipedia and answering questions about biology that I don't even know. But there are also components of the world that the human brain has rendered and contains that ChatGPT does not, because of our ability to make hypotheses and intervene and learn the causal structure of the world. And I think that is the dividing line. But I wouldn't say it's because the human brain knows the objective state of the world and ChatGPT does not.

• 3:18:40Tim Scarfe?

  Max Bennett, it's been an absolute honor to have you on MLST. Everyone at home, you need to buy his book immediately. It is a wonderful, wonderful book, Max. You did such an amazing job of bringing all these things together. And we've now spoken for four and a half hours going through the last two chapters. And my God, it's been an honor. Thank you so much.

• 3:19:00Speaker 01

  It's been my pleasure. Thank you for having me.

• 3:19:02Tim Scarfe?

  Beautiful. Okay.

• 3:19:07Speaker 02

  Now, more than ever, we need someone on our side. But Rob Whitman? Whitman's votes drove up costs, cut Medicaid, and risked 350,000 Virginians' health care to pay for tax breaks for himself and his donors. He's not on our side. Shannon Taylor will fight corruption, cut taxes, and bring down costs. This November, vote for Shannon Taylor. She's for us. Paid for by HMP, the House Majority PAC dot com. Not authorized by any candidate or candidate's committee.