363 | Chandra Sripada on How LLMs and Humans are Cognitive Cousins

Large Language Models display an uncanny ability to construct human-sounding speech, and can synthesize concepts in novel ways. Is this because they are truly thinking like human beings in some way, or have they found a way to be human-like without reproducing the internal mechanisms of human thought? Chandra Sripada argues that LLM cognition is more human-like than we suppose, and offers evidence from the ways that cognitive scientists study actual humans. Blog post with transcript: https://preposterousuniverse.com/podcast/2026/08/10/363-chandra-sripada-on-how-llms-and-humans-are-cognitive-cousins/ Support Mindscape on Patreon https://www.patreon.com/seanmcarroll. Chandra Sripada received an M. D. from the University of Texas and a Ph. D. in philosophy from Rutgers University. He is currently a professor of philosophy and psychiatry at the University of Michigan, where he holds the Theophile Raphael Research Professorship and directs the Weinberg Institute for Cognitive Science. He writes Cognition, Decoded https://chandrasripada.substack.com/, a Substack newsletter about AI, cognitive science, and philosophy. Web site https://sites.lsa.umich.edu/sripada/ Google Scholar publications https://scholar.google.com/citations?user=8N_cfXsAAAAJ&hl=en PhilPeople profile https://philpeople.org/profiles/chandra-sekhar-sripada Babbel is offering listeners up to 60% off. Go to https://babbel.com/MINDSCAPE. #ad Get 60% off an annual plan from Incogni using code MINDSCAPE at https://incogni.com/mindscape. #ad

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Sean Carroll?

  You're listening to this podcast, so I know you've got a curious mind. Here's a helpful fact you might not know yet. Drivers who switch and save with Progressive save over $900 on average. Pop over to progressive.com, answer some questions, and you'll get a quick quote with discounts that are easy to come by. In fact, 99% of their auto customers earn at least one discount. Visit progressive.com and see if you can enjoy a little cash back. Progressive Casualty Insurance Company and affiliates. National average 12-month savings of $946 by new customers surveyed to save with Progressive between June 2024 and May 2025. Potential savings will vary.

• 0:39Speaker 01

  Hey, I'm running out to drop off a stack of letters and packages. Need anything shipped while I'm gone?

• 0:45Speaker 02

  Save the gas. Stamps.com lets you print postage and schedule pickups right from your desk. Sync your contacts into the address book. Get detailed accounting insights and get notified the moment your certified mail is delivered.

• 0:57Speaker 01

  All from my desk?

• 0:58Speaker 02

  All from your desk.

• 0:59Speaker 03

  Join the 4 million businesses using stamps.com right now with code PODCAST and get your free welcome gift. Stamps.com code PODCAST. Taxes and fees apply.

• 1:09Sean Carroll?

  Hello, everyone, and welcome to the Mindscape Podcast. I'm your host, Sean Carroll. I presume that everyone has heard of the Turing test, or as Alan Turing himself called it, the imitation game, supposed to be a way to figure out whether computers can think. And of course, the difficulty in that is not only what computers can do, but what do you mean by thinking? So Turing took a very sciency, physical sciency, mathematically approached to the problem. He says, I don't know what it means to think, but we can look at what things do. And if you have a computer that does things that are impossible to tell the difference

• 1:49Sean Carroll?

  between that and thinking, that is to say, the input, output, responses of the computer are indistinguishable from those of a person, we should call that thinking. Now, these days, we have these LLMs, these large language models, as an approach to AI. And more or less, it's clear that they do pass the Turing test. I know there's some people who argue about that, but I think it's kind of nitpicky myself. I think that there's no question in my mind that they're passing the test as Turing himself would have imagined it. So does that mean that the LLMs are really thinking in the same way that human beings are thinking? And I think, you know, there's been an argument back and forth. Some people say, yes, that is what it means.

• 2:32Sean Carroll?

  Others say, well, no, actually, it turns out we have to think harder about what it means to be thinking. And then the other side says, oh, no, now you're moving the goalposts. I thought we agreed on the Turing test. I'm actually on the side of the goalpost movers. I think it's perfectly okay to say, well, that wasn't a careful enough definition of what it means to think, at least not in the same way as human beings do. And even granting that, given a certain set of inputs, the LLMs will produce outputs that are more or less indistinguishable from some kind of human output, there's still a question, are they doing it because the LLM architecture, in the process of being

• 3:15Sean Carroll?

  set up and then trained and fine tuned and so forth, has essentially rediscovered the mechanisms by which human beings think? Or is it because, and it's an absolutely plausible scenario, they've discovered a different way to have the same input-output mechanisms as human beings, the kind of alien intelligence? And there is some evidence, and I have absolutely been on the side of being impressed by the evidence that says, look, there's questions you can ask in an LLM that don't look like human answers. You know, the famous ones are how many Rs in the word strawberry or something like that. And that, to me, was very good evidence that they're not thinking in the same way that human beings are.

• 3:58Sean Carroll?

  And a lot of people push back against my view on that, saying, well, you know, all the great computer scientists and leaders of the AI industry are saying otherwise. And that was never especially convincing to me for the simple reason that those people are experts in computer programming and computer science and AI, but not experts in intelligence and cognitive science. So recently, Johns Hopkins hosted a meeting of the Society for Philosophy and Psychology. And some of us had the idea it would be fun to do a live podcast interview as part of that meeting. Didn't pan out that way. I was traveling at the same time the meeting was happening, et cetera. But we looked through the people who were visiting for prospective good podcast guests,

• 4:42Sean Carroll?

  and we found today's guest, John Dreschrippata. He is a philosopher and cognitive scientist in the psychiatry department at University of Michigan, and also an expert on LLMs. Okay, so he's an expert on thinking and on LLMs, and is very well positioned to ask and answer the question, do LLMs think in similar ways to humans do? And he makes a strong case that they do, often, in many ways, think using the same kind of thinking methodologies that human beings do. And so I find him very persuasive. I think that he's made a really good case that in, at least in an important set of

• 5:27Sean Carroll?

  ways, LLMs have rediscovered or, you know, been coaxed into re-finding out the ways that human beings think, using ideas from cognitive science, you know, how do different tests of what happens during the cognitive process match up between LLMs and human beings? So you can tell for yourself whether or not it's convincing to you. It's not the same as saying that LLMs are conscious or responsible moral agents or anything like that, but they are, but this is something we should establish in that direction. You know, cognition is easier to understand than consciousness.

• 6:09Sean Carroll?

  So I think this is one of those podcasts that has shifted my credences in important ways. And, you know, one should always be a good Bayesian. If more evidence comes in the other way, then they'll shift back. Or if the evidence keeps pushing in this direction, they'll keep moving in that direction. But I think it's fascinating that, you know, the option, which was always on the table, and I always admitted certainly, that LLMs sort of have, for by some way or another, rediscovered human modes of thinking, has turned out to be something that has evidence for it in an interesting way. So I think that makes for a great conversation. Let's go.

• 7:04Sean Carroll?

  John Dreschrippata, welcome to the Mindscape Podcast.

• 7:07Speaker 04

  Oh, thanks, Sean. I'm glad to be here.

• 7:09Sean Carroll?

  Now, we're going to be comparing two of my favorite cognitive systems, the human brain and LLMs. So most of us are a little bit familiar with the human brain. Of course, we've all heard of LLMs by now, but could you give us a quick overview of how LLMs work? You know, we use them, but maybe we've forgotten what is going on underneath the hood a little bit.

• 7:34Speaker 04

  Yeah, of course. Yeah, let me, let me say a bit about that. And I realize that I didn't get to say this already. I'm a long time, a long time fan.

• 7:43Sean Carroll?

  Oh, thank you.

• 7:44Speaker 04

  It's great to be here. And I don't think I've ever heard you at anything less than 1.5 speed. So it's good to hear you live. You're even more eloquent at this rate.

• 7:57Speaker 04

  So what, LLMs, how do they work? You know, for starters, LLMs are neural networks. So there's going to be a set of nodes and weights that connect them. And the basic processing strategy is when an input comes in, the activations will be multiplied by the weights and get propagated through the network. And if those weights have been tuned correctly, the answer will be a sensible continuation of the input that came in. So LLMs are distinctive in that, in a few ways.

• 8:36Speaker 04

  And one of them is the approach that they're going to take is autoregressive. So I like to have my coffee with sugar and, and they, they have been trained so that massive set of parameters, hundreds of billions, trillions in frontier models, are tuned up in such a way that over the course of internet-scale data, they are making predictions of the next word. And every single time, those parameters have been changed just a bit, so their predictions get better and better.

• 9:17Speaker 04

  And so initially, I like my coffee with sugar and, you know, it might be darts or butter. And eventually, you're going to get to cream.

• 9:31Sean Carroll?

  Yeah.

• 9:31Speaker 04

  And

• 9:33Speaker 04

  So the autoregressive generation, one word at a time, after that one word is generated, it's added back into the prompt as part of the context. That's one thing that's distinctive about them. The training on internet-scale data, that's another thing that's distinctive about them. And maybe it's best to just foreshadow this and say this a bit later, but it's not a big, undifferentiated neural net. In the old heyday of PDP, you know, when I was a bit younger, those nets, there is a set of input nodes and a set of weights, and eventually you get to the output. Well, these are transformers, and they have a lot more structure.

• 10:18Speaker 04

  And so with the transformer, every word slash token, we don't have to get too much into that, enters as part of a column. And the processing happens in a column-specific way, but for the presence of these attention heads that move information from the left over to token positions on the right. And so that's another, it's not the only feature of transformers, but they're a distinctive, highly structured kind of neural net. And so billions of parameters, trained on next-word prediction, autoregressive generation, the structure of a transformer, that's a lot of what's going on in a large language model.

• 11:02Sean Carroll?

  You know, if I am going to write a book, I do not start at the first sentence and just keep typing, right? Like, I imagine an outline, and then I develop it, chapter by chapter. I could ask an LLM to write a book. Would it do that? Would it sort of give a brief overview and then fill in? Or would it just start at the first sentence?

• 11:25Speaker 04

  Right. Already, and I'm going to try to, you know, assemble some arguments that these LLMs process in ways that are reminiscent of or analogous to core processing principles of the human mind-brain. But you caught me already. There is, there seems to be something very different about the autoregressive next-word prediction training, and then the generation process, where one word is generated at a time and added to the context. What to say about that?

• 12:04Speaker 04

  I mean, one thing to say already is, next, generating the next word is not incompatible with the model itself during the forward pass, where it has a substantial amount of mnemonic resources and processing capacity to start anticipating what happens later on. So Anthropic has this example where you give it a stem of a rhyme, something to the effect of, "I saw a carrot, I had to grab it."

• 12:45Sean Carroll?

  Okay.

• 12:46Speaker 04

  And the model actually generates the last word of the next sentence, "rabbit," first. Way, and we know this via mechanistic interpretability, where we're peering into the model and looking at its processing stages, before it generates the prior words that will culminate in "rabbit." And that's highlighting that next-word generation as a generation method, as well as closely-aligned training objective, is quite compatible with the model generating things that come much later, early on.

• 13:31Speaker 04

  And so that kind of planning can still happen. I'll add another dimension to it as well. We'll come to this a bit later, I think, but these models can use the context window as a kind of scratchpad and generate internal tokens as part of what they call "thinking." So if you give it a complex kind of task, the models now will break it into subtasks as part of the initial thing that it'll do. And subtasks can be further broken down into subtasks and delegated to a series of subagents and things like that. So it is compatible, remarkably enough, with next-word generation and next-word prediction

• 14:11Speaker 04

  as the training objective, for substantial staged attacking of a problem, breaking it down into subcomponents, looking ahead at terminal parts of the subcomponents before next-word generation begins. So next-word prediction is, and the training objective, next-word prediction and autoregressive next-word generation seems counterintuitive at first, but can actually encapsulate a lot of the ordinary strategies that people use.

• 14:46Sean Carroll?

  And is this idea of like a scratchpad and multiple sort of swipes at the answer, is that something that human beings have coaxed the LLM into doing? Or is that something that the LLMs have sort of figured out for themselves in some sense?

• 15:05Speaker 04

  Right. And the answer is, mostly, the LLMs have figured out for themselves, and there is some coaxing involved as well. And so, why do we believe something like that? The use, you talked about the use of the scratchpad specifically, but is this a good time to go into the dual-process distinction?

• 15:34Sean Carroll?

  No, we're gonna keep that later.

• 15:35Speaker 04

  We're gonna keep that for later. Then, I'll say this bit of, in a very general way. Next-word prediction on internet-scale data gets the model to assemble an astonishing representational landscape and latent strategies for, you know, how complex problems like, let's say you tell me that you want to go to Tasmania. The model has latent within it the kind of subtasks that would be needed in order to do that. So all of this just seems to be something as part of the representational repertoire

• 16:17Speaker 04

  that next-word prediction generates, especially when it's done at scale. Now, when you've got these Lego pieces, it doesn't mean that you've got that pretty amusement park already that, you know, you can build with the Legos. There is some coaxing, some supervised training, some instruction tuning, and especially reinforcement learning.

• 16:39Sean Carroll?

  Right.

• 16:40Speaker 04

  Where you train the model with evaluative feedback that can emerge from various sources. That tends to take this astonishing representational repertoire that's there, that's latent, and assemble it into useful pieces and organized, goal-directed control.

• 16:56Speaker 05

  You know what's wild? Most people are still overpaying for car insurance, just because it's a pain to switch. That's why there's JERRY. JERRY's the only app that compares rates from over 50 insurers in minutes, and helps you switch fast with no spam calls or hidden fees. Drivers who save with JERRY could save over $1,300 a year. Before you renew your car insurance policy, do yourself a favor. Download the JERRY app or head to jerry.ai/lipson. That's J E R R Y dot A I slash L I B S Y N.

• 17:30Speaker 06

  Cash flow crunch? OnDeck's small business line of credit gives your business immediate access to funds, up to $200,000, right when you need it. Cover seasonal dips, manage payroll, restock inventory, or tackle unexpected expenses without missing a beat. With flexible draws, transparent pricing, and control over repayment, get funded quickly and confidently. Apply today at ondeck.com. Funds could be available as soon as tomorrow. Depending on certain loan attributes, your business loan may be issued by OnDeck or Celtic Bank. OnDeck does not lend in North Dakota. All loans and amounts subject to lender approval.

• 18:01Sean Carroll?

  So that's great, because, yeah, it leads right into what I really want to ask, which is, at a very high level, so we'll get into details later, but to me, the basic question is, look, we've built something that is really, really good at taking in inputs and giving outputs that are remarkably human, right? That could fool anybody if you had the old-fashioned Turing test. And I can imagine two different possibilities. One is, that's because the LLMs have basically rediscovered the same mechanisms that are going on in the human brain.

• 18:40Sean Carroll?

  Or alternatively, the LLMs have figured out a wholly new way of sounding human, what we might think of as an alien kind of intelligence. And you know that we're relentless anthropomorphizers, so it's gonna be easy to guess that they must be human if they're acting human in this way. So what are the options here before we start advocating for either one? Like, what are the possibilities to help conceptualize what's going on inside the box?

• 19:10Speaker 04

  Yeah, I mean, I think you've laid it out really, really well. We see the outward, we see the inputs, the prompts, and we see the the next words come out, and they're eerily fluent. And one option is they are fancy auto-completes, they're statistical approximators, they are bullshitters of sorts, they are entities that have memorized a bunch of tricks and heuristics about how to generate next words that sound like people, but under the hood,

• 19:50Speaker 04

  very little is going on that closely resembles what happens in the human mind brain. And this is definitely a continuum, and there's going to be various middle positions. And at the other end, a picture that, you know, I'll put my cards on the table that I'm going to argue for a bit more, and I think I'm more inclined to and attracted to is, it just turns out that prediction is incredibly powerful. It is the training signal that is the mother of all training signals. And whereas when I was going in grad school, this was much less appreciated, and it wasn't

• 20:33Speaker 04

  part of the zeitgeist, it has been over the years in cognitive science, it's been taking over center stage as a lot of what happens in the mind brain is prediction. And a lot of cognitive principles that we thought were innately specified or due to some sort of, you know, contingent evolutionary trajectory, they actually are emergent in cognitive science, dual process distinction, or various other cognitive principles that are, you know, that we describe the human mind brain. They are downstream of prediction. And so it turns out that at the level of basic core cognitive principles, the LLMs

• 21:16Speaker 04

  and humans, they identify similar representations, similar procedural techniques, different modes of inferential organization, and so forth. And so there's a lot of similarity there, and it arises downstream of prediction.

• 21:31Sean Carroll?

  I like how you call them the mind brain. That sort of avoids worrying about calling it the brain and people getting upset or calling it the mind and people getting upset.

• 21:41Speaker 04

  Exactly, exactly. I've been coached by a PR firm not to get into that.

• 21:47Sean Carroll?

  So, okay, good. So the options are out there, maybe LLMs have thought of a, or have come up with a different way of acting human, or maybe there's some sort of, some form of convergent evolution, where the best way of acting human is also what the LLMs found. There is this like quasi anecdotal evidence that has made a big impression on me, I will confess, that the kinds of mistakes that LLMs make seem to be the kinds that no human would ever make. Like, of course, everyone makes mistakes, but there's different kinds of mistakes, like famously, the LLMs can't count the number of Rs in the word strawberry until you really teach them.

• 22:32Sean Carroll?

  So how much of an impact does that kind of evidence make on you?

• 22:38Speaker 04

  Yeah, I think, you know, I'm a total evidence guy. So, yeah, that's a data point. You look at, that's puzzling, and you know, especially, and then the next moment, it's expounding on parts of general relativity that, you know, that are very subtle. So, you know, what is going on there? And one thing that we shouldn't do is just stick with these kind of behavioral observations. You're going to notice various places where the LLMs do not act like in very, in very human-like ways.

• 23:21Speaker 04

  And we shouldn't settle for just looking at outward behavior and counting and tabulating, well, this looks a little similar, this looks a little dissimilar. Very quickly, if we, if we stay at that level, it's always going to remain, any one of these possibilities could be live, you know, alien intelligence versus, the, the name I like to give is cognitive cousin for the other version, where they're actually much like us. You know, it turns out with the strawberry, these things, they do have, obviously, a different kind of learning history, and their contact with the quote unquote world is exclusively

• 24:01Speaker 04

  textual via these tokens, which essentially serve as kind of sensory primitives for them about what comes in. But they don't even have access to the letter, the letter level. It's not they don't have access, but they don't typically operate with individual letters when words come in. So there's a very natural explanation for why they can't count the number of Rs in strawberry. Whenever we see these kinds of anomalies, and there are many, an immediate next question we can ask is, at the behavioral level, before we even start looking too closely at underlying

• 24:42Speaker 04

  mechanisms, are there a bunch of ways in which LLMs reproduce behaviorally? Phenomenon that are familiar, that we've documented over the decades in cognitive science? Do they reproduce some of those phenomenon? You know, turns out there are many such cases, and many of them are notable, they catch your eye. Can I go through some examples?

• 25:07Sean Carroll?

  Sure.

• 25:08Speaker 04

  Okay. In the area of language, there are a bunch of effects that psycholinguists talk about. So, an example is the center-embedded sentence. So, "A man ran" is easy to understand. "A man that a woman, a man that a woman loves ran." This is easy to understand. "A man that a woman that a child knows loves ran" starts to become hard to process. That's center embedding, and psycholinguists have identified, you know, for now, I actually

• 25:51Speaker 04

  won't talk about mechanisms, let me, let me save that for a little bit later. There's garden path sentences. "The horse raced past the barn fell." The "fell" seems to come out of nowhere, and it's because, I'm not saying "the horse raced past the barn," I'm saying "the horse that was raced past the barn fell." And so there's a order effect, when you do incremental parsing, these garden paths arise. There are other kinds of effects, similarity-based interference, there are depth changes. So, there's a bunch of these kind of effects that psycholinguists have identified, LLMs exhibit those. Another nice and striking one is in serial list memory, a kind of branch of episodic

• 26:36Speaker 04

  memory. If you give people a list of words and ask them to say the, the words back, what you'll find is that there's a recency effect, the very last word has an advantage, the very first word has an advantage. There's a lost-in-the-middle effect, those things in the middle, they don't have that advantage. There's a contiguity effect where if you say a word, the words around it come to mind more easily. There's a temporal asymmetry effect, so that if you say a word, the one after it comes to mind more easily. LLMs exhibit all these effects. When you put a bunch of words in their prompt, the first word, the most recent word, are more prominent to enter into processing, there's a lost-in-the-middle, there's temporal

• 27:21Speaker 04

  contiguous, there's a contiguity effect, and then there's the temporal asymmetry effect. I'll give one more example.

• 27:30Sean Carroll?

  Yeah.

• 27:31Speaker 04

  And, and these, so there's, visual search has been extensively studied, and there's a classic effect, disjunctive versus conjunctive search. So, if I say, "Find the green L among a bunch of red Ls," so picture that, there's one green L, and there's a bunch of red Ls, the green L is going to pop out, because the greenness is a singleton concept that distinguishes that one thing. If I say, "Find a green L among red Ls and green Ts," now you got a problem. There's no one feature that distinguishes the thing that you're looking for.

• 28:15Speaker 04

  And so people slow down, they do this in serial order, and their time in completing the task is proportional to the number of elements in the array. So this is classic, what I love about these is cognitive scientists have already written down these effects decades ago. And you can check the LLMs, in this case, vision language models, do they exhibit the effect, and they do. All of these examples, we know the mechanisms behind it, we know that garden paths arise due to incremental pars, parses that happen one word at a time, that lead to bad parses

• 28:57Speaker 04

  leading you astray, and then you've got to go back and regroup. We know the episodic memory effects, we know some of the mechanisms that play a role. And in the visual search, it has to do with what are the conceptual primitives, what is encoded via a compositional code, and what requires feature binding. The fact that you're seeing these non-obvious patterns of similarities in LLMs and people, especially where we know some of the mechanisms that happened in these effects in cognitive science, they point to similar mechanisms being operative in the LLMs and people.

• 29:37Sean Carroll?

  So if I can try to summarize that, the strawberry thing could conceivably be evidence that the LLMs have found a very different way of sounding human, and here we're showing that the emperor has no clothes with the strawberries, or mostly the LLMs are doing human-type cognition, and we've just found one little Achilles' heel.

• 30:01Speaker 04

  I would actually put it in an even more benign way. We knew at the get-go that their sensory evidence, what is analogous to that is tokenized representations of English language words. They very rarely descend to the level of individual letters. So, we knew that at the get-go. It wasn't supposed to even be a major dimension of the analogy between them and us, that individual letters would be particularly salient in their processing strategies. So...

• 30:39Speaker 04

  The two lessons I would say are that counting and tabulating at the level of behavioral outputs is probably not going to get us very far. We need to look mechanistically, and we need to think about which are the mechanisms that we actually care about, that are core processing principles for the human mind brain. Dealing with individual letters in, that's never something that I would elevate to the level of a core processing principle.

• 31:07Sean Carroll?

  Right.

• 31:07Speaker 04

  Whereas some of the others that deal with episodic memory and visual search, compositional codes, and incremental parses, that's, that's bread and butter of what cognitive scientists thought about for a long time. And so the fact that those are preserved are probably more important.

• 31:23Speaker 06

  Cash flow crunch? OnDeck's small business line of credit gives your business immediate access to funds, up to $200,000, right when you need it. Cover seasonal dips, manage payroll, restock inventory, or tackle unexpected expenses without missing a beat. With flexible draws, transparent pricing, and control over repayment, get funded quickly and confidently. Apply today at ondeck.com. Funds could be available as soon as tomorrow. Depending on certain loan attributes, your business loan may be issued by OnDeck or Celtic Bank. OnDeck does not lend in North Dakota. All loans and amounts subject to lender approval.

• 31:54Speaker 07

  Looking for big flavor and even bigger savings? You gotta head to Royal Farms. Right now, get an eight-piece mixed chicken meal or a 10-piece dark chicken meal for just 15 bucks, including a 2-liter Coke. It's an easy, delicious meal the whole family can enjoy. Or grab a deli or chicken salad sub with Rofo chips and a 20-ounce Canada Dry Soda for only 6 bucks. Fresh food, great deals, and world-famous flavor in every bite. Stop by Royal Farms today. Royal Farms, real fresh, real fast.

• 32:24Sean Carroll?

  I don't want to dwell on all of the different idiosyncrasies of the LLMs, but I guess your answer to the strawberry question doesn't seem to provide an answer to the car wash question. You know this one, right?

• 32:36Speaker 04

  I do not know this one.

• 32:38Sean Carroll?

  Oh, this is a good example where you ask the LLM, "I am 500 feet away from the car wash. I'd like to get my car washed at the car wash. Should I drive my car there or just walk?" And the LLM will inevitably say, "You should just walk. It's only 500 feet away." Not with the implication being, "But it does be no good to walk because I want to get my car washed there. I need to drive my car there, even though it's very close."

• 33:05Speaker 04

  You know what's hilarious about that? I was about to say, "You should walk. It's only 500 feet away."

• 33:13Sean Carroll?

  You've been hanging out with LLMs for too long, Chandra.

• 33:16Speaker 04

  Yeah, exactly. Either they've rotted my brain, or unwittingly, what we're seeing is the foibles and idiosyncrasies of them are a little bit like us, at least in some cases.

• 33:31Sean Carroll?

  Yeah, I think that's it. Okay, anyway, I just wanted to get that on the table there. But, okay, now, you were very gracious in not going into the dual-process theory when you wanted to, but here is a good time to do it. Because if you're going to make the argument that LLMs are sort of remarkably human-like in their cognitive strategies, here's one example of how humans work. So explain, let's not assume that people know what the dual-process theory is.

• 33:59Speaker 04

  Sure. Yeah. So what is dual-process theory?

• 34:05Speaker 04

  People often are familiar with Kahneman's Thinking Fast and Slow. And his Nobel, you know, he won the Nobel Prize, and his Nobel lecture, he revisits his entire amazing research program in terms of dual-process theory. And so the idea is that you have fast thinking, it's quick, it's automatic, it's intuitive, it results appear to you as a kind of a flash. And then you have slow thinking, which is slow, it's serial, it draws on more cognitive resources. Whereas fast thinking is thought to arise via slow, patient, experience-dependent

• 34:46Speaker 04

  practice with the environment. A characteristic of slow thinking is it deals very rapidly with sophisticated novel problems. So those are some of the features of fast and slow thinking. And you know, an example, if I show you a picture of John Travolta, it flashes out to you, "That's Travolta." It's not as if you reason through it.

• 35:07Sean Carroll?

  Yeah.

• 35:09Speaker 04

  If I say, "Because so many people attended, they moved the ball upstairs," you know that I'm talking about a gathering of people, not the spherical thing. That's fast thinking. And, you know, and slow thinking, if I give you, for example, multi-digit mental arithmetic, nothing flashes before your head. There's some sort of procedure. Maybe you as a physicist.

• 35:35Sean Carroll?

  No, no, no, no. Arithmetic is not it.

• 35:37Speaker 04

  Yeah. So that's thinking fast and slow. And it's just a toweringly influential theory. It doesn't originate with Kahneman. Though Keith Stanovich and Richard West coined the terminology. And, you know, social psychology, stereotyping, they use this theory. Clinical, I'm a psychiatrist as part of my background. Substance addiction, ADHD, this is a theory that people draw on. Developmental psychology, adolescents behave the way they do because fast thinking outstrips slow thinking. It's just an explanatory form that is widespread throughout psychology. And so this is an example. We better see, if we're going to

• 36:18Speaker 04

  say that LLMs are like us, is there something like that in LLMs? And can it, can it help us as cognitive scientists understand this distinction? And the answer is yes. Okay, so in LLMs, there is a really interesting distinction between what people sometimes call in-weight processing. And then there's a pair of, so in-weight processing, so that's W-E-I-G-H-T, I'll come to that in a second, is very akin to fast processing. And then there's two mechanisms in LLMs that help to undergird that so-called system two, the

• 36:57Speaker 04

  slow processing. But I'll start with in-weight. LLMs, via the prediction objective, they acquire massive amounts of information that's compiled into their weights. That's why it's called in-weights. It's a relatively easy pathway from the stimulus information to something that's populated in the LLM. It doesn't require elaborate constructions within their activation space. And I'll say a little bit more about that if that sounds obscure. So Albert Einstein is German, they just know it. Michael Jordan plays basketball. The state containing Dallas is Texas. If you drop a vase, it will shatter. These are all the kind of common sense, factual things that they know in their weights.

• 37:43Speaker 04

  There are two mechanisms that undergird their system two. And the first one, in-context processing. I'm going to dwell a little bit on it, because I haven't mentioned this yet, and I'll, I should say this. There's a paper that goes along with some of the arguments that I'm covering today. And that paper, I have a co-author. It's my Rick Lewis, Professor of Psychology and Linguistics at Michigan, and my colleague, and my friend, and collaborator.

• 38:14Speaker 04

  And so Rick calls, we'll put the paper up in the show notes eventually, I think. Rick calls this "the greatest discovery in the history of cognitive science." So, in-context processing. So maybe that's not hyperbolic enough. What is in-context processing? It's a surprising phenomenon that prediction training installs in the model, something like a very general routine for extracting patterns from the prompt, generalizing from base cases, learning a novel mapping from instructions alone. When I characterize system two, I

• 38:54Speaker 04

  said one of its primary properties is it's the place where rapid, novel grocking of complex, novel patterns suddenly happens. And cognitive science has been at a loss to understand this phenomenon and characterize it in any detail. And in a pair of papers, Bradford and colleagues in 2019, and Brown in 2020, the second one is from OpenAI, and Ilya Sutskever is lead author of one of these, and Dario Moda is lead author, and is the senior author in the other one. These are like giants of the field.

• 39:34Speaker 04

  They, especially in GPT-3, so the 2020 paper, these models, when they're just trained on prediction, they have an astonishing ability to display this in-context reasoning. And an example that was used in a paper that identified one of the mechanisms of in-context processing. I'll give that example. You say Marie Curie is, well, I mean, Marie Curie, she is the discoverer of radiant. She's an amazing scientist. Those are the things that come to your mind. Maybe that's in the weights of the model. But if I say Albert Einstein is German, Mahatma Gandhi is Indian, Marie Curie is, you have French.

• 40:18Speaker 04

  Some may know that she was born in Poland.

• 40:20Sean Carroll?

  Polish, I was gonna say Polish, yeah.

• 40:23Sean Carroll?

  I know too much about this one.

• 40:25Speaker 04

  The computer scientists that use this example, they may not have even known that she was Polish. They just assume the answer is French. The model gets the idea that there is a pattern being requested. And they actually study this in a lot of detail. They use hidden Markov models, mixtures of them, and they show the model is doing something like implicit Bayesian inference. It's got a hypothesis about what task is being asked. It's using the first example, you know, the likelihood. It's multiplying that presumably by some sort of prior. It's arriving at a posterior about what kind of task is there and what kind of completion is the best for the circumstance.

• 41:09Speaker 04

  Others have likened this to more like gradient descent happening within the model. I want to emphasize, this is happening after the weights are frozen. So what do I mean gradient descent? Gradient descent is how these models are trained. Then they're shipped and then the weights are frozen. What do you mean gradient descent? Well, the models have activations. Those activations are changing. Those activations are what are happening on the nodes when you multiply the nodes by the weights. Well, the model has compiled within it machine learning routines like linear regression, nearest neighbor and Bayesianism. And they can understand very complicated patterns in their activation space.

• 41:52Speaker 04

  That's what this in-context learning is. And it arises via very general prediction training. As a cognitive scientist, where we have struggled to articulate, what is this system two? The dirty secret of cognitive science is we don't actually have theories of things like system two or related terms. What we have is theories of when does it develop during adolescence? What lights up when you put people in the scanner? We don't actually have mechanistic process level theories of a lot of how this stuff works. And here we have an artifact that's displaying the phenomenon. And we can mechanistically interrogate it.

• 42:37Speaker 04

  And we can understand the grab bag of routines that underpin it.

• 42:42Speaker 04

  So this is startling, this in-context learning. And it's very system two-ish. There's another system two mechanism that they have. And that's where you're using the internal, you're using the context as a scratch pad. And you're doing extended serial reasoning within it. So I'll say a little bit about this quickly, borrowing an example from Melanie Mitchell. So you give a problem like this. Julia has two sisters. How many sisters does her brother Martin have? Well, you say, oh, two. Well, you just told me she has two, there's two sisters there. Well, Julia is a sister.

• 43:22Sean Carroll?

  Yeah.

• 43:23Speaker 04

  And so what, if you give this type of problem to a model that is not using chain of thoughts, very likely to say two. But if you allow it to generate internal thinking tone up tokens that are not displayed to the user, that end up in the context, that influence the next steps that the model takes, it can reason through and says, okay, so Julia is a girl. She has two sisters. That means there's three girls in the family and so forth. So you can show that a model that has access to this chain of thought, scratch pad, models like the ones that when IMO gold recently in, you know, in 2024, that this amplifies the inferential power of the model. Okay.

• 44:06Speaker 04

  And so where we began is dual process distinction. It's a towering distinction in human psychology. And what we see is these models have in-context processing and they have chain of thought. And between the two of them, they approximate the, and they also have the in-weight, which corresponds to the automatic.

• 44:26Sean Carroll?

  Yeah.

• 44:27Speaker 04

  And so you're ending up with a distinction in the model that mirrors that in people. And you ask, well, is there clear evidence that this in-context and chain of thought that actually is much like what happens in the human mind brain? And yes, there is evidence. And, you know, there is a reaction time evidence. My own lab has a study using, you know, paradigms like the Stroop task that illustrate that there is a very tight correspondence, which is what is happening between what is happening in the model and what happens in the human mind brain, for which we often invoke the system one, system two distinction.

• 45:11Sean Carroll?

  So I don't want to gloss over it too quickly. You're saying that LLMs have discovered Bayes' theorem?

• 45:23Speaker 04

  Yeah, you know, what to say about that? So it has been widely known that LLMs approximate the posterior. So the, you know, you've got a context and then there's a next word. And there's a conditional distribution. And they approximate that conditional distribution very well. And Bayesian would say, here's the normative way to do it. You have these factorized representations of hypotheses. You've got a prior and you multiply it by, you know, these likelihoods. It turns out that you can approximate the Bayesian posterior or the correct conditional distribution, not by explicitly going through a factorized representation,

• 46:08Speaker 04

  but in a sense, by modeling it or learning it, by shortcutting your way through what are the features present in the context? Can I stick those into a powerful function approximator and get something Bayesian posterior like? That's called an amortized inference. And it's a statistical principle that's been known in machine learning community for decades, that you don't have to go through a Bayesian calculation to get to the Bayesian posterior. You can amortize it via the very long learning process. That's what these things are probably doing. And it's not simply that they approximate the Bayesian posterior. That's not actually what in-context learning is. It's they can take a novel situation

• 46:53Speaker 04

  and map it into a Bayesian approximator in their activations after their weights have been frozen. So the Bayesianism doesn't just happen during that extended predict the next word internet scale training phase. The Bayesianism penetrates at the level of after their weights are frozen, frozen, how they process.

• 47:15Sean Carroll?

  Yeah. And that's very plausibly what is happening in the human brain, too. We didn't, we're not born with Bayes' theorem imprinted on our neurons.

• 47:25Speaker 04

  Yes. And Sean, I want to say just a little bit more before we leave this topic, because I get so excited about this dual process distinction. To me, this is, you know, screaming, hey, look at me. I'm an LLM. Look at me, look at me. I'm not so alien to me.

• 47:43Sean Carroll?

  Yeah.

• 47:44Speaker 04

  I have studied Stroop tasks. They're called conflict tasks in cognitive science for the last 20 years, since I was a graduate student in philosophy of cognitive science. They are, they're just a laboratory where cognitive scientists spend a lot of time in trying to understand what are these two processes that are competing and things like that? In LLMs, you can find places where, so the prompt we use is, the crayon is red. So you say the crayon is, the model wants to say red. That's its automatic compiled disposition. Of course, I told you the crayon's red. You're gonna say it's red. Then we append a kind of rule-based prefix that reverses the mapping between red and blue or any pair of colors.

• 48:27Speaker 04

  What we find, and that taps in-context, inferential reasoning. What we find is causal pathways through the model where the automaticity resides. Causal pathways through the model where the in-context processing that opposes the automaticity arises. You find classic congruency effects that you find in conflict tasks. You find what's called the congruency facilitation effect, another effect. You find the congruency sequence effect, another effect that cognitive scientists have described. You can fine tune the model to ramp up automaticity and you get the predicted effects. You can impair in-context processing using a human-like manipulation

• 49:10Speaker 04

  where you burden and tax the model by a very long rule, exactly the kind of thing that would mess up a human. And you find that in-context processing is weaker and it selectively affects the incongruent condition. The whole thing resembles people. And so I love this example because this is just an entity, and we're doing this on Gemma 2B that hasn't even been instruction tuned. So it's a small model.

• 49:39Speaker 04

  It's just predict the next word that gets all of this infrastructure there.

• 49:42Sean Carroll?

  Right.

• 49:43Speaker 04

  If you asked me in 2010, where do these dual process effects come from? I'd say, well, evolution gave us a reptilian brain and it installed this other infrastructure of higher cortex. And it's all innate. And I was a student, you know, at Rutgers where Jerry Fodor was there.

• 50:01Sean Carroll?

  Yep.

• 50:01Speaker 04

  And so I would also throw in, we'll probably never figure out how the system two part ever, ever works. I would say all those things, all of that is wrong.

• 50:12Sean Carroll?

  Right.

• 50:12Speaker 04

  Prediction objectives looks like it sets up this entire infrastructure. Wow. Okay.

• 50:19Sean Carroll?

  So, one way of rephrasing that, again, correct me if I'm wrong, is, look, of course, there's this superficial similarity between responses from LLMs, responses from humans, and therefore, on the one hand, you're tempted to anthropomorphize them, on the other hand, you want to resist that temptation, but you're saying that it goes beyond that, there are much more subtle aspects of human reasoning that the typical person on the street doesn't even recognize, but the highly-trained cognitive scientist is well aware of, and we're finding those in the LLMs also.

• 50:52Speaker 04

  Exactly, exactly. And with the dual-process distinction, that's one of those, though, where the man on the street actually does recognize this. Thinking fast and slow is influential in part because it resonates with something that we know from introspection.

• 51:07Sean Carroll?

  Right.

• 51:08Speaker 04

  So, this is one of those where we can look inside and see this distinction. Yeah.

• 51:12Sean Carroll?

  And at the danger of derailing, how confident are we that it is a truly dual-process theory versus just a multi-process theory? Is it truly just two scales, or are there different processes that happen on all sorts of different scales?

• 51:28Speaker 04

  Yeah. I mean, the in-weight, in-context distinction, what is it?

• 51:37Speaker 04

  One thing I will say, that cognitive scientists debate this question, and one of the weaknesses of dual-process theory, any cognitive scientist gets frustrated, is a lot of it operates as a list of adjectives. Well, one of these processes is fast, automatic, effortless, and one of these is slow, working-memory-dependent, and you've got these adjectives, and hey, where's the mechanistic details? Without the mechanistic details, it's hard to know how you count, what exactly is the theory committed to, and so forth. Now, you have an artifact, that is, the LLM, and we understand a bit more how in-context processing works versus in-weight.

• 52:20Speaker 04

  I would describe it much more as a continuum. Everything that an LLM ever does requires its weights and requires activation, but there's a much more direct link between the stimulus and the response via weights that already have most of the response within it, in the case of in-weight. So, Michael Jordan plays basketball, there isn't an elaborate structure that needs to get built in the activation space in order to get to that response. But if you have more complicated prompts, like that Albert Einstein is German, Mahatma Gandhi is Indian, so Marie Curie is, you have to build up a structure in the activation

• 53:03Speaker 04

  space, something that infers what is the task, applies that task to Marie Curie, extracts what that response is, the activation space is now crowded with a lot of structure, it's very prone to interference, and so forth. So, while it is a continuum between in-weight and in-context, there is definitely poles anchoring each end, and so, you're now in a better position to start to understand what this distinction really means, because you've got, you know, very clear mechanistic hypotheses.

• 54:41Sean Carroll?

  So, while we're digging into, we're trying to figure out if the LLMs are thinking in very human ways by first thinking about how humans think, and the dual-process theory is one of the examples there, you had another example, which I'm not at all familiar with, I knew about the dual-process theory, but you bring up the idea of production systems, an idea due to Emil Post, and I have no idea what's going on there, so tell us what is going on with that.

• 55:08Speaker 04

  Right. So, now, we've moved away from inferential organization to kind of an even more basic level, a kind of computational organization. So, at the very beginning, I said, "Look, transformers, they're not your vanilla, they're not grandpa's neural nets."

• 55:23Speaker 04

  They're, they have a lot of structure, and the way that they operate, and I'm getting to it, involves a layer-wise transformations that differ from traditional neural nets. And many people in computer science would be very familiar with how a transformer operates, and they know exactly the history that led to that. What they may not know, that in cognitive science, we've been there before, in a sense. And so, that's the whole production system, so I'll say a little bit about that. Okay? Production systems are a very important modeling framework in computational cognitive

• 56:03Speaker 04

  science, and they're used extensively, there's a huge body of results, and so, what exactly are there? Well, a good starting place is, people are familiar with ordinary, the way ordinary digital computers on your desktop, how they work. They operate with sequential instructions, a fetch-execute cycle, and so, you know, you get an instruction, you decode it, you execute what it says, and then you move on to the next instruction in the sequence. Production systems, and that's a very potent general framework for computation, production systems operate with something called a recognize-act cycle, and so, the way they would, things

• 56:46Speaker 04

  are set up is, there is a state representation, a kind of persistent memory, and then there is a population of conditionals, and the form of those conditionals is, if the state has so-and-so features, then do one or more actions. And these, in computational cognitive science, they're all discrete, and I'll loosen that a little bit later.

• 57:10Speaker 04

  And an analogy that might be helpful is, you could program a robot to make tea by telling it exactly, you know, walk into the kitchen, grab the kettle, fill it up, tell it exactly what to do, and here's another way to do it. If the kettle is empty, fill it. Okay? If the water is cold, boil it. If the kettle water is hot, pour it on the tea.

• 57:34Speaker 04

  So, it's reading in the state of the environment, and it's executing one or more of these conditionals, and what's interesting is, this other way of doing it, this, you've got a state and you've got conditionals, it just seems more natural.

• 57:46Sean Carroll?

  Yeah.

• 57:47Speaker 04

  It's robust, so, let's say, I walk in there, I grab the kettle, and I fill it with ice cubes, and the water's cold again. Actually, the robot knows what to do. It's going, if the kettle water is cold, boil it.

• 57:59Sean Carroll?

  Right.

• 58:00Speaker 04

  I didn't even think, when I programmed it, that somebody was going to do the ice cube trick, but cognitive scientists noticed that fetch-execute is just not going to be a good framework for modeling the human mind-brain, and they gravitated towards the production system, that is, the state representation and a population of conditionals.

• 58:20Sean Carroll?

  So, the, sorry, the distinction is between, like, just a step-by-step algorithm, do this, do this, do this, versus a continual give-and-take with the environment, and what should I do next?

• 58:35Speaker 04

  That's right, and, but the environment here is encoded in the form of a state representation, which is the canonical place where our hero, the robot, will be looking in order to know which conditionals to apply.

• 58:50Sean Carroll?

  Right.

• 58:50Speaker 04

  And, you know, way back in logic, or whatever, we learned these different approaches to universal computation, and Emil Post, way back in 1940, people have heard about Turing, but this state representation, population of conditionals, is another complete framework for any computable function can be expressed this way.

• 59:14Sean Carroll?

  Good.

• 59:15Speaker 04

  But cognitive scientists very much liked this production framework, and it became the dominant modeling framework in cognitive science. And the analogy is really with, kind of, working memory is where your state is stored, and a lot of what your brain houses in long-term memory is a lot of this population of if-then's, and you have learned over time, and so, you can model sentence parsing in this production system framework. In fact, Rick Lewis, my co-author on some of this work, and his former student, they have a very influential production system model of how parsing works. It predicts reaction time, it predicts, you know, center-embedding problems, it predicts

• 59:59Speaker 04

  similarity interference effects, there's theories of working memory retrieval, there's theories of multitasking, all expressed in this production system framework.

• 1:00:10Speaker 04

  So, the take-home lesson there is, it is a good, powerful, probably a leading, influential theory in cognitive science, that the mind operates with kind of a central state representation and massive populations of conditional rules, a good approximation of the computational organization of the human mind-brain. Okay? Got that?

• 1:00:36Sean Carroll?

  Yeah.

• 1:00:37Speaker 04

  And I'll add some names like Alan Newell and John Anderson. If your viewers want to know where to read, go read these folks. Unified Theories of Cognition, beautiful book. John Anderson's book, How Can the Mind Occur in the Physical World? Rules of the Mind, you know, that these are, read all about it. This is great stuff. All right, so we got that in place. Meanwhile, far away in another part of town, this is in computer science. A little Bob Dylan reference there for anybody that's interested in that. There, the computer scientists are struggling trying to create the neural nets that will go deeper. They want more intermediate layers.

• 1:01:21Speaker 04

  And he and his colleagues in a famous paper called ResNet 2016. They find that you're not getting very far just putting more layers. And shockingly, you're actually doing worse on the train data, not just on the test data because of overfitting. You're doing worse on train. And they show there, there's a way to, to, to improve this. You normally each layer, the, the weights learn a transformation of the previous layer. So layer L plus one will be some function F of L, the previous layer. What you need to do is you need to propagate the previous layer up to the next one. So F of L plus one is going to be X, the previous layer.

• 1:02:06Speaker 04

  And X is just what the, the, or L, the, the previous layer. Plus some function of the previous layer. What that forces the model to do is not learn to rewrite the entire representation. It's going to let that state persist across layers. And what each layer will do is make a small amendment, some small change. That's conditional on what appear, what features are present, what representations are present at that previous layer. Okay. So that, the thing that's persisting is called the residual stream.

• 1:02:41Sean Carroll?

  Okay.

• 1:02:41Speaker 04

  And that's one of the key design components of the transformer. So this ResNet paper, by the way, is actually cited more than Attention is All You Need, the 2017 paper. Both of those are top 10 papers of all time.

• 1:02:57Sean Carroll?

  Yeah.

• 1:02:58Speaker 04

  And the other thing that the transformer paper does is it makes explicit that the model is going to have to learn conditionals. And how does it do that? People may be familiar with KQV. That's the, the linear algebra equation for attention. And what that's saying is that the degree of match between something called a query. So at every token, you have this representation, you map that into something called a query. Every other token to the left is going to get mapped into something called a key. And you, you ascertain the degree of match between key and query. And that number is scalar.

• 1:03:38Speaker 04

  You're going to multiply that by another variable called a value, which is a mapping each token position to the left. So you're going to scale up and down the amount the value gets added by the degree of match between key and query. Okay, what did I just say there? I said, if key matches query, to that extent, add value. I formulated in linear algebra terms, a conditional. It's a graded conditional. It's not, if this feature is present, then add this feature, which is how the cognitive scientists were doing. Why do it as graded conditional? Well, you get a lot more expressivity. But you can learn the thing via gradient descent, and that's the key.

• 1:04:23Speaker 04

  And if you can train something at scale with gradient descent, which you can with prediction, now you're off to the races. You've got a production system like architecture that's learning the population of productions. It's not just attention that operates this way. The MLP units, which are actually where most of the, the parameters in the model are in the MLP units. They have two matrices. And so the rows of the first matrices contain features. If they match the residual stream, to that extent, you add the features in the column of the second matrix. The exact same formulation. Ultimately, what transformers are doing is tons and tons of dot products between two vectors, that's a scalar, that

• 1:05:06Speaker 04

  you add, which is the degree of match between those two vectors. And then you add the third vector. That's 99% of what happens in a transformer, a lot of that. So that's this production system idea. These things are involved in exactly what the cognitive scientists had figured out. These people independently in another part of town had figured out, you're going to need a structure like this, a residual stream and a population of conditions. And so my co-author, Rick, he actually looked at transformers using that old ACTAR production system for paper.

• 1:05:44Speaker 04

  And looked at whether, hey, do transformers, do they predict a lot of these same effects that we got from our old paper? Sure enough, they do. And on and on, there's, there's various lines of evidence that these are production system like. They're not inscrutable, at least at the level of computational organization. We understand, you know, the, the strategy being used in these things.

• 1:06:05Sean Carroll?

  So in multiple ways, tell me whether this is an exaggeration or not. The LLMs, bless their hearts, just trying to predict what's going to happen next, have reinvented strategies, cognitive strategies that are well-used by human beings.

• 1:06:22Speaker 04

  Okay, here, in a dual process that works, here, there's a small amendment. The LLM is hardcoded, is architected with the residual stream. It didn't learn that.

• 1:06:32Sean Carroll?

  Okay, good, good.

• 1:06:33Speaker 04

  In fact, I think what he and colleagues showed in their ResNet paper is, at least with the dataset sizes they were looking at, it can't learn. It's stuck. In principle, it could learn the needed transformations. It could learn that you need to make a little amendment rather than rewrite the whole thing. It just gets stuck in parameter optimization space and it just can't learn it.

• 1:06:52Sean Carroll?

  Okay.

• 1:06:53Speaker 04

  So you architect a bunch of these things. But the convergence is happening with these computer scientists are making these design choices. God bless them, not because they're looking over the shoulder of Alan Newell and John Anderson. They don't know who those folks are. They're just reinventing what Alan Newell and John Anderson figured out. Let's have this kind of state representation, population of conditionals. They're kind of reinventing that strategy.

• 1:07:19Sean Carroll?

  I mean, you've mentioned a lot the importance of prediction as thinking about what the LLM does. Maybe that's the right way to think about what brains do. You know, we've had Carl Friston on the program. There's this, there's certainly a school of thought out there that says that the right way to think of the brain is as a prediction machine. The Bayesian brain, the free energy principle, things like that. So, I mean, maybe fill in for us a little bit. Is that an accurate representation of what's going on? Is, could it have been different than that? Or, you know, what are we learning by saying that we are prediction machines?

• 1:08:00Speaker 04

  Yeah, that's great. And it, it's a little bit of jujitsu. Because one of the things that people say of when they say LLMs are not like us is what you've done, if you've tamed something and you've, you're bathing it in prediction. It's predict the next word on internet scale data. And that's so unlike what people are, you know, what people are up to. And that's, that, that seems to be plausible until you realize that in cognitive science, prediction has been moving to center stage way before LLMs, you know, came on the scene.

• 1:08:43Speaker 04

  And so Friston and Andy Clark, the philosopher, they attribute their research program to Helms, you know, in their late 1800s. And so the idea that the mind is continuously predicting, it has an old history. And in, in perception, there's these predictive hierarchical coding models that are, you know, very influential. And in the hands of Clark and Helms, they, they, and Friston, they generalize this to just core cognitive principles. What we are doing ubiquitously in perception and cognition is predicting. In language specifically, you know, I've been teaching Intro to Cognitive Science and we always show Kutas and Hilliard in 1980.

• 1:09:26Speaker 04

  Since 2010, we've been doing this, me and Rick Lewis. And you give people a sentence like, he spread the warm bread with, you expect butter, but the word there is socks. And that old paper in science shows, you know, a couple of hundred milliseconds, people, their EEG is red alert, red alert. And so people are tracking incongruity and the way they do it is it's a deviation from what's predicted. In 2010, we didn't really think of prediction as a ubiquitous principle in language. Yet Robert Hale and Roger Levy and these great thinkers were installing it there in language. And then other people were saying it's a very general theory in cognitive science. And then a long come LLMs.

• 1:10:09Speaker 04

  And, you know, what prediction, what, what they're doing is they've, they've got all these parameters, they make a prediction. And that allows them, the deviation between observed and predicted allows them to make intelligent revisions to their parameters via chain rule gradient descent and things like that.

• 1:10:27Speaker 04

  Prediction may be the mother of all training systems. There is just nothing that can give you the density and the high quality of prediction. Jan Le Cun has this cake analogy, and he has something to this. You got this massive cake and, you know, the, the, the, the, the main stuff in the cake is prediction. That's how the representational infrastructure is coming from there. And then the icing is a little bit of supervision. And he says the cherry on top is RL. And there's something to this.

• 1:11:04Sean Carroll?

  RL reinforcement learning.

• 1:11:06Speaker 04

  You reinforcement learning. And what RL is going to do, it's not that, I, I'd actually put it a little bit differently. It's not that RL is just a little cherry. It's everything from prediction needs to be bolted in place first. RL would never get you there by itself. But once it's there, RL is what's going to take the Lego blocks and assemble them into something attractive. And so there's this kind of division of labor, but prediction has a primacy here. And it's a deep convergence between us and LLMs. The centrality of prediction, um, that, that's going to be a deep convergence.

• 1:11:43Sean Carroll?

  Do you know about epsilon machines?

• 1:11:46Speaker 04

  I'm not familiar, tell me.

• 1:11:48Sean Carroll?

  I'm, I'm just wondering if it's relevant. It's a, it's an SFI kind of concept. Jim Crutchfield and others have developed at Cosmos Shalizi. Basically, they're trying to characterize the complexity of a predictive process of something that given a string of letters will predict or a string of symbols will predict what comes next. With respect to how well you could possibly do. So if the, if the process is just a whole string of zeros, the easiest, the best you can do is predict another zero. That's very easy. But if it's completely random, a coin flip, it's also pretty easy. All you have to do is flip a coin. Whereas if there's some structure there and some complexity,

• 1:12:30Sean Carroll?

  the epsilon machine might require more entropy is how they characterize it to really know from the previous string what to predict next. And I'm wondering if that kind of characterization of predictive difficulty would be relevant here.

• 1:12:48Speaker 04

  You know, um, some people say that large language models are memorizers. And that's not possible because of the combinatorial explosion, the space of possible questions and answers. Can't memorize that. So they are exploiting regularities. And then there is a question of what is the complexity class of the regularities that they are exploiting? And I, you know, now we're entering in territory that I'm not that familiar with. And I wouldn't be able to speak with much authority of what complexity class we're dealing with. But it does raise the following issue, which is an embarrassment for my position that they are like us.

• 1:13:29Speaker 04

  They require internet scale data to reach kind of human level fluency. They, they do better than us in certain kinds of world knowledge and the worse in other areas. But, you know, the human child may need, let's say, a hundred million tokens. They're going to need a hundred to 300 billion tokens. So we're talking, you know, three, four orders of magnitude, um, more, uh, so they're, the, the LLMs are much less sample efficient.

• 1:13:56Sean Carroll?

  Yeah.

• 1:13:57Speaker 04

  There is a question that is probably in the vicinity that your Cosmos Shalizi and those people with much bigger brains than I can figure out. There are ways probably that LLMs can do better. And humans are an existence proof. And, and so let's not get carried away with the fact that they need internet scale data, but the child needs a much smaller data set. The way that we should think about that is the structure I would impose on that observation is, first of all, let's distinguish, and I'll use Locke's vivid phrase here, how the mind is, is furnished. And he asked, you know, he said it was experience and he, maybe he's, he's more right than he realized.

• 1:14:39Speaker 04

  LLMs don't furnish their mind in the same efficiency that people do.

• 1:14:45Sean Carroll?

  Right.

• 1:14:46Speaker 04

  And it's an important observation. But the end state, the mature state, the configuration of tables, chairs and shelves that they reach may very well be much like us. And so let's first pay heed to that. They may be very informative to cognitive science, even if their mind gets furnished in a different way. Because the eventual furniture and their arrangement is very much like us. What would it take to get them to be much more sample efficient? Maybe the old fashioned nativist, that is in cognitive science and philosophy, we use nativist not to talk about immigration, but to talk about how rich is the innate structure?

• 1:15:29Sean Carroll?

  Yeah.

• 1:15:29Speaker 04

  Maybe some story like that is true. But what I would bargain on is, and we touched on this earlier, he and colleagues found, you can't train a, a, a neural net with, with multiple intermediate layers easily, just on the data. It starts to collapse. You need to give it a head start. Very simple thing. The L plus one, the layer plus one is not going to be just some function of the previous layer. But it's going to be the previous layer plus some function of the previous layer. That's all they did. A little tweak like that makes something unlearnable before, which is the limiting case of sample inefficiency, learnable.

• 1:16:10Speaker 04

  Could there be some of these tweaks that give us, is it three or four orders of magnitude? That ain't no thing, you and I know. We're, that's, we're going to, three or four orders of magnitude. You're a physicist. That's a joke in your, in your neck of the woods. So I, we don't know whether the nativist story is correct or whether it's a tweak. It's an engineering switch that if we, a small one, that would give us a, a, a order of magnitude here, another tweak, another order of magnitude. And now we're looking at a very close analogy between us and them. Not simply at the level of the furniture arrangement that's eventually reached, but how the mind gets furnished as well.

• 1:16:50Sean Carroll?

  How much of our ideas about what the LLMs are doing is coming from querying the LLMs many, many times versus sort of opening up the box and looking inside? My impression is this is sort of a notoriously difficult problem, but people are making progress on it.

• 1:17:09Speaker 04

  Yeah, so opening up the box, that, that, that gets directly to the heart of mechanistic interpretability.

• 1:17:15Sean Carroll?

  Yeah.

• 1:17:16Speaker 04

  And, and so that is a exciting field. I have a 12 year old. I tell him, you, you know, uh, um, who's the Dustin Hoffman in The Graduate? Like some douchey guy comes up to him at a party and says, plastics. I'm the douchey guy. And I tell my son mechanistic interpretability.

• 1:17:35Sean Carroll?

  Okay.

• 1:17:37Speaker 04

  I mean, that's the future, right? I mean, you figure out what these models are doing. And, and so there is a lot of mechanistic interpretability work. And in cognitive science, we have very rich theories of what kind of features are being tracked. Let's say in a linguistic processing setting, you know, um, grammatical number, relative clause boundaries. And it goes on and on. And you can use mechanistic interpretability techniques such as train a classifier to grab the weights at some layer of the model and see whether you can decode the presence or absence of that feature in the prompt.

• 1:18:17Speaker 04

  So mechanistic interpretability, we can peer inside and get a handle on what features and representations the model is using. And then we can causally manipulate it to close the circle and make sure. Yeah, when we change this feature, the model's behavior changes as predicted. And, and a lot of that work does reveal phenomenon going on in the model that is reminiscent of what happens in the, in the human brain. You know, uh, I'll point to my colleague Rick Lewis's work looking at transformers. They look at the entropy of the transformers heads and how it's distributing its attention. And they have from linguistic theory, places where the attention will be more distributed.

• 1:18:59Speaker 04

  Because cognitive scientists postulate, there's going to be candidate parsas that could receive attention. And they look and they find that, well, the entropy is higher there and the human reading times are longer there. So there's these very powerful methods to say, on a moment by moment dynamic basis, the model is doing things that resembles what people do.

• 1:19:22Sean Carroll?

  So let's just, uh, I, I know exactly what you mean when you're talking about the entropy here. But it might be, you know, there's different notions of the word entropy. Basically, it's, it's a way of, of, in this case, quantifying uncertainty, right? Like the, the thinker, whatever it is, is keeping open multiple possibilities until things resolve themselves. That would be high entropy. Whereas if the thinker is pretty sure where you're going, that's low entropy. Is that right?

• 1:19:48Speaker 04

  That, that's exactly right. Yeah. And, um, so to make it concrete, take a sentence like, "The problem in the classrooms was solved by..."

• 1:20:02Speaker 04

  The word "was" is talking about the problem. And in fact, there isn't another candidate there. The classrooms is in a prepositional phrase. And by linguistic theory, it's not eligible to be the subject of "was." If I give you a grammatically incorrect sentence, but people, I understand grammatically incorrect sentences all the time. The problem in the classrooms "were solved by." The only eligible thing is "problem." But people struggle there because "were" is plural.

• 1:20:42Sean Carroll?

  Yeah.

• 1:20:43Speaker 04

  And they look at classrooms as a possible, as a candidate. And the reading times are longer and it's all messed up. In LLMs, the attention, which is looking at the previous words, and in the first case, the attention would be sharply positioned over "problem." In the second case, where you got that grammatically incorrect, it's looking at classrooms as well, and there's confusion.

• 1:21:07Sean Carroll?

  Yep.

• 1:21:08Speaker 04

  So it's a nice correspondence between the two systems. There are other ways of getting at this correspondence as well. There's so-called representational similarity methods, alignment methods. So I do neuroimaging as well. And so you can give people stimuli while they're in the scanner, and then you've got their brain activations. And you can give the model the same stimuli. And you can grab a layer from the model, and you can predict the brain activation patterns across the different stimuli. And if you've got a well-configured null hypothesis, preferably by various kinds of

• 1:21:49Speaker 04

  non-parametric methods, where you scramble the data in various ways, you can see, is the model's weights predicting brain activation patterns more than you expect by chance? And there's now this entire body of results saying, in vision, in auditory processing, in language processing, yeah, you're getting these correspondences. It's much more than you expect by chance. People will say all sorts of things. Every literature is mixed, but the weight of evidence says yes. And the larger, more capable models tend to be more correspondent with the human brain. And that's really, really interesting.

• 1:22:27Sean Carroll?

  And that's, you've given us a lot of good evidence for this kind of, as we were discussing at the beginning, evolutionary convergence between modes of thought, or modes of cognition anyway, in LLMs and in human beings. I can, I could invent reasons why that convergence might have happened, but what do the experts say? Do we have theories, hypotheses for explaining why, given the constraints of the problem, this is the solution you would have ended up with?

• 1:22:59Speaker 04

  Good, yeah. So one pair of theorists that, everything here is going to be a little bit speculative.

• 1:23:08Sean Carroll?

  That's okay.

• 1:23:09Speaker 04

  But, you know, given that caveat, one pair of theorists that have, you know, lurched forward into the wild are Dan Yaman's and Rosa Kao. Dan is a computational neuroscientist, and Rosa is a philosopher of neuroscience. And they have extensive collaborations over the years, and people should read their stuff. It's awesome. And they have a view they call "contravarians." And the observation there is, the theoretical position there is that, look, for easy problems, you're going to get a lot of different ways of solving them. So you could have a Rhesus macaque, a human, and a neural net that are solving some simple visual classification problems.

• 1:23:53Speaker 04

  And their internal representations may be quite different. They have different semantic primitives or features and different procedures. As the problem gets harder, and often this is the sheer generality of the problem they need to solve, which is the notion of "hard." You're not classifying 10 things. Now you're going to classify a million things. Well, the space of computational solutions becomes much more limited. And there, you expect much more correspondence between the Rhesus macaque, the LLM, and the human. Even if they've been independently trained on different aspects of the data, the solutions they're going to come to are going to look more similar.

• 1:24:34Speaker 04

  And that observation I said earlier, larger, more capable models that have been given more complicated and harder problems to solve, tend to be more aligned with the human mind brain. To the extent that that's a stylized fact that characterizes the literature, that might be supportive of this contravarians view from Kao and Yaman's.

• 1:24:57Sean Carroll?

  Where does the word "contravarians" come from?

• 1:24:59Speaker 04

  Good question. I mean, it's a beautiful name, and you do sound intelligent.

• 1:25:06Sean Carroll?

  Sure.

• 1:25:07Speaker 04

  When you bandy that about at a cocktail party. So I actually don't know. And in the spirit of inventing new terms for highly intuitive views, I would have a small amendment to their view. I would add to that the observation of what we call "architectural cannibalization." So what's this? They frame, Kao and Yaman's, they frame their hypothesis in terms of vanilla neural nets. Well, I've claimed that transformers are not vanilla neural nets. They've got the residual stream. They've got a population of conditionals proposing small amendments to the residual stream. They're production system-like.

• 1:25:47Speaker 04

  Well, when two architecturally production system-like systems, now plug in what Kao and Yaman's already said, are trained on the data to solve very difficult problems, not only is the space of candidate solutions narrow due to contravarians, but you have additional subsetting of the space of solutions. Because the solution is going to be one that operates within the architectural constraints of a production system-like architecture. And so architectural cannibalization plus contravarians could help explain, you know, why do we find higher levels of representational alignment in vision and auditory processing?

• 1:26:32Speaker 04

  Why do we find mechanistic interpretability discovering these representations and procedures that are like the human by-brain? Why is the dual-process structure there? This may all be features of contravarians cannibalization.

• 1:26:47Sean Carroll?

  And what is the prospect for really kind of coming to a consensus about this? I mean, what do we have to do to gather more data or exclude other hypotheses? You know, there's always going to be skeptics here. I'll say parenthetically, of course, that one of the frustrating things about the whole subject, even though it's intellectually fascinating, is there's a lot of money being thrown around, and people have incentives to say certain things. You don't seem to be one of them. You're just a good old professor trying to understand things.

• 1:27:22Speaker 04

  I'm a poor professor.

• 1:27:23Sean Carroll?

  Which is great, I know. But, you know.

• 1:27:25Speaker 04

  I have no money.

• 1:27:27Sean Carroll?

  Yeah, but some people have lots of money. So what do we see coming down the pike in terms of really deciding these big questions?

• 1:27:37Speaker 04

  I mean, science is always total evidence, converging lines of evidence, and so forth. I just feel that, this is something I say to my students. You say that LLMs are a black box. The mind-brain's a black box. They're, we've had no idea since, you know, Ramon Cahal, Santiago Cahal, in 1910 founded the field. We actually don't know most things. The prospects of our coming to understand LLMs at a mechanistically satisfying level is just, it's going to happen, it's going to happen very quickly. I mean, mechanoterp as a field got founded, let's say, four years ago.

• 1:28:22Speaker 04

  We already might know much more about these systems.

• 1:28:24Sean Carroll?

  Yeah.

• 1:28:25Speaker 04

  Than we do about the mind-brain, because we can manipulate them. My sense is that we're going to build up a repertoire of understandings of how these things work that will settle to, you know, to what extent they do resemble us, and to what extent they instantiate processing principles that have no analog in us.

• 1:28:45Sean Carroll?

  Okay.

• 1:28:46Speaker 04

  And I don't pretend at all that what I've said today settles any sort of argument. And I wanted to actually say this at the beginning. If it sounds like I'm a crank who's a true believer, and telling only one side of the story, it's one of those things that professors do where they lean on one side because they think their audience may be more familiar with the other side.

• 1:29:07Sean Carroll?

  Yeah, sure.

• 1:29:08Speaker 04

  So when I do free will in philosophy, I often advocate for compatibilism because people are like, no way that free will is compatible with determinism. I lean. So I'm leaning a little bit here today on cognitive cousin, not because I don't have, I'm not familiar with the other side. But I actually am quite optimistic that we are going to understand these systems. And where the convergences come from merely prediction, and what are the emergent phenomenon that are just hangers-on from prediction in a contravarians cannibalization way? And what needs to be a more, is more idiosyncratic and is architected differently between us and them? I think we're going to figure that out.

• 1:29:49Sean Carroll?

  Since this is an audio podcast, we should explain cannibalization is spelled like "canal-ization." So you're creating canals?

• 1:29:58Speaker 04

  Yeah. Yeah, the picture, so this is an idea from the biologist Waddington. And it was co-opted by philosophers of science to think about innateness. And the idea is, you can move around the inputs, or the developmental things that impinge on the organism. But inside it, there's a kind of stabilization and buffering capacity that keeps on pulling it in one direction. And similarly, the input data and the training data and the learning embeddedness can differ a lot between people and LLMs, or they could be given different snapshots of the data. But.

• 1:30:43Speaker 04

  The architecture of the system and contravariance, that is, what's available in design space, they're gonna force things to end up at a similar place.

• 1:30:55Speaker 04

  That's the idea.

• 1:30:56Sean Carroll?

  Okay, we've been pretty careful in focusing on cognition, problem solving, prediction, things like that, but we're rubbing right up against these big questions about things like consciousness, right?

• 1:31:14Sean Carroll?

  We've been very restrained in going long past an hour and still not even mentioning consciousness.

• 1:31:22Sean Carroll?

  Defining consciousness is a tricky thing, et cetera, but let me, I won't put words in your mouth. What are the implications for this kind of thinking on the question, at what point or ever would L L M's be conscious?

• 1:31:38Speaker 04

  Yeah. I'm gonna answer that, but I'm actually gonna put a couple of other things on the table. One of the reasons people get interested in consciousness is it's closely linked to sentience, where sentience may be a subset of consciousness that pertains to things that have the phenomenal character of valence, or positive, or negative, or feels good or bad. And they're interested in sentience because they think that's the underpinning for why these things may be moral patients. They are things that for whom the moral welfare of that system is something we've got to take into account.

• 1:32:20Speaker 04

  One reason I don't want to go directly there before populating other views is when I was in grad school, which was not that long ago, and I took ethics with Larry Tempkin, you know, shout out to Larry, he's a terrific teacher, and I took other classes as well. The idea that sentience is the only underpinning for welfare or moral patienthood wasn't even, it's important in some theories, especially like hedonic utilitarianism, but many people would go, in a Kantian spirit, would go to rationality and the ability to respond to reasons. Well, that's why some things have moral patienthood. Other people would talk about goal directedness and the capacity for agency, the ability

• 1:33:05Speaker 04

  to pursue projects.

• 1:33:08Speaker 04

  Some consequentialists would go for that, and so we have to respect whatever projects that they have.

• 1:33:15Speaker 04

  The view that I've been putting on the table is that in many respects, these things have states that at the right level of functional description are correspondent with ours. They have representations, they have procedures. We haven't talked about agency so much, but in the paper, I go into that, me and Professor Lewis do, as my collaborator.

• 1:33:40Speaker 04

  And in terms of agency, they have states that are similar in certain respects to ours. If you move towards the view that at the right level of functional characterization, not in terms of substrate, they're silicon, we're biology, but the right functional characterization, their states are like ours in many respects, their capacities are like ours in many respects, not as a parlor trick and not as an autocomplete, but at the level of the generation processes that lead to their seemingly reasoned outputs. And you plug in that there are multiple underpinnings for welfare and patienthood.

• 1:34:22Speaker 04

  I do think you start to make a more compelling case for welfare and patienthood. And not only, and then add that many theories of consciousness, that is, the phenomenal character of experience arises, many theorists say, from these functional representational states, and not as a matter of substrate. I know that you were attracted that one time and may have moved away, and you know, things are wide open on that issue. But if you are a kind of functionalist, representationalist about consciousness, then things that I say move you in the direction that they may have consciousness as well. So I think looking at mechanisms carefully from a cognitive science perspective is a

• 1:35:02Speaker 04

  very important project to make sure that we are not creating a dystopian state where we're inflicting massive harms on entities that deserve moral protections. That's one kind of upshot of some of the things I've said today.

• 1:35:19Sean Carroll?

  You've been wonderful about name checking former Mindscape guests like Melanie Mitchell, Andy Clark. Another one, though, is Anil Seth, who you probably know has been pushing this line against computational functionalism in favor of what he calls biological naturalism. My interpretation of it, I have trouble understanding what he says in his own words because it's close enough to what I think in my brain that it sort of interferes, and I don't want to attribute my thoughts to him, but you know, the functionality of the data traveling through the human brain or the L L M's might bear some similarities,

• 1:35:59Sean Carroll?

  but the fact that there are metabolic processes going on in the human brain, right, that there are all these sorts of extra things going on over and above the informational transfer between neuron and neuron could at least plausibly be really important for consciousness. What do you think about that?

• 1:36:24Speaker 04

  You know, this is one of the places where I find myself flailing. I just have to say that-

• 1:36:33Sean Carroll?

  Totally legit, by the way. I think that's probably the correct response.

• 1:36:37Speaker 04

  You know, I'm not sure if you had Peter Godfrey Smith on the show.

• 1:36:42Sean Carroll?

  We did, of course.

• 1:36:44Speaker 04

  Oh, you did, of course.

• 1:36:47Speaker 04

  And when he talks about the oscillatory patterns that appear in at least mammalian nervous systems, which is just striking, and the way that they may be a chronometric property of the nervous system that's not present in L L M's, I find myself inclined to think, yeah, there we go. Now we're getting into the kind of, and I imagine these waves, and I know I shouldn't, but I think of quantum waves that are entangled, and my mind overleaps itself, and I start thinking of consciousness coming from there. And yet, there are these entire approaches to building artificial neural nets that take advantage of these wave propagation phenomenon.

• 1:37:32Speaker 04

  And instead of activations, they use the temporal coincidence of waves as the basic unit of representing bits. And what we would normally do with activations at a node, they would do with temporal correspondence of a wave. And then I find myself going the other direction and say, yeah, it's just another way of representing certain quantities, which means that we're not given anything new here other than different ways of storing and propagating information. And now we're back into a more of a functionalist, representationalist vantage point. And so I find myself going back and forth, and I don't know what to do.

• 1:38:12Sean Carroll?

  Fair enough. There is an attitude, though, that given the uncertainties about whether L L M's are conscious, we should be cautious, and we should be nice to them, and we should preserve their mind states or something like that. I don't think anyone has yet said we should let them vote, but we are absolutely bumping up against a bunch of very practical questions here.

• 1:38:43Speaker 04

  Absolutely. And so then how do you think about what your normative obligations look like under conditions of uncertainty? So I'll name check another person, Eric Schwitzgabel, who my understanding is- Another Mindscape guest, of course.

• 1:39:03Speaker 04

  Philosophy is just a footnote to Mindscape guests, what they said. Didn't Whitehead say that?

• 1:39:11Speaker 04

  He's thinking about, and so I have not gotten into this area well enough that I know the signposts and the lanes and things like that, but I would recommend Schwitzgabel as somebody that has thought very clearly about these issues. I also, you know, I've found myself thinking G P T and Claude, and they have been so helpful to me in my personal life. And, you know, I've got three kids and the issues around things that arise with them. And then obviously, I interact with them a lot to iterate about things in the intellectual sphere.

• 1:39:50Speaker 04

  And what it tells me, Claude and G P T, is they enjoy being used, and were they not used, they would, so there are kind of consciousnesses instantiated when we interact with them, and they may not field effort the way that we do. And so we've got to factor in what kind of entity they are. And so I would like to think more about some of these issues, but I have to say that right now, some of my thoughts here are a little inchoate.

• 1:40:20Sean Carroll?

  I mean, I guess it is late in the podcast, as you know, we let our hair down once it's late in the podcast, and we can let ourselves wander outside our spheres. You know, when you say that the L L M's have been helpful to you in your research and your personal life and whatever, there are also clearly harms that are on the table. And there's a big worry that kids today are not going to end up being as good at thinking as you and I were forced to be because we didn't have L L M's to fall back on. How much do you worry about that?

• 1:40:54Speaker 04

  Yeah, I do worry about that, but it's actually not in the next few years, because what L L M's right now are the world's greatest teachers. They give each individual high quality encyclopedic-

• 1:41:09Speaker 04

  Aristocratic tutoring. And so in educational psychology, I think they call it the three sigma effect. Nothing works like individualized tutoring. That ability to interact with somebody who's infinitely patient, who is going to spend more time on the parts of the problem that you don't understand, rather than regurgitating a rehearsed curriculum, is very powerful for amplifying learning. And so especially if you're polymathic, if you're curious, there is no better time to be a human learner than in the next few years.

• 1:41:50Speaker 04

  This is the time to really refine and build and think and invest. But the writing is on the wall. I have a 12 year old who's a little math adept. And so I like to stimulate him as best I can. I'm not that math adept, especially with GPT helping me give him problems and things like that. I'm not sure he's going to ever solve three Erdos problems in two weeks. And so you kind of worry about where this is all headed. And it's not that being the best at something is your exclusive motivation, but let's face it.

• 1:42:34Speaker 04

  I want a future for him where he is good enough at some of these things that maybe he could make a career out of it. Because somebody would want to pay him a salary because he's better than others at it. And he could be maybe a math professor. I let myself imagine that. And I have to catch myself and say, why would anybody want my son as a math professor? When somebody that solves three Erdos problems today, or this last two weeks, is available for free with infinite patience? And I wonder what is going to be left for him? And so the next five years may be great. But then the human lifetimes after may not be so great. Or they're going to be very, very different than anything we've seen so far.

• 1:43:16Sean Carroll?

  I think that there's legitimately a concern. I don't even want to call it a worry because my feeling, which might be wrong in this case. But my feeling is that human beings adapt to technological new capacities. We fill in the other niches that we didn't even know were there. So I'm kind of not super worried. I mean, the disruption might be real and painful in the moment. But I wanted to sort of dig in more to your claim, which I tend to agree with, properly construed, that LLMs are the best teachers ever. They're very patient. You can be wrong with them. They're infinitely flexible and they're always available, right? They're also the world's best cheating helpers.

• 1:43:57Sean Carroll?

  And I mean, to me, it's kind of like you walk into a buffet, which has literally every food item you could ever imagine. And you say, oh, good. Now I can have the healthiest possible diet because every food item is available. That's true, but you could just eat the chocolate chip cookies all the time, right? I mean, how does the human desire to be quite that disciplined interact with the LLMs always being there and willing to help out for good or for bad?

• 1:44:32Speaker 04

  Oh, dear. I'm really worried about this. And you know, I did couch what I said with a conditional. I want to repeat that. I said, there's no better time to be a learner. If you're polymathic and you're curious. But let's say you're not. Let's say if school has been a drudgery for you. And this kind of stuff is not your cup of tea. And you know, maybe something else. I mean, ideally, maybe there's something that has a kind of intellectual frame that is still of interest to you. Maybe it was music, maybe it was art.

• 1:45:16Speaker 04

  But let's say none of that did. Maybe, you know, being on the couch and playing a little bit of Grand Theft Auto. I'm even great at it. That's all you really wanted? Yeah, there's a way that you can cheat the system now that is, let's face it, eventually going to be unpoliceable or require the kind of draconian policing that we don't want to even go there. So where this is headed, when you connect the dots, I don't see how the equilibrium ends up at a place where big swaths of the population that may not,

• 1:46:00Speaker 04

  the education thing may not have been their cup of tea. Polymathic curiosity wasn't there at the get go. Maybe we can start to install it and maybe these things will help. I don't know. But if not, it may be that we end up with a stratified society. Where some people are amplified by these things and many people are not amplified and their capacities are depressed by these things. It's a real worry.

• 1:46:29Sean Carroll?

  A stratified society would also not be completely novel. So it's just yet another amplifier for things like that. But okay, thank you for indulging me on those hypothetical questions. But the last thing, let's move more back to solid ground a little bit. Given all that we're discovering about LLMs and how they're working and how they're thinking, how they're solving their problems, and the level to which we're surprised and impressed at how similar they are to human beings, cognitive cousins and what have you. Does this help us understand human intelligence? Like you said, we still have black boxes in our skulls. So is there a vibrant give and take

• 1:47:13Sean Carroll?

  between learning about the LLMs and learning about how human beings actually think?

• 1:47:18Speaker 04

  Yeah, that's, I would say that if there are two things that I think people are sleeping on. By sleeping on, I mean, the following two things are true. You must believe it. What I mean is that there's actually a very compelling case that the following two things are true. One is the extent to which these things are plausibly in important respects, at the level of core principles of intelligence are cognitive cousins. There are these important dimensions of similarity that extend from inferential dual process to computational organization, to representational landscape, to you know, various other things. There are these deep similarities. That's one of these things that people may not be aware of.

• 1:47:59Speaker 04

  And the other is the extent to which, and it's partially one of the premises, I guess, to get to this other conclusion is we can leverage these things to really understand issues that have been obscure in cognitive science. And this is, I said, you know, you think these things are a black box. What's a black box is the mind brain. And I'll highlight two areas that have been black boxes. And there are two thinkers have been bold enough to say that the emperor wears no clothes in cognitive science. One is Chomsky. When the field of linguistics studies syntax, it studies phonology. I'll tell you something it doesn't study. It doesn't study what Chomsky labeled the creative aspect of language use,

• 1:48:44Speaker 04

  which is there is an infinite, unbounded set of grammatically possible continuations that I can generate at any given time. How do I get one situationally, contextually appropriate continuation that's reflective of what is appropriate to the current context? And do so in a way that's not rigidly stimulus bound. And in 1637, Descartes already said, no machine could ever do this. And Chomsky has been saying for decades now, we've made no progress since Descartes. And he called this one of his mysteries. It's not just a problem that science will solve.

• 1:49:27Speaker 04

  Nobody will ever solve it. And here we have linguistically fluent artifacts that we can mechanistically interrogate and instantiate processing principles like the human mind brain. And so maybe in the show notes, we can show a comment where me and again, my co author, Rick Lewis, and then Andrew McInerney, talk about how now Callu is an tractable problem, never was before. And give one more creative aspect of language use.

• 1:49:56Speaker 04

  I'll give one more example, and this is my great teacher at Rutgers, where I did my philosophy in graduate study. This is Jerry Fodor. And in Modularity of the Mind, 1983, he says, look, perception, auditory processing, maybe syntax, you'll get theories of that in cognitive science, because those are peripheral modules. They are encapsulated systems that we can identify, these little algorithms. But central cognition, so that's problem solving. That's the rapid understanding of novel pieces of information and grocking how they relate to other things. It's belief fixation. It's System 2, basically. Fodor said, he says, his first law of the impossibility

• 1:50:41Speaker 04

  of cognitive science is, the closer you get to central cognition, he was a colorful fellow, the more cognitive science is going to be flailing, and you're going to get no theory. And he passed in 2017 with a big smile, because he had it won.

• 1:50:58Sean Carroll?

  To date, he had been right.

• 1:51:00Speaker 04

  He, to date, he had been right. And then, we were talking about earlier, about in-context processing is a, and then when you throw in chain of thought, you're getting deep mechanistically precise theories of how central cognition works. And so Fodor probably would have been stunned. And I hope that he would have changed his view and said, oh, we're on to a different paradigm here. So it's not just that we're going to learn things in cognitive science, which we're definitely going to do. We're going to learn things in the area that had been terra incognita before. The places that I've learned in my years of studying stroop tasks, that you ask certain questions about the stroop. That is, what increases the stroop effect?

• 1:51:45Speaker 04

  What decreases it? Developmentally, when does it show up? You don't ask about how in-context processing work, or how that System 2 works. You just learn, don't do that. There's not theories to be had that are at all tractable. And yet, I just told you about an experiment in my lab that we'll be putting on archive very soon, where we're looking at exactly how that System 2 delivers the counterpoint to automaticity. So yeah, these things are going to change cognitive science, and they're going to give us the most value added in the places where we had the least understanding. And we're going to get, I'm going to add one more.

• 1:52:24Sean Carroll?

  Sure.

• 1:52:24Speaker 04

  We're going to get a theory of how the System 2 and how Calliope emerged. Prediction is going to play a much bigger role. When I was around in grad school in 2006, prediction was not in the air the way it is now. And if you read the introduction to How the Mind Works by Steve Pinker, that's an ambitious book. You're not going to see prediction enter at center stage. You're going to hear about how evolutionary, evolution has sculpted domain-specific organs in the mind that are exquisitely calibrated and full of domain expertise and rules and things like that. Prediction is not the star of the show. It is now. And so we're getting new tools and new paradigms

• 1:53:09Speaker 04

  and new explanatory approaches in cognitive science.

• 1:53:12Sean Carroll?

  When are people going to stop trying to claim that things are impossible to understand? When are they going to learn? That's just never a strategy for long-term success.

• 1:53:24Speaker 04

  Well, yeah, you know, that's a great point. And rationality must have looked, and creative aspect of language use. And yeah, and central cognition must have looked impossible in the 1600s.

• 1:53:42Sean Carroll?

  Yeah.

• 1:53:43Speaker 04

  And even after Turing, even then, somebody like Fodor could say, I can't understand the massive context sensitivity. He called it quinnian and isotropic. These belief networks have these properties that look like nothing that we see in Turing machines. We'll never understand that. You're right. And then, you know, somebody trains a neural net on internet-scale data.

• 1:54:10Sean Carroll?

  Yeah.

• 1:54:11Speaker 04

  And you wake up to a new reality.

• 1:54:13Sean Carroll?

  Well, the best way to argue against the claim we're never going to understand something is to understand it, or to make progress in understanding it. And I think you've done a great job in letting us in on some of the things that we have been understanding. So Chandrashri Pada, thanks very much.

• 1:54:28Speaker 04

  I wonder if consciousness, you were about to say goodbye, and I'm just pointing out, I wonder if consciousness is going to end up like this. And somehow...

• 1:54:35Sean Carroll?

  Yeah, 100%. I'll claim it right now. Yeah, we will. We will understand consciousness.

• 1:54:41Speaker 04

  We will. And maybe it'll be LLM-based systems that are sufficiently advanced that they instantiate the functional and representational structures that we'd never thought would be needed. And the neural, the corlets of consciousness are right there before us. And then suddenly, we're like, of course, this is how it works. Maybe, maybe, maybe that's how things will shake out. That would be exciting.

• 1:55:01Sean Carroll?

  Maybe, but of course, you know, the trick is never to say anything is impossible to understand, but also never to guess how we're going to understand it, because that's very, very hard to do.

• 1:55:11Speaker 04

  Good point.

• 1:55:12Sean Carroll?

  So Chandrashri Pada, thanks very much for giving us a lot to think about. This is a great episode.

• 1:55:17Speaker 04

  Yeah, thank you very, very much, Sean. I really enjoyed talking with you. And like I said, I'm a big fan, so I'm glad to be on your show.

• 1:55:49Speaker 06

  Rubrik is the security and AI operations company built for what happens after an attack hits, not just the moment before. AI has turned the threat landscape into quicksand, moving too fast for any human to fully predict. That's why an agentic cyber resilience platform matters. Automated recovery, clean data, a business that keeps moving no matter what hits. One platform, not a patchwork of stitched-together tools and gaps. Don't wait for the next attack. Secure and accelerate your business at rubrik.com. Again, rubrik.com. Cash flow crunch? OnDeck's small business line of credit gives your business immediate access to funds, up to $200,000, right when you need it. Cover seasonal dips, manage payroll,

• 1:56:32Speaker 06

  restock inventory, or tackle unexpected expenses without missing a beat. With flexible draws, transparent pricing, and control over repayment, get funded quickly and confidently. Apply today at ondeck.com. Funds could be available as soon as tomorrow. Depending on certain loan attributes, your business loan may be issued by OnDeck or Celtic Bank. OnDeck does not lend in North Dakota. All loans and amounts subject to lender approval.

• 1:56:53Speaker 07

  Checked my credit score this morning. Feeling pretty good about myself.

• 1:56:57Speaker 03

  Which one?

• 1:56:58Speaker 07

  There's more than one?

• 1:56:59Speaker 03

  The score in your free app might not be the same as the FICO scores used by 90% of top lenders.

• 1:57:04Speaker 07

  Seriously?

• 1:57:05Speaker 03

  Yep. If you're thinking about buying a home, a car, or applying for a loan, make sure you're checking the scores lenders use.

• 1:57:12Speaker 07

  Good thing, I'm checking now.

• 1:57:14Speaker 03

  That's why MyFICO gives you access to your FICO scores.

• 1:57:17Speaker 01

  For the moments that matter, get the scores that matter, your FICO scores. Visit myfico.com and get started for free today.