Large Language Models and Emergence: A Complex Systems Perspective (Prof. David C. Krakauer)

Prof. David Krakauer, President of the Santa Fe Institute argues that we are fundamentally confusing knowledge with intelligence, especially when it comes to AI. He defines true intelligence as the ability to do more with less—to solve novel problems with limited information. This is contrasted with current AI models, which he describes as doing less with more; they require astounding amounts of data to perform tasks that don't necessarily demonstrate true understanding or adaptation. He humorously calls this "really shit programming". David challenges the popular notion of "emergence" in Large Language Models (LLMs). He explains that the tech community's definition—seeing a sudden jump in a model's ability to perform a task like three-digit math—is superficial. True emergence, from a complex systems perspective, involves a fundamental change in the system's internal organization, allowing for a new, simpler, and more powerful level of description. He gives the example of moving from tracking individual water molecules to using the elegant laws of fluid dynamics. For LLMs to be truly emergent, we'd need to see them develop new, efficient internal representations, not just get better at memorizing patterns as they scale. Drawing on his background in evolutionary theory, David explains that systems like brains, and later, culture, evolved to process information that changes too quickly for genetic evolution to keep up. He calls culture "evolution at light speed" because it allows us to store our accumulated knowledge externally (in books, tools, etc.) and build upon it without corrupting the original. This leads to his concept of "exbodiment," where we outsource our cognitive load to the world through things like maps, abacuses, or even language itself. We create these external tools, internalize the skills they teach us, improve them, and create a feedback loop that enhances our collective intelligence. However, he ends with a warning. While technology has historically complemented our deficient abilities, modern AI presents a new danger. Because we have an evolutionary drive to conserve energy, we will inevitably outsource our thinking to AI if we can. He fears this is already leading to a "diminution and dilution" of human thought and creativity. Just as our muscles atrophy without use, he argues our brains will too, and we risk becoming mentally dependent on these systems. TOC: [00:00:00] Intelligence: Doing more with less [00:02:10] Why brains evolved: The limits of evolution [00:05:18] Culture as evolution at light speed [00:08:11] True meaning of emergence: "More is Different" [00:10:41] Why LLM capabilities are not true emergence [00:15:10] What real emergence would look like in AI [00:19:24] Symmetry breaking: Physics vs. Life [00:23:30] Two types of emergence: Knowledge In vs. Out [00:26:46] Causality, agency, and coarse-graining [00:32:24] "Exbodiment": Outsourcing thought to objects [00:35:05] Collective intelligence & the boundary of the mind [00:39:45] Mortal vs. Immortal forms of computation [00:42:13] The risk of AI: Atrophy of human thought David Krakauer President and William H. Miller Professor of Complex Systems https://www.santafe.edu/people/profile/david-krakauer REFS: Large Language Models and Emergence: A Complex Systems Perspective David C. Krakauer, John W. Krakauer, Melanie Mitchell https://arxiv.org/abs/2506.11135 Filmed at the Diverse Intelligences Summer Institute: https://disi.org/

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:01Speaker 00

  Into the wild. The Calthea Panther encounters a shimmering crocodile, a lonely lovebird, a breathless zebra, and a restless tiger.

• 0:15Speaker 01

  I love football season. The games, the rivalries, the food, getting everyone together. No game day feels complete without a trip to Total Wine & More. I can find everything, from cold beer and canned cocktails to wines and spirits for the whole crew. And if I want to bring something different to the watch party, the Total Wine team always has ideas. Plus, it's nice knowing I'm getting the lowest prices. Get your game day lineup ready at Total Wine & More. Spirits are not sold in Virginia and North Carolina. Drink responsibly. B21.

• 0:43Speaker 02

  The brain is an organ, like a muscle. If I outsource all of my thinking to something or someone else, it will atrophy just as your muscles do. There's nothing confusing about that. It's just a fact of physiology. So I'm David Krakauer, and I work on the evolution of intelligence and stupidity on planet Earth. Science is a humanistic endeavor. The purpose of science in the universe is to make the universe intelligible to us, not to control it, not to predict it, and not to exploit it. Science is no different from poetry

• 1:28Speaker 02

  is that we're trying to make sense of the world, trying to give it meaning in relation to our own existence. Superintelligence is only interesting to the extent that it makes me more intelligent, not to the extent it makes me more stupid or more servile or more dependent.

• 1:48Speaker 02

  Oh, good.

• 1:48Speaker 00

  Oh, has the camera just stopped?

• 1:50Speaker 02

  That was actually, that was sympathy. That was sympathy.

• 2:00Tim Scarfe?

  Amazing. Well, I think the first paper of yours I read was a couple of years ago with Melanie. And we interviewed Melanie about the debate of understanding and language models, I think it was called. And at the time, there was this fervor, you know, this hype around, you know, the Sparks of AGI paper, for example, and they had early access to GPT-4 without RLHF. And they were saying, isn't it amazing that we have these emergent capabilities?

• 2:26Speaker 02

  Yes. No comment. Right. Well, there's just this whole question of, as you know, I mean, what emergence is, what intelligence is.

• 2:41Speaker 02

  And we can talk about it all. My particular interest is the evolution of intelligence, and which I consider extraordinarily varied. So I consider bacteria intelligent, as you probably know. So.

• 2:56Speaker 02

  And my basic, I guess one way of framing a lot of this is that,

• 3:02Speaker 02

  for me, intelligence manifests most clearly when you can do a lot with very little in terms of input.

• 3:13Speaker 02

  And I'm less and less impressed when you manifest so-called intelligent behaviour, when you have more and more and more information at your disposal.

• 3:24Tim Scarfe?

  Yes.

• 3:25Speaker 02

  And unfortunately, the way that AI has evolved is in the direction of confusing being very knowledgeable with very intelligent. And I think that's, in some sense, encapsulates my critique.

• 3:41Tim Scarfe?

  Yes. You said in your talk yesterday that all of the reasonable definitions of intelligence, if anything, tried to marginalise away the contribution of knowledge. It's about, you know, adapting to novelty, adaptivity in general. Your evolutionary perspective is very interesting. You're kind of pointing to this yesterday when you were saying about the history, preserving and accumulating information, but preserving that history is very important, not only phylogenetically, but also ontogenically. Can you explain what you mean by that?

• 4:11Speaker 02

  Yeah. So, I mean, it's a really interesting question, right? I mean, one of the big confusions.

• 4:17Speaker 02

  In the intelligence world is whether we're allowed to call ingenious adaptations intelligent. That is capabilities, adaptations, is that an intelligent thing? Or is that just what evolution gives you? And there are people out there who would say, no, that's not intelligent. Because intelligent isn't, isn't a capability. It's the ability to acquire capability.

• 4:46Speaker 02

  The capacity to acquire capacity, as Woodrow puts it. And Francois Charlet and others like that have adopted that perspective. So the question is, where does that come from?

• 4:59Speaker 02

  And you can actually derive this mathematically. In the 1970s, a very prominent theoretical chemist who won the Nobel Prize, actually, for high speed chemistry, Manfred Aigen, started developing a series of theories that were finally formalized in the 80s called quasi-species theory. And to cut a very long story short, what that theory gives us is a fundamental bound on the rate at which information can be acquired in any evolutionary process.

• 5:42Speaker 02

  And it turns out that the fundamental speed limit is established by the generation time.

• 5:54Speaker 02

  So it's essentially one bit per selective death. That even has a name, it's called the Muller principle. So the way to think about it is you had lots of variants, each of which has a different hypothesis about the world. Selection kills all the ones that have the wrong hypothesis and keeps the one that has the right one. And it turns out that you can only maintain one bit because you don't know which bit would be responsible if it was many. So it's one bit per genome, per generation. Now, if you're a large multicellular organism.

• 6:27Speaker 02

  All of the adaptive information exists at frequencies that are higher than the generational frequency. So what do you do? So you have to build a system that's extragenomic.

• 6:40Tim Scarfe?

  Yes.

• 6:41Speaker 02

  And we call them epigenomes, or we call them brains. And these are systems that can acquire high-frequency information that goes beyond the selective dynamic. So that's the basic idea, right? And so you can actually make a qualitative distinction, and that boundary is called the error threshold, where you have to acquire an inferential organ mechanism to extract high-frequency information from the world. For some people, that's intelligent. I don't share that view because it's somewhat arbitrary, but it's principled.

• 7:18Tim Scarfe?

  And you said yesterday that, of course, you know, the nervous system is an example of one of these things, but you said culture is evolution at light speed. What do you mean by that?

• 7:29Speaker 02

  Well, it turns out, okay, so I don't know how technical we can get here.

• 7:34Tim Scarfe?

  Very technical.

• 7:34Speaker 02

  Okay, we can get technical. So one way to think about this, right, is that imagine a genome as being a point in a configuration space. It's called sequence space. And as you evolve, you move around in this space. You can think of it as a graph, where adjacent nodes are connected through mutation, which tends to be local, not always. Okay. So the Igen theory tells us how much information it can be preserved at a certain rate of mutation, or what's the fastest speed you

• 8:18Speaker 02

  can move adaptively in sequence space and not lose everything you've acquired about the past. Okay, and this is the error threshold.

• 8:28Speaker 02

  If you're in a position to take the information that you've acquired in the past and store it, refrigerate it, quite literally, put it in a library, put it in a book. Now you can move on that graph much more quickly because you're not corrupting the information that you have hitherto accumulated. And so culture actually has no upper bound, because as long as I store, like saved to my hard drive, saved to my library, the information that I've acquired up to now, I can just randomly produce variants at any rate. As long as one of those is good, I add it to my library. So culture actually breaks evolutionary light speed and is a qualitatively different process of evolution to organic evolution because of that.

• 9:13Tim Scarfe?

  And David, you're at pains to distinguish capabilities from intelligence. So why is culture or a library a form of storing, I don't know whether you would call it intelligence?

• 9:25Speaker 02

  Well, okay, again, I don't think so. I wouldn't. I make a big distinction between knowledge and intelligence.

• 9:32Tim Scarfe?

  Yes.

• 9:33Speaker 02

  And we talked about this, which is that smart people don't need a lot of knowledge to solve a problem. Not so smart people do. And that's our experience in life. You know, as you go to a friend who's swatted up on all of the solutions to the problems and you ask them and they answer, you're not as impressed as that friend of yours you spent the last week, you know, at a pub or on a hike, not in the library. Those are the people who impress us more. And I think we've lost track of that, actually. I think that, as I like to put it, when it comes to intelligence, less is more, not more is more.

• 10:11Tim Scarfe?

  Yes, indeed. Yesterday, you said that, yes, so intelligence is doing more with less. And you actually said stupidity is doing less with more, which was very interesting.

• 10:21Speaker 02

  Well, no, again, I mean, so, okay, so the whole, so let me give you the background to that. So at SFI, we're very interested in this idea of emergence. And Phil Anderson in 1972 wrote a very famous paper called More is Different. And Phil was reacting to high-energy physics. High-energy physics deals with very small things where the symmetric laws of physics...

• 10:55Speaker 02

  Produce a corresponding symmetry in the configuration of states in the physical system. But as you make things larger, and he gives the example of NH3 and PH3, ammonia and phosphine. The underlying laws of physics and protein folding are symmetric. But when you have a very large structure, it gets stuck in a potential well. And so you break the symmetry. So it doesn't matter if the law is symmetric. It's like saying I have a ball, rolls down a hill, rolls into a very deep valley, it gets stuck. And there's an energy funnel. And it doesn't matter that the law is symmetric, because now the only thing that's going

• 11:39Speaker 02

  to tell you what state you observe is the initial condition, where you started in the landscape. And his point, this is called a broken symmetry. And Phil's point was as you get larger and larger, hence more, in this particular instance, in terms of the atomic mass of the molecule, you're more and more likely to have to use an additional parameter to tell you where you are. And the consequence of that is that if you're going to have a theory now, which isn't just a description, a list of initial conditions, in other words, you have to take averages, you have to do coarse screening.

• 12:23Tim Scarfe?

  Yes.

• 12:24Speaker 02

  And that's the foundational principle of complex systems, that in order to go from a very non-parasimonious microscopic description to a parsimonious macroscopic one, you have to take averages in a clever way. And that's the essence, by the way, of emergence.

• 12:42Tim Scarfe?

  Yes, indeed. And we should talk about your papers. So recently released, Large Language Models and Emergence: A Complex System's Perspective. And as you were just alluding to, emergence is more is different. And in a minute, we'll talk about scaling, criticality, compression, novel bases and generalization as being the principles of the methods. But just rewinding a tiny bit, many Focus in the LLM literature, famously Jason Way. I was speaking with Daniel Hendricks last night and he actually told me that he said it first, even before Jason. And they have this rather cartoonish version of emergence, which is something along the lines of when there is a sharp discontinuity in capabilities. We can say that, and they gave examples, didn't they, of, you know, three digit multiplication and so on.

• 13:26Tim Scarfe?

  And because you have these scaling laws and then you see these these divergent appearances of capabilities. You're having none of that, David.

• 13:33Speaker 02

  None of that. Well, the joke there, there's a few things to say. Exactly. You just put it very well and so don't need to repeat it. But yeah, so this three digit addition that goes from being, I can't remember the percentages right. But let's say that they're under 50% when you've got 100 billion parameters and they go to 80%. You've got 175 billion or something like that. And you think, well, okay, I can do three digit addition very effectively on an HP 35 calculator with a 1K ROM. But that's an order of a billion times smaller memory footprint. So you think, okay, so you can engineer a solution into a tiny little memory footprint very efficiently.

• 14:18Speaker 02

  Using approximations that we are familiar with. Or you have to train it with all the data in the world at considerable expense. It does lots of other things in addition to, right, that's the interesting point to three digit addition. And you call it emergent. I would simply call that really shit programming.

• 14:37Speaker 02

  Right. In other words, I think that is the right way to talk about it. And if you're doing really shit programming with natural language, you need loads of it to achieve the goal of interest. The fact that it's discontinuous is neither here nor there. I think that's almost an irrelevance and it's never really had much to do with the emergence debate. Except as an analogy to what are called first order phase transitions, which are sort of

• 15:03Speaker 02

  where the essentially the first derivative of the free energy of the system in relation to some, let's say, control parameter, let's say, temperature.

• 15:15Speaker 02

  Is infinite. And so there are these technical definitions in relation to theory of phase transitions, but this is nothing like that. And we can talk about that because a phase transition is characterized by a demonstrable change in the internal organization of a system.

• 15:30Tim Scarfe?

  Hmm.

• 15:31Speaker 02

  That's really what it's about. It's not about the discontinuity. That's that's superficial.

• 15:36Tim Scarfe?

  Yes, yes. So broadly speaking, you think it's about this coarse graining where there is an entirely novel description using novel bases to you gave the example in the paper, actually, of, you know, a microscopic description could be of how molecules interact with each other. A macroscopic version would be something like Navier-Stokes, so fluid fluid dynamics. And this is an entirely kind of aggregate description of the system, which only works at a certain scale.

• 16:02Speaker 02

  Right. And notice that's completely non discontinuous. You know, how, when does something cease being the atomic theory of gases or H two O molecules? And when are you really at the hydrodynamic limit, where you can write down Newton's laws of compressible or incompressible fluids, Navier-Stokes equations? And, well, there isn't, there isn't this magic discontinuity. It's just a, it's a limiting process. And, and yet, to this idea of intelligence, which is, you know, about doing more with less, now all of a sudden, you don't have to track all those molecules anymore. You can just look at the average densities.

• 16:43Tim Scarfe?

  Yes.

• 16:43Speaker 02

  And so the key point here of emergence for many of us is, there's a sufficient change in the internal organization of the system, that you can get this more parsimonious description of its behavior. And which screams off, that's a term of art, the contributions of the molecular degrees of freedom. You don't need to track them. They won't help you. They're surplus to the prediction that you're trying to make.

• 17:11Tim Scarfe?

  Yes, indeed. So for you, David, when we saw, when we would see a break with scaling laws, so right now, we are memorizing everything, capabilities and scaling laws are commensurate. There have been studies, Dan Hendricks last night said it's something like 96% correlated. You would expect to see a deviation. So when we can have intelligence without scaling, that would actually imply that some kind of coarse graining has been established.

• 17:38Speaker 02

  Yeah, that's interesting. I think, yeah, because sometimes it's called the breaking of scaling. So scaling laws are not evidence of emergence.

• 17:49Tim Scarfe?

  Yes.

• 17:50Speaker 02

  So the most famous example that we work on at SFI, my colleagues Jeff West and Chris Kempis and others, is analytic scaling. It's good that we're in St Andrews, right? Because Darcy Thompson, the great Scottish mathematical biologist, essentially invented the field in his beautiful book on growth informally, published in 1917. I went to see his collection This Morning Here. So that's an aside. But this theory tells you that at all scales you'll have a fixed relationship between mass and heterobolic rate. Okay. And it comes out of an optimization principle that applies at all scales, which is very interesting. There's nothing emergent about it. Emergent means there's a change in the organization

• 18:34Speaker 02

  which requires a new scaling law.

• 18:37Tim Scarfe?

  Yes.

• 18:37Speaker 02

  And, again, there's quite a lot to say about this, but minimally, right, to start investigating a potential case of emergence, you would want to see that the scaling law is broken. Then secondarily, you'd have to go inside the system and ask why. And this is the big, I think, objection that Melanie John and I have to emergence claims is they're based only on the external manifestation of a task and not on the corresponding internal microscopic dynamics, which you want to somehow map onto the macroscopic observable. That's

• 19:20Speaker 02

  the essence of emergence, the micro to macro map. This is just macro.

• 19:26Tim Scarfe?

  Yes, indeed. And I'm a massive fan of Melanie, probably her biggest fan, actually. And she did this copycat work for her PhD thesis. And because I suppose the question is, what does coarse graining look like for language models? What would it look like? Analogy is a wonderful candidate. And more broadly, I think what you're pointing to is that these models Learn fractured, entangled representations, they're microscopic representations. And what we would expect to happen if they were emergent would be that they would learn these factored, unified representations which correlate with the world, which carve the world up by the joints. And perhaps you would extend that and say, actually, respect the phylogeny, things like symmetry.

• 20:10Tim Scarfe?

  You know, they respect the way that the world works, which means that they would be able to take creative intuitive steps because they know how they got there.

• 20:18Speaker 02

  Yeah, I think that's a very good point. I think that's exactly right. I think that

• 20:24Speaker 02

  I mean, it's a deep question, right? But I think there's a sense in which our nervous system and our bodies respect certain fundamental constraints in the physical world, is what you're alluding to. And, you know, and this has been borrowed, of course, by neural networks in things like convolutional neural nets. Because then you're saying, there is a structure in the world which there tends to be some covariance in visual scenes. And convolutional networks capture that covariance very naturally. So that's an example of using a stronger prior which respects something you know about reality. There is a little bit, as you know very well, in the neural net community, a bit of an allergy

• 21:06Speaker 02

  against building priors into models because you want to do it all kind of inductively, which is a little bit of a confusion, it seems to me, given that the entire structure derives from 1943 McCullough and Pitts, which was a nervous system inspired concept. So why not use a bit of world inspiration too?

• 21:25Tim Scarfe?

  Yes, indeed. You mentioned yesterday the Road to Reality by Penrose. And funnily enough, I interviewed Michael Bronstein, who is, you know, one of the founders of this geometric deep learning idea. And it is basically Platonism, right? So it's the, you know, let's imbue all of these symmetries because we think that the generating function of the world is constrained by these symmetries. And it's an abstraction which doesn't leave out any detail. Are you amenable to that?

• 21:52Speaker 02

  I mean, I'm not because it's so anti-complexity. I mean, I might be, to be honest, I'm not familiar with it. Sometimes the way I like to put this is, you know, in 1918, the great mathematician physicist Emmy Noether.

• 22:05Tim Scarfe?

  Yes, conservation laws, yeah.

• 22:07Speaker 02

  Right, one of those sort of hidden figures by virtue of sexism in society.

• 22:13Speaker 02

  publishes a very, very important piece of work where she shows a relationship between the symmetries of the action functional, yes, the symmetries of the Lagrangian, the principle of least action, that captures the principle of least action, and showed, right, that shifting the time or space coordinate corresponded to the conservation of a particular physical observable. Change time and you conserve energy. Change space and you conserve momentum.

• 22:54Speaker 02

  And Darwin is in some sense anti-nota. The Origin of Species says, change time, everything's different. Change space, everything's different. And this gets to the Anderson broken symmetry idea, that it's unlike physics, because it's not symmetry dominated and consequently various observables are not concerned. So I actually think the power of symmetry simply ceased to be the important organizing principle once life came into existence, and certainly intelligence. So I'm not sure what he's claiming. That's different, by the way, to saying that there are symmetries in the world that perception exploits.

• 23:39Speaker 02

  That's clearly true. And I don't, again, I don't know if that's the.

• 23:43Tim Scarfe?

  How convergence do you think evolution is? If we ran it in a parallel universe a thousand times, you know, did you think there's this kind of real ontological coupling, or is it quite divergent?

• 23:54Speaker 02

  Oh, it's a good question. Depends who you ask.

• 23:58Tim Scarfe?

  Right.

• 23:59Speaker 02

  Well, Simon Conway Morris, he'd say, very. If you ask Gould, very not, or not very. I think.

• 24:09Speaker 02

  It sort of depends on the resolution of your measuring device. Because at a very coarse-grained level, alometric scaling theory says everything scales in the same way. It doesn't matter how many times you run it. The scaling of essentially energy to mass is dependent entirely on the dimensions of space. The three-quarter scaling law is three dimensions of space over four dimensions of space, where the extra dimension is fractal. That has nothing to do with life. I mean, it has something to do with life, of course, but it doesn't depend on a contingent history of life. So at that level of granularity, if that's all you could see,

• 24:50Speaker 02

  all you could measure, like the average mass of an organism, there would be no variation, right? Very little. But if you zoom in, then, of course, you see more and more variation, which is a, if you like, a kind of fossil record of its unique history. And biologists do tend to zoom in. Psychologists do tend to zoom in. And I think that's almost dispositional. Physicists like the granular, the very non-granular, averaged. Biologists like the very natural, historical. It's not surprising that those two traditions come to different answers.

• 25:30Tim Scarfe?

  Yes, indeed. Could you explain what you mean by this distinction of knowledge out versus knowledge in with emergence?

• 25:36Speaker 02

  Yes. Okay. So first of all, let's just establish that with emergence, we mean that there is a description of a system that's more parsimonious, that a given level of observation does not benefit from more microscopic detail. We gave the example of fluids, right? And there are many.

• 26:05Speaker 02

  So how does it come about? How does emergence come about? And there's an infinite number of examples. But physicists tend to be interested in how many identical particles, all subject to the same context or environmental inputs, experience emergence. I gave that example, add molecules, change the temperature. That's just one dimension, the average energy. When it comes to evolution or learning, there isn't one input that every molecule is experiencing.

• 26:50Speaker 02

  You're not just changing the temperature uniformly. Everyone is experiencing a different, unique parameterization. So the theory of emergence was developed mainly in the physical domain, where you had large numbers of identical things with a global signal. And now emergence claims are being made for large systems of non-identical, all experiencing a unique signal. So there's already this question of whether or not we can even talk about emergence. And we don't typically, by the way, because we talk about evolution. We talk about learning, development and engineering. No one would say an iPhone is emergent.

• 27:34Speaker 02

  Because you say it's somewhere, there's a plan that tells you exactly what to do with every component. Just as there's a plan that tells you what to do with every cell in development. That's a complex distributed plan, part genome, part environment. But nevertheless, there's an abstraction in which you could talk about it in that way. So that's, so knowledge in are systems where essentially you have to, to get the structure of interest, the pattern of interest, you have to parameterize each component individually, more or less. Knowledge out is the example of physics, where you say, "God, all I did was change the temperature and I got a solid from a fluid. I got a plasma. What happened?

• 28:17Speaker 02

  This is a completely different state of matter and all I did is change one thing." And so that's the big distinction. So the challenge, I think, for biology and machine learning is how to talk about emergence.

• 28:32Speaker 02

  When you've sort of violated the prime directive, which is you've been allowed to have local modification of each component.

• 28:40Tim Scarfe?

  Yes.

• 28:40Speaker 02

  And I think you can, but that's the essential distinction between knowledge in and knowledge out.

• 28:47Tim Scarfe?

  One thing that fascinates me about emergence is, we'll get on to agency. I think of that as a, as a form of causal shielding or causal disconnectedness, certainly from the perspective of an observer. But there are philosophers like George Ellis, I believe, who said that causation is kind of shielded between the levels and causation happens between the levels. But I mean, to you, what is the relationship between causality and emergence?

• 29:11Speaker 02

  Oh, that's interesting. I mean, I think in a way, one way to say it, right? Everything I've said sort of trivially is that you do have a new coarse-grained set of effective mechanisms that you can talk about as being genuinely causal in the interventional sense of pearl, not in the fundamental Newtonian sense. And, and these are complementary conceptions of causality, right? But I think, I would say it would be completely reasonable to restate the emergence claim as a more parsimonious causal mechanism.

• 29:50Tim Scarfe?

  Yes.

• 29:51Speaker 02

  Right, that there is an aggregate observable that conforms to your preferred neo pearl framework that sort of works with the do operator and the appropriate treatment of conditional distributions. And so I think that is appropriate to, to think of it as a new form of causality.

• 30:13Tim Scarfe?

  Fascinating. And, and on this notion of, of agency, how would you define it in relation to things like intelligence? I mean, just me, just very personally, there's, there's the, the autonomy and the causality thing, but there's also this intentionality and directedness thing. So it's, it's not about just autonomy in one direction. It's the ability to set your own direction. And yeah, just, what, what does that mean to you?

• 30:37Speaker 02

  Yeah, so I've actually been developing a framework for thinking about this.

• 30:43Speaker 02

  And the way I think about it is in three different, more sophisticated conceptions. The first is the conception of action from physics.

• 30:57Tim Scarfe?

  Yes.

• 30:57Speaker 02

  Balls roll downhill.

• 30:59Tim Scarfe?

  Yes.

• 31:00Speaker 02

  You wouldn't call that agentic. Some people probably do, unfortunately, but okay. That's the simplest. We have a perfectly good Newtonian, Ehrger and Hamiltonian theory. We don't need to use these words, agentic for that, some people. Okay. Then we have this Darwinian concept, adaptation. Well, adaptation is a bit more than a ball rolling down a hill because it says that the ball gets better at rolling down the hill because it has an internal record of its past efforts. And the general term that Kant gave to that and John Holland gave to that and Murray Gilman gave to that is a schema. Okay. It's a, it's a, you can think of it as a simple lookup table. It says, if I find myself in this situation, I should do that.

• 31:44Speaker 02

  If I find myself in that situation, I should do that. A bit different from the moon, a bit different from the ball. And then the most sophisticated to me would be the agentic. And that adds something to the adaptive, which is a policy. It says, this is what I want to do. Notice adaptation could just be reactive. You could say, if I find myself here, then do that. Policy actually says, I would like to do this into the future. So I think there's a very nice spectrum that takes us from fundamental conception of theoretical physics through to a conception in evolutionary theory, through to a conception that seems to be appropriate for a psychological or cognitive system.

• 32:23Speaker 02

  And, and they're adding, if you like, complication to the internal schema to accomplish that particular objective.

• 32:32Tim Scarfe?

  And does agency entail this kind of emergent coarse graining that we were speaking about before? It certainly seems to. We look in the natural world and, you know, we, we, we see planets, we see things, but then there are these agents that have a level of sophistication which is much greater than that. And would that be an example of this kind of coarse graining you're talking about?

• 32:52Speaker 02

  I think for us to understand them, um, the answer is definitely yes. There's a secondary question of what we sometimes call endogenous coarse screening.

• 33:06Speaker 02

  And that's where it gets more complicated philosophically. Because when we reason, let's say I think I'm reasoning with language, but I'm still recruiting millions of neurons to do it. And so at the level, if I was a solitary agent, it would be very difficult to know whether I'm really doing endogenous coarse graining. The data for endogenous coarse graining comes from communication. So if I tell you, you know, Tim, this is what it means to integrate by parts or something. This is what a Fourier transform does. And you look at it and you say, oh, thank you. I know how to do it now.

• 33:47Speaker 02

  Now I've actually communicated to you a very low dimensional symbolic scheme, which you can then kind of use to program, if you like, your neurons. And at that level, I think no doubt, because there we've actually done coarse graining quite demonstrably, and it's actionable on what your neurons are doing. So I think this is Wittgenstein's private language problem. If all you get to do is observe a system in isolation, I would never know. But I think in this social collective intelligence setting, I think it goes without any shadow of doubt that there's evidence of emergence.

• 34:24Tim Scarfe?

  Can we go over to ex-bodiment? And by the way, I'm a big externalist myself, but I hadn't come across the term ex-bodiment, which has a very specific meaning.

• 34:34Speaker 02

  Yeah, so I was very interested in this idea of embodiment, which we're all familiar with, which is that

• 34:44Speaker 02

  we can outsource computation to constraint.

• 34:53Speaker 02

  We can use the reduced degrees of freedom of a limb to reduce the complications of our policy. Okay, that's, everyone knows that.

• 35:04Speaker 02

  But what about an object in the external world? What about a pencil? What about a fork? What about an astrolabe? You know, what about a Rubik's cube? And there, it's not embodiment.

• 35:18Speaker 02

  It's something else. And the something else there requires contributions from culture, right? In other words, no one person built the chessboard, lots of people did together, and they collectively discovered a very effective representation of a certain kind. That's one part of it. So let's make a distinction. The ex-bodied, which is collectively arrived at, versus embodiment, which is your characteristic. The second point about embodiment is that interests me, is how we have to go via an external material vehicle to get back into the brain. And the example I like, and there are many examples, let's give some simple ones. A map.

• 36:01Speaker 02

  So we collectively make a map of this city of St Andrews. But you can give that to me, and I can memorize it. And I can burn that map. And as we discovered this morning, I'm not very good at navigating. And I'll be able to navigate freely without a physical object. And this feedback between the collective construction of an ex-bodied artifact and its internalization into the individual mind, I call the ex-bodiment helix. And it just gets better and better and better, right? Because now I have a map, I can explore more of the space, I can contribute to that collective script, which you can then internalize into your mind.

• 36:41Speaker 02

  This is a very under-theorized process, and we've been working on it very carefully in the last several years in relation to problem-solving artifacts, like the abacus, like the soma cube, like the Rubik's cube. How does that dynamic work? What does the physical object do that your brain cannot? And we can talk about that, but we can quantify that quite exactly.

• 37:07Speaker 00

  In deepest jungle, Cartier's panther spies a wandering bird that circles the skies. A lovebird, lost and missing his mate, is far from his path in lonely state. Dry your tears, now there's a start. Follow my steps to heal your heart. Panther stalks through the leaves with silent grace. Through the power of scent, she finds the trace. Behold, two lovers reunite in the skies. A most precious thrill sparkling in their eyes. Together forever, they shine each day, bound by the beauty of Cartier.

• 37:51Speaker 04

  I'll admit it, I'm one of those people who gets excited for football season way before the first kickoff. That's why a trip to Total Wine & More has become part of my game day routine. I can get the cooler ready with my favorite beers and ready to drink cocktails. I'm usually picking up enough for everybody, so it's nice knowing I'm scoring the lowest prices. This season, make Total Wine & More part of your game day tradition. Spirits are not sold in Virginia and North Carolina. Drink responsibly. B21.

• 38:22Tim Scarfe?

  David, your institute has done some fascinating research on collective intelligence in general. I know it sounds like a contrived question to ask, but what is the relationship between, or maybe what is the boundary of the individual mind to the collective?

• 38:35Speaker 02

  I think that's a really hard question to answer.

• 38:41Speaker 02

  So I've worked on these mathematical formalisms for finding boundaries. Many people are interested in them. I call it the informational theory of the individual. There are theories of Markov blankets that you're familiar with. So there are many of them interested in this question of where do you draw it, the boundary? And I think the reality is you can draw many. And so let me just give an example from biology and then come back. So what's an individual ant? You know, well, all the workers are clones. So you say, maybe that's not really the individual, that's a constituent of the individual. That's like a cell in you.

• 39:24Speaker 02

  But there is a level of observable resolution where you could say a cell is individual. It is dividing semi-autonomously.

• 39:33Speaker 02

  So a lot of it has to do, I think, again, back to that really critical conception of scale of observation. The formalism that we started developing was one that was scale-dependent. It says, "At a given scale of observation, the key characteristic of the bounded object is that it contains sufficient information to propagate itself into the future. You don't have to look elsewhere." And clearly, that's true for a cell over a given number of divisions at the resolution of the cell. So it's individual. It's not entirely true for the body, because, of course, that doesn't really propagate into the future.

• 40:17Speaker 02

  It's true for your genome, which does partially propagate into the future. It's true for ideas, which are partially carried by many minds. So if you asked me to propagate the work of Einstein into the future, you'd get a fairly incomplete jigsaw puzzle. But if you took a group of people, it would probably be quite complete. So it really depends, again, on the complication, the size of the object, the artifact, the concept, how many informational contributions you require. And the theory tries to calculate that over a particular time scale and a particular

• 41:02Speaker 02

  space scale. So mind is so fascinating, because it would depend, what you said, if it was, you know, my preference for a certain flavour of ice cream, you don't need many, you don't need much collective intelligence for that. But when it comes to things that we actually care about, like, you know, affine or projective geometry, ah.

• 41:26Speaker 02

  Unless you have a coxeter who can do it on his own, you need a bunch of us to do it together. So even that's interesting, right? Because there's heterogeneity in this individuality with respect to what can reliably propagate information into the future. It's very difficult.

• 41:43Tim Scarfe?

  Yes. You reminded me, my friend Ken Stanley, I'm sure you're familiar with this.

• 41:49Speaker 02

  He visited us. He was on sabbatical as a fine.

• 41:52Tim Scarfe?

  He's my hero.

• 41:52Speaker 02

  Okay.

• 41:53Tim Scarfe?

  We've just had him on the show. And he was talking about the need for evolvability. So certainly when we have representations, it's not about where you are, it's how you got there, but also where you can go from there. And agency has a little bit of a shared principle in the sense of, you know, it's about controlling or modifying the environment to meet your goals and so on. And here you're talking about this general principle of information propagation into the future.

• 42:19Speaker 02

  Yeah, no doubt. And that's a really interesting point because you're not only propagating the variation, the information in the Shannon sense, but the variational operator.

• 42:31Tim Scarfe?

  Yes.

• 42:32Speaker 02

  Right? Which could be mutation, it could be different mechanisms of imagination or novelty generation.

• 42:40Tim Scarfe?

  Yes.

• 42:41Speaker 02

  And so that's actually a very good point, that even within genetics, right? You don't only propagate the genes, but the mechanism of genetic segregation.

• 42:52Tim Scarfe?

  So David, you've looked at various forms of sort of, you know, representation and computation, organic, inorganic, cultural, et cetera. Can you tell me about that?

• 43:02Speaker 02

  Yeah, one way to think about this, and it gets to our opening conversation about the limits of evolution and why brains and minds had to evolve to capture information that couldn't be captured by natural selection. Another way to say that, right, is that we've transitioned from a more mortal style of doing information processing, bacteria and so on, which are very dependent on life and death for the information to be captured and stored.

• 43:37Speaker 02

  Versus us, where in some sense, much of the information is stored in culture itself. So it has more of an immortal flavour.

• 43:47Speaker 02

  And if you think of us as analogous to cells in a body that turn over quickly, we turn over relatively quickly, relative to the edifices of knowledge that we construct. So I actually think you could argue that evolution itself, as a process, has moved from more mortal styles of computing, and I mean information processing in the organic setting, to more immortal-like things which we're familiar with, with software and hardware. And so, and you might argue that technologies, and I have argued this actually,

• 44:26Speaker 02

  why do they exist? And the way to think about this is, we have technologies for everything that we're bad at.

• 44:35Speaker 02

  We don't play tennis well with our hand. People do, but it's not quite as exciting as watching Roger Federer or what have you. We don't calculate well, hence calculators in the abacus. We walk well, right? And we sing well. And so these are things that we don't really have technologies for. They're also harder to build. And there's a sense in which computers and software and archives more generally, bodies of knowledge, represent an inevitable

• 45:14Speaker 02

  contribution that complements all those domains in which human reasoning is deficient.

• 45:21Tim Scarfe?

  I mean, that's one way of thinking about it. My biggest fear is not that,

• 45:27Tim Scarfe?

  you know, AI and ChatGPT and all of this will degrade our thinking and creativity. It's that it already has.

• 45:33Speaker 02

  Well, that's another point, right? Okay, so that's again, an interest of mine. So I just wanted to make that point that these technologies exist for a reason. And they're there to, in some sense, complement deficits. I'm not saying they're more intelligent than us at all. That's quite the opposite. I think they're intelligent the way a calculator is. So they're compensating for things that we don't do well.

• 46:01Speaker 02

  There's this other question which is really important and you can actually generalize the Eigen theory, and this is a little bit maybe technical again, which is that you can show that in the same way that we outsource to the body and we outsource to artefacts as a means of increasing fidelity and reducing energy replements, we will outsource everything, right, if we can. And we see evidence of this all the time. And the most obvious example

• 46:41Speaker 02

  is the difference between outsourcing to maps versus outsourcing to GPS, where if you have guaranteed a system that's more effective at navigating than you, there is no reason for you to acquire that skill. And I think we're already seeing evidence, and there's lots of studies, actually, some recently came out, suggesting that human cognition is attenuating by virtue of outsourcing it to calculators. And I actually, it's one of the reasons why I'm not very optimistic about technology, because up until now, you

• 47:20Speaker 02

  could argue it was net positive, right? No one's going to argue with a sextant, right, or with a slide rule or something. You think, yeah, that's great, you know, I couldn't do it otherwise. But I think you now can argue with the idea, be a proponent of the idea, that the long-term implication of technology is the significant diminution and dilution of what it means to be a human. I now receive on a daily basis, tens of emails written as collaborations between humans and other lamps, and it's 100% rubbish.

• 48:08Speaker 02

  And I think that the ratio of the human to the L L M is just going to keep going down. And eventually, everyone will sound the same. So it's not to suggest that the tool can't be useful, but this evolutionary drive to increase fidelity and reduce energy is so strong in us that we will eventually outsource ourselves. And I think I don't know why people are excited about the technology if that's the likely omega point of the evolution of the species.

• 48:38Tim Scarfe?

  Yes. Could we expand more on this diminution of, you know, agency and creativity and thinking? I mean, in universities, will we need to almost create separated areas for the students to actually think and read books and do work without A I? Another thing is, you know, we don't believe in this superintelligence idea, but if superintelligence were invented, because the folks in Silicon Valley, they are saying that, don't worry about labor anymore, the old way of, you know, doing useful economic tasks. Now we just need to buy lots of G P U's and we need to, you know, sort of defend the

• 49:20Tim Scarfe?

  G P U's because people might want to bomb the G P U's because they're the source of all power. And that's very much their view. Would that change if we did have superintelligence?

• 49:29Speaker 02

  There is a world where I would delegate the gym to you. I'm not going, there's no point me going because you can go for me. I'm not going to get up and walk around because you can get up and walk around for me. The consequence of that would be, as you know, significant trauma to our physiological and mental well-being. Everyone understands that.

• 49:58Speaker 02

  The brain is an organ like a muscle. If I outsource all of my thinking to something or someone else, it will atrophy just as your muscles do. There's nothing confusing about that. It's just a fact of physiology. And now you might say, just as long as I have people who can carry me everywhere whenever I need to be carried, then it doesn't matter that I lose my legs, you know. And, um, okay. So there is a sense in which we become, you know, the male deep sea anglerfish that's just a testicle, uh, that's reduced to its absolute minimum for retaining individuality

• 50:43Speaker 02

  in the sense of propagating information forward in time. Um, it's not a future that I consider desirable. So superintelligence is only interesting to the extent that it makes me more intelligent, not to the extent it makes me more stupid or more servile or more dependent. And it just seems to me so glaringly obvious a remark that I don't understand why people are falling for the moonshine.

• 51:10Tim Scarfe?

  David, it's been such an honour having you on the show. Thank you so much for joining us.

• 51:14Speaker 02

  Thank you. It's been fun. So I'm David Krakauer. I'm a faculty member at the Santa Fe Institute. I'm also the president of the Santa Fe Institute and I work on the evolution of intelligence and stupidity on planet Earth.

• 51:30Tim Scarfe?

  Incredible. David, this has been great. Thank you so much.

• 51:36Speaker 02

  All right.

• 51:37Tim Scarfe?

  One of my favourite ever interviews.

• 51:39Speaker 02

  None of yours. What the fuck's he talking about?

• 52:03Speaker 05

  Life is all about making decisions. Sometimes you make questionable ones. Like buying a new TV that's too big for the wall because the electronics store employee was a little too good at their job. Then, sometimes you make great ones. Like protecting your home and auto with Progressive. You get customized coverage options so you can be sure you have the right coverage. Get your quote today at progressive.com. Progressive Casualty Insurance Company affiliates and other insurers. Not available in all states.