Michael Timothy Bennett: Defining Intelligence and AGI Approaches

Dr. Michael Timothy Bennett is a computer scientist who's deeply interested in understanding artificial intelligence, consciousness, and what it means to be alive. He's known for his provocative paper "What the F*** is Artificial Intelligence" which challenges conventional thinking about AI and intelligence.**SPONSOR MESSAGES***Prolific: Quality data. From real people. For faster breakthroughs.https://prolific.com/mlst?utm_campaign=98404559-MLST&utm_source=youtube&utm_medium=podcast&utm_content=mb***Michael takes us on a journey through some of the biggest questions in AI and consciousness. He starts by exploring what intelligence actually is - settling on the idea that it's about "adaptation with limited resources" (a definition from researcher Pei Wang that he particularly likes). The discussion ranges from technical AI concepts to philosophical questions about consciousness, with Michael offering fresh perspectives that challenge Silicon Valley's "just scale it up" approach to AI. He argues that true intelligence isn't just about having more parameters or data - it's about being able to adapt efficiently, like biological systems do. TOC:1. Introduction & Paper Overview [00:01:34]2. Definitions of Intelligence [00:02:54]3. Formal Models (AIXI, Active Inference) [00:07:06]4. Causality, Abstraction & Embodiment [00:10:45]5. Computational Dualism & Mortal Computation [00:25:51]6. Modern AI, AGI Progress & Benchmarks [00:31:30]7. Hybrid AI Approaches [00:35:00]8. Consciousness & The Hard Problem [00:39:35]9. The Diverse Intelligences Summer Institute (DISI) [00:53:20]10. Living Systems & Self-Organization [00:54:17]11. Closing Thoughts [01:04:24]Michaels socials:https://michaeltimothybennett.com/https://x.com/MiTiBennettTranscript:https://app.rescript.info/public/share/4jSKbcM77Sf6Zn-Ms4hda7C4krRrMcQt0qwYqiqPTPIReferences:Bennett, M. T. "What the F*** is Artificial Intelligence"https://arxiv.org/abs/2503.23923Bennett, M. T. "Are Biological Systems More Intelligent Than Artificial Intelligence?" https://arxiv.org/abs/2405.02325Bennett, M. T. PhD Thesis "How To Build Conscious Machines"https://osf.io/preprints/thesiscommons/wehmg_v1Legg, S. & Hutter, M. (2007). "Universal Intelligence: A Definition of Machine Intelligence"Wang, P. "Defining Artificial Intelligence" - on non-axiomatic reasoning systems (NARS)Chollet, F. (2019). "On the Measure of Intelligence" - introduces the ARC benchmark and developer-aware generalizationHutter, M. (2005). "Universal Artificial Intelligence: Sequential Decisions Based on Algorithmic Probability"Chalmers, D. "The Hard Problem of Consciousness"Descartes, R. - Cartesian dualism and the pineal gland theory (historical context)Friston, K. - Free Energy Principle and Active Inference frameworkLevin, M. - Work on collective intelligence, cancer as information isolation, and "mind blindness"Hinton, G. (2022). "The Forward-Forward Algorithm" - introduces mortal computation conceptAlexander Ororbia & Friston - Formal treatment of mortal computationSutton, R. "The Bitter Lesson" - on search and learning in AIPearl, J. "The Book of Why" - causal inference and reasoningAlternative AGI ApproachesWang, P. - NARS (Non-Axiomatic Reasoning System)Goertzel, B. - Hyperon system and modular AGI architecturesBenchmarks & EvaluationHendrycks, D. - Humanities Last Exam benchmark (mentioned re: saturation)Filmed at:Diverse Intelligences Summer Institute (DISI) https://disi.org/

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Progressive's Accident Response can contact emergency services if you're in a major accident. The only thing worse than someone you love being in an accident is not being able to help them. So Progressive is here to help when you can't. So get peace of mind knowing your loved ones are protected with Progressive's Accident Response. Learn more at progressive.com. Progressive Casualty Insurance Company and Affiliates. Limited availability in California. Driver is responsible for emergency and towing charges not covered by the policy. Coverage subject to policy terms and conditions.

• 0:31Speaker 01

  This episode is brought to you by Google Chrome. You think you know a browser, but Gemini and Chrome, that's new. It can help you with practically anything on the web, like restoring a vintage motorcycle from a 50-page restoration block, or finally break down that long article you've had open for weeks. Gemini and Chrome is here for it. Ready to make anything online make sense? There's no place like Chrome. Check responses set up required. Compatibility and availability varies 18 plus.

• 0:56Tim Scarfe?

  How close are we to AGI? I mean, it's interesting how much it has stuck around. We have just replaced the pineal gland with a touring machine. You're a big fan of what I would call biologically inspired intelligence.

• 1:07Speaker 03

  Biological systems with a tiny fraction of the energy and learning data can do so much more.

• 1:13Tim Scarfe?

  Is that fair?

• 1:14Speaker 03

  Because whatever that software does has to pass through an interpreter and the interpreter decides what it does.

• 1:18Tim Scarfe?

  Consciousness is basically an illusion.

• 1:20Speaker 03

  One of my supervisors accused me of writing libertarian biology because one of the results of one of my theses is called "The Law of the Stack",

• 1:30Speaker 03

  which I like dramatic names.

• 1:32Tim Scarfe?

  MLST is proud to be sponsored by Prolific. This is Enzo Blindao.

• 1:37Speaker 04

  Yeah, that's kind of the goal that we're working towards. So we're trying to make human data or human feedback, or actually any kind of feedback at that. We treat it as an infrastructure problem, right? We try to make it accessible, we make it cheaper. You see this pattern in almost any company and even in academic research as well. Every academic researcher cares about the quality of their data. Let's abstract it away, let's put a nice API around it to make it just like the same way you also do CICD or you do model training pipelines. We effectively democratize access to this data.

• 2:18Speaker 03

  Yeah, okay, I'm trying to think of my beliefs. Give me one second.

• 2:24Speaker 03

  What were they again?

• 2:25Tim Scarfe?

  I'm entirely cognizant of it.

• 2:27Speaker 03

  Uh, should I look at the camera? Should I look at you?

• 2:29Tim Scarfe?

  Look at me.

• 2:30Speaker 03

  Okay.

• 2:30Tim Scarfe?

  Yeah.

• 2:31Speaker 03

  Um, my name is Michael Timothy Bennett. I am a computer scientist who has to use his middle name because there are too many Michael Bennetts in the world. I am interested in understanding AI, uh, intelligence, um, life, uh, the universe and the nature of existence. Uh, and I spend all my time doing that. And I have a side hobby trying to build AI.

• 2:56Tim Scarfe?

  I, I got in touch with you, actually. It was quite a few months ago. It was when your paper called What the F is Artificial Intelligence. Did I get the name right?

• 3:04Speaker 03

  Yeah, I mean, it's, it's a couple of extra letters, but yeah.

• 3:07Tim Scarfe?

  Okay, yeah. That was doing the rounds and I flicked through it at the time. I've now just spent the last couple of hours like reading it word for word and it's actually brilliant. I do recommend that folks at home read that, especially for folks in the MLST audience, because, uh, we're a little bit eclectic in our tastes. We are ideas collectors and we like, you know, different approaches to AGI and also hybrid approaches and a little bit of philosophy and consciousness and whatnot. So, um, certainly in that respect, you might be the perfect guest.

• 3:35Speaker 03

  Thank you.

• 3:36Tim Scarfe?

  But this is all very good. This is all very good. Um, in that paper, you were talking about what intelligence is and various approaches to AGI and also approaches to categorizing them. Tell us about that.

• 3:50Speaker 03

  Okay, so, uh, intelligence is a hotly debated topic for a long time. Um, I sort of started off with the Leghuther definition of the ability to satisfy goals in a wide range of environments. Uh, but as I, you know, delved more into biological intelligence and other things, I sort of arrived at a definition as, um, the ability, uh, as the efficiency of adaptation. So how sample and energy efficient you are. And then later I found, uh, Pei Wang's definition, which preceded mine by several years, is adaptation with limited resources, which I think is really succinct and clear. Uh, and of course, there's myriad other definitions, but my favorite's Pei Wang's.

• 4:35Tim Scarfe?

  Me too. I, I could, Pei, if you're watching this, I'm sorry, we haven't published your interview yet. I- Basically, the reason is I I loved his defining artificial intelligence paper so much that we'd be meaning to make a special edition on it. So I kind of held back his material for quite a while and we still haven't still haven't done it. But even though it's my favorite definition as well, and of course it informs his NAS framework, his his non axiomatic reasoning framework, you know, some might say it's almost a bit tautological, you know, just to say, well, yeah, intelligence is about adaptation with with insufficient resources. And in his paper, he went, he was at pains to kind of say, well, we need to have simplicity, we need to have fruitfulness in the definition. It's actually a really difficult thing to get a handle on.

• 5:15Speaker 03

  Yeah, and I think a lot of people who sort of look at what is intelligence, they end up writing very long and complicated definitions. And then you spend more time trying to figure out what the definition means than actually thinking about intelligence. That's another reason I like pace, I guess.

• 5:31Tim Scarfe?

  Yes.

• 5:31Speaker 03

  Clear.

• 5:32Tim Scarfe?

  So as you know, I'm a big, you know, fan of Francois Charlet. For much of MLST history, there was a rule that I was only allowed to mention Francois's name once per show. We've relaxed that a little bit recently. But maybe that's a that's a good place to start. So how does Charlet define intelligence? And you also said that Charlet was kind of inspired a little bit by Legenhatter, certainly in terms of the use of like, you know, Kolmogorov complexity.

• 5:58Speaker 03

  Yeah, I mean, his his work is very much descended from that sort of way of thinking. He defines it in terms of the ability to acquire skills, which is, you know, I suppose perhaps a more benchmark focused version of Pei's definition, which makes sense because Charlet was proposing a benchmark in that paper. So he was looking at each, perhaps he was looking at each test question as a skill and the ability to acquire it is what it was testing, which makes a lot of sense. But his formalism, which almost seems an afterthought in that paper, is is where you can see much more of Legenhatter's influences.

• 6:42Speaker 03

  It's once again uses Kolmogorov complexity. It sort of frames things in the same terms, even though in that paper, Charlet goes on to say that he doesn't think compression is sufficient for intelligence. He was very clear about that in the paper and then he uses it. So.

• 6:57Tim Scarfe?

  Oh, I think that might be because, yeah.

• 7:02Tim Scarfe?

  He describes like a meta generating process, which is the intelligence and that produces skill programs. So the programs are a compression, but the process which created them is doing something more, maybe?

• 7:17Speaker 03

  Yeah. I felt like that part was, um, my, uh, I chose to focus on the test part at that point. I'm like, oh, this isn't the bit that he was really focusing on. I don't know. I'd have to ask him, uh, I guess.

• 7:31Tim Scarfe?

  Yes, but you did say something interesting. So yeah, maybe a distinction with Sholay is that, and, and he thinks of L L Ms as being a kind of interpolative collection of skill programs. So, you know, he thinks programs are the output of an intelligence system, not the intelligence itself. And with, um, you know, let's say, um, AICSI, and we should, we should talk about what that is. Um, I don't think the concept of a program was an explicit output artifact. It was more a definition of an agent which can succeed in an environment. Is that fair?

• 8:02Speaker 03

  Yeah. I mean, I suppose a lot of this is kind of semantics a little bit, but the, the AICSI model is a general reinforcement learning agent. So it just takes the standard reinforcement learning thing and tries to make it as, um, sort of a, uh, what you might call an upper bound or a super intelligence based on that using Solomon off induction. Uh, and Solomon off induction is, uh, you could think of it as a formalization of Occam's razor. Uh, it's just if I have two explanations, I pick the simpler one. And so the idea is that I actually achieves this upper bound on intelligence because if you, if you accept that Occam's razor is, uh, some sort of optimal heuristic that you can use and does this using Kolmogorov complexity, which is sort of the optimally compressed version of a

• 8:47Speaker 03

  model. So if I can compress something more than it's simpler. And so if I just take the most compressible models, then I can get the simplest ones. Uh, and this is, you know, useful for thinking about what a super intelligence might do. And because it's a general, it's a reinforcement learning agent, we can sort of model it out, we can, we can build approximations of it. Um, I disagree with some of the theoretical foundations. Uh, and I, you know, a lot of my publications are about like what we could do better, but I find the overall idea of I actually very compelling and it has informed a lot of my work.

• 9:21Tim Scarfe?

  Yes, because I, I guess for you, reading, reading your work, you said that. If you know, if we wanted to create AGI, it would be, it would be something that looked like a scientist. Because you know, if we, if we frame it at the right level, a scientist can generate hypotheses. And you know, they're, they're an agent, they can act in the world, you know, they're embedded in, in environment. So you're framed at, at a sort of sufficient level of embedding that you can actually capture the dynamics of the system. And, and in that respect, I'd see it is an agent. And it has this principle of compression. And actually, maybe you can contrast it to active inference, because that's quite similar. It's about this agent that, you know, balances energy and entropy. So sort of like, you know, predictive control and simplicity in some, you know, so-called natural way. How is that different from ICSI?

• 10:06Speaker 03

  Oh, I mean, they're very different formalisms. I would love to see someone try to do active inference ICSI.

• 10:14Speaker 03

  But it, it's just, I guess, active inference has sort of, it has a simplicity bias built into it. You know, there's like a, there's a regular, there's, but it's, it's sort of a, the focus is more on

• 10:30Speaker 03

  explaining something else, like explaining, I mean, and also I think active inference is more. Okay, so there's a whole bundle of ideas there. You've got the free energy principle, you've got active inference, you've got all this stuff about Markov blankets and maintaining the border of an organism and having an internal and external world and.

• 10:50Speaker 03

  So the targets are kind of different. But.

• 10:55Tim Scarfe?

  I, I, I think that, I think that's fair. We, we have an agent and the agent is doing prediction in the environment and it can act and so on. And they are both in some sense, ICSI and active inference, trying to produce a simple model, right? So there's this assumption or principle, if you like, that simplistic models, if they predict well, must be good.

• 11:15Speaker 03

  Yeah, yeah. And this is a very popular, almost orthodox assumption to make, because Occam's razor does kind of work. But even as far back as 10 years ago, there were people pointing out that this assumption is based on, you know, but, I mean. Something like Solomonoff induction is it it performs reliably within within bounds based on the original assumptions. But once you put it in an interactive setting like with ICSI, now you've got the subjective notion of complexity that the agent has, which is used for its version of simplicity, because it's it's it's sort of perceiving the world through an interpreter.

• 11:58Speaker 03

  In the case of ICSI, a universal Turing machine. But you could think of it as just just think of it as an instructionally, like a language. When I when I say something in a language, how long it takes me to say it depends on the language I use. If I have some memetic, single syllable word to describe a complicated concept, then the length of that concept is one in my language. And in in sort of my subjective world, that's fine. But if I have an external world that assesses complexity, in the case of ICSI, this is sort of like, like looking at Leghutter intelligence, which is sort of a measure of intelligence, measuring the intelligence of an agent based on the complexity of the model it comes up with,

• 12:41Speaker 03

  it's got a different concept, a different sort of, you know, it the you can make these, you can make it perform arbitrarily well or arbitrarily poorly by sort of shifting the goalposts of interpretation. You can make it so that if if I am to, you know, well, yeah, you can essentially make simplicity completely disconnected from performance, if you like. Not that that actually happens in reality. That's like, it's not so cut and dried, but it's it's certainly not optimal, which is, I think, what a lot of people were hoping for with the original.

• 13:18Speaker 00

  Progressive's Accident Response can contact emergency services if you're in a major accident. The only thing worse than someone you love being in an accident is not being able to help them. So Progressive is here to help when you can't. So get peace of mind knowing your loved ones are protected with Progressive's Accident Response. Learn more at progressive.com. Progressive Casualty Insurance Company and Affiliates. Limited availability in California. Driver is responsible for emergency and towing charges not covered by the policy. Coverage subject to policy terms and conditions.

• 13:49Speaker 01

  This episode is brought to you by Google Chrome. You think you know a browser, but Gemini and Chrome, that's new. It can help you with practically anything on the web, like restoring a vintage motorcycle from a 50-page restoration block, or finally break down that long article you've had open for weeks. Gemini and Chrome is here for it. Ready to make anything online make sense? There's no place like Chrome. Check responses set up require compatibility and availability varies 18 plus.

• 14:14Tim Scarfe?

  Yes. And of course, it's so interesting that actually when you dig into this deeply, it just becomes apparent how difficult this problem actually is. You know, like many people might just think, oh, yeah, defining intelligence. We've got that now with ages ago. I think there was a distinction as well that certainly Legan Hutter, they were very focused on this simplification of the model in Occam's razor. And I think Charlet did overcome one hurdle, which is task generality as well, this developer-aware generalization. Yeah, so Charlet's definition is not a general definition of intelligence. So it's very much a specialized definition. So it's intelligence relative to a scope of tasks. Oh, yeah. And then the generalization difficulty is like the relative entropy

• 14:59Tim Scarfe?

  from that scope of tasks for, you know, from the wider scope of tasks. Yeah. And if I'm understanding correctly, like, I think that it's been years since I read that 2007 paper by Legan Hutter. But it was about an agent minimizing combi-gro complexity, which can like do well on a, the expected performance on a wide range of environments.

• 15:17Speaker 03

  The definition of task here is important. I agree with Charlet that the tasks are what's important. I just disagree about what constitutes a task. So like something like Legan Hutter's definition with the environments and the goals and the sort of, it's the reinforcement learning framework. You've got things like actions. These are all like high-level abstractions that we humans use to simplify the world. And as the problem of relative complexity in an interactive setting kind of illustrates, if I use a different set of abstractions to achieve the same ends, I can make it, I can make something very difficult or very easy.

• 15:59Speaker 03

  So if I'm trying to talk about tasks, then I need to talk about embodiment as well. I can't just rely on this idea of a software mind, because whatever the software mind does depends on the interpreter or hardware. You have to look at the system as a whole. And in cognitive science, they've got this idea of inactive cognition, which is not just embodied, but in the environment as well. Because if you change the environment, then the same goals change difficulty, which is kind of an idea that you can see in Legan Hutter's definition. But it is something that needs to be formalized as part of the process of intelligence. Because if you just sort of assume you have a set of actions or assume an environment,

• 16:42Speaker 03

  you're kind of bypassing a lot of what intelligence needs to do to solve a task. And so.

• 16:52Tim Scarfe?

  It doesn't make sense to think of a computer program, you know, in an absolute sense. Programs have purpose. They are situated in a context, in a world, in an environment. And I guess more broadly, you're a big fan of what I would call biologically inspired intelligence, which is that we should create intelligence which, you know, has properties like self-organization and delegation and causal learning and all of this kind of stuff. You know, because that's much more like how it works in the real world.

• 17:25Speaker 03

  Yeah. So to solve the, so I, you know, had Hutter very briefly as a supervisor during my masters and then sort of continued working on that sort of thing as I progressed through my PhD. And I wanted to address this sort of subjective complexity, subjective performance thing. And that turned out to be about defining this process of coming up with an abstraction layer. If you think of the Turing machine on which, with respect to which IXC is computed as a, or Legan Hutter intelligence.

• 18:02Speaker 03

  If you think of that as an abstraction layer, then you've got a software mind on a hardware abstraction layer, and then that's sort of interpreted by physics. And in a conventional computer, you've got like Python interpreted by a C program interpreted by, and it just goes all the way down to hardware, but doesn't really stop at hardware, because hardware is sort of a state of a physical world, and it's interpreted by whatever physical laws according to which that world runs. And you could then say, well, our knowledge of physics is kind of incomplete. So where does the abstraction end? And if you just, if you really want to make an objective claim, or a claim about objective behavior, to be more exact,

• 18:42Speaker 03

  or exact, you need to formalize what must be true of all abstraction layers, not just a sort of a fixed subset assuming some basic layer that you can identify. Because we, we're sort of all interacting with the world through our own abstraction layers anyway. So if we want to make claims that generalize to other abstraction layers, it sort of helps to have this framework. Wait, where were we at the start of this?

• 19:08Tim Scarfe?

  No, no, that, that's great. That makes sense. Let's bring in the causality component, right? So in your, in your paper, you were describing almost like single direction arrows of causality. So you know, we have the, the hardware, we have, you know, like the C compiler, the interpreter, all of this kind of stuff. And so part of what we were saying is that, you know, to, to build a living, breathing, lifelike system, you, you need to sort of like, respect the causality. You can't just, you know, take something out on, on its own. But I was also more broadly interested in, is it always the case that the causality goes in, in one direction? Or is it actually kind of like quite multi-scale, bidirectional?

• 19:47Speaker 03

  It is definitely multi-scale, bidirectional. Because in the same way that, so cells, cells can network, right? Each cell has its sort of own goal-directed behavior. This is in biological systems. And they can network with other cells in their sort of perceptual field, if you will. And, and then they are constrained by the collective of cells of which they're part, in the same way that a human within a legal system is constrained by the behavior of the other humans around. They're not going to suddenly run down the street naked. So it's

• 20:24Speaker 03

  Yeah, definitely. There is a top-down causation. We can see it in our own multi-scale architecture that we're a part of as a species.

• 20:33Tim Scarfe?

  One really cool thing you did in your paper was you drew a plot and described what it is, but you had abstraction on the Y-axis and delegated control on the X-axis. And you gave an example of like, you know, like a centralized, you know, form of governance would be in the top left and like a free market would be in the top right. Tell me about that.

• 20:51Speaker 03

  Yeah, I actually have a much better version of that graph that is coming out in the final version when that because it's provisionally accepted. So it's that'll that'll be much clearer. But so it's like, it's the new version of the graph has got like, like food stamps versus UBI to illustrate something that is a different levels of delegation of control. So these things both distribute resources to members of an entire population, but the food stamps central doesn't delegate control. It only delegates some of the resources. People are very restricted in in what they can do with that. And so that illustrates the difference between sort of

• 21:36Speaker 03

  decentralization and delegation of control. And then you can think of every system as a stack of abstraction layers, not just so a computer is typically arranged into like this hardware, you know, machine code, assembly, C, all this the stack. But so are human organizations. We've got in a military organization, we've got like soldiers, then you've got a squad, a platoon. And so you've got these different levels of abstraction at which you can look at the system. And each layer is sort of the behavior of the parts of the layer below. And in biological systems, the the in the same so cells, their behavior can be an organ.

• 22:18Speaker 03

  The behavior of organs can be an organism, and the behavior of a set of organisms like humans could be language. So you just can move. And so you can use this framework of abstraction layers to understand some of the relative advantages that biological systems have.

• 22:35Speaker 03

  Should I keep going?

• 22:36Tim Scarfe?

  Well, does that imply that the abstractions are real? Right? So, you know, like, I see you as an agent and I see you as factorized quite neatly into organs and brains and eyes and whatnot. But if we want to build an artificial intelligence, one approach is we we handcraft the abstraction hierarchy. Another one is that we adaptively learn it.

• 23:01Speaker 03

  Yeah, and adaptively learning it is definitely the way to go because we we have learned the abstractions we have in order to because these things are useful to us. So a chair, for example, is useful to me. It is something that that is a cause of valence to me, or is a step on the way to causing some positive or negative valence. It has utility. Table, same thing. I don't have the concept of half a chair that I think about. Like, it's just not useful to me to think. I have to combine this other concept of half to even describe it. And the world is in is in every aspect divided into these simplifications, these classifiers. And these are if, you know, we want to go back to talking about an intelligence or something that builds programs.

• 23:45Speaker 03

  I'm building all these classifier programs for television, light, chair, you, table. Like, this is all stuff that matters to me. I don't build classifiers for things that don't matter. And so you can even tie this in with things like the Fermi paradox of like, why we don't even notice something that might be classified as intelligent. Its behavior is just not relevant to anything that causes us valence, which would tie in with Mike Levin's work on mind blindness and. Yeah.

• 24:12Tim Scarfe?

  Very good, very good. And you spoke about the need for actually learning causal relationships between things as well. Tell me about that.

• 24:20Speaker 03

  Right. I started with a bit of Pearl stuff because I was reading this book of why and looking at optimal agents, of course. I was thinking, well, if it's going to be optimal, it has to learn some sort of representation of its own interventions in the world. So that's to say that if I want to know whether, if I want to be able to get food and navigate my environment, I need to be able to tell the difference between if I'm a fly on my shoulder, for example, and the world moves around the fly. There's two reasons that could have happened. Either my shoulder moved or the fly moved. Fly needs to know which or it's going to get switched. Same with humans, same with insects all the way up. This is, you know, others have already suggested this is sort of important to the notion of

• 25:02Speaker 03

  subjective experience because you need a subject to have experience, right? You need to have an eye to know that I did something. And so I started looking at how you would arrive at, how you would construct that as a self-organizing system. And this came into this, this alternative to simplicity that I was working on for like optimal learning, I call weak, weak policy optimization or weak constraints, or even after feeling particularly cocky, called it Bennett's razor. That one hasn't caught on, but I'm working on it.

• 25:36Tim Scarfe?

  You never know.

• 25:37Speaker 03

  Maybe, if anyone.

• 25:40Tim Scarfe?

  Yeah, Bennett's razor, everyone.

• 25:44Speaker 03

  So, so we can get this, so the point was if we can define an optimal agent that learns optimally, it must construct this sort of representation of itself, sort of a, I call it a causal identity for self. It's like I have an identity that I associate with the causal effects of my actions. But it's also like, if I don't start off with a world divided into objects, then I would also do that for other things, like a chair, a television. And it wouldn't just be a passive classifier in the sense of like, we tend to think of things as value neutral because it's simple. But these are things are not value neutral.

• 26:23Speaker 03

  If you have a system that is sort of, you know, impelled by sort of attraction and repulsion from the ground up, right? It's not like, it's not like it's going through an interpreter and having valence attached after the fact. It is, you know, in the case of like a biological organism, a network of cells, each of which are being attracted and repelled, each collective of cells within that is sort of being pushed and pulled by attractive and repulsive forces. And the organism as a whole is being attracted and repelled. So you get, think of this as sort of a tapestry of valence.

• 26:56Speaker 03

  And it's developing representations and classifiers of the world, not just of itself, but of other objects. And all of these objects would be sort of the causes of that valence. So, or in some way causally relevant to what causes valence.

• 27:14Tim Scarfe?

  So there was there was scale maxing, which is the San Francisco thing.

• 27:17Speaker 03

  Yeah.

• 27:17Tim Scarfe?

  And then there's a simp maxing, which let's make it simple.

• 27:21Speaker 03

  Yeah.

• 27:21Tim Scarfe?

  So like ICSI would be an example of that.

• 27:22Speaker 03

  Yeah.

• 27:23Tim Scarfe?

  And then there was the W maxing, and W, I think, means world.

• 27:26Speaker 03

  Uh, well, it meant weakness, but I just thought it was funny because it was like win-maxing.

• 27:32Tim Scarfe?

  Oh, interesting.

• 27:34Tim Scarfe?

  What was what was the interest? Because you were talking about things like, you know, in active cognition, you know, where we have like weakness of the whole environment and everything. Is that roughly?

• 27:44Speaker 03

  Yeah, so it's like, if if ICSI is a sort of dualist, and when I say dualist, right, I'm referring to like, uh, the, for anyone who's unfamiliar, there's Cartesian dualism is the idea that you have mental substance and physical substance. And he was trying to come up in the 16th century with an explanation of the mind that can form to church doctrine. And so he said that you had mental substance, which are all our thoughts and things, interacts with the physical substance of the world through the pineal gland and the animal spirits around the pineal gland. It it's sort of, you know, bumps it and then it bumps us and then we act. Now, this, even in the 16th century, came under some criticism, but it sort of stuck around.

• 28:27Speaker 03

  And it's interesting how much it has stuck around because we've kind of done the same thing with AI. We have just replaced the pineal gland with a Turing machine. And in active cognition is the idea that your cognition is in the world, right? It's not a mental substance of sort of, it's not just embodied in the sense of like, I'm not just a body, but I am part of the world around me. I store into my memory extends into the world. I can write things on a piece of paper. And I enact my cognition by interacting with the people around me and the objects around me.

• 29:02Tim Scarfe?

  Yes. I'm so I'm so glad you brought this up because the other day when we were chatting, you were talking about the pineal gland. And I thought, oh my god, what the fuck is he talking about? And yeah, so you're saying in the 1600s, we had this Cartesian dualism, you know, the mind and the body, two different ontological substances. And people at the time thought that the pineal gland was almost like the mediating thing in the brain.

• 29:25Speaker 03

  Yeah, yeah.

• 29:26Tim Scarfe?

  So obviously, you're not saying that the pineal gland is the mediating thing. Maybe you are, I don't know.

• 29:30Speaker 03

  No, I'm definitely not saying that.

• 29:33Tim Scarfe?

  Okay, good, just to get that straight. But the very interesting analogy is that you're saying that there is this thing called computational dualism. And that, and I want to press on this a little bit, because I don't know, are you saying that any form of functionalism or computationalism is computational dualism? Or are you saying this use of a Turing machine in some formalisms is computational dualism?

• 29:57Speaker 03

  I'm saying, I'm using, I use computational dualism more to poke fun at the idea of defining just a software intelligence. Because if we just have software by itself, and we just, we don't say anything about the hardware, then it can't really be intelligent if intelligence is measured in terms of performance in the environment. Because whatever that software does has to pass through an interpreter, and the interpreter decides what it does. So we can just make it arbitrarily stupid if we want.

• 30:28Tim Scarfe?

  Yeah.

• 30:29Speaker 03

  And so it is really, it was, I used that term computational dualism as almost like a rolled up newspaper to whack people on the nose, because I was getting frustrated with repeating myself.

• 30:40Tim Scarfe?

  Oh, I like it, I like it. How does this idea relate to, I mean, I spoke with a few folks about mortal computation, right? You know, so Turing machines and program, you know, these are great because they allow us, and also, you know, people talk about pan computationalism. It's a really neat idea to think about computation in the abstract and think about programs in the abstract. You know, these are things which can run on any computer, potentially on different substrates and whatnot. And the world isn't really like that.

• 31:08Speaker 03

  No, no, there are finitely many copies of every piece of software we make.

• 31:15Speaker 03

  I love the branding mortal and immortal computation, but there's no such thing as immortal computations. There's just finitely many copies of the software we make. And I get that people might quibble about, but yeah, you can copy the thing. But in the context of trying to define intelligence, it is a Terrible concept. There's just mortal computation. Let's not complicate the matter by adding in an extra concept that doesn't apply.

• 31:43Tim Scarfe?

  Yes.

• 31:43Speaker 03

  Yeah.

• 31:44Tim Scarfe?

  Yeah. And then just to help folks understand, so mortal computation is that the the stuff is the computation. There's like no meaningful disconnection between like the program and the stuff which does it.

• 31:56Speaker 03

  I've seen a few people give different interpretations of it. The best is probably that of Aurobia and Friston where they talk about.

• 32:04Tim Scarfe?

  Oh, yes, I interviewed him, by the way, Alex.

• 32:06Speaker 03

  Oh, right.

• 32:06Tim Scarfe?

  Yes, yes, cool guy.

• 32:07Speaker 03

  Yeah, so they've got quite a rigorous definition that extends over several pages. The most common definition I've seen is the internet version, which is just people going, it's immortal. But and I think it was first, the the term was used as sort of an afterthought by a hint in at the end of a paper that was mostly about feed forward neural networks.

• 32:29Tim Scarfe?

  Oh, yeah, was it his new, his new proposed article? Was it like the feed forward forward from twenty twenty two in Europe?

• 32:36Speaker 03

  I I think so, yeah.

• 32:38Tim Scarfe?

  I can't remember the name of the paper, but yeah.

• 32:40Speaker 03

  But I think it it's sort of, and I remember I saw that, and I'd already been writing about like embodiment and how like you you've got to like take into account the abstraction layer if you want to like have any claims that hold up about performance. And so and my response to that was to come up with the term computational dualism and write an irritated paper. But.

• 33:04Tim Scarfe?

  My grumpy papers are my favorite papers, which is amazing. Okay, very good, very good. And so in in this kind of, you know, categorization of different approaches, we haven't we haven't spoken about, you know, the Silicon Valley scale maxing and so on. I mean, maybe like what what drives you? I mean, how close are we to AGI? I mean, the folks over over there think that we've already done it, right? But just by scaling.

• 33:30Speaker 03

  Well, I mean, if you want something that can do jobs and automate a lot of the economy, then sure, we've got some form, that form of A G I. If we, if you want something that's actually intelligent, like a human, though, we do not have that. And a lot of that is because, um, like even just interacting with, and I like, I use things like, you know, Grok and Chat G P T. It's not like I'm not interacting with this stuff. But if it was, if it was anywhere near as intelligent as a human, I wouldn't have to do all the work that I do. I would, um, be able to offload a lot of, it's not sample efficient. I know, um, there's claims about the ARC test, but just interacting with these agents,

• 34:14Speaker 03

  you can see that it's not really sample efficient. I don't know what they did there to get those results on ARC one. And I don't know what, uh, what the claims are about ARC two. It's very hard to tell what is true when people don't release the code and the results and everything that you can sort of puzzle through and work out. So, uh, I think we're probably a good ways off something that really resembles human intelligence. And we need to look at something that isn't just like a software innovation, but hardware innovation. The reason I think that is because we're using these abstraction layers in the form of like, you know, the, the hardware we have purpose built for very useful standardized software applications that we can roll out and copy and run on many different computers. Um, but if we want something that's as adaptive as a biological system,

• 34:59Speaker 03

  we need something that is sort of modular and cellular and efficient, like a biological system. We can't be expecting some, I mean, a biological systems with a tiny fraction of the energy and, uh, and learning data can do so much more. Yes, um, yes. Which, you know, is great because I like having a job, so that's, uh.

• 35:20Tim Scarfe?

  Absolutely. By the way, hot up the press. Did you hear that Elon released Grok 4?

• 35:25Speaker 03

  Yeah.

• 35:26Tim Scarfe?

  I don't know who's released it, but Greg, you know, from Arg Prize, he posted this morning.

• 35:31Speaker 03

  Yeah.

• 35:31Tim Scarfe?

  Greg's a good guy. And apparently it's scored about 16%, which is sotter. Even because my friends at Tufer, you know, like Mohammed and Jack, I think they're around 15 and a half percent at the moment. So yeah, Grok 4 is now in the lead. I mean, how do you interpret that? Did you think it's just sort of like dint of memorization?

• 35:51Speaker 03

  I don't know. I mean, I'll have to interact with it, right? Maybe it'll be really impressive. Maybe it'll. But I suppose the proof's in the pudding. Like if if this starts to do a bunch of really useful jobs across the economy, then we can say with certainty that we're closer to it. But these benchmarks, even like, and the Arc AGI benchmark is a great benchmark, right? I've been looking at that my whole PhD is it was a it's it's great, but it's not perfect. None of the and even Shilette would not claim it to be perfect. And 16% is a great result, but I want to see if it can add long numbers. That probably.

• 36:30Tim Scarfe?

  That would be a good start.

• 36:31Speaker 03

  I don't know. It's like, this is my usual thing. I sort of, oh, it's a new toy. Let's see if it can add long numbers. And it almost always can't.

• 36:39Tim Scarfe?

  Well, I mean, devil's advocate. Yeah, no one's going to argue with you. If you say vanilla LLMs are, you know, basically data based. No one's going to argue with you, right? But you can add tools to them. I mean, so you were starting to talk in your paper about there are some very interested, interesting hybrid approaches. Or, you know, obviously on the LLM side, you add tools. And there's an interesting discussion to be had there, whether, you know, you've used, you train them with stochastic gradient descent, they use tools, what can they do? But you're also talking about some other interesting, like, you know, there's the non-axiomatic reasoning system from Pay. And there's like the Hyperon system from Ben Gertzel. And my honest, like, I don't know anything, I don't know much about those systems.

• 37:21Tim Scarfe?

  But honestly, when you were describing them, it seemed a little bit like they were everything but the kitchen sink. So they're like, you know, they can do a bit of Bayesian inference over here and they can do some neural networks over there. And. I mean, that that could work, but what's your assessment?

• 37:34Speaker 03

  Yeah. So, oh, should I go around? Should I go over that like tool?

• 37:38Tim Scarfe?

  Oh, yeah, yeah, yeah. Just just sort of weave weave a path.

• 37:41Speaker 03

  So I sort of said there, like, taking the inspiration from Sutton's bitter lesson where he talked about sort of search and learning. I sort of divided it. We got sort of two basic tools, right? We got approximation, which is what the L L Ms are. I mean, and by definition, they're inexact. And that's really great for like, you know, trawling through large amounts of data and coping with noisy data because it's an approximation. You can do a lot with that. And then with the computational resources we have, approximation works beautifully. And then you've got search, which is like iterating through a flow diagram or something like that. And that is great for precision or things like navigation on your phone.

• 38:23Speaker 03

  And when you combine these things, you get something like AlphaGo or or.

• 38:30Speaker 03

  Yeah, I mean, so you can use the approximation part for a sort of a heuristic to guide the search. You can you can combine these in many different ways. And these hybrids allow us to create much more more effective intelligent systems of some one form or another. So in the case of, you know, there's well known examples of this or things like AlphaGo or AlphaStar. But there's also, you know, the sort of more comprehensive architectures that are meant to kind of emulate the versatility of a human mind. So something like like NAS is a system that sort of can integrate many different components.

• 39:15Speaker 03

  And I've seen some of the experiments involving NAS and L L Ms that were at the 2023 A G I conference. And like Hyperon is like an inherently modular system that is just meant to allow you to sort of plug and play lots of, well, that's lots of different modules. It's meant to be decentralized and adaptable, so you can plug all these things in as they develop. And yeah, that's that is definitely. Uh, you know, like it can include the kitchen sink if you want to plug that in. But I guess.

• 39:46Tim Scarfe?

  Yes. I think it was, yeah, sorry, it was more hyper on, I was thinking about the kitchen sink. Pays one, maybe you'd do a better job than me at this, but roughly speaking, it's about like building up a whole bunch of reasoning about something which I don't know much about. And then adapting and modifying it over time until which I can make, you know, deductions and inferences about things.

• 40:09Speaker 03

  And I remember there was some stuff with time constraints in there. So if it got stuck on something, it would move on and it would rank things by, it would take into account the resources it had. So in practice, it would actually be quite useful. You can put it on a little robot, have the little robot run around the room and do stuff.

• 40:23Tim Scarfe?

  Yes, yes.

• 40:25Speaker 03

  Which is cool. Uh, but, and they, they seem to have every year come up with better and better benchmarks of results, but it doesn't seem to get much attention in the sort of mainstream machine learning space.

• 40:37Tim Scarfe?

  Yeah. What do you think about benchmarks by the way? Cause Grok, um, I actually interviewed the guy who created humanities last exam the other day, Dan Hendrix. And I think it was at twenty six percent today with Grok four, it's about forty six percent. You know, what, what is the point of benchmarks if they're so easily saturated?

• 40:52Speaker 03

  Great marketing. Um, I mean, it's, it's like, we love measuring sticks. I, this is just, it's a, it's a, it's nice to have measuring sticks. It lets us know that we're progressing in a direction, whatever direction we point the stick in, I guess. But like, humans are adaptive. If we set up a measuring stick and it's less than perfect, we will find a way to exploit that. I think this is, I, this isn't necessarily a bad thing. It's just that I think people should interpret benchmarks as what they are, which is measuring sticks.

• 41:27Tim Scarfe?

  Yes. Talk to me about consciousness.

• 41:31Speaker 03

  Okay. Uh, well, that whole spiel about I X C and abstraction layers kind of led me down a very long and winding rabbit hole with the abstraction layers thing. Because after doing that, I mentioned before, the idea of coming up with a causal representation of the self and this idea of tapestries of valence. And if you keep scaling up the ability to sort of learn these causal, uh, causes of valence, then you don't just get like a, uh, sort of a, a do operator for the self or a representation of the self, uh, which others have already, oh, actually, I should start there, that self thing.

• 42:13Speaker 03

  Um, when I was looking at that in the original paper, I, uh, that I put that in, I sort of thought, well, if you've got this self and it is sort of has, is inherently valenced and it's sort of made up of sensory motor activity, then wouldn't that be the sort of the basis for subjective experience and explain something of consciousness? So I, I put that in the paper and people liked it and I thought, this consciousness thing is great, I'm going to keep doing this. Um, I like this and people aren't laughing at me. So, um, I kept going and then one of my supervisors said, hey, that's like my theory. I did a causal self thing, uh, and pointed me at his paper. And his paper was about something called reafference in, uh, the insect central, uh,

• 42:56Speaker 03

  central complex, which is also, uh, he was trying to show that flies have subjective experience. And this tiled back to some work from like 20 years ago, where someone was saying that, well, this is, uh, this, this sort of representation of the self is like where human subjective experience, uh, comes. We have, uh, something called reafference in the mammalian midbrain. Um, and so many animals have this. It's what enables us to tell when I am pressing down on the chair versus the chair pressing up on me. And this is very useful for causal relations. And then I kind of started thinking about consciousness more generally and how, well, we could make up our subjective experience with these causes of valence I was talking

• 43:40Speaker 03

  about, you know, things like the television, the chair, whatever. Um. And someone uh started um beating me over the head with a copy of David Chalmers work on the hard problem of consciousness.

• 43:51Tim Scarfe?

  Oh, yes.

• 43:51Speaker 03

  Um. And after a lengthy argument uh over that, I decided, well, now I have to write about it. So I started writing a uh what it in ended up as a seventy page paper, much to the um uh my supervisors at the time were were pleased, they're like, please stop, just graduate, just just just finish the thesis. Um. But I kept writing this and then it got cut down to a uh some much shorter paper. I ended up writing uh bringing on one of my supervisors collaborator, found another new supervisor who was sort of expert in consciousness to come on and help me finish that and talk about the hard problem of consciousness.

• 44:31Speaker 03

  Um. And I proposed to solve it by showing that uh what's called a philosophical zombie is impossible in every uh conceivable world. And this is because if you go down all the abstraction layers, you can say, well, every conceivable world is um is sort of it must uh whatever it is, right? Um. It must include just change or difference. Otherwise, there's just a sort of universal oneness. And as I put it in my thesis, becoming one with the universes beyond the scope of my work. Um. The so you've got a set of states and you can build up a formalism from that.

• 45:14Tim Scarfe?

  Yeah.

• 45:14Speaker 03

  And um and this formalism, I argue, describes all conceivable worlds. So if you can show us a philosophical zombie that is, and just to explain what that is, um, it's something that's like you or me, but not conscious. It's but in every way identical to you or me. So it's as efficient energetically, it's as a it's as smart, does everything we do, just missing consciousness. And I I got really into philosophical zombies when I read the work of of Peter Watts, uh, who is this science fiction author that writes cosmic horror about philosophical zombies. Um.

• 45:50Tim Scarfe?

  Do you want to do you want to continue? I've got I've got a couple of questions.

• 45:53Speaker 03

  No, no, let's do the questions because I can go on ranting for ages.

• 45:56Tim Scarfe?

  Well, no, no, I mean, this this is brilliant. Yes. So philosophical zombie. I mean, I love Chalmers, we've had him on. All of the function, dynamics and behaviour without the little bit extra about the phenomenal component. And so I might be a phenomenal zombie. So I think what you're saying is because there's this kind of lowercase s and capital S subjectivity. And if I understand correctly, you're saying that the capital S subjectivity can be discounted. There are no phenomenal zombies. I don't know whether you're saying it's because there is no hard problem of consciousness, like that consciousness is basically an illusion.

• 46:37Tim Scarfe?

  So we should think about subjectivity in terms of purposeful, perspectival representations, right? But we shouldn't assume that there's some magical extra realm on top, which is called consciousness.

• 46:53Speaker 03

  I suppose there's a lot of, lot to interpret there. I mean, if we're saying, is there like a mental, anything non-physical that is, if we take physical to mean something that interacts directly with the physical world and is like not sort of. I suppose the advantage of doing that all conceivable worlds thing is I don't need to talk about physical and non-physical. And I like that because then I can entertain sort of ideas of magical worlds or whatever, but beside the point. I'm not saying that consciousness isn't a thing or that it's an illusion. I'm saying that if we can sort of enumerate a bunch, explain a whole lot of the features

• 47:38Speaker 03

  of consciousness as a necessary consequence of, you know, just the state of the environment changing from one to another in all possible worlds, all conceivable worlds. And if one accepts that my description of a conscious organism, without having said that anything is just physical or whatever, is compelling. And I think it is. Then there is no conceivable world where you can have something that is as intelligent and acts the way that I do without, without being conscious. The conscious is a necessary adaptation. This is, and that the idea of information processing without consciousness is.

• 48:24Speaker 03

  Is implausible.

• 48:25Tim Scarfe?

  Yes, very good, very good. So. You are saying basically that it is, it is just something that happens when you have configurations along the lines that you are describing.

• 48:39Tim Scarfe?

  I, I guess, in a way, I can't challenge you because you're already saying they need to be biological and, and like, you know, physical and real and so, because, I mean, my, my obvious retort to that would be, well, you know, like a, a computer simulation of those things obviously wouldn't be conscious, but you're, you're not really saying that, are you?

• 48:57Speaker 03

  No, and I, I, there's one question that at the end of my thesis, I get it, I sort of am and are about, because it's like, uh, I don't, I like this question. I like that I couldn't figure this out because I, it's fun to think about. So that tapestry of valence I mentioned with like all the cells getting pushed this way and that, and you could sort of say that, you know, a conscious state is a tapestry of valence. Now, if I simulate a bunch of cells and give it a tapestry of valence and all the necessary ingredients like the first and second and third order cells that I only talked about the first order self just before, but you can keep scaling a system up and getting predictions of predictions of predictions of, uh, stuff.

• 49:38Speaker 03

  And if you, you get all the necessary ingredients for consciousness and you just simulate it in a computer,

• 49:45Speaker 03

  there is two possibilities. One, either a conscious state has to be realised at a point in time by one state of the environment, or it doesn't. If it does have to be realised by a state of the environment, that is to say, all of the parts of the conscious state are at the exact same moment there. And in the, in the formalism that has, it is a much clearer statement, but.

• 50:15Speaker 03

  Well, um, then it's just humans and organisms and highly distribut- you could make a nanobot swarm that's conscious, but you can't make like a single-thread CPU conscious because it's not really doing all this at once, right? It's kind of like looking at a thing, program counter, sort of load something into a register, does some stuff, shuffles some stuff around. It's smeared over time. And so the idea is that smearing consciousness over time would kind of kill it. The other possibility is that, how would we know? Am I allowed to swear on this?

• 50:49Tim Scarfe?

  Yeah, of course.

• 50:49Speaker 03

  Yeah, how the fuck would we know if we're being simulated? We don't, we have no way of knowing. I think I'm existing at a point in time. Um, and I could rerunning, this whole thing could be running on a single-thread CPU in some kid's basement. Now, I'm not endorsing the simulation hypothesis. I'm just saying, I mean, I like the idea that, so everything is clearly, there's like an infinite stack of abstraction layers. Maybe those abstraction layers end in some kid's basement? I don't think they do. Doesn't matter, that's, I'm making fun of the simulation thing, but, um, if it's smeared across time, if I can have a single-thread CPU load stuff into a register, do all, simulate all the stuff that is to do with consciousness, then I could make like populations of humans

• 51:29Speaker 03

  or what are called liquid brains of, and so a solid brain is like something like a, it's a human brain with a persistent structure that supports like, uh, you know, in the case of a human brain, a bioelectric abstraction layer that can do information processing. Very useful, but it requires a certain stability. It's, it has to be, maintain its form. A liquid brain is something like a population of humans. It doesn't have to maintain its form. Its computation is not in the form of sort of electrical signals, but people moving around, doing actions and interacting with the world. Now, maybe we'll network ourselves up, I don't know, but that's a liquid brain. And Corning is another liquid brain. According to my theory, if, if a consciousness has to be at a point in time, then liquid brain, not conscious. If you could smear it across time, then you could have a conscious liquid brain, which

• 52:14Speaker 03

  is cool, but, uh-

• 52:18Speaker 03

  Has some weird implications.

• 52:19Tim Scarfe?

  Very cool. By the way, we'll be clipping that part. So you'll see that you'll see that part on there on Twitter. There is just one potential objection, which is that, you know, what a lot of theories of consciousness do is is they kind of, they they brush it to one side and they treat it as something which is epiphenomenal, which means it's not like causally embedded in in the system. And we did ask Friston about this and he had a bit of an interesting response. He said that, you know, phenomenal states are just parts of the generative model, you know, it's all it's all in there. It's all kind of like part of the causal nexus or whatever. What do you think about that?

• 52:57Speaker 03

  So I very explicitly argue that a phenomenal state is a tapestry of values, that that the sort of what the that I am not just doing a representation that's value neutral, that these that my classifiers of the world, like television and so on, is me being attracted or repelled from a physical state in at like different levels of abstraction, at different scales, all happening at once. So there's a lot of sensations going on there, which why it's not that it's not just loading a file. I feel something as I try to process this information because I'm being impelled by it.

• 53:34Tim Scarfe?

  But isn't consciousness something immaterial, something unobservable? So for me, it makes sense to say that it's a correlate of something physical in your modeling, but it has to be kind of somewhere else outside of the system.

• 53:49Speaker 03

  Yeah. And I guess so that's the idea of like a first person ontology. There's something that I know by being conscious that you can't know by watching me. But

• 54:03Speaker 03

  but if if I mean, that's the point of the abstraction layer thing, I guess, is that if I if I do this formalism of all conceivable worlds and I have the luxury of saying that I I have this like gods I view into everyone's head. So in the formalism, it's nice and neat like that. From a sort of an explanation point of view here, yes, I am stuck in my first person ontology and I can only say what I'm interacting with through my abstraction layer. But it's also kind of Um, kind of like saying that, uh, you know, it's if we if we buy into the idea that we can't say anything about things outside of abstraction layer, we're basically giving up on science altogether. And I think that why would we draw a different line for consciousness than for everything else?

• 54:47Speaker 03

  When um, we clearly, we could draw the same line for everything else. That's another thing. I'm not saying that it's not a line we we could draw. We could say, I don't know the television's on. It's just, it might be on. I don't know. Uh, prove it to me, uh, according to my first person ontology. And I, um, and so, you know, I can continue to hold whatever belief I like, really.

• 55:10Tim Scarfe?

  Very good, very good. I've enjoyed this.

• 55:13Speaker 03

  I've enjoyed it too.

• 55:15Speaker 03

  Yeah, so Dizzy, uh, Diverse Intelligence's Summer Institute, uh, was something I found out about, about um, three days before the application deadline. I thought, oh, I should do that. Um, cold weather and talking about intelligence. Uh, so I have come here and met a lot of people who, uh, do everything from machine learning to biology to philosophy and had very intense discussions about these things for three days. I, uh, I'm not sure my brain still functions, but I know it's ticking away there with something. We're supposed to come up with a project. Uh, I, um, I have a weird problem. There's too many projects. There's too many good projects.

• 56:00Tim Scarfe?

  Yes.

• 56:00Speaker 03

  Um, but it's, it's interesting because it sort of ties in with all this. Uh, there's a lot of people who are also big fans of Mike Levin's work and

• 56:08Tim Scarfe?

  Oh, yeah, love him.

• 56:09Speaker 03

  And I, uh, well, I've been writing more recently. I've been writing a lot with the Tapestry of Valence stuff and this talking about abstraction layers. Everything I'm doing has become about, uh, that because it's just what I'm enjoying at the moment. I've been looking at, say, AI safety is, uh, for a while. One thing is like, oh, well, it's not about the AI in isolation. It's another, um, it's another swarm architecture in which we are a liquid brain into which the AI plugs in. It's just a part of the organism. And so rather than worrying about Aligning a policy or whatever, it's really about just designing the system as a whole

• 56:49Speaker 03

  to accommodate these different components and make use of them in the same way that that a biological system sort of, you know, makes use of resources available to it and the network sells together to form stuff.

• 57:03Tim Scarfe?

  That's actually very interesting. Yeah. So, you know, like, the way biology works is that it is very decentralized and there's this delegation and cannibalization. And weirdly, it seems quite orchestrated, even though it is so decentralized. So, humans maturation in similar ways, we have similar behaviors and so on. But we also have a degree of agency and freedom on top of that.

• 57:29Speaker 03

  Hmm.

• 57:30Tim Scarfe?

  I mean, if you were to design like an artificial or augmented AI system, how could you kind of have your cake and eat it and have something decentralized and steerable?

• 57:40Speaker 03

  So, one of the results, one of my supervisors accused me of writing libertarian biology because one of the results of one of my of my thesis is called "The Law of the Stack", which I like dramatic names. But it's basically that if you've got a high level of abstraction, say, you know, some software running in a computer and it's learning, its ability to learn and adapt hinges on the ability to adapt at lower levels of abstraction. And so like the hardware level, learning and adapting there. Like, in comparison, if you compare

• 58:19Speaker 03

  biology and computers, and so there's a paper coming out soon called "Are Biological Systems More Intelligent Than Artificial Intelligence?" And it is basically, it says, "Well, in addition to the W-maxing thing that biology seems to do better, it seems to delegate adaptation down the stack, whereas computers are like an inflexible bureaucracy that makes decisions only at the top. And if we want to make something as adaptive and efficient as a human body or whatever, we need to emulate this sort of delegation." And if we want to, and, and then I go on to say, um, cause I, I watched way too many Mike Levin interviews.

• 59:00Speaker 03

  He was talking about cancer and how cancer is something where, uh, can be seen as where you've got a cell. It becomes isolated from the informational structure of the collective of which it's part. And it reverts to like primitive, uh, transcriptional behavior, which basically means it just starts, you know, reproducing and eating itself. Eating and like expanding and cancer. Um, and so you could think of it as like, well, what, what, under what circumstances does part of a collective system become isolated from the informational structure? So I formalized that in this stack thing and said, well, each, each cell is like a little tusk with little policy.

• 59:38Speaker 03

  And if there are, are not any correct policies available for the overall collective, then the only way for the thing to continue is the, the overall, like higher level thing to continue is to break off some of the parts until there are some correct policies that exist. There's two ways this could happen. One, you impose lots of nasty stuff from the outside, make the thing hard, uh, make it too hard to come to have a correct policy. It's just too difficult, right? So in the case of biological organism, like lighting it on fire. Um, another way to do this, uh, would be to just impose too much top down control and sort of just sort of cut yourself off at the ankles and eliminate otherwise good policies by can, over constraining the members of your collective.

• 1:00:22Speaker 03

  And if you think of this in terms of like humans, like, uh, this would be like an overly, um, this is why my supervisor was making jokes about libertarian biology. Because I was saying that it's just like, if you over constrain the members of the collective, then they're going to break off and, and do crazy shit. Um, so it's like having too much of a totalitarian state or something like that, too many laws, too much restrictions. And you could, I used to live in Italy and I could see this in the way people used lines. Um, they didn't line up, they just kinda.

• 1:00:51Tim Scarfe?

  Yeah.

• 1:00:51Speaker 03

  Yeah, went towards the front of the shop. There were rules for everything in Italy, so nobody obeyed any of them.

• 1:00:56Tim Scarfe?

  Yeah, other than that, it's a great place.

• 1:00:59Speaker 03

  Yeah, I loved living there. Um, as soon as I got over the whole, you know, nobody standing in line thing, it was fine.

• 1:01:05Tim Scarfe?

  Yeah, and don't drive.

• 1:01:06Speaker 03

  Yeah, well, I did a lot of that. That was terrifying.

• 1:01:09Tim Scarfe?

  Yeah.

• 1:01:09Speaker 03

  Um, I had to stop at the side of the road and my boss, uh, who was Italian and he was like yelling at me, he's like, why aren't you driving? I'm like, I had to stop. Everyone's crazy. They're all driving like maniacs.

• 1:01:21Tim Scarfe?

  I know, I know, it's insane. Yeah, go on.

• 1:01:23Speaker 03

  The AI, if you put too many constraints on it, it's like you're more likely to end up with that. You want to just constrain it in the areas you actually need it to be constrained if you're trying to sort of avoid some sort of dangerous behaviour. Don't sort of like constrain something to a set of impossible circumstances, it's just going to break.

• 1:01:40Tim Scarfe?

  Yes, yes. Because I'm interested in this concept of what it means to be alive, right? Because we're kind of dancing around this a little bit. We want to create artificial systems that have the vibrancy, you know, and we could have perspectives on what that means, you know, maybe it's sort of like certain patterns of information processing diffusion and whatnot. And there are some existence proofs, like you look at Conway's game of life, you know, artificial life and millennia, and you look at these things, you think, wow, that seems very lifelike. I can't really put into words why it is, but it seems like it is. And then on the other side of things, I love building, even with computers, distributed agent-based systems using the actor pattern. I love, I love sort of like separating things out into autonomous little units of computation that can run distributed.

• 1:02:24Tim Scarfe?

  But they're very much not alive. I mean, they have some cool computational properties, but you know, they're brittle. You know, maybe I could make them do meta-programming. So, you know, have a little L L M and it can kind of heal itself and update itself. But still, still not really like, you know, it's knocking on the door of it, but it's not really where we want to go. There are some interesting approaches like neural cellular automata, where you sort of do this emergentist sort of optimization, where, you know, it's self-organizing and it can heal itself from the bottom up. But that's quite domain-specific. So we're dancing around this idea of creating systems which are alive.

• 1:02:58Speaker 00

  Progressive's Accident Response can contact emergency services if you're in a major accident. The only thing worse than someone you love being in an accident is not being able to help them. So Progressive is here to help when you can't. So get peace of mind knowing your loved ones are protected with Progressive's Accident Response. Learn more at progressive.com. Progressive Casualty Insurance Company and affiliates. Limited availability in California. Driver is responsible for emergency and towing charges not covered by the policy. Coverage subject to policy terms and conditions.

• 1:03:29Speaker 01

  This episode is brought to you by Google Chrome. You think you know a browser, but Gemini and Chrome? That's new. It can help you with practically anything on the web, like restoring a vintage motorcycle from a 50-page restoration block, or finally break down that long article you've had open for weeks. Gemini and Chrome is here for it. Ready to make anything online make sense? There's no place like Chrome. Check responses set up required. Compatibility and availability varies 18 plus.

• 1:03:54Speaker 03

  Yeah, okay. There's, there's two things I want to talk about there. Yeah. One is you mentioned healing, right? Yeah. So I was interested in why simp maxing sort of works when it could, in an ideal world, not work at all. And so I was looking at, well, we live in a spatially extended universe. Stuff gets, there's only so much stuff you can cram into a bounded system or finite space. And so if I want my system to, say, I've got, you know, a biological system made of, just coming up with abstraction there is to process information, then as it delegates control, in order to make efficient use of space, it's going to make weaker constraints take simple forms. And so if you delegate as much as you can,

• 1:04:35Speaker 03

  you're generally going to get simp maxing correlating with W maxing as you sort of scale things up and do this. So this means it's something that maintains homeostasis, like a self-repairing organism, is going to need to delegate control a lot in order to do this. And that, and so self-repair and being alive almost requires this kind of combination of simp maxing and W maxing. And I was thinking about this because I wanted to know why things are alive. And I figured this would be a fun side gig with my thesis. So I am waiting for examiner feedback on this. But I thought, well, a rock is something that simp maxes

• 1:05:20Speaker 03

  and just kind of persists through simp maxing. As in, by just simp maxing, as the universe sort of transitions from one state to another, it's going to destroy some objects and preserve others. And if something is simple, why would it persist? Well, it would persist because something that's simple is more likely to sort of stumble into a kind of a sort of a weak constraint because there is this sort of basic correlation thing we've got going on. And so rocks and lots of objects like that in the universe persist by being simple.

• 1:05:52Speaker 03

  But then if you take something that self repairs, it's doing the opposite. It is becoming more complex than the same thing if it didn't self repair. It is increased complexity and massively increased its ability, its, its sort of ability to embody weak constraints. And so I would argue life is that which uh which W maxes at the expense of simp maxing. Whereas things which are not alive just can simp max.

• 1:06:18Tim Scarfe?

  And we were just talking about healing and, and like self-organization and so on. And it doesn't just come from within, it also comes from the outside.

• 1:06:27Speaker 03

  The interactions with the surrounding environment.

• 1:06:29Tim Scarfe?

  Exactly, because I think like a lot of what agency and intelligence is, is about storing a history of information. It's like, you know, it's almost like you've got, like there's, there's a, there's a hard drive of human knowledge, which is our culture. And, and this is all being stored and it survives and, and it's kind of like ontogenically imbued into all of us during our lifetimes. And, and, and this is why David Krakauer said that culture is evolution at light speed. Right. Cause we've, we've like language and culture help us to transgress the physical limitations of DNA and physical evolution. So it's, there are just many different ways of thinking about how this like machine works.

• 1:07:13Speaker 03

  I sent an abstract of my thesis to David Krakauer last night and I don't, I don't know what he thinks of it yet. I, I, I will wait and see. He, he might think it's terrible. Um.

• 1:07:24Tim Scarfe?

  I think he would be honest with you if, if it was.

• 1:07:26Speaker 03

  Yeah. But um, but I think these ideas are compatible. I mean, I'm, I'm, I sort of, I don't know, with the start of my PhD, I was, I sort of saw the free energy principle, thought, what is this? I don't know about this. And then I read more and more and I'm like, oh, okay, I could come around to it. Now I like it. So, um, so maybe it is, maybe there's something I've missed, but I, I feel pretty confident in this. And just to be clear, the, the, the life is W maxing without simp maxing thing is not something that has gone through peer review yet. I have submitted it for peer review and we'll wait and see what sort of hate mail I get in return.

• 1:08:03Speaker 03

  But it's definitely something I would want to run past Friston if I get the opportunity in the future and hear what he thinks of it.

• 1:08:13Tim Scarfe?

  Yeah, very cool, very cool. Well, Michael, what I will say is you should join our Discord server.

• 1:08:18Speaker 03

  All right.

• 1:08:19Tim Scarfe?

  I think the folks would love it if maybe one Friday afternoon, we all got together and, you know, shoot the shit for a little while.

• 1:08:25Speaker 03

  I'd love that. Yeah, that sounds great.

• 1:08:27Tim Scarfe?

  You'd have a lot of fans on there. Michael, this has been amazing. Thank you so much.

• 1:08:31Speaker 03

  Thanks so much for having me on.

• 1:08:32Tim Scarfe?

  Awesome.

• 1:08:33Speaker 00

  Progressive's Accident Response can contact emergency services if you're in a major accident. The only thing worse than someone you love being in an accident is not being able to help them. So Progressive is here to help when you can't. So get peace of mind knowing your loved ones are protected with Progressive's Accident Response. Learn more at progressive.com. Progressive Casualty Insurance Company and Affiliates. Limited availability in California. Driver is responsible for emergency and towing charges not covered by the policy. Coverage subject to policy terms and conditions.

• 1:09:04Speaker 05

  What's the secret to being a great host this season? Keeping things deliciously simple, like mixing Simply Cranberry into a festive drink as guests arrive, or serving up a warm mug of Simply Apple on a chilly night. That means less time behind the scenes and more time in the moment with everyone. Crafted with real fruit, Simply juices and fruit drinks make any holiday drink effortless, inviting, and completely delicious. Simply. Say yes to simple.