The Fractured Entangled Representation Hypothesis (Kenneth Stanley, Akarsh Kumar)

Are the AI models you use today imposters? Please watch the intro video we did before this: https://www.youtube.com/watch?v=o1q6Hhz0MAg In this episode, hosts Dr. Tim Scarfe and Dr. Duggar are joined by AI researcher Prof. Kenneth Stanley and MIT PhD student Akash Kumar to discuss their fascinating paper, "Questioning Representational Optimism in Deep Learning." Imagine you ask two people to draw a perfect skull. One is a brilliant artist who understands anatomy, the other is a machine that just traces the image. Both drawings look identical, but the artist understands what a skull is—they know where the mouth is, how the jaw works, and that it's symmetrical. The machine just has a tangled mess of lines that happens to form the right picture. An AI with an elegant representation, has the building blocks to generate truly new ideas. The Path Is the Goal: As Kenneth Stanley puts it, "it matters not just where you get, but how you got there". Two students can ace a math test, but the one who truly understands the concepts—instead of just memorizing formulas—is the one who will go on to make new discoveries. The show is a mixture of 3 separate recordings we have done, the original Patreon warmup with Tim/Kenneth, the Tim/Keith "Steakhouse" recorded after the main interview, then the main interview with Kenneth/Akarsh/Keith/Tim. Feel free to skip around. We had to edit this in a rush as we are travelling next week but it's reasonably cleaned up. TOC: 00:00:00 Intro: Garbage vs. Amazing Representations 00:05:42 How Good Representations Form 00:11:14 Challenging the "Bitter Lesson" 00:18:04 AI Creativity & Representation Types 00:22:13 Steakhouse: Critiques & Alternatives 00:28:30 Steakhouse: Key Concepts & Goldilocks Zone 00:39:42 Steakhouse: A Sober View on AI Risk 00:43:46 Steakhouse: The Paradox of Open-Ended Search 00:47:58 Main Interview: Paper Intro & Core Concepts 00:56:44 Main Interview: Deception and Evolvability 01:36:30 Main Interview: Reinterpreting Evolution 01:56:16 Main Interview: Impostor Intelligence 02:11:15 Main Interview: Recommendations for AI Research REFS: Questioning Representational Optimism in Deep Learning: The Fractured Entangled Representation Hypothesis Akarsh Kumar, Jeff Clune, Joel Lehman, Kenneth O. Stanley https://arxiv.org/pdf/2505.11581 Kenneth O. Stanley, Joel Lehman Why Greatness Cannot Be Planned: The Myth of the Objective https://amzn.to/44xLaXK Original show with Kenneth from 4 years ago: https://www.youtube.com/watch?v=lhYGXYeMq_E Kenneth Stanley is SVP Open Endedness at Lila Sciences https://x.com/kenneth0stanley Akarsh Kumar (MIT) https://akarshkumar.com/ AND... Kenneth is HIRING (this is an OPPORTUNITY OF A LIFETIME!) Research Engineer: https://job-boards.greenhouse.io/lila/jobs/7890007002 Research Scientist: https://job-boards.greenhouse.io/lila/jobs/8012245002 TRANSCRIPT: https://app.rescript.info/public/share/W_T7E1OC2Wj49ccqlIOOztg2MJWaaVbovTeyxcFEQdU

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  So it goes back to these observations from PicBreeder, which is this picture breeding website where neural networks, which were a special kind of neural network called compositional pattern producing networks, were bred or evolved by people to produce images. And this is a really unusual and exotic form of training that's very different from modern deep learning. It's completely different from it, which is one of the important points about it. And so when people were able to actually breed interesting looking pictures, there's a lot of lessons we learned from that. Actually, the novelty search algorithm and things like that came from those lessons. But there was this one dangling lesson that like just hasn't gotten out. And that's what this paper does.

• 0:45Speaker 00

  And that that lesson was that the representations, the underlying representations of these images, which are basically represented encoded by these neural networks, are absolutely incredible, amazing. And there was like no good explanation for how they could be as good as they are. And we noticed this very early on that like they have unbelievable modular decomposition, which means that it's almost like it was engineered by a person. You know, it's like there's a there's a there's a network that that generates the image of a skull. And it the network has decomposed it such that like there's a component of the network that's responsible for the mouth and can do things with the mouth, like open and close the mouth. Or there's another dimension that can make the mouth smile.

• 1:29Speaker 00

  And things like that, which are not, it's what's really amazing is not data driven. Like, we're talking about dozens of iterations in terms of like the amount of search that's involved to get a representation like this. And so I've always there's we've had many discussions over the years like, why, why is the representation so amazing in these PicBreeder images? Why does it have this unbelievable modular decomposition? And and extremely efficient and compact? And so one thing that like at one point I did with Joel Lehman was we were kind of just playing around to see like, well, what if we tried to get SGD to produce? Use the same images, like what would it be different internally?

• 2:13Speaker 00

  The image, the way that it represents these images, and it was just dramatically different. But then we did nothing with that, like we just kind of like left it. Both Joel and I knew this thing, but we didn't really do anything with that information. But I've been just mulling it for years that like there's something there that's really important. That like with this weird kind of open-ended search process, you get these incredible representations. And then with conventional like objective-driven SGD, which is like the backbone of all of machine learning right now. You get a completely different kind of garbage representation, just total spaghetti.

• 2:52Speaker 00

  And we used, we came up with some terminology that we put in the paper to more like clearly articulate what these differences are. But basically you're talking about just amazing versus garbage. And so like the question that the paper addresses is what does this mean? Which is something I think that has endless repercussions and potential implications. Like the fact that we're basing the entire field on something that produces this complete garbage under the hood. And does this mean anything? Now, I think if you just saw this, by the way, the fact that it's images makes it easier to see. Like that the, I mean, that the representation is garbage, like the fact that it's a network that produces one image.

• 3:32Speaker 00

  Because what that means is that we can visualize every single neuron, what subcomponent it's actually computing. It's very easy because it's just a single image. So it makes it very easy to see all at once how it actually computes the image. So we can see the internal representations. And if I just showed you that for SGD, like if I said, here's a skull, I made this network reproduce this skull. And the internal representation is crap. I think it would be not like hugely resonant in the field. Because it'd be like, well, it's not super surprising that it's kind of hard to understand this like really complicated and tangled representation. It's just emergent from how SGD climbs these gradients. But the thing that I think makes this really intriguing, like the reason that the paper is, because beyond just that.

• 4:15Speaker 00

  Is that it gives you something that otherwise could never exist, which is a counter. That there actually do exist networks that don't have that issue. You would think that that's just intrinsic to neural representation, that somehow they just look like kind of entangled messes. And that's just the way life is, but clearly is not how life has to be. And so, because we have now actual concrete examples from Pickbreeder that show beautiful internal representations. And so I think that the paper's trying to get at, what does that mean? Like, how should we interpret this? Should we perhaps aim for algorithms that actually do achieve these kind of, you know, really amazing kinds of modular decompositions?

• 4:58Speaker 00

  Or should we, or should we just say, well, no, that's, that's just not important. Let's just forget it and dismiss the fact that we know they exist. I think that would be, that seems to be a pretty premature move to say that. So, so there's something here, I think, a deep, deep lesson for us to learn in the field about internal representation, which just is totally off the radar. It's way, way off the radar of anything that's being discussed. You know, mechanistic interpretability is not on the same page with what we're showing here. And so, of course, it can play a role in, in further like interpreting this result. But this is a new thing to look at and for us to contemplate and try to understand what the implications are.

• 5:42Tim Scarfe?

  Don't we just need to have humans in the loop? I mean, wouldn't it be great if we could just have these evolving topologies, right? Without humans, wouldn't it be great if the systems could themselves learn the categories, the abstractions? And is it just a matter of missing data? Or is there just a fundamental gap in capability?

• 6:04Speaker 00

  Yeah, so it's clear that part of the explanation for these amazing representations in Pickbreeder is that humans were in the loop. So that's part of the explanation. But the question is, what's the deeper lesson? Like, what did the humans do that caused this to happen? It clearly wasn't intentional. I mean, the humans weren't thinking about the underlying representation at all. They're just choosing things they like. So why does that lead to this virtuous type of representation? And I think, you know, there's one really interesting lesson that this shows is that it matters not just where you get, but how you got there. You know, and that's like something missing right now, you know, because we tend to just

• 6:46Speaker 00

  care where you get, like, we look at the benchmark score, you know, in the field and it's like, that's the result, it's doing really well, it just passed the Math Olympiad. But what if it matters how you got there? And you say, you could say, well, why does it matter how you got there? If it performs well, it's like the same either way. And that's what the paper is all about, basically, is that like, you know, you can have these two things, these two networks, they both output a perfect looking skull, exactly the same skull. One was found in this completely different, open-ended way with people guiding the search. Another was found objectively through S G D. And so very different trajectories through the search space we're taking, radically different, and which means that under the hood is radically different representation. And so you say, well, what does it matter then?

• 7:30Speaker 00

  What's the underlying representation? Well, it can matter a lot how you represent the world. And so, so then, like, the question is, with humans, like, is that the essential ingredient? And I think, can't be. It can't be that it has, like, the only way to get good representations is to have a human guiding the search. There's a deeper explanation, like, why were the humans successful at getting to these amazing representations? And it has something to do with the underlying, open-ended nature of what they were doing. Like, in other words, on the road to getting an image of a skull, they were not thinking about skulls. And so, like, when they discovered a symmetric object, like, an ancestor to the skull, they

• 8:13Speaker 00

  chose it even though it didn't look like a skull, but that caused symmetry to be locked into the representation. You know, from then on, symmetry was a convention that was respected as they then searched through the space of symmetric objects. And somehow, this hierarchical locking in over time creates an unbelievably elegant hierarchy of representation. And it's plausible, I think, at least for me, that there are algorithms that don't have humans that could take similar trajectories. Through search space, maybe not quite as like perfect as the human trajectories. You know, I wouldn't be surprised if we can't actually hit that ideal. But there's probably a continuum where you could come closer. And then what are the implications?

• 8:54Speaker 00

  You know, if the order that you encounter, the principles that lead to your final understanding of the world matters for how you represent the world. And therefore, for your ability to be creative in the future. Then does it matter? The order in which we allow these large models to encounter the different principles that they encounter on the road to total understanding of everything in the universe. And I mean, I would guess like this implies probably, yes, it probably matters. And that opens up a huge range of possible creative opportunities for alternative ways of thinking about training that would lead to better representations.

• 9:32Tim Scarfe?

  You brought in the open-endedness aspect, which is fascinating because you're saying they weren't looking for the skull. Right? So what were they looking for? They were composing these primitive basis functions that they have in their mind. So they know that symmetry is good. Where did they get the symmetry idea from? Well, it must be somehow gleaned. And so actually, our function space is restricted in some very important way. We know that we have certain things we can compose, and we know that we can compose them in certain topologies. And we know that invariably, if we follow that trajectory, we will land on interesting things, even though we don't necessarily know exactly what we will land on.

• 10:09Speaker 00

  Yeah, yeah, that's how this happened. I mean, the people, people in general have intuitions about potential, not just about where we are right now, but that this might lead to something. And in fact, it's very complex because like the more people play with pickbreeder, the more they get intuitions about pickbreeder itself. So they start to understand what might lead to what what's promising, which is different than what's aesthetically pleasing in the moment. And so they're both enjoying the image and predicting like, well, where might this go? And it's not like they're predicting they're going to get a skull, but they're predicting symmetric things are really interesting and beautiful and let's see. What other kinds of symmetries come out of this, like they're thinking that.

• 10:52Speaker 00

  And so that helps to to get this kind of virtuous ordering. Which causes like a a sequence of lock ins of different conventions of of increasing complexity. Which then creates this like amazing representation underneath the hood. But I want to I want to point out, though, that like this view of the world. Is so radically different from the data driven view that we live in right now. You know, it's what's really fascinating to me about it is not data driven. Like, you know, we we think about, you know, when you say like, eventually, if you experience enough of the world, like you say. Of course, you might expect that your representations start to mirror just the way that the world is.

• 11:31Speaker 00

  And you get this kind of isomorphism, maybe between like the the the organization of your brain and the organization of outside reality. But that's a data driven view, like in basically almost everything we think about is data driven. I mean, the bitter lesson is sort of like a data driven philosophy. But like, this is like, what's so interesting is this is totally contra bitter lesson. You know, because like what you're saying here is like, we see almost nothing of the world, like Pickbreeder knows nothing of the world. There's no pre-training at all. Start with some blobs. Over a few dozen iterations, I mean dozen is crazy, like it's like peanuts, like we're used to millions, billions, like we don't do dozens. That's not what we do in our field. But here we have dozens of iterations, not enough to be exposed to almost anything.

• 12:15Speaker 00

  It actually somehow finds within this. Can newly constrained space of dimensions that has been discovered by humans. Things like the difference between opening and closing a mouth or smiling and not smiling. And like those dimensions exist now in this space of the network, but not because of data. They're whole cloth de novo discoveries, which are not data driven, so there's no bitter lesson. It's just out of nothing. And there's even more crazy ones, like there's like, like the apple one, which is in the paper, it's in the appendix. This apple has this unbelievable weight, a single weight in the apple representation. Which is a single continuum that if you move along that continuum, you swing the stem. Of the apple back and forth, like from left to right.

• 13:01Speaker 00

  Maybe someday we'll put an animation over this so we could show, like I could give you the animation. But you can see the swinging stem is one dimension. And it's it's like three dimensional, you know, it's not just like a two dimensional thing like it moves like you would expect in a rotation in three dimensional space. It has a shadow underneath it. It's like a green leaf. The underlying apple, which is a symmetric object, is not disturbed at all. It's totally independent. It's been decomposed. And then there's this one thing, which is the stem swinging. And so, you know, what I'm saying is it's absolutely incredible. It's mind blowing that if you think about that as a world model. Like it's an actual true hypothesis about the world. This is the way that stems look when they swing.

• 13:40Speaker 00

  But this model has not been trained on anything in the world. It's never seen swinging stems, let alone apples at all. I can almost guarantee you that in the training trajectory itself, there was no swinging of the stem. After all, if the stem started swinging, that would mean it already had that ability. So it's just circular to argue that. This is just something that arose out of the fact that the that the representation is so elegant. That it somehow has an internal hypo, I think of it as a hypothesis about the world, which is correct. And you have to ask yourself, how many of our hypotheses are like that? Instead of data driven hypotheses, because we do sometimes have these unbelievably elegant underlying representations of the world that often are unique.

• 14:23Speaker 00

  Unique to an individual, not necessarily universal across human beings, like everyone's representation is unique. And so this is a totally different way of thinking about representation and knowledge and how it comes to be.

• 14:37Tim Scarfe?

  Yeah, and the thing I'm trying to understand is we can agree that humans have this incredible abstract model of the world. We have a language of thought. And the way we understand things is through these. Compositions, these topological compositions of these little basis functions, right?

• 14:59Speaker 00

  Yeah, I mean, I think it's true that pickbreeder is a kind of psychology experiment. That is a dimension of what we're seeing. And it's true that intelligence and compression are often equated and clearly have a relationship. I think in the paper, we talk a bit about this and even speculate that there may be something more to say than just compression, you know, in the sense of like the factored aspect. How you factor matters. It's not just that it is compressed. It's like if I know that a face is composed of eyes, nose and a mouth, and I actually factor those out. Even if you had a greater compressed version of the face that didn't factor those out, I would prefer the factored version. That's still in some sense better. Then I can, you know, generate new faces in a principled way.

• 15:44Speaker 00

  And so I'm not sure that it's always like the maximally compressed version is like the most so-called intelligent, depending what we mean by that. There's multiple factors to consider, but obviously compression is a virtue. You know, and so these extremely inefficient representations that you see in just like a regular SGD is obviously that's part of the problem. And so I think though that, like speaking about the human aspect of it, just to go back to that for a second, it's just one thing that I think is important to think about is just that you can extrapolate outside of pickbreeder.

• 16:22Speaker 00

  Like this principle of searching through regularities or finding good isomorphisms with the world in some kind of sequence. Like that's a just general way that people do discovery. So in the sense like, so in other words, it's not just in this kind of like very almost psych test like environment that that kind of stuff happens. Like that's just a general aspect of human exploration, intellectual exploration. And so when we do that, you know, you can imagine like the difference between somebody who learns calculus from a textbook versus someone who invents it for themselves because they were curious. But they both end up doing The same thing, you know, so they both take the test and they both get a good score on the test.

• 17:06Speaker 00

  But it's just pretty obvious that the person who found it through their own independent exploration is probably going to be doing much more interesting math like after that test. But why? It's because like they they went through a pick breeder like exploration process. Of course, this is easier said than done. Most people won't do that. But it's just like the way that you got there matters a lot, like all throughout life.

• 17:26Tim Scarfe?

  If we did have a large language model that could understand abstractions at the level we do. Why couldn't we just run it autonomously and and why couldn't it step by step? Know what's interesting and just traverse this phylogeny and find interesting things and self reflect. You know, it it feels to me that the missing link is having the correct level of abstraction and being able to do this iterative, open ended search. They can't do that because they simply don't have the abstractions. We have the abstractions, but could we actually build an autonomous, open ended system? That could find the abstractions the way we do, or do you still need the humans?

• 18:04Speaker 00

  Is there still something missing in this respect from large language models like their ability? To leverage abstractions, to to really think creatively out of the box and so forth. There's different ways you could say this, but that's basically the big question, I think. And I basically agree that I don't think that what we have now. Can really match the very best of human creativity. Like, I think that's safe to say. I think a lot of a lot of people would probably agree with that. It can do some level of creativity, what I would call derivative creativity. You know, which is sort of like the the bedtime story version of creativity. It's like you ask for a bedtime story. You get a new one. It's actually new. No one's ever told that story before. But it's not particularly notable. It's not going to win a literary prize.

• 18:49Speaker 00

  It's not inventing a new genre of literature. Like, there's basically nothing new really going on, other than that there's a new story. And so that's derivative, like in my nomenclature, I call it derivative. And so that's that's pretty much, I think, where we are. And so when we do things like try to leverage these models to explore really important.

• 19:11Speaker 00

  In creative spaces, I think that's a, that's a, um, an obstacle. It doesn't totally stop us because there are ways of getting out of distribution even with that limitation. Which like, for example, wrapping evolution around it like, like we did with evolution through large models. Like, uh, recently, DeepMind does with Alpha Evolve. Like, you can actually get out of its own distribution. But I think it's an inefficient way, you know, compared to the human mind, which leaps through levels of abstraction that it, that it's encoded, basically, in its representation, which then points back to, I think, the lesson in this paper. Which is that it could be that part of why we're sort of trapped in this box of derivative creativity

• 19:53Speaker 00

  is because we, we don't have these really, uh, nice style of, we call them, uh, unified factored representations. In other words, like, concepts are unified and it's well factored into the different components that actually corresponds to what's interesting within the domain. So they don't, maybe they're lacking, so they, in other words, they have, what we use the other term for the bad kind of representation, which is a fractured, entangled representation. Concepts are fractured into pieces and entangled with each others in ways that are inappropriate. And so, maybe it's because SGD naturally does produce these fractured, entangled representations, at least conventional SGD, the way it's being applied here. Not necessarily saying that no version of SGD can actually create the more elegant form.

• 20:36Speaker 00

  But the way that we apply it conventionally, it tends to produce this fractured, entangled representation. Maybe that's part of why it doesn't have access to the abstractions, as you put it. That are necessary to do truly transformative type of, uh, creative ideation. That would require like a very well factored version of the world, which is just lacking. And there's lots of evidence of the problem, uh, with, um, having entanglement and fracture in representation. Like, we, we go through some of it, in one section, we give a lot of examples from more recent large models. We don't just look at picture and images. And we give hints of it because it's hard to get direct evidence because you'd have to look under the hood at the actual representations. Which is quite complex to do, of course, like, for things that are not just images. Like, it's concepts and knowledge.

• 21:21Speaker 00

  Of course, is very complex and the whole field of mechanistic interpretability is trying to grapple with this. But we give some kind of hints that like there is evidence of this kind of entanglement and fracture. And so, so maybe it's a result of the fact that we're doing this conventional type of objective driven search that Jen kind of gives you the lowest common denominator version of representation, which then is going to be a creative straitjacket eventually when you're trying to get outside of the box in a way that it isn't for pickbreader networks, which can imagine things that are almost unbelievable, like the swinging stem of the apple, even though it's not part of a world that it's ever experienced. And so I think there's a lot to learn in representation

• 22:05Speaker 00

  that can help to address what we intuitively perceive as a limited ability to be creative and abstract.

• 22:13Tim Scarfe?

  Basically, there is a spectrum of representations. The problem with neural networks is it can't distinguish the good ones from the bad ones. And what Kenneth is basically saying is that we don't have to do it the the lottery ticket hypothesis way, which is where you start with an infinite number of degrees of freedom and then like a block of clay, you kind of carve it down afterwards. The really smart thing to do is to build it up from first principles at the beginning. So, you know, you actually build these neural networks piece by piece. But, you know, he's advocating for a non-stochastic gradient descent, possibly an evolutionary method of doing this. But honestly, Keith, the thought that I was left with

• 22:54Tim Scarfe?

  is he thinks that there is an algorithm which can do this. And the reason why it works in the real world is because we have this epic panoply of computation. The universe is executing all of this computation at the, at the sort of like miniscule level, right? And that is, that is the physical process which gives rise to evolution and intelligence and all of the interesting phenomena we want to capture. And even though Kenneth is absolutely on the mark when he points out the reason why neural networks aren't creative and indeed what creativity is, no one has better intuition of evolution than Kenneth.

• 23:37Tim Scarfe?

  But he still seems to be slightly off the mark when he believes that we can construct an algorithm completely disconnected from how the physical world works or largely disconnected. And we could still capture some of the phenomena.

• 23:50Speaker 02

  Right, well, I mean, in the spirit of his title, you know, questioning representation optimism, I think, you know, Kenneth has engineering optimism. And, and I, like, to a degree, I share that optimism. So, I mean, you know, because, because you and I both have spent enormous amounts of time in our Discord server arguing about things like this. So I'm well intimately at a gut level aware of just the absolutely cosmically insane amount of computation that goes into powering the, the algorithm of evolution with a capital E. You know, it's happening on the earth in many forms, not just life, but memetics and information and,

• 24:33Speaker 02

  you know, whatever else, right? Higher order phenomenon. So I'm just saying there are probably very clever hacks, at least I hope, maybe Kenneth hopes, that there are clever hacks that will get us somewhat towards a better method, you know? Akash had a very good take on, on kind of, you know, some of these evolutionary aspects in particular. You know, it, it not being a, um, an optimization, but a constraint. And so it's good to see, you know, new, new researchers, right? Like, kind of taking up, taking up that mantle. And I think a big part of what the point of this paper is, and their point is, we need to think more about these processes. Because I think there are, you know, like, you brought up a lot of interesting,

• 25:16Speaker 02

  you know, sort of takes just now and intuitions on how Poet and these algorithms might work. And it's, I think the building up is, is essential. So I think what we need to do, like, I imagine some future training neural networks will, will look like this. Like, you'll start with a neural network that has 100 parameters. And you'll train it on some, some subset of the data, some simple subset. And then the next phase, you'll train one that has 1,000 parameters. Um, and in some way, it's kind of expanded from that lower level network. Like, we'll have algorithms that will identify sub-networks in there that do something in particular. But say these 12 neurons, and what we'll do is, like, we'll now make those 120.

• 25:59Speaker 02

  Twenty neurons and then train them and and keep them sort of isolated as a group, right? So that they grow that abstraction like sort of further. Um. And then it'll be ten thousand, a hundred thousand, like it'll kind of keep scaling it up, right? And then what you'll end up with is almost like a neural network that has a very specific kind of, not sparsity, but modularity, like a very specific kind of modularity because it was grown from like seeds, you know, up rather than starting at the massive scale and allowing it to do all this like really obnoxious and and counterproductive entanglement.

• 26:39Tim Scarfe?

  Yeah, and this is the, I mean, I think we're getting to putting the pin in in the center of the dartboard here. So um, he said the the problem is that neural networks have too many degrees of freedom. They're like a pile of sand, right? And we are different from that because, you know, the the very basis of how we think is correlated to how the world works. So there's something about building up, you know, you said building up, not not tearing down. And and he said, it's not about what you know, it's about how you got there and how evolvable the knowledge is, so where you can go with that knowledge. And it's not necessarily that stochastic gradient descent is a bad thing in of itself.

• 27:22Tim Scarfe?

  It's simply that it's statistically intractable to use anything other than stochastic gradient descent. Otherwise, it simply doesn't work. So yeah, he's not necessarily saying, I mean, maybe he is, but he's saying that these evolutionary algorithms are really, really good when you're building up, when you have a very, very sparse search space. And and then the other thing which is very important, which we didn't get to, is this whole building up thing, like the neat algorithm, and you can bring that in. Is this idea that we need to have a form of training or learning about the world, which monotonically increases information and complexity? So we're we're adding this thing and we're adding this thing. And every time we compose things together, we're looking at the evolvability. So we're not just saying, can you do the thing now?

• 28:07Tim Scarfe?

  We're saying, is it actually likely to be able to deal with future things that I might encounter? When children are born, they still have two legs, basically. So you shouldn't just think of it as a random, you know, crossover operation. It's an operation that respects the the provenance, the topology, the structure, right? So there are, you know, in practice, there are crossovers and mutations within certain topological frames, but some things are held constant because they need to be held constant. But on your comment about degrees of freedom, I think Kenneth was saying that we actually want to reduce degrees of freedom. So he's not kind of saying, oh, we just need degrees of freedom for stochastic gradient descent, but degrees of freedom are OK.

• 28:50Tim Scarfe?

  I think he was saying that there's actually a magical Goldilocks zone when it comes to degrees of freedom. So we want the degrees of freedom to be at least a representation of how the world works, but necessarily there should actually be more flexibility than how the world works. But if there's too much flexibility, you get a weird kind of mode collapse and you don't see the emergence of evolution.

• 29:12Speaker 02

  Degrees of freedom, the way I think about is like, it's like a memory with an algorithm. You want to use as much as necessary and no more. And it's really hard to, it's really hard to figure that out, so.

• 29:29Tim Scarfe?

  Yeah, I mean, you just imagine Einstein. Of course, there's an element of serendipity, right? You have to be in the right place in the right time, but there are still just mental degrees of freedom. And simply, if there are too many degrees of freedom, then relativity wouldn't be, wouldn't be conceived of, right? You need to have that, that, that spark.

• 29:48Speaker 02

  What's funny is, so he added one additional parameter, the cosmological constant, right? And then was kicking himself for like lots of his life that that was his greatest blunder, right? So, so he considered his greatest blunder adding an additional parameter. Like, think about that, like, you know, how? Like simplicity is such a driver of of successful models. Now, it turns out, though, in the spirit of just as many parameters as you need and and no more, now we need the cosmological constant, you know, for very valid scientific reasons, right?

• 30:22Tim Scarfe?

  On on your comment about degrees of freedom, separating them from training and how things are enacted in in the real world. I like thinking of them more in the same frame. This is actually related to what we've been talking about when we when we discuss agency and autonomy. I mean, Kenneth is using the frame of of creativity, but it's it's all the same idea, right? It's when we have agents that are acting in the real world, we need them to make intuitive decisions that make sense. And that is precisely the reason why current AI doesn't do intelligent things when it's autonomous. That is the reason why we have to supervise current AI every single step of the way, because it would do stupid things.

• 31:07Tim Scarfe?

  So I think if you did reduce the degrees of freedom to actions that actually made sense, the probability of it working autonomously would be much higher.

• 31:18Speaker 02

  Yeah, that's the I think maybe that's a bit unknown for me. Um, I'm not sure, because I think there's there's trade offs there. Um, if if there if it's reduced, a- again, it's this Goldilocks thing. It's like, um, if you reduce it too much, then it ends up sort of not being able to do the things that you need it to do. Um, and if there's too many parameters, it's able to do the things you need it to do. Plus, it does a bunch of stuff like you definitely don't want it doing, you know? And so there's this this weird, you know, this weird balance in there. And and yeah, so I think that's a really difficult question and and very hard engineering problem.

• 32:01Speaker 02

  But this paper is pointing in directions that we need to go. And actually, one really great thing about this paper, which you mentioned earlier, is how visual it is. So like if you look, for example, at the, um, at the, the intermediate layers for the evolved, you know, compositional neural network versus the SGD trained one. Okay, like, I mean, come on, folks, like, I don't care how much of an SGD fanboy you are out there. Like, just look at the images and come and tell me with a straight face, okay, that there's any merit whatsoever to the entangled, convulsed, contorted, convoluted nonsense

• 32:45Speaker 02

  that like you end up with, like with SGD, like, seriously, like, I don't even know if we have a conversation if you think there's like value in there, but if you, because if you look at the evolved ones, they're like a stencil that you could imagine an artist would have. Like, well, here's an oval thing and some, you know, gradient lines and, and this sort of thing. And the point is, it's not just about appeal to humans, okay? It results in extremely powerful transformation capabilities, like these parameter sweeps, where there's a parameter that controls the mouth opening and closing, or the mouth being wider, or winking, or the apple's, you know, leaf literally rotating as if it was on a 3D axis, okay?

• 33:27Speaker 02

  So these are neural networks that were trained solely on 2D images, and they have evolved a concept of like 3D rotation. Like, if you're not impressed by that and kind of blown away by how cool that is, and if you can't appreciate that that's relevant to the real world and agents operating in the real world with an effective world model, like, I don't know, I can't really pull you out of that hole.

• 33:54Tim Scarfe?

  Yeah, and I wrote some software to visualize that so we can show that on the screen. But this actually comes back to the famous episode 61 of MLST when we had Randall Bellistriero and Jan LeCun. It was called interpolation extrapolation something else, I can't remember now. But yeah, we showed this neural network visualizer, you know, it was the TensorFlow playground, I think it was called. And it would show you like the evolution of training, you know, for all of these different types of neural networks with different activation functions and learning rates. And one of the data sets was a spiral manifold, so it showed you a spiral and you could kind of see it, and it was trying to fit the thing over time. And one of the reasons why it can only interpolate when inside the training support, which basically

• 34:38Tim Scarfe?

  means roughly speaking, inside the, you know, there's this high dimensional space of data and it can roughly, you know, go inside the space, is because the activation functions are ReLU, right? They're piecewise linear functions, which means they just chop up the space. Now, what's interesting about the evolved CPPN, the Compositional Pattern Producing Networks, is that the functions can actually be trigonometric functions. These are continuous functions, because most of the time in a neural network, like if you if you give it a test sample, which is outside of the training support, you're in no man's land. It gives you nothing. Whereas with these CPPNs, if I understand correctly, they actually do extrapolate outside of the training support, because these trigonometric functions, they just they just keep going,

• 35:23Tim Scarfe?

  right?

• 35:24Speaker 02

  The problem isn't just the activation functions, it's the training methodology. And what would happen, for example, let's say we just keep ReLUs. I think if you still found a way to do this kind of evolutionary building up from simpler, you know, mappings, what you would end up doing is, there would be there would be kind of a a simple higher order layer that, say, took the spiral and chopped it up into four quadrants, like nice, nice quadrants, and then one that did sort of like some 45 degree angle chopping and whatever. So it would still end up being A piecewise linear division of the space.

• 36:07Speaker 02

  But rather than if you go and do this over at the TensorFlow, you know, site, it's all messed up, weird, you know, lines, funky angles and whatever. Instead of that, you would end up with the type of chopping up that you and I might do, right? Like quads and then hexagons and like whatever else. So the point is, whatever activation functions you have, there's a better way to train them.

• 36:31Tim Scarfe?

  Yeah, I mean, there's a few points here because I guess like what I was, where I was going with this before is you take y equals x squared. And the reason why we think of it is robust is for any value of y, it kind of does something, it does something reasonable, right? It's not, it's not just memorizing little, little regions in, in the training support. And we often speak about the, the limitations of neural networks in a variety of different ways. So there's, there's a complexity limitation, there's a computational limitation, there's a training limitation. But it's quite interesting to play with this and pull it apart a little bit. So yeah, as you say, the SGD thing, that is actually a serious limitation because there are neural networks that can do a hell of a lot more.

• 37:12Tim Scarfe?

  If they are not trained with SGD, I mean, Schmid Huber gave an example of, you know, the, um, it's possible, it's possible to have an R N N, which is Turing complete under certain conditions, but it's not trainable with, um, with SGD. And there's also the other matter that when you start using these like weird and wonderful activation functions, you can do things outside of the training support. And then there's the other matter that when you start doing non SGD training, you can actually build hybrid systems as part of the training process. So imagine if you, if the way we recognize faces, we, we do some kind of composition, splitting, um, iteration, recursion. What you could just train that entire system with another form of training process. So it rather opens up the whole game.

• 37:54Speaker 02

  Right. Yeah, a hundred percent. I mean, like, and you can handcraft neural networks, like for example, um, you know, there was that, uh, was that, um, a DeepMind paper, right? The, the one where they handcrafted sort of a universal. Turing machine in a in an LLM form, but you also have to add to it, you know, expandable memory. So they had to do extended generalized autoregression rather than just, you know, autoregression. So that little technical detail aside, like you're saying, you can handcraft things, you know, you you can construct them, but they're not trainable. And so it's the core thing we need to improve is the training methods. And I think nobody

• 38:38Speaker 02

  knows what they look like, really, like at the moment, like there are some hints, right? Like poet, maybe what you're talking about, these kind of hybrid systems where things are trained in different ways, or the curriculum learning, or some type of building up, you know, from seeds that are somehow kept slightly isolated. I don't know, like I I we don't know, because everybody's just stuck on SGD, SGD, scale, scale, scale's enough, SGD attention's enough. It's like, until the industry kind of fails enough to achieve anything real, like with those methods, I don't think they're, I don't think sufficient resources are gonna go into the alternatives.

• 39:17Tim Scarfe?

  Yeah.

• 39:17Speaker 02

  But they're getting close. They're getting close, because even the doo even the doomers are having conversations about like, why aren't we all dead yet? Like, we thought, you know, once GPT four point five came out, or four point whatever, it's like, I thought we were doomed then. And like, what's what's going wrong? So they're kind of reevaluating their, you know, apex of rationality over at less wrong to figure out what went wrong. You know, why aren't we dead?

• 39:42Tim Scarfe?

  I know. But even this, actually, I think this paper is the perfect, it's the perfect thought pump for people to understand why we are not doomers. Right? Because we're not, we're not horrible people. If we actually believed as the doomers do that this technology is autonomous, that it has agency, that it could just go and do things on its own, we would be doomers as well. I mean, we would be regular guests on Liron's YouTube channel. But it just doesn't work like that, right? That it's not even close to working like that.

• 40:12Speaker 02

  Well, let me just, let me caveat something there. So, excuse me, I'm on the record as having like, you know, a and again, it depends on how you define doom, but sort of let's say an all cause doom of, you know, one third or whatever it is. And I also am extremely concerned about AI harm. Like AI is causing harm today. Massive harm. It's causing massive harm. It will continue to cause massive harm. People are using AI to do heinous things and cause, cause lots of harm. And I absolutely think we need to protect ourselves from that. I'm just not like, I'm not a AI triggered, AI is going to become super intelligent and kill us all

• 40:54Speaker 02

  because we don't yet have the right architecture for that. So like, we haven't discovered any architecture that in my opinion is machine trainable that will lead to AGI.

• 41:06Tim Scarfe?

  Yeah, I agree with all of that. I mean, the way I think about it is there are two types of AI. There is the agentic, monomaniacal, you know, separately intelligent form of AI that Leron is talking about. I don't think AI is intelligent because to be intelligent requires creativity. This is what Chollet says. It's about being able to invent new knowledge, given a novel situation. AI doesn't do that. But the AI that we have now, it requires humans, right? It's, it's, it's a tool that humans use and and it makes us more creative and smarter and it has all sorts of ramifications. And I agree that there's a lot of harms to be discussed there, but I do make that clear separation between the two.

• 41:50Speaker 02

  Yeah, and I make a similar separation. I just use different words. So I talk about the AIs we have now as being narrow intelligences, not general intelligences. And, and I considered those two things like categorically different, not, not just on a spectrum different. So, and I know we have some differences on that and, you know, lots of, but the similar things.

• 42:12Tim Scarfe?

  We discovered this incredible book, Why Greatness Cannot Be Planned, and it's got this butterfly on the front cover. And the reason Kenneth put the butterfly on the front cover is There was this phylogeny which was created by Pickbreeder, which was basically a cross between Tinder and Flickr where you log in and you you select two images that you like and you breed them together and behind the scenes it's using the NEAT algorithm which is a way of evolving neural network topologies. So essentially the the topologies of of the two CPPN neural networks are mixed together and you get a new image. And a CPPN is is simply a neural network which has a broader array of activation functions, so trigonometric

• 42:54Tim Scarfe?

  functions like sine and and you know cosine and whatnot. And it also takes an input of an X and a Y and also a couple of other things, and it has an output of a hue, a saturation, and a luminance. So if you want to generate an image with it, you just basically enumerate a bunch of pixel values and it's resolution independent, which is quite cool. So you can like make very high resolution images or low resolution images. And and it will just generate you an image of something. It might be a skull or an apple or whatever. And yeah, so essentially the whole system is you could, humans could supervise this breeding process. And when you looked at the phylogeny, even though the humans weren't looking for anything in particular, so you you you got these weird intermediate steps

• 43:39Tim Scarfe?

  that did not resemble the amazing thing that was discovered. But what's interesting is that amazing things were discovered in surprisingly few steps.

• 43:46Speaker 02

  It's really fascinating on on so many on so many levels. I think, you know, I highly encourage people to to check out the NEAT paper and Pickbreeder and a lot of related things. I think, and you you pointed this earlier, the crux of the the difficulty is, all right, cool, we did Pickbreeder, we got a skull and a butterfly, and lots of other things, but I don't need a skull and a butterfly. I I actually, I need something that that generates, you know, I don't know, knights' helmets, or or or dragonflies, like not butterflies, dragonflies. And therein is the crux of the problem, because we did this open-ended

• 44:28Speaker 02

  cool thing. And I mean, part of the, let's say, downside of an open Open-ended search is you don't know where you're gonna end up. And I think in the interview, you know, Kenneth referred to this like cone of inevitability, right? Where, where as, as you progress further and further along the, the, let's say the, the time dimension or the step dimension of evolution, where you end up becomes less and less certain. And the crux of the problem is, how can we do open-ended evolution? Um, in such a way that we end up with the things that we need, like we need something that detects school buses and something that finds, you know, pedestrians in the crosswalk

• 45:12Speaker 02

  and something that can generate, um, movies about Samurai. And, you know, like there are things we, we need. And so there's this, this, this, uh, conflict, right? Between, we do have goals, like we do have certain goals. We need to protect pedestrians and, and monitor school buses and like whatever else. So we do have these kind of goals and yet we wanna try and achieve them with open-ended evolution. You know?

• 45:37Tim Scarfe?

  It's such a, it's such a paradox that, you know, you, you, you find what you need when you're not looking for it. But even, there are so many, um, forms of deception in open-ended search. This deception word is, is very interesting. And it's, it's all about what you think you want isn't actually what you want, right? And even you just said now, um, we want school buses to be safer. And there are many discoveries in the search space that would make school buses obsolete. In fact, YouTube is now, it's completely taken over Hollywood. You know, if, if you're trying to be a, a traditional filmmaker, you don't make any money anymore because everyone's watching it here on YouTube. But my God, YouTube could be out of business in, in a few years. We could easily discover something that, what that wipes YouTube away.

• 46:19Tim Scarfe?

  So it's about this epistemic gap between what we really want and what we think we want.

• 46:25Speaker 02

  Yeah, yeah. That's a good point. It's kind of like, um. Why do we need all the kids to go to a physical location every day of the week? You know, maybe, maybe there's some other pure virtual or, or more local, you know, system. You know, you're totally right. I mean, and that's, that's the, the really the crazy thing about life and the universe and we need more variety. We need not all our eggs in one basket. We need to explore alternate, alternate methods, folks. GPUs were invented for like video games. They're also happen to work really well for a certain kind of machine learning. So let's just do that.

• 47:06Tim Scarfe?

  Yeah. And, and that, by the way, in the hardware lottery, that is also a manifestation of Kenneth Stanley's Why Greatness Cannot Be Planned. Because nobody thought when they were developing the GPUs that it would have anything whatsoever to do with AI. And YouTube started as a video dating website. It had nothing whatsoever to do with, you know, a new type of media. Wonderful. Well, Dr. Duggar. Thank you very much.

• 47:30Speaker 02

  Yeah, it was a pleasure. It was, it was so great to be, to be with you guys again. And, you know, it's four years ago that we interviewed, I think about four years ago that we interviewed Kenneth the first time. And it was eye opening then and, and, you know, it's, it's opened a lot of great, open ended exploration for me personally.

• 47:47Tim Scarfe?

  I think he, he won you over in the end, didn't he?

• 47:53Speaker 02

  Sure. I, I, I'll, I'll concede to that.

• 47:58Tim Scarfe?

  Guys, welcome to MLST. Today we have Kenneth Stanley. And Kenneth is my hero, as folks of the show will, will very much know. And also Akash Kumar from MIT. Akash, since you haven't been on the show before, can, can you give us a bit of an introduction?

• 48:15Speaker 03

  Yeah, definitely. Yeah. My name is Akash. I'm a third year PhD at MIT. I'm working with like Philip Izzola. And I'm, I'm interested in researching like emergence, open endedness, and all the cool stuff around like artificial life and meta RL, these kinds of, anything that's really emergent process that creates intelligence or creates some complexity. I'm very interested in and I'm super excited to be here.

• 48:41Tim Scarfe?

  And you've had the great fortune to be writing a paper with Kenneth Stanley. I think not many people would, would have that accolade on, on their resume.

• 48:51Speaker 03

  Yeah, I'm super privileged. I'm super happy to be with them, be here with them.

• 48:55Tim Scarfe?

  As I understand it, you guys have just written this paper called "Questioning Representational Optimism in Deep Learning: The Fractured, Entangled Representation Hypothesis." And the basic idea is that the types of representations that neural networks learn are chaotic. They're just like spaghetti, right? And what we want to have in artificial intelligence is a way of carving up the world by the joints, right? A way of actually understanding how the world works at an abstract level. And it seems to be the case that us humans, by some kind of osmosis or magic or god knows how we do it, we have a very powerful abstract model of how the world works, and neural networks don't do that.

• 49:38Speaker 00

  Yeah, it does seem like we do this in a very sophisticated way. But I think, you know, the paper isn't so focused on saying, "Hey, look how good humans are," and then, "Look how bad LLMs are," for example. It's not really that story. It's more that there's an observation of the underlying representation that you get from SGD, and just how terrible it looks, which is just really visceral. Like, you don't need quantification to see that there's something wrong there. And I think that observation on its own would make only half a paper, you know, because like a lot of people would think, including maybe

• 50:23Speaker 00

  I would think, that's just the way it is. Like, it looks the way it looks, but it works, so it's fine. And of course, people quibble about like, you know, what are the differences between human-level cognition and LLMs? And I don't doubt that humans do something beyond what LLMs today can do. But you can't really look at that by itself and just that when you see you observe the underlying representation and just say, "Oh, clearly it's messed up," or something like that, because we don't know what it should look like. There's no baseline to compare it to. But what's really, I think, unique here in this paper is that we do have a comparison point from a very unusual Set of circumstances that led to a very different looking representation. So actually, you can compare it to something.

• 51:07Speaker 00

  And indeed, the thing that we compare it to looks like it should look intuitively. It's not necessarily clear that like this is how it looks in the human mind or in the human brain. Um, we don't know, there's no way to go in and check. But it definitely would be nice if it did, like it looks the way you would think it should look. And so it just really, I think it just raises questions. You know, we could still, you could still take a position that, oh, well, I'll just dismiss that. And I still think it's fine the way it is with all this horrible looking entanglement. But it definitely raises questions like, should it look like that? I mean, now that we know that there are representations that don't look that way. And so I think what's really valuable about this is not necessarily that it shows that there's a fundamental flaw.

• 51:50Speaker 00

  It may or may not, but that it raises really deep questions that we can pursue now that we wouldn't otherwise have pursued. So there's lots of questions that we can now go after to try to understand the nature of the representation. And maybe if it is potentially flawed in a way that could be rectified.

• 52:07Tim Scarfe?

  So, um, Akash, I mean, Kenneth was just saying that, um, this shortcut rule or good heart's law. You know, we refer to it via many different terms. I mean, the reward hacking and things like that. That there's this problem in machine learning that you put an objective into the system. And generally, the objective is created by us. It's a proxy because we don't understand reality. So we create a proxy. Initially, it works very well, and then it actually gets much, much worse if you continue to train on it. And that is a really fundamental problem, right? Throughout all of machine learning. So I just want to break it down like you mentioned S G D a minute ago. It's why is it S G D that's the problem as opposed to something else?

• 52:46Speaker 03

  So we, in our paper, we don't term it S G D. We term it conventional S G D. And we do that on purpose because we're not trying to, we're not a hundred percent sure if it's S G D. But it's like if we just need to get rid of S G D or what the exact problem is. We're just saying the current paradigm, which encapsulates a fixed architecture, a fixed objective. And S G D as a target, as the thing that's doing the chasing of the target. That entire paradigm, there's something wrong with it because it's susceptible to shortcut learning. And people have known this for a long time. Like I think Melanie Mitchell talks about this a lot, which is basically just good heart's law for representation learning. Right? It's basically just you can, if you try to solve a task, you're going to perfectly solve that task. But what you really wanted was a good representation of that task.

• 53:30Speaker 03

  And as our paper kind of shows, there's many different ways to solve the task. You can solve it with a bunch of heuristics and a bunch of if statements, or you can solve it with the right abstractions. And uh there's the, there's, those are the two ways to solve it, right? And what you pick, uh which one you end up like employing and using, it really matters for what you really want, which is adaptivity, uh generalization, creativity, out of distribution, generalization, and continual learning especially. Um and we can, we're gonna talk about that a lot more later, but yeah, I guess the point is, we don't just care about the training loss. That's not like, we trained it on the training loss, but that's not what we care about. What we care about is the da- all the downstream stuff that we're gonna use it for later,

• 54:12Speaker 03

  which is much harder to quantify, much harder to formalize um than uh just a training loss, yeah.

• 54:19Speaker 00

  And uh let me add one thing. Um I think one of the really interesting things about the observation in this paper is it pokes a hole, I think, in a very deep assumption that we have. Um that if the results are good, then what's underneath the hood is also good. It's a kind of assumption that's unstated uh most of the time. Um but it's clear here, like when you look at the underlying representations in this paper, it's not necessarily the case. Um and so like you can't just look at benchmark performance, which is most of what we look at when we talk about is this model good, and know that actually things are okay under the hood. Um and like Akar said, that does have implication, because you may think, well, who cares? If it does well on the benchmark, then it's good. What, what else matters?

• 55:04Speaker 00

  Like, why should I care? But as Akar says, there's all kinds of downstream implications if your underlying representation is terrible. Um and I'm sure we'll get into those. Um but so like just this idea that like you can't be confident just because things look good on the surface um is thought provoking, I think. It, it leads to a lot of questions about what's really going on.

• 55:27Speaker 03

  Uh one thing I really wanna add quickly to Ken's statement is, I think at the end of the paper, we have a quote from, or like something uh, like a quote or a paraphrase of Ken, Ken's dad. And I think he says something along the lines of um, two mathematicians, they can both like ace like a math exam. And one can go on to become like a great mathematician that discovers a lot of things in the field, and the other one can go on to discover nothing. So the test is just like, it doesn't give you a picture of what we really care about, which is downstream, like um how they influence the field and how their research progress carries out.

• 56:00Tim Scarfe?

  Yeah, I mean, this might be a good time, Kenneth, to talk about the story of your book a little bit. So on the front cover of the book, there was this butterfly. And I'd love for you to tell the story of the butterfly. But in the book, you went to great lengths to talk about this concept of deception, which is simply that when you have a complex objective, again, there's this epistemological gap, right, that we were saying why we have proxies with good hearts lawyers, because we don't actually understand the thing that we're modeling. So we have a proxy. So anything that's ambitious, we create a proxy, we optimize towards it, and we actually get stuck in this deceptive search space, and we find anything but the thing that we actually want to find. What's the story of that butterfly?

• 56:44Speaker 00

  Right. So, you know, this is, a lot of people know my work for this point that's related to the butterfly, which comes from this system called a pick breeder, which is also relevant to this paper. But just for context, like, we're actually going to draw a new lesson in this paper from the usual one that people have heard before from me, if you're familiar with my work. Like, the original lesson had to do with the fact that it seems to be the case that there are some things that you can only find by not looking for them. And so, setting an objective and saying, like, I want a butterfly, for example, can actually cause you to fail and not be able to get to the thing that you want. And

• 57:28Speaker 00

  this was true in this pick breeder system, which was basically a big crowdsourced experiment in people searching through, exploring, image space generated by little neural networks called CPPNs or compositional pattern producing networks. And this, this, this pick breeder system allowed people to effectively breed pictures. And it was, we found inside of the system that the people who would decide they want a certain image and try to evolve that image would fail. And then people who are not looking for anything in particular would discover all these amazing things, like the butterfly was kind of the symbol. That we used, because we put it on the front of the book,

• 58:12Speaker 00

  of one of these serendipitous kinds of discoveries. And this led to this idea that, well, you know, there's many things in the world that we're not going to be able to find if we directly search for them, because of deception. And that's the underlying reason. And deception means that the stepping stones that lead to these interesting artifacts that you might want to find don't resemble them. Which means that if you have a, if you have an algorithm that's trying to follow a gradient by matching closer and closer and closer to the objective, getting a higher and higher score, or a lower and lower loss, that would mean that you're going to get stuck in a dead end because of deception. Because the things that lead to the thing you want actually don't look like the thing you want.

• 58:56Speaker 00

  And this is true in the lineage of many of these images in Pickbreeder. And so it led to, you know, the novelty search algorithm, which was an idea that, well, hey, if that's the way Pickbreeder users succeed, maybe there could be an algorithm that's actually not driven by where it's trying to go and instead just tries to do something new and interesting all the time. Which then led to the field of quality diversity. You see quality diversity inside of major systems like now, like for example, very recently Alpha evolve. It has MapElites under the hood, which is a quality diversity algorithm. So, and these are systems that try to get out of deception and try to illuminate spaces of many, many interesting things. And so it led to a lot of stuff. This insight, and then the butterfly is kind of symbolic of this insight.

• 59:40Speaker 00

  But there's one further insight that was missing from all of this, which is another deep insight, which just shows like this, this crazy Pickbreeder experiment had all of these really unexpected discoveries underneath the hood. And it was this, this, this point that, what about if you do get to something as an objective? You know, so like, what was interesting with Pickbreeder was we did experiments later to confirm the hypothesis that you can't get to these things by trying with the same algorithm. The underlying algorithm in Pickbreeder was the neat algorithm. So like, someone found a skull, for example. The skull is used all over this new paper. So someone found a skull, and so we, we...

• 1:00:19Speaker 00

  Then said, okay, well, we have this hypothesis that like you wouldn't be able to find the skull by actually looking for it. So we tried to automate the um the the neat algorithm to just try start from a random position, a random configuration of the network, random weights, and then to just try to get closer and closer and closer to the skull. Um, and this generally would not work. Um, it was too hard for the neat algorithm to do that because it is deceptive. The things that lead to the skull don't look like skulls, so image matching is not a good heuristic. However, there was this one caveat to that, which was kind of interesting, which was that with relatively simple images,

• 1:01:00Speaker 00

  like some of the easy images from PicBreeder, like a crescent or something like that, just something relatively simple, we could evolve to it as a target. Um, but we noticed something weird about it when we would succeed, which was that it was always triple the complexity of the network. And it's it's important to note that with the neat algorithm, the networks grow in size as they optimize. And so like we could see the difference in size of the networks. And so if there was a success, it would be much worse because it was obviously inefficient compared to the original discovery. This was a just a kind of a um a side observation. We didn't really push on it too much, but it's really interesting

• 1:01:40Speaker 00

  that it seemed like the open-ended process, like the process where someone's not looking for something would lead to a more efficient representation. Farther down the line, when deep learning, you know, became dominant, you know, we we started to think, Joel Lehman and I started to think, you know, let's just see what SGD would do. Um, if we just try to train a network to output a single PicBreeder image. Um, and then we could see that like it's not, you you don't get a bigger network in effect because like you just you could get whatever size network you start with. I mean, it doesn't grow, but we we found you couldn't start with the same small network that was found through PicBreeder. That there's too small, too few degrees of freedom for SGD because it's deceptive in that space. So like you need a lot more degrees of freedom.

• 1:02:22Speaker 00

  This is one reason that you can train these huge networks like and and to do amazing things. And so if we started with a big enough network with S G D in this case, we could reproduce Picbrieter images, no matter how complex they are, we could do it. But when we look under the hood, the representation is terrible compared to the underlying representation that we see in the original discoveries from Picbrieter. This is a really weird principle if it actually generalizes, because what it means is that the story doesn't end with the question of whether you solved the problem. It's that if you solve the problem in one way, it's worse than if you solve it in another way. Like, we never talk about this in this field. Like, if you solve the problem, like, to be able to output the picture of the skull

• 1:03:06Speaker 00

  in a serendipitous way, it's a beautiful underlying representation, which is what the paper shares. But if you solve the problem through brute force, like, I'm just gonna move deliberately towards that image through stochastic gradient descent, it's terrible in comparison. You may say, "Well, how do I know how terrible? Why do I say it's great versus terrible? Like, why would I know this so well?" Well, the really cool thing about 2D images, like Picbrieter images, is that it makes it easy to just see directly, explicitly, what is represented at every single intermediate node. You know, because we can query every node in the network, not just the outputs, but every single neuron, we can ask for every X, Y position, "What are you computing?"

• 1:03:47Speaker 00

  So we can draw a picture for every single node in the network and literally see what the representation is, how it builds it up, piece by piece, to get that final image. And it was just stark that what you see from S G D is horrendous. So, in the sense that, for example, it has no recognition of the symmetry of the images, like the butterfly or the skull, it has no idea that they're symmetric. It computes these horribly entangled and fractured representations where pieces of the left side and the right side have no coherent organization and have to be rerepresented multiple times when they should only be represented once, because it's effectively symmetric. And you can just see that explicitly, and you can see in the version from Picbrieter,

• 1:04:32Speaker 00

  this beautifully captures the underlying representation. It's not just that it captures symmetry, it has an unbelievable modular decomposition. Like it knows what a mouth is, like there's separate controls of the mouth, the mouth can open and close, which is absolutely incredible, mind blowing to me. Because it's like there's a world model of what a mouth is there without being data driven. Like, how is that even possible? It's it's like contra the bitter lesson, like there's not a lot of data here, but we're getting world models out of this thing. And so I think um like this this observation that it matters how you got to the solution, how it's represented under the hood, just hasn't gotten the light of day until now. Um, and it's it's kind of a companion to the old insight from the book,

• 1:05:16Speaker 00

  you know, which is that sometimes the only way to find something is by not looking for it. But now there's this caveat, but even if you do by looking for it, you may pay a steep, steep price in terms of the underlying representation.

• 1:05:30Tim Scarfe?

  Yeah, to to to play back a couple of things there. So there was this paper called the lottery ticket hypothesis, which was by Jonathan Frankel. And and he said that S G D only works when you have many, many degrees of freedom. When you start with this kind of dense monstrosity, you train it with S G D because it can use those degrees of freedom and then you sparcify it afterwards. And we should be clear to say that this isn't a limit of neural networks as a representation. I mean, there exists a neural network representation which does, I mean, a C P P N is is a neural network. But the question is, how do we start sparse and build up rather than starting dense and using S G D and and coming down? And then I want to pull apart a couple of things. So you're framing this just as an optimization problem, right?

• 1:06:13Tim Scarfe?

  And I think there are other interpretations for what's going on here because it might not be contra bitter lesson. It might be the fact that us humans, we've had the benefit of billions of years of evolution. So we've had all of the data, we have learned all of these abstract motifs about how the world works. And through a form of interactive imitation learning, I certainly think this is what happened in pickbreeder. We're unwittingly imbuing the the model with the representations that we know about the world. And that explains why in so many intuitive, you know, so few intuitive steps. If you look at the phylogeny for pickbreeder, you can get to the butterfly in something like, was it five or six steps? So is the optimization process just kind of cleverly?

• 1:06:58Tim Scarfe?

  Like, is it telegraphed from the humans or is it actually learning it in some fundamental way?

• 1:07:04Speaker 00

  So I've had this discussion, or you might even call it debate, with a number of people. And I don't want to claim to know the answer, because I think there's a deep, almost philosophical question at the heart of this, where the answer is actually not known. But I think at least it's important to convey that there is another side to that argument that's valid, I think, at least reasonable to consider, which is that, no, the human has not supplied enough information for that to be a satisfactory explanation. Like, the humans, generally, we're talking about from dozens of steps to hundreds in the entire run. I mean, think in terms of deep learning, we think about millions or billions of steps, like, this is absolute peanuts.

• 1:07:48Speaker 00

  And so, like, the dozens or hundreds of steps, like, it was dozens to get to the skull, for example. There just isn't enough time to show it things like a mouth opening and closing, whatever that would even mean. It would be a strange, also, it's a strange trajectory to take, like, the human is not thinking, that they're training, it's not the thing in their mind. So, like, you know, you would think, like, from a training perspective, you would show examples of closed mouth, open mouth, closed mouth, open mouth, like, let's show the space so you can understand that dimension of freedom. They would not do that, that's not how people are thinking. People are choosing things because they look good. And so, like, we don't know for sure, like, maybe, you know, it could be that inadvertently,

• 1:08:30Speaker 00

  somehow, miraculously, they just sort of, like, trawled it through the space of all of these dimensions of freedom in just the right way. But I think that that's probably implausible, and there needs to be another explanation. I think it's actually that the open-ended process causes the human to actually move instead through a hierarchy of regularities, like, starting with things like the discovery of symmetry, just because humans like symmetry. They see a symmetric object, they like the symmetric object that locks symmetry into the system, then from there, they continue to search and find symmetric things that now vary on the symmetric convention that's been established, and add more to that. And that it's a side effect of that process, because it develops this hierarchy of represent-

• 1:09:13Speaker 00

  ...which is very virtuous in its structure. A side effect of that is you get some kinds of degrees of freedom or dimensions that just happen to align with things in the real world, almost like miraculously. And we see multiple examples of that in Pickbrieter, underlying representations, like, my favorite is this, is the apple. Like, the apple, which is in the paper in the appendix, has a stem, which has a single weight, one weight in this network of, I think in this case, hundreds, hundreds of connections. One of the weights actually can control the swing of the stem. It looks like a 3D projection of a stem and moves back and forth like this. There's a shadow that moves underneath it.

• 1:09:55Speaker 00

  And this is a single dimension that you can sweep to cause this stem to move back and forth. Again, I think of it as like a hypothesis about the world or a world model, which has absolutely no data behind it. So, it's insane. You know, it's like the exact opposite of Move 37. It's like play millions of games, and then, you know, maybe it's not to surprise that something might happen that humans have never seen before that's actually quite valuable. It's an origin of a creative insight. But this kind of insight that we're seeing here is not from millions of trials or seeing all of the possible ways the world could work. It's from very, very limited amount of interaction. And just somehow, it's a coincidental alignment with the way reality actually works, but it's happening over and over and over again.

• 1:10:39Speaker 00

  You know, so I think it makes me wonder, like, you know, when humans have really deep insights, something like theory of relativity, something like this, you know, which one is it more analogous to? Is it more like Move 37, like Einstein did 10 million simulations in his head of what might happen when certain physical interactions take place? Or is it just that something about the order of the hierarchy, of the things that he integrated into his model of the world over his lifetime, just happened to precipitate this in the right place at the right time, that actually does align with the world? And many times it wouldn't, so he's kind of lucky, and all, we all are when we have a good idea. But nevertheless, because of these, this virtuous ordering and the way that integration, the way that representations integrated...

• 1:11:24Speaker 00

  It can lead to these amazing serendipitous alignments once in a while, which just come out of thin air. And that that might be the kind of thing that we're observing here, which is a different kind of thing. It's not the bitter lesson style of thing. It's a wholly other type of thing, which I think we need to dig into more, now that we've exposed it, to understand better what it actually means. Because I don't think we have a good understanding yet.

• 1:11:47Speaker 02

  And let me, let me jump in, because I didn't get, so it's, it's great to, great to see you again, Kenneth.

• 1:11:54Speaker 00

  Likewise.

• 1:11:54Speaker 02

  It's been, it's been a very long time, and Akash, nice, nice to meet you.

• 1:11:58Speaker 03

  Nice to meet you, too.

• 1:12:00Speaker 02

  And the reason I want to jump in here is, is, first of all, this, this paper, I mean, it's, it's brilliant, it's insightful, it's important, it's visually one of the most beautiful papers I've seen in a, in a long time. And I think even people just looking at this will get the insight that you're talking about right now, like with Einstein, you know, because, I mean, sure, we're not Einstein, but I think anybody who introspects the way they think, the way they think about solving problems, the way they think about a skull, the way they think about an apple, you know, they're going to find that the images in this paper completely reflect the way we think about the world, the way we model the world. Right?

• 1:12:44Speaker 02

  And so by this kind of open-ended search and interaction with humans, like you said, the neural network is, is able to grow in such a way that it captures these fundamental world model aspects. I mean, it's really cool. It's really cool.

• 1:13:00Speaker 00

  Thank you. Yeah, and it's great to see you, Keith, too. Like, this bring back good memories to that, that old interview that, that was really fun. So, yeah, appreciate your comments. Yeah, and I, I, I, I'm not going to agree with the part about the, the beauty of the paper, because I want, that's for others to judge, but, but obviously, I agree with that, that, like, this is something we need, we need to think about, like, what this means. That's, that's why we put it out there.

• 1:13:28Speaker 03

  Yeah, I wanted to say something about Picbreeder more concretely, uh, really quickly. It's basically this, uh, just to drive the point home a little bit. These people who are on Picbreeder, they were selecting only images via, they only saw like fifteen mutations and they just clicked one. That means they're selecting, they're injecting like what, like a few bits of information into the system? And you end up with a butterfly after a few dozens of, uh, generations. So like, just from an information theory point of view, you're not, you're not able to put enough information to, like, put, like, describe the regularities of the butterfly, like symmetry and the skull. And I think the thing that really makes this, uh, Picbreeder work, and it goes back to your point, they're talking about in general, the creative space of like all human outputs.

• 1:14:10Speaker 03

  But in Picbreeder, I think what's especially, what was at play, was like the evolution of evolvability. Because people were only selecting for what they wanted, right? What they would look good. In turn, in the general sense, in an open-ended sense, what they thought would be, uh, cool to follow, pursue. But implicitly, there also, there's also implicit selection for, uh, selection pressure for evolvable things. So if there's like two versions of the skull, which is one is like spaghetti and one is like very modular and composable. After a few generations of, yeah, evolution, the one that's more evolvable will be the one that wins out, right? Just like in natural evolution, the evolution of evolvability. And this evolvability combined with the serendipity is what I think gives you these nice representations.

• 1:14:53Speaker 03

  Cause if you just try to get a skull which moves its, uh, mouth up and down, right? Like you're just wanting that one type of regularity. You're not gonna be able to, one, you're not gonna be able to find it. And two, even if you do, the representation is gonna be like overfit to that thing, right? Uh, what you really want is evolvable in any direction of interestingness, kind of. And that's where the serendipity really comes in, is you have to embrace whatever the system gives you. And if it gives you a regularity of symmetry versus it gives you the moving of the mouth versus if it gives you the eyebrows winking. You don't know which one it's gonna give you. You just have to see what it gives you and go from there. And I think the combination of those adaptability, evolvability and in general, adaptability,

• 1:15:37Speaker 03

  combined with serendipity is what's giving you these nice representations. And I think that has a lot of, um, analogies to natural evolution and scientific discovery as well. Yeah, and building on top of that also, uh, really quickly is parsimonious. I guess in deep learning, people are obsessed with the idea of compression, right? They think compression is intelligence, compression is everything. And in some ways, it kind of seems like it has to be true. But in other ways, I'm not sure that compression captures this idea of adaptability fully. Because you can compress the hell out of the skull and get a very nice compressed representation. That doesn't mean it's going to be adaptable or evolvable in the way that you want it, right? In the same way, you can probably compress all the knowledge about the world

• 1:16:21Speaker 03

  and your mental model of the world in a very like small zip format or like very compressed thing. But that doesn't mean it's adaptable in the sense that it's going to be, it's going to give you creative leaps and the way to connect the dots, if that makes sense.

• 1:16:35Speaker 02

  Yeah, sure. So we'll be looking forward to your next paper. Compression is not enough, right?

• 1:16:42Speaker 00

  That sounds good.

• 1:16:44Tim Scarfe?

  Copyright Keith Duggar. But some people say, I don't know, like polysemiticity or Grokking or scale. And you know, it just appears like the neural network isn't Grokking it. But actually it is, and maybe if you just trained it a little bit differently.

• 1:17:00Speaker 00

  Part of what's important is just to point out that we've identified a really interesting phenomenon, but we don't have answers to all these questions yet. And sort of part of what we're doing is saying, let's go find out the answers to these things. It's possible that some of them help. It's possible some of them don't help. I think one thing just to think about, just to observe upfront is that the difference that you see visually is so dramatic and stark. I mean, anybody who looks at the figure of the two different versions of the skull, it's hard to believe, although we don't have the evidence yet, that there's something that's going to fix this that's just real simple, like a Grok. Like, I imagine Grokking does something good, but it's hard to believe it ends up making something that looks like the CPPN version of the skull from Pickbreeder. That's hard to believe.

• 1:17:44Speaker 00

  But nevertheless, it's possible that things like Grokking and all kinds of other things that we do, like mixture of experts, maybe convolution helps in some ways, although that's not in the LLMs, but it could be in other networks helping. Like, they're all just questions that are not answered. But I think one thing to think about with respect to things like Grokking is just... Even if you, like if you're trying to rationalize and save your faith in how these things work right now and think, oh, well, that's okay. Grokking will fix this. Just think about this, like, wouldn't it be nice if you didn't have to do grokking? Like what grokking basically means is that you get this absolutely horrible entangled mess and you clean it up later. Like, once you start figuring things out, start deleting all the excess and redundancy, all the fracture gets fixed.

• 1:18:28Speaker 00

  Wouldn't it be nice if you don't have to do that and it's just good in the first place? Like, I mean, like the, the, um, what, what we see because of the pick breeder CPPN is that that actually is possible. Like, without this example, without this weird example from pick breeder, almost no one would believe or buy that that's even possible. It's like, well, I mean, yeah, of course, I'd love miracles to happen every day, but it's just not gonna happen. But the thing is that it did happen and it happens consistently. So it raises this question is like, even if grokking is helping, is there a better way or maybe a complimentary way? Because maybe both factor into why, like, you know, human brains get to where they get to. Um, so like, these are unanswered questions. This is not to say that grokking doesn't help. Um, we need to check and find out the degree to which it helps.

• 1:19:12Speaker 00

  But one other kind of interesting thing to think about is that if you think about, um, the efficiency of training, you know, how expensive it is, like rebuilding, spending, you know, billions, hundreds of billions of dollars, like creating these, these giant data centers to do giant kinds of training runs. Um, like, if, if it's true that, like, there's an incredibly more efficient way of doing things. You know, it's like, so for example, like, it could be that, like, yes, S G D from where the brute force will do basically almost anything, at least, um, in terms of being able to get the training data. Um, like, obviously, there are implications for generalization from what we're observing here. But at least, like, you know, if you cover almost all of human knowledge, you could do a lot of useful stuff.

• 1:19:54Speaker 00

  Um, and so you just brute force it into the system and you get this F E R representation, which sucks, but it still works. Um, but the thing that this raises is the question is like, but the, it's really, really hard if you're in a point in search space where all the regularities are broken, to get to the next point in terms of like higher level. You might be able to do it, but the the the representation is not doing you any favors in doing it efficiently. Right. Because all the degrees of freedom that you have are the wrong ones. And so you're constantly overcoming and making up for and overcoming and making up for, how much cheaper would this whole thing be if we just did these things in a way that actually worked the way we're seeing in these CPPNs? Like if it was actually like always getting the right dimensions or close to the right

• 1:20:37Speaker 00

  dimensions that actually align with the modular decomposition of the way the world actually works, then this could be multiple 10x, 100x, like more efficient in many ways. So even if like you still believe we're gonna get there with just SGD, this could at least have implications from an efficiency point of view. I suspect it has more implications than this just that, because when we go beyond efficiency to things like creativity, you know, what you're seeing is that the dimensions that have been discovered in the skull, for example, align with new skulls, imagining new things in the world. You wouldn't be able to imagine these new things if you didn't have those dimensions already grocked, so to speak. And so your ability to imagine something new is highly compromised.

• 1:21:21Speaker 00

  And of course, like the whole thing, like, you know, the conversation, the public conversation right now, is turning to exactly this issue of creativity. Like people are saying, well, where's all the new knowledge? Like, isn't that next on the menu? Like, we're getting all, we're acing all these tests and it's like amazing, Math Olympiad, like beating the PhD level people. The people don't get PhDs because we want them to score well on tests. They get PhDs because we expect them to invent something. They're supposed to come up with new knowledge. Where is this gonna happen? And so this suggests that there could be serious implications here, because the ability to invent something new requires that as a prerequisite, you understand the underlying dimensions of reality.

• 1:22:02Speaker 00

  Even if you're gonna break those dimensions, like some really innovative people do intentionally, like break a rule, it's because they know the rules before they break the rules that it's interesting. If they just don't know anything and don't understand how the world works at all, they're not going to be brilliant and have these amazing insights. And so I think, I think there's like really big implications here, um, in terms of Both efficiency, creativity, generalization and continual learning. You know, because what I said about like, if you want to keep learning, it depends on how you represent the world, how efficient that's going to be. So like, I don't think we can just say, oh, Grokking might fix it and like, everything's going to be fine.

• 1:22:38Speaker 02

  Let me just jump in on two points. So one is, and I want to know if you think this analogy is fair. This issue of the problem with fractured representations, I kind of think of it as, you know, if you set out to build a nice UI for an application or something, and instead of starting with a stencil that has triangles and squares and ellipses and whatever, you start with a jigsaw puzzle, you know, it's going to be a lot harder, right? To try and build a nice UI out of fractured, weirdly shaped, you know, components. Like, that's a fair analogy, right?

• 1:23:09Speaker 00

  Yeah.

• 1:23:09Speaker 02

  Okay. And I guess I wanted to ask you, like, I really see, you know, connections between this kind of path dependence that you talk about. Like, it matters a lot how you got to your state of knowledge, how you got to your representations. I think there's a direct or deep connection with, you know, poet, like the work, you know, your earlier paper, right, on this kind of increasingly complex curriculum and environment, where you start off training in simple cases and make them more and more complex. I mean, there is a connection there, right? And maybe that's a simple tool that can be utilized.

• 1:23:44Speaker 00

  For sure, yeah. I mean, then that's, you know, one of the other factors we talked about that might help to mitigate the issue is, you had the kind of open-ended search, where, yeah, it sort of naturally guides the process through a set of tasks of increasing complexity, but in a way that's divergent. I mean, that's like what's happening in pick breeders, so like, we could extrapolate that that's actually partly, or maybe largely, responsible for what we see in these representations. But it raises, you know, questions about, you know, are there very different training paradigms from just dumping in all the data in the world in a kind of a batch?

• 1:24:29Speaker 00

  Which are more intentionally focused on the chronology. Being intuitive, or at least aligned in some way with building good representations. I mean, it evokes ideas about curriculum and things like that. But like, there's also a kind of concept of a natural curriculum where like, the human being themself, by their nature, tends to learn things in an order that's actually useful for building good representations. Like, for example, like, you know, you take little kids learning arithmetic and if you start trying to teach them calculus, they're just gonna ignore it. They're not gonna start absorbing it. I mean, that's not the same as LLMs. Like, they'll take anything you feed them and start to make the connections, which arguably is really unhealthy.

• 1:25:11Speaker 00

  You know, because if you start to learn calculus before you've learned arithmetic, you can actually create some kind of heuristic version of arithmetic. You can't avoid it. At the same time as you're learning arithmetic somewhere else in your brain. And this is what causes this kind of redundancy or fracture. You get like, multiple representations of the same thing. Some of them diminished in their capacity.

• 1:25:30Speaker 02

  Well, that's shown in the Mechantr, you know, paper. For example, the biology paper, right, about where you find arithmetic that's done in some bizarre way. Like, I'm gonna add thirty-nine and multiply there, subtract fifty-three and add to, and, you know, it's exactly what you're saying, which is a fractured representation.

• 1:25:48Speaker 00

  But I just wanna, wanna, yeah. I mean, one other really important point I just wanna add to this. This may also apply to humans. So, I don't wanna seem like I'm saying that all humans have unbelievably beautiful, you know, unfractured representation. We also, I think, are victims of going through things in a bad order sometimes.

• 1:26:06Speaker 02

  Absolutely. And, yeah. Well, I can give you a personal example. I can give you a personal example of that. So, in high school, when I went to sign up for physics, for Physics One, they put me in the one that was for people who had not had calculus. And I had had calculus, right? And so, I'm in this class and I'm like, "What the heck are we doing? We're just memorizing all these like, long lists of equations, you know, for a cannonball in this situation." And they're like, "Oh, actually, we put you in the wrong class. You have calculus. You're supposed to be over here." So, after a week, I switched to the other one. It was so much easier. Because I knew calculus, so I didn't need to have a formula for this specific cannonball situation. I just could derive it, or just calculate it directly. You know, it's radically different, different learning mode.

• 1:26:49Speaker 03

  And that's, that's exactly the- Oh yeah, I was gonna say, I was just gonna comment on the biology paper that you mentioned. That's kind of like how it's doing arithmetic. It's like memorizing all these random heuristics. Like if you look at like one of the uh circuits they found, it's like they're trying to add like twenty three and fifty seven. It's like, oh yeah, it's around fifty five plus around twenty five. So it's like gonna be around this. And at the end, it all magically works out, but it's just like a bunch of like if statements and passwords that gets you there. Yeah, so if you if you learn arithmetic like that, how how can you adapt that to like new situations, right? So.

• 1:27:25Tim Scarfe?

  And and even with that, I mean, we're not gonna get into the computational argument because Keith will go off on one about Turing machines. But you know, there there are actually limitations of what you can do. So so the neural network has has to learn some fractured version of multiplication or whatever it is, because it's it's a finite amount of computation. And we were just talking about Grokking, and the way I understand that is as the training process progresses, they start off learning quite simplistic um sort of low frequency um representations. And then you train and train and train to Grokking, and eventually you learn very high frequency representations. And it just so happens that many of those high frequency representations are more aligned to the natural factorization of the world.

• 1:28:06Tim Scarfe?

  But that's basically coincidental. There is no principled way to distinguish good representations from bad representations. I tell them never use G P T to generate anything, because it's it's always obvious. But what you can do is you can write something and you can discriminate with a G P T model, right? And so so so it's good at discriminating, not good at generating. Would it be possible to have our cake and eat it? Could we build some kind of a sort of like a bottom up algorithm that does something a little bit similar to pick breeder? And every step of the way, it's asking a language model that's been trained on everything in the world, does this look good? Does this look good? Would something like that work?

• 1:28:46Speaker 03

  Yeah, like a pick breeder for A I and intelligence, right? Yeah, I mean, like, creating an open-ended algorithm which solves this issue. I think Jeff Clune on our paper, he calls it like the trillion-dollar question or the trillion-dollar algorithm. Because that's basically like how you, if you think that the UFR, like the Unified Factor representations are akin to like a human, then that's basically like creating like a human representation, right? Um, so yeah, I mean, I guess the answer from my side is that uh, some algorithm should exist and we're gonna try to find it, but right now, we're, we're still, you know, just in the preliminary stages of this kind of thing.

• 1:29:23Speaker 00

  Yeah, I mean, it doesn't seem like there's any principle that says you can't do this algorithmically. I mean, presumably, it's all algorithmic. Um, so hopefully, this is just a nudge to have us start looking into it. Um, and, you know, you, you can go on your merry way down the path that we're going down and just make things bigger and have more and more data. Um, and ignore this, but then you're at risk of disruption, you know, because if somebody does actually take this seriously and it works, obviously, would have profound implications. I mean, you just look at the two pictures of the underlying representations in the skull. If that actually would be translated into the world of giant L L Ms, there's gonna be implications for that.

• 1:30:06Speaker 00

  So I don't think we can just ignore this question, um, and, and the question of like these magical types of algorithms is on the table now.

• 1:30:14Speaker 02

  Yeah, so how, how, how can I get in with you guys on the ground floor of this trillion, answering this trillion-dollar question?

• 1:30:21Tim Scarfe?

  Andre Capathie posted on Twitter, and, and he said VO3 has just been released. And loads of people have been releasing videos of like, you know, hairy gorillas in the tundra, doing ASMR videos and all this crazy stuff. There, there's this ideation and curation. People came up with these incredible ideas and then they, they generated images with, with VO3. Now, Andre Capathie has said, well, the obvious next step is that we kind of do gradient search over the creative space. So what if Zuck built Instagram version two? And now the ideation and curation were basically optimized using stochastic gradient descent. And the images would still be like in the the mixture of collaborative filtering.

• 1:31:04Tim Scarfe?

  So the users would be kind of like saying, I like that, I don't like that. My intuition is that would mode collapse. And maybe it would work better if it was mixed in with with human creations. What say you?

• 1:31:16Speaker 00

  That, yeah, this is a very complicated question. I think, you know, you have to disentangle the degree to which the human is part of what you're calling creative and the degree to which the model is. So like with the VO three, you know, the the human came up with the idea of the the ape doing the ASMR or something like that. So so who's getting the credit for being creative? It's not that it's not impressive. I mean, certainly the model did something impressive, but where's the creativity? And so I think, though, in in the larger picture, you know, if you want to ask like, can the model totally autonomously on its own be genuinely creative, whatever that means? Because people are going to disagree about what that means.

• 1:31:58Speaker 00

  Then, you know, is it just a matter of some kind of gradient following algorithm? That's going to uncover some really interesting, insightful types of creative scenarios that that would be that would be really valuable for all of us. And, yeah, I think I think it depends on the underlying representation. I mean, that's like the answer. It's like, what, how is the world represented inside the model? Because that's going to be what determines what are the adjacent points where the gradients can actually get you from where you are. And if the model doesn't represent the world in a coherent, parsimonious way, then you're going to find your options more limited, not not so limited that you won't be impressed at all. You'll probably still be impressed.

• 1:32:41Speaker 00

  But but more limited than like a really creative auteur or something who's coming up with amazing new genres and a new way of thinking about film or something like that's not probably going to fall out of this. And so it's sort of like depends on the degree of of what of this creativity that you want. And I do think that, like, you know, I distinguish between I call like derivative creativity and transformative creativity. You're going to get a lot of the derivative style, but a lot less of the transformative style if you have a bad underlying representation. And so that's the trade off that we're working with. It's still going to impress a lot of people, but the mode collapse problem, I think, is a real. Concern here, you know, because as we sort of like freeze pop culture in the year

• 1:33:25Speaker 00

  twenty twenty five and just live inside of that bubble for the rest of eternity, like it's gonna get more and more tiresome. And so like the ability to have like pathways out of that, like again, will depend on how we represent the world. And there's a danger with this road that we're going down that that we're gonna be trapped. And we're already seeing this. I mean, it's like we're seeing that, uh, like it looks like like what's on the radio doesn't sound as different from twenty years ago as it did twenty years before that. And this kind of convergence could be just accelerated by the kind of stuff we're talking about here. And ultimately, I think it all boils down to representation.

• 1:34:04Tim Scarfe?

  It doesn't work autonomously. It's not creative because it's not built on the foundation of a representation that describes the world well. So as soon as you leave the thing on its own, doing anything with any flexibility, it'll just kind of go crazy. But I guess the question is, does it matter? Like, we have AI and we use it in tandem and we can do great things with it. Is that a problem?

• 1:34:29Speaker 00

  I mean, I totally agree that in terms of creativity and the amplification of what humans already can do, AI is hugely valuable in its present form. Like, it can amplify things in really interesting ways. But it's still ultimately the germ of the ideas is coming out of the human mind. And so, you know, you could be satisfied with that or not. You could say, this is good enough. I'm happy. But I think like in terms of what like what we can't do because of that, that we might someday in the future be able to do, the things we can't do are the things where the human mind isn't having the ideas. You know, like we're worried about things, you know, from a scientific perspective, for example, and like the kinds of things like the new place that I work now, Lila Science

• 1:35:14Speaker 00

  is thinking about like automating the wheel of science, where we would like these ideas to come faster, to solve the problems of the world. But I mean, it also applies to art as well. You know, like, I mean, it's like, when are we gonna get the next big idea? In music like that's gonna like really shake things up. It's like it's taking a long time here. And so of course, we can just wait for humans to come up with these things. And wait long enough and it might happen. But the question is, can AI accelerate that process? Both in the sciences and the humanities. And you know, this is separate question of is that a good thing, but it's just an interesting question. Can it be done? And you know, if it can be done, then then that's not what's happening with current models yet.

• 1:35:58Speaker 00

  And so like I think like when we talk about representation, it starts pointing us in the direction of how that will ultimately be done. Is by taking seriously the underlying representations. And understanding that to actually see something novel in the world. In a genuinely interesting and deep way. Requires you to decompose the world in a special way that's highly organized and unique. And that's just not what this fractured and tangled representation seems likely to do. At least as far as we've understand it right now.

• 1:36:30Speaker 02

  And actually, this question came up, Ken, in your doom debate, which is. You know, is survival an objective or a constraint? And I think, you know, Ken, I think, and by the way, I want to say that was a brilliant interview. I thought your performance was. Excellent. I really enjoyed watching that debate. I saw it from start to finish. I think, you know, your communication of. Open ended search and and kind of these this take on it has. Has been sharpened, you know, significantly since like four years ago when we talked, I found it like really compelling. So I thought that was a great. A great conversation. And then there you say, look, you know. Survival of the fittest is is not an objective. It's a constraint.

• 1:37:13Speaker 02

  It's like it's a constraint on the system. You have to survive if you're gonna. Propagate, but subject to that binary, you know, survive or not survive. It's not part of any objective. You know, I think is the way you kind of communicated that.

• 1:37:27Speaker 00

  Yeah, yeah, I think that's an important point here. Like, I don't think that the right mental conception of evolution, biological evolution, is as an optimizer. And this causes endless confusion, you know, because like, I mean, in the field of AI, it causes a lot of confusion. Because unfortunately, you know, early genetic algorithms, essentially, were using selection for explicit optimization. Like, the metaphor was broken, in my view. Like, that, like, basically, we had the metaphor for selection, but the metaphor is still wrong, because like, that's not overall what evolution is doing. It's not selecting just to get to a single target. A single target in the search space that we're trying to get to, like, the solution to one problem.

• 1:38:13Speaker 00

  But the problem is that, in the field of AI, like, people think of genetic algorithms as effectively like, an appropriate metaphor for what evolution is. You know, it's sort of, and it's sort of a bad optimization algorithm. Like, people think of it as like, the poor man's optimization algorithm. And so, get like, really dismissive of evolution as a useful metaphor. It's like, ah, we played with that in the 80s, but like, it is nothing really down that path. But I think that's why it's been damaging. That, like, it actually is a very deep metaphor, if you think of it, like you said, as a constraint. Like, you think of survival as a constraint, is not as an objective. Like, there are many, many things that we could have thought of as objectives, like, flight, you know, or, or, or like, photosynthesis.

• 1:38:58Speaker 00

  Like, these could have been objectives for machines. There would have been single runs, where if they succeeded, we would have celebrated and said, "This is amazing achievement. We discovered the ability to fly, the ability to process sunlight." Like, these would have been achievements that were objective, but they're not the objective of the system, in the usual sense. And the reason that the system is discovering is not because survive obviously leads to these things. It doesn't follow at all directly. They say, "Well, we have an objective and we're optimizing it. It's called survive, so of course we're gonna get photosynthesis." It obvi-, I mean, no one could predict this at the beginning. It makes absolutely no sense at all. It's an orthogonal issue. It's sort of like a side effect. The point is that the things that we care about are actually the side effects of the constraint. It's not directly a consequence of the constraint.

• 1:39:43Speaker 00

  But we have to understand it as like the side effect is actually the main event. Like, we care about the side effect. And so, what, that, there's a really important... So when we look at it in that light, like, we change the metaphor. You know, and we think of it as evolution is actually an open-ended divergent process without a final objective. Then, we, we can see that, like, there are huge consequences for, again, things like representation. I think this is a very important point. Like, because, like, it gives us hope for this idea that there might be an algorithm, which isn't an evolutionary algorithm, which is more like, like what deep neural networks do, but some kind of, like, learning algorithm in large neural networks that's more analogous to evolution

• 1:40:25Speaker 00

  that then would get these unified, yeah, these unified factored representations. Because evolution has done that. You know, so I claim that the underlying representations in DNA are incredible in a similar way to the skull in Pik Breeder. And the reason is because what you see when you have offspring is that fundament, that you get changes clearly, like, you know, your, your child is not like a clone of you. So you get changes. But the dimensions of variation that are searched almost always preserve the underlying most important regularities. Like, in other words, like, humans always have bilaterally symmetric children.

• 1:41:06Speaker 00

  The problem is that selection is captured by a genetic algorithm. It does do the selection part. But what it doesn't capture is this divergent aspect. Because genetic algorithms are traditionally optimization algorithms, which are trying to use selection to converge to a point, which is what optimization algorithms do. But it's a huge digression from what actually natural evolution does, because it doesn't converge to a point, and it's not trying to. It's just diverging, subject to the constraints of survival. So we can think of it as a constraint. And then the things it actually finds, which would have been objectives if this was a genetic algorithm, like flight or photosynthesis, amazing achievement.

• 1:41:50Speaker 00

  Are just side effects. They're not actually the goal. They just happen to happen as a side effect of having this constraint of survival. It's a very different algorithm from an optimization algorithm in that sense. And the thing that's interesting about this is that it then expresses an algorithm, which is much more like PicBreeder in the way that the PicBreeder users are actually searching the space of images in PicBreeder. So PicBreeder, in effect, is a much better metaphor for evolution in nature than a genetic algorithm. And that's important because we also notice, I believe, that representation in nature

• 1:42:30Speaker 00

  is similarly unified, factored, the way that we see it in PicBreeder. In other words, amazing. And the evidence in nature is more indirect, because I can't just look under the hood and see the underlying representations. But there's evidence of how amazing the representation is from the kinds of variations that you see from one generation to the next. And so, for example, the fact that a human being always has a bilaterally symmetric child. I mean, if you think about it, this is really remarkable. Like, it seems like such an obvious thing, like, it's not even that interesting. But it's actually really remarkable from a genetic algorithm perspective, because preservation of regularities like that is not at all a foregone conclusion.

• 1:43:13Speaker 00

  Like, in a regular genetic algorithm, you're messing things up on every single mutation. You don't get this really high probability of preserving some deep underlying regularity. Like, that would be incredible if you were having such a fortuitous set of offspring, that they all preserve some really important aspect. But what's happened in the representation in nature, the way DNA has become organized, the way that the genetic regulatory network works, you know, as a hierarchy, is that it's been, and biologists use the word "canalized" or "canalization", It's been cannibalized, it's like it dug a trench into a mountainside and created a canal. So that if there is a change, like a mutation or an earthquake in the analogy, you still get the water to run down through the same canal.

• 1:43:56Speaker 00

  And that's the developmental pathway, like with the bilateral symmetry and the other regularities. Like you have arms, you have legs. It's true, obviously, there's a miscarriage rate. But it's incredibly low compared to like the like general destruction of the phenotype that you see in a traditional genetic algorithm. And there are things you never see, like you never see trilaterally symmetric offspring and things that could happen, like if you were using like an indirect encoding or something like an L system in a traditional genetic algorithm, like which can express body plans. Like you can get, it's very easy to go from bilateral to trilateral or five fingers to 10 fingers, like in a single jump, like it never happens in biology.

• 1:44:37Speaker 00

  So biology has learned underlying regularities, just like the underlying pick breed or genome for the skull. There's a strong analogy, and why does that matter? It matters because it shows as a proof of concept that there are forces in nature that are not guided by humans, that similarly, because of the divergent aspect of the search, the serendipitous aspect, open-ended aspect of the search, similarly, similarly also do end up with representations that arguably are approaching what we're calling a unified factored representation in the paper. And so it gives hope that there may be algorithmic interventions that are possible, independent of human users, like making selections themselves, that could do something

• 1:45:20Speaker 00

  similar to representation, because we also see in nature, in a sense, in an automated environment, but not an objectively driven one in the traditional sense.

• 1:45:31Speaker 03

  A lot of people in ML think that we perfectly understand evolution and that it's just like, you know, a genetic algorithm. I think Ken would agree with the statement that if we really were to understand it, then we should have, then there should be, exist like an algorithm that we can scale up right now, that can recreate all the glamour of evolution, right? All the flights, photosynthesis. There's not a single algorithm that can, that we can scale up, that we can definitively say that will like do what evolution did in the current state of genetic algorithms in ML. Would you agree with that, Ken?

• 1:46:03Speaker 00

  Yeah, yeah, I mean.

• 1:46:04Speaker 03

  So. So if that's true, that means that we don't really fully understand and we understand aspects of it, like Ken's amazing work on like open-ended serendipity, but we don't have a full picture of evolution. And that's why genetic algorithms are nowhere close to a full picture, and we don't really understand everything that evolution is doing.

• 1:46:23Speaker 00

  Agreed, yeah.

• 1:46:24Speaker 02

  Yeah.

• 1:46:26Speaker 03

  Just for that bell audience out there, who's claimed that we understand evolution fully.

• 1:46:31Speaker 02

  And I wanted to kind of jump in, you know, along this line of not understanding evolution completely. I want to put in another dimension, which is, which is very well-known in just almost anything in the natural world, but certainly with life, which is energy constraints. You know, there are these very heavy constraints on resources, on energy, on material, et cetera, that life has to endure in order to survive. And so again, I'm going to beat on my parsimony slash simplicity sort of drum here and just get your thoughts on that, which is that it isn't just open-endedness, it's not just divergence. I think there are other aspects to natural evolution, one of these being the constraint to use less resources, less energy, simplicity of solution. Like, do you think that's fair? Like, is that another

• 1:47:16Speaker 02

  important dimension?

• 1:47:19Speaker 00

  I think it's, it's, you know, part of the survival constraint. So I think you could think of it as, as part of the function of, that computes survival, whatever survival is. Like, to the extent that efficiency is important for survival, then you're going to stay in regions that are relatively efficient. And so that, that is having some effect on representation. But, you know, similarly, like in pick breeder, you know, there's less of this, perhaps, concern for efficiency that's at least not as explicit. But it could be implicit in the sense that, that, you know, users perceive evolvability and like

• 1:48:01Speaker 00

  it. So like, users get a sense that, oh, this thing is really malleable. And so I'm going to keep playing and exploring in this area. And maybe that's a side effect of being efficient in some sense, that it's, that it's so evolvable. But it's an, it's an indirect correlation. I mean, it's, it's not like exactly explicit that necessarily efficiency equals evolvability. It could be, though, something implicit and indirect. And so I think, I think overall, it's, it's like Akar said, there's still some unknowns here about what actually, what actually matters, what doesn't matter. Especially with respect to the representation, you know, because like we're having kind of two conversations at once. One is like, what is the best metaphor for biological evolution? But then it's actually not the key question.

• 1:48:45Speaker 00

  The key question is just what leads to these really nice representations. It might not actually have to be a perfect metaphor. So it's a little bit confusing to disentangle these things.

• 1:48:55Speaker 02

  Well, that's why I was kind of suggesting that because there's other lines of evidence and a lot of discussion and just the, let's say, evolution, you know, the literature on evolution itself, on evolution as a general meta-algorithm, not just biological evolution. That, you know, in order to achieve the degree of variety and diversity that you actually get in life, there has to be an element of shock and challenge and sort of, it can't just be lots of resources around. Because then the earth would have just been, like if the earth had just started off as a massive ball of, you know, consumable cheese or something like that, it would have just been covered by gray goo that never had any incentive to actually diversify,

• 1:49:39Speaker 02

  crystallize into different solutions and various forms.

• 1:49:43Speaker 00

  Well, you know, I'm not as sure because the, like, you know, that, like, you don't want too much pressure. I mean, this is get, maybe this might be going into the weeds, but like, you don't want.

• 1:49:53Speaker 02

  Well, yeah, it's a Goldilocks.

• 1:49:54Speaker 00

  Yeah, yeah.

• 1:49:55Speaker 02

  It's a Goldilocks zone.

• 1:49:56Speaker 00

  So it's like when, when you, if, yeah, if there's too much constraint, then you can't do divergence. Like, you have to be safe, you have to play it safe.

• 1:50:03Speaker 02

  Right.

• 1:50:03Speaker 00

  So you want to have some opportunity to be opportunistic. But yeah, if everything survives, then you go to the other extreme. And then uninteresting things start surviving. Like, we've said in some of our publications, you know, about like open-ended evolution, that like we think that, we call it the minimal criterion, like the minimal thing you need to do to be able to pass on to the next generation, that it has to be non-trivial. It's like that we think is important. Like, once it's trivial, like if it's just like, if you hit a minimal mass, then you get to have an offspring. This is a thought experiment, it's impossible, but like if somehow like God would intervene and give you a child just because you hit some mass.

• 1:50:43Speaker 00

  It might just create a situation where, you know, that the world, yeah, would fill up with just like inert blobs. Because all they need to do is get bigger. And so you do want something non-trivial in these constraints enter into the non-triviality of the minimal criterion. So it's a complex, it's a complex question about how to set something up like this. And then now we're like talking about transferring these insights into the way you train a neural network. That like, because there's another leap then of sort of like cognitive complexity for us to think through. It's like, what is the analogous thing like in just a training session where you're being exposed to data?

• 1:51:27Speaker 03

  I like that Goldilocks analogy a lot for evolution because I think Joel and Risto had a paper where it's like you need like some sort of catastrophic events to happen in order to get adaptable solutions. Extinction events, like near extinction events. So that's like one extreme, but on the other way, like Ken said, you can't have global competition just everywhere. Or else you just end up doing like local hill climbing, which is what genetic algorithm does. You need some sort of local competition and not global competition.

• 1:51:58Speaker 02

  I have to thank you, I have to thank you actually Kenneth, because you know, when we talked four years ago, up until that point, I mean, ironically, I was, I was trained a lot in biology and biomedical engineering. And but up until that point, I really hadn't thought that much about evolution and you know, and since, since the conversation with you, I've been thinking about it more and more and more and reading more and more. And just becoming increasingly fascinated at how powerful and amazing this, this algorithm of evolution is, you know?

• 1:52:29Speaker 00

  Yeah, I appreciate that. That's great to hear. Yeah, evolution is, is just wildly powerful and amazing. I mean, yeah, it's like to think it's a GA is so underrating it and so missing what it is. It's a real shame, like in our field, that in AI especially, you get people making this analogy a lot. I see it on social media. And it's like GAs have done a disservice to our field's understanding of how profound this process is. I mean, it created all of living nature and it begs an explanation, like it's biblical, literally.

• 1:53:07Speaker 02

  Yeah, and I mean, and also, I think there's a vast underappreciation of the fact that evolution occurs in many scales and many systems, like it occurs in psychology, it occurs in memes, it occurs in the information space, like it's a very general, almost meta-algorithm, right? And I would encourage people to look into whatever domain they're in, whether there's any kind of evolutionary perspective on it, because it can be very fascinating. Like, a lot of human behaviors that people dismiss as irrational are actually not irrational if you consider it from the perspective of an algorithm operating on a ecosystem of

• 1:53:48Speaker 02

  species, or individuals, or things like that. So it's just fascinating.

• 1:53:53Speaker 00

  Yeah, and the pickbreeder is like that, you know? I mean, pickbreeder has a lot of evolutionary properties, and here it is creating a neural representation that has these fascinating properties internally, and that can't be dismissed. I mean, the evolutionary properties of what the human beings are doing in pickbreeder is obviously part of the explanation. But it's not that we want to have to depend on humans guiding the search process, it's whatever they're doing that's somehow analogous to what evolution did, which allows it then to discover these incredible internal representations. We want to abstract that. So we want to find, like, yeah, the correct level of abstraction to allow this to happen in an autonomous fashion,

• 1:54:37Speaker 00

  so that the training regimens of these large models leads to a similarly amazing underlying representation. And I think that the question that looms is, like, how far is the representation currently from this ideal? Because it's not totally known. Like, we show in the paper a number of pieces of evidence. Indirect evidence that there is fracture inside of fracture and entanglement inside of today's large models. We go through literature, we also show new examples. But you know, the word, reason I use the word indirect is because unlike these pic reader images, we can't just go in and look at a neuron and know what it does explicitly. Because that's what's so nice about pic reader images, because they're 2D.

• 1:55:20Speaker 00

  You can just look and see exactly what's represented. It's more implicit in like a real giant multi, you know, multi-billion connection network. And so the easiest thing to do is just look at indirect evidence of fracture. But there's this overriding lesson here that even when you're getting answers right, you could be messed up under the hood, which then, you know, further confounds our ability to analyze what these models really know. Because like, even if they got everything right, they could be just like the network that outputs the skull, which is basically almost perfect, or it is perfect. And so it could be perfect and every question we ask is perfect. And it could still be an absolute mess underneath the hood, which

• 1:56:05Speaker 00

  means that this indirect evidence, you know, could diminish over time, like as the models get so-called better. But it might not mean that they're fixing the underlying representation to be less fractured and entangled.

• 1:56:16Speaker 02

  But I think it's super important that you name this concept impostor intelligence in the paper. So I think it's important that we that we point that out, like the idea that if, sure, if you have enough fractured, entangled representations, you can perfectly reproduce an output, you know, that looks identical to the one that you get from unified, oh shoot, what are they called again?

• 1:56:40Speaker 00

  Factor.

• 1:56:41Speaker 02

  Unified, factored representations, right? Can look identical on the surface. The output can can look identical. And that's impostor intelligence, correct?

• 1:56:50Speaker 00

  Yeah, yeah, it just seemed like a like a good, another good metaphor is to think of it as an impostor. It's like this, you know, the the representation of the skull. Is just somehow a farce. Like, it's not what you expect. Like, if you just look at the output, it's great. It looks exactly like a skull. But underneath the hood, it's not capturing any of the underlying components or the regularities. And so, in some sense, it's not really a skull. It's an imposter underneath the hood. It's very misleading. And so, if you think of the skull, again, as a metaphor for all of human knowledge, because that's what an LLM is trying to capture. It's not just a single image. It's like an image of all of human knowledge.

• 1:57:35Speaker 00

  For any input, it should output something that's convincingly human. Then, like, it could be just similarly an imposter. It could be the same. Underneath the hood, everything could be organized wrong. Not the way you expect. It's like a giant charade. And again, this is very confusing and counterintuitive, like, for people, because, like, you know, people are naturally, including even me, I would react like, but should I really care? Like, when you say it's an imposter, but it's getting everything right in its human level? Like, what are you objecting to? But the point is that it can still be an imposter, because it's like, what we care about here is not just that it's going to get answers right, like, get good test scores, like, seem to be plausibly human when you talk about things that are in distribution. We want it to be able to go outside.

• 1:58:20Speaker 00

  Like, to do things that are creative. To be able to continue to learn, like, to get to the next level, including learn on its own and get to the next level. I mean, these are like the next frontiers for the field. If it's an imposter underneath the hood, then these kinds of things are going to hit a wall or become insanely expensive. It could be that you can always push through that wall, but the expenses just go up and up like crazy. Exponential, worse, I don't know what it means, but it could be something terrible. We might already be seeing that. Like, you know, the amount of money that we're spending here raises questions, like, is it necessary? Like, does it have to cost this much in energy and in money? And so, being an imposter is not a good thing,

• 1:59:04Speaker 00

  even though the results look good. I mean, that's what it means to be an imposter. And we would like to know if that's the case. We don't have direct hard evidence yet. What we have is some evidence from these pic breeder images that suggests there's a concern here. But we don't know how bad it is. And so, it's something that's, you know, good behooves the field to actually go off and figure this out.

• 1:59:26Speaker 03

  Speaking on the evidence part of it, this goes into mechentrip research. People have been trying to dissect like how these models think and operate for the longest time now, right? And it's like notoriously hard because they look into these like models and you have like stuff like polysemiticity where it's like one neuron is like activating via like adding circuits and also like black holes or something random. And it's like, it's like, it's so, that's why it's so notorious. I think this is one of the fractured and tangled representations are one of the reasons why mechentrip is so hard is because how are you supposed to even interpret something that messy, right? It's like, it's just a hard problem. And this might, I mean, I guess it begs the question, is neuroscience isn't easy either,

• 2:00:11Speaker 03

  right? If you just try to look into the human's brain, it's not like it's like smooth sailing when you look at a human's brain either. But I guess the question is, is there like a better organization for neural network, which is also interpretable? Maybe it's not, I think we say in the paper, it's like an aspirational ideal. Maybe it's not possible, but it's a good like direction to go towards, right? At least more going more unified and factored.

• 2:00:36Tim Scarfe?

  One thing that is lingering in my mind a little bit, and we talk about this a lot in many different ways on MLST, is in the real physical world, we seem to have this emergent phenomenon of intelligence and even evolution, actually, I mean, you know, like we could all agree as materialists that you have these physical rules and you have these particle interactions and you have these dynamics that emerge and that's evolution. And it's beyond our cognitive horizon. We simply can't conceive of it. So what we do is we create these abstractions and we think about it with algorithms and these are idealizations or abstractions and it's just, it's too, it's unimaginably complicated. But there's always this question of

• 2:01:18Tim Scarfe?

  Does it even make sense to think that we could, in a disconnected electronic way, recreate something which had many of the properties of the thing in the real world? Even with Picbreeder, for example, I think we'd been couching it as an algorithm. But actually, it is also a constraint. The magic comes from humans, and humans are still embedded in the physical world to a high enough fidelity that, you know, we see the phenomenon that we're interested in. But even with Picbreeder, does it really have the evolvability that you're talking about? Like, if humans stopped using it, wouldn't it just mode collapse? So there's always this question of, don't we just need the world, right, to actually give us the type of intelligence that we want?

• 2:02:03Speaker 00

  I mean, I think that it could be that, like, a process that yields representations as good as presumably what humans have at their best would require some interaction with the real world. But I don't know if that's really, like, that prohibitive. Like, I mean, we are getting to the point where, effectively, computers are directly interacting with the world. Like, I think, you know, it's interesting, like, in the field of AI, that if you go back 10 years or so, like, most of the interactions of AIs with dynamic training environments would be in simulations. Like, it's not the real world directly, the environment is an abstraction of the real

• 2:02:47Speaker 00

  world too. But we're now, like, getting to a point where these models are actually directly interacting with real-world data. And so I think it's conceivable that they, like Picbreeder users, could simply explore the world itself. I mean, of course, they're not, I mean, unless they're robots, they're not literally out there exploring. But the internet is a proxy for the world. And so they could explore the internet, they could gain knowledge in some more natural way, which is more like a Picbreeder user. And, you know, the important part of it is that allows them to discover an order or a chronology on their own. The way we do, to a large extent, you know what I mean?

• 2:03:30Speaker 00

  We develop curricula for students, obviously that's what school is. But it doesn't account for the first three or four years of life, I mean, which are obviously very formatively important. And then there's lots of life outside school, too. You know, and so like if you take a person who like invented mathematics independently of going to school, presumably that person is going to be a better mathematician than somebody who just read the textbooks and memorized it and gets the same exact score on the test. So it's like, what led you to knowing this is going to affect what you're going to do in the future? These are things that involve real world interaction. And so I think it's conceivable that now with internet access, you could imagine a world where there's more control on the model side in terms of how it goes off and experiences

• 2:04:15Speaker 00

  the world. It's not, it's not easy, it's not like, it's not as easy as, oh, well, we'll just dump in all the world's data, like, and just like train. Like, obviously, that's what we do, and it's because it's easier to think of, it's harder to conceive of what I'm saying, but I don't see it as impossible, and it could end up a lot cheaper, you know? Because like, I mean, one of the, one of the offshoots of that could be that it's multiple orders of magnitude more efficient to happen in a natural way. And so while we might think it sounds like a tough haul, it might actually be easier in some sense, because it is so much more efficient.

• 2:04:51Tim Scarfe?

  To that point, then, do you think that the, the structure of organization of the internet, or even language is a great example, it's incredibly diverse and different cultures have different words for different concepts and so on? You know, we spoke about this before, Kenneth, but one school of thought is that there are kind of natural categories. And give it, you know, if you, if you could press play on evolution thousands of times over that you would see this, just as we have morphological convergence in evolution, there would be a kind of informational convergence. Do you, do you subscribe to that?

• 2:05:25Speaker 00

  You mean like if, if more than one person explored through the world that they would converge to similar categorizations? That's kind of the idea?

• 2:05:34Tim Scarfe?

  Well, I suppose like if we if we had a thousand parallel universes and we just watched the evolution of language and the internet and all of those different universes, would we see a kind of convergence towards natural categories which are just representative of the world or, you know, like the other school of thought is it's very constructive and it's remarkably disconnected from the physical world. But I feel that even though there are more degrees of freedom in the virtual world, it's very difficult to imagine how this works because of the complexity of emergence and whatnot. But there seems to be some kind of grounded.

• 2:06:09Speaker 00

  So I think that the closer you are to the origin of an open-ended process, the easier it is to predict what might happen. You know, so it's like from the very beginning of evolution, whatever that means, like the first cell or something like that, you might be able to say like if we reran this this film again, then, you know, there'll be something like photosynthesis is likely. Like there's some early things that we might get, algae, something like that. But like the farther you go, I think the less likely it is you're going to get the same thing again. So like almost certainly not humans again. I mean, you might get something as intelligent, but not not what we are. It'd be something different in some different part of the space. And so like sometimes, you know, it's like we've actually in my old research group, we had a term for this.

• 2:06:53Speaker 00

  It's not a known term. It's just like it was an eternal term, but we called it the cone of inevitability. It's like the more closer you are to the origin, the more inevitable things are in an open-ended process. And as you go outward, the cone expands and there's more possibilities. And then it becomes less and less inevitable as you go up. And so this would be true, starting at any point in an open-ended process. It's like the invention of the internet. What do you think will happen like in the next couple of years? Probably it's somewhat predictable and similar to what did happen. But then you say, well, what would happen in 10 years, 20 years, 100 years? I think it becomes less and less predictable over time, and it will diverge in different ways. And that's a general principle, I think, for open-ended processes.

• 2:07:34Speaker 03

  Really quickly, we know that like eyes evolved multiple times in animal species, right? So there are some stuff where it's like just so fundamental to like how the world works, where it's like it kind of is inevitable how it comes about, like eyes. But I guess I would still argue that there's so much other stuff that's just, I guess it seems unpredictable. I guess I can't say for sure, but. I mean, good luck predicting like peacocks would exist, right? Because that seems like it's just like, like their beautiful feathers are just like a one-off in evolution. That's like, yeah.

• 2:08:05Speaker 00

  There's also, you know, one thing we said in our, in our, our book is we, we talked about the fact that you can think of searching for novelty as a form of information accumulation. And so, you know, so this applies to any kind of divergent search, like something that continually finds new stuff. And so like to find something new, you have to exploit some property of the universe that hasn't yet been exploited. Like that, that seems to be a principle. And so, you know, it's like eventually to, humans keep doing new stuff, but eventually we have to get off the planet to actually do something new, like we exploit what's here. So we have to go somewhere new. And so like as you, as you wait a long period of time in something like evolution,

• 2:08:48Speaker 00

  you're getting a kind of encyclopedia of everything that's possible, given the physical structure of the universe. So like you have eyes because there's photons, you know, and you have ears because there's sound waves. And so it's like, it's almost like this is like our, our structure is a documentation of all the degrees of freedom that exist in the universe as is constructed. And, and it's, it's, it's almost like an encyclopedia in accumulating information. And so with that philosophical viewpoint, you might say that, you know, there is some kind of predestination to it like that. Like, well, I mean, we're gonna, we're gonna see over time, as long as it's a divergent process, some exposure of everything that's possible, like some kind of, but it's not necessarily the, the same configuration.

• 2:09:30Speaker 00

  Like there's probably many configurations that can express these different degrees of freedom in, in functional ways. But to some extent, those are the fundamental things that exist and can exist. And so we're going to expose them if you wait long enough. So there, there's, there's some degree of predestination there.

• 2:09:49Speaker 03

  And even if you do, this reminds me of library learning a lot. But another thing I want to say is, I mean, everyone's been, everyone in physics always talks about emergence, right? You have like these simple laws, like underneath the kaleidoscope, it's like just super simple programs and stuff. That doesn't mean that the compositions and like the higher level emergent phenomena is going to be predictable, right? Even if the simple rules are there. So in physics, I mean, the standard model of physics, it's like relatively simple, right? I don't understand it, but it should be like theoretically super simple. But the fact that biology emerges from that and that chemistry, biology, sociology, all emerges from that is, I think, it's unpredictable.

• 2:10:30Speaker 03

  So the fact that it's like not reducible in like intuitive sense means that just because there is like underlying simplicity to it, doesn't mean that it's always like exploited, exploitable.

• 2:10:42Speaker 02

  You know, maybe if, if we think about the current state of machine learning, you know, attempts to develop AI, right? I would like each of you to both, if you would, give a brief, brief recommendations for how you'd like to see, you know, the research paths change, like what would you change about current research paths? If you wanted to, to push us more rapidly towards AGI, assuming we want to get to AGI, you know, let's, let's, let's assume maybe that's, that's goal. So like Kenneth, what, starting with you, like what would be your, your recommendation to the, the field?

• 2:11:18Speaker 00

  Um, well, we, I think, you know, consistent with this paper, uh, we need to understand underlying representations better. Um, and uh, figure out the degree to which um, they actually are imposter representations that don't really capture the things that we expect them to. Um, and then we need to see what the mitigations are. Um, and, and there's a lot of possible mitigations and we don't really know what their implications are. So we need to do a lot of exploring. And then even beyond the paper and that, I think, um, you know, it's, it's really important to look at the question of creativity and open-endedness, um, and to take it really seriously.

• 2:12:03Speaker 00

  Um, like what I find really interesting in the field is the current focus, or at least up to very recently, has been on chains of reasoning, um, reinforcement learning. Improving the ability to reason so that you can solve specific problems. And just want to point out that doing that is useful. There's no doubt about it's useful and impressive. But it's antithetical to creativity. I mean, once I say that what you need to be good at is if I define where I want you to go and then you can get there, then I'm basically training you not to be able to be smart if you don't know where you're going. Like, but that's what creativity is.

• 2:12:47Speaker 00

  It's about being able to get somewhere and be intelligent even though you don't know where your destination is. And people just struggle with this notion. I mean, I think that's why it's not on the radar very much. It's like people think often like if I say that, um, let me just. People will just say, uh, you know, that's if you don't know where you're going, then you can't be intelligent. Like you have to be random then. What kind of intelligence is that? But I totally disagree. I mean, that's what the most interesting people are doing is that they are saying, actually, based on my life experience, if I take this path, I have a gut instinct and I can justify that gut instinct.

• 2:13:32Speaker 00

  It's not completely implicit, like some of it I can explicitly explain. But what I have a gut instinct about is that it will open up new possibilities. It's not that I know what they are yet, but it opens up a new playground and I understand this. And by training things to just solve directly a problem, we're training it not to be good at that, which is really funny. Because that's like the most interesting aspect of human intelligence. And it's really, really underrated that that's like what we're good at. You know, we're good at saying something is interesting. You know, to the point where people think it's almost preposterous that you could even think in a situation like, like, I mean, if you put a kid in a playground, that's what they're doing. They don't have a goal for the day.

• 2:14:13Speaker 00

  It's like, and we believe that this is not part of human intelligence. And so how are we going to train it to have a chain of thought when it doesn't know where it's going that's still intelligent? A completely different thing.

• 2:14:27Speaker 02

  That's all, that's the great irony of, of, of your work and like the title of your, your book, right? Greatness cannot be planned. So I mean, I think that's the, that's the core, the core irony.

• 2:14:37Tim Scarfe?

  Yeah.

• 2:14:38Speaker 02

  So Akash, like what would you, um, what would you recommend to the field? What should we all be doing differently to, uh, progress more?

• 2:14:46Speaker 03

  I think, uh, the one of the high level things we should be doing is not putting all our eggs in one basket, right? That's like the main point of the open ended this lesson is obviously there should be people scaling up these L L M's to see how far the current paradigm can get us because I'm also curious like to see how far it takes us, right?

• 2:15:02Speaker 02

  So you're applying, you're applying your principles to the, the field itself.

• 2:15:06Speaker 03

  Yeah, I mean, like you can't, I, I asked Risto this whenever I first like got into AI, uh, Risto Micalin. And when I got into AI research and I was like, what's your path to AGI? Is it evolution or is it like the current deep learning? And he was like, I don't know, and I don't, I don't want to give you a recommendation because the point is that none of us are, we're supposed to all follow what we think is the coolest thing and not just like decide like, oh, we need to put a hundred percent of our resources into this one thing. So in that light, I think that there needs to be, I think there are more people should look into, you know, artificial life, pick breeder and the ideas from our paper and, uh, see, because I think it's a very promising direction. And one thing that's like, uh, I'm really curious about, even if like the current paradigm is like very cool, you really like it.

• 2:15:49Speaker 03

  It's very different than how humans operate and how evolution operates. It's very different. No one will deny that, right? No one, we don't just train on I I D batches of data for like a million steps, right? So no one will deny that. And I think for academia, we really should try to understand and, um, the evolution and artificial lifestyle of things a lot more. So I think a lot more investment should go there.

• 2:16:12Speaker 02

  Great.

• 2:16:13Tim Scarfe?

  Awesome. Well, um, gentlemen, it's been an absolute honor having you on the show. Thank you so much for joining us today.

• 2:16:19Speaker 02

  An awesome paper.

• 2:16:19Speaker 03

  Thank you.

• 2:16:20Tim Scarfe?

  Thank you. Thanks to both of you.