The Mathematical Foundations of Intelligence [Professor Yi Ma]

What if everything we think we know about AI understanding is wrong? Is compression the key to intelligence? Or is there something more—a leap from memorization to true abstraction? In this fascinating conversation, we sit down with **Professor Yi Ma**—world-renowned expert in deep learning, IEEE/ACM Fellow, and author of the groundbreaking new book *Learning Deep Representations of Data Distributions*. Professor Ma challenges our assumptions about what large language models actually do, reveals why 3D reconstruction isn't the same as understanding, and presents a unified mathematical theory of intelligence built on just two principles: **parsimony** and **self-consistency**. **SPONSOR MESSAGES START** — Prolific - Quality data. From real people. For faster breakthroughs. https://www.prolific.com/?utm_source=mlst — cyber•Fund https://cyber.fund/?utm_source=mlst is a founder-led investment firm accelerating the cybernetic economy Hiring a SF VC Principal: https://talent.cyber.fund/companies/cyber-fund-2/jobs/57674170-ai-investment-principal#content?utm_source=mlst Submit investment deck: https://cyber.fund/contact?utm_source=mlst — **END** Key Insights: **LLMs Don't Understand—They Memorize** Language models process text (*already* compressed human knowledge) using the same mechanism we use to learn from raw data. **The Illusion of 3D Vision** Sora and NeRFs etc that can reconstruct 3D scenes still fail miserably at basic spatial reasoning **"All Roads Lead to Rome"** Why adding noise is *necessary* for discovering structure. **Why Gradient Descent Actually Works** Natural optimization landscapes are surprisingly smooth—a "blessing of dimensionality" **Transformers from First Principles** Transformer architectures can be mathematically derived from compression principles — INTERACTIVE AI TRANSCRIPT PLAYER w/REFS (ReScript): https://app.rescript.info/public/share/Z-dMPiUhXaeMEcdeU6Bz84GOVsvdcfxU_8Ptu6CTKMQ About Professor Yi Ma Yi Ma is the inaugural director of the School of Computing and Data Science at Hong Kong University and a visiting professor at UC Berkeley. https://people.eecs.berkeley.edu/~yima/ https://scholar.google.com/citations?user=XqLiBQMAAAAJ&hl=en https://x.com/YiMaTweets **Slides from this conversation:** https://www.dropbox.com/scl/fi/sbhbyievw7idup8j06mlr/slides.pdf?rlkey=7ptovemezo8bj8tkhfi393fh9&dl=0 **Related Talks by Professor Ma:** - Pursuing the Nature of Intelligence (ICLR): https://www.youtube.com/watch?v=LT-F0xSNSjo - Earlier talk at Berkeley: https://www.youtube.com/watch?v=TihaCUjyRLM TIMESTAMPS: 00:00:00 Introduction 00:02:08 The First Principles Book & Research Vision 00:05:21 Two Pillars: Parsimony & Consistency 00:09:50 Evolution vs. Learning: The Compression Mechanism 00:14:36 LLMs: Memorization Masquerading as Understanding 00:19:55 The Leap to Abstraction: Empirical vs. Scientific 00:27:30 Platonism, Deduction & The ARC Challenge 00:35:57 Specialization & The Cybernetic Legacy 00:41:23 Deriving Maximum Rate Reduction 00:48:21 The Illusion of 3D Understanding: Sora & NeRF 00:54:26 All Roads Lead to Rome: The Role of Noise 00:59:56 All Roads Lead to Rome: The Role of Noise 01:00:14 Benign Non-Convexity: Why Optimization Works 01:06:35 Double Descent & The Myth of Overfitting 01:14:26 Self-Consistency: Closed-Loop Learning 01:21:03 Deriving Transformers from First Principles 01:30:11 Verification & The Kevin Murphy Question 01:34:11 CRATE vs. ViT: White-Box AI & Conclusion REFERENCES: Book: [00:03:04] Learning Deep Representations of Data Distributions https://ma-lab-berkeley.github.io/deep-representation-learning-book/ [00:18:38] A Brief History of Intelligence https://www.amazon.co.uk/BRIEF-HISTORY-INTELLIGEN-HB-Evolution/dp/0008560099 [00:38:14] Cybernetics https://mitpress.mit.edu/9780262730099/cybernetics/ Book (Yi Ma): [00:03:14] 3-D Vision book https://link.springer.com/book/10.1007/978-0-387-21779-6 refs on ReScript link/YT

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  In the past ten years, I think the question about the intelligence or artificial intelligence has captured people's imagination. I'm one of them, but it took me about ten years to try to really understand, can we actually make understanding intelligence a truly scientific or mathematical problem? To formalize it, you probably will get some of my opinion and also the facts about it. And probably will change your view what is intelligency is, which is also very searching process for me. How do we clarify some misunderstandings, the common misunderstandings about the intelligence? Through this journey, maybe we'll gain an entirely new view about what we really have done, right?

• 0:45Speaker 00

  In the past ten years, the practice of artificial intelligence, the mechanism we have implemented, the all the because behind all the large models, deep networks, and the large models are truly are. And the true natures and hence understand their limitations. And also what it takes to truly build a system that has intelligent behaviors or capabilities. I think we have reached the point, right? We'll be able to address what's next for understanding even more advanced form of intelligence. What's the difference between compression and abstraction? Difference between memorization and understanding?

• 1:27Speaker 00

  I think for future, those are the big open problems for all of us to study.

• 1:34Tim Scarfe?

  MLST is supported by Cyberfund. Link in the description.

• 1:39Speaker 02

  The idea of having to traffic in squishy people in order to make our systems go is not immediately appealing, let's put it that way.

• 1:48Tim Scarfe?

  This episode is sponsored by prolific.

• 1:51Speaker 03

  Let's get few quality examples in. Let's get the right humans in to get the right quality of human feedback in. So we're trying to make human data or human feedback, we treat it as an infrastructure problem. We try to make it accessible, we make it cheaper. We effectively democratize access to this data.

• 2:08Tim Scarfe?

  Professor Ma, it's amazing to have you on MLST. Welcome.

• 2:12Speaker 00

  Thank you. For having me.

• 2:14Tim Scarfe?

  So normally, I ask guests to introduce themselves, but given your stature in the field, I think it's best that I give you an introduction. So Yimao is a world-leading expert in deep learning and artificial intelligence. He's the inaugural director of the School of Computing and Data Science at Hong Kong University, and director of the Institute of Data Science at the University of Hong Kong. He's also a visiting professor at UC Berkeley, where he previously served as a full professor in electrical engineering and computer science. He's an IEEE Fellow, ACM Fellow, and CM Fellow, whose pioneering work on sparse representation and low-rank structures has fundamentally shaped modern computer vision and machine learning.

• 3:00Tim Scarfe?

  His recently published book, Learning Deep Representations of Data Distributions, proposes a mathematical theory of intelligence built on two principles: parsimony and self-consistency. This framework has led to white box transformers, known as crate architectures, where every component can be derived from first principles rather than empirical guesswork. So Professor Ma, tell me about your book.

• 3:26Speaker 00

  You know, about, you know, eight or seven years ago, you know, deep networks, deep learning has been pretty, you know, changed the practice of machine learning or artificial intelligence in the past decade. About a years ago, I had a chance to get back to Berkeley and gave me a chance to look into these topics more deeply, try to understand it from a more principled approach. And hence, the book is kind of summarizing the kind of progress we've made in the past, you know, eight or beyond years, myself, my group, as well as many colleagues,

• 4:07Speaker 00

  trying to understand the principles behind the deep networks, explain it from a first principle. And in that journey, we also seems to embark on a little bit beyond that. Find something probably more general behind that is the intelligence, at a certain level of intelligence. And hence, when I get back to join Hong Kong U about two years ago, had a chance to design or redesign some of the curriculum to reflect those, some of the progress updates in our field, the rapid progress in our field.

• 4:48Speaker 00

  So my students and colleagues decided that maybe it's time to systematically organize this body of knowledge and reflect as a textbook, as well as a new course, which I'm teaching this semester. And likely will be offered as well at Berkeley next semester. So this is actually a first time probably we try to provide a more principal approach to explain the deep networks, as well as some principle of intelligence.

• 5:21Tim Scarfe?

  And these principles are parsimony and self-consistency. So it's an ambitious idea that these principles could explain natural and artificial intelligence. What do you mean by that?

• 5:33Speaker 00

  Intelligence, artificial or natural, or whatever adjectives you add to intelligence, we have to be very specific. It's a very loaded world, right? I mean, even intelligence itself may have different levels, different stages, right? So it's high time we clarify that concept scientifically and mathematically, right? Then we, so that we'll be able to talk about, study intelligence, the mechanism behind it at each level. There are some more unified principle behind even different stages of intelligence. There's, there's something in common, there are also things are different.

• 6:17Speaker 00

  So it's high time we, we, we do that. One of the intelligence at the level is that common to animals and human, right? Human, we are animals. That level of intelligence is what we think are very common to all life.

• 6:34Speaker 00

  Which is how memory, how we learn knowledge about the external world. And then memorize as part of our memory. And use that to predict, to react to the world, help us make decisions, predict and make better decisions with for survival and so on, of course. That's very, very common. And this is the level of intelligence we're talking about, very much for the book as well, right? And hence, for this level of intelligence, how our memory works. Today, we also have a fancy word for memory, we call it a world model, right? And it's how this, how we develop such a memory, such a world model, and how the model gets evolved and how we use it. That's actually, this is the level of we talk.

• 7:18Speaker 00

  So, we actually believe that for this level of intelligence, for how our memory formed and how they work. Is precisely the two principles like incredibly important. And we believe they are necessary. Is that? Memory or knowledge is precisely trying to discover what's predictable about the world. Hence, for all understand, all such information are have intrinsically very low degree of freedom. We call it a low dimensional structures. And hence, the way to pursue such knowledge is precisely through trying to find the most simple representation of the data.

• 8:00Speaker 00

  And hence, compression, denoising, dimension reduction is actually all just different words to pursue such knowledge, such a structure. And hence, that's the word captured by the word parsimony. Finding, you know, explain, making things as simple as possible, but not any simpler, right? So this is Einstein says this word, this, this sentence Einstein used to describe science. Actually, that's also what the intelligence, at least at this level, is precisely doing the same thing. The second part of a sentence, not any simpler, precisely says.

• 8:36Speaker 00

  Consistent consistency. Make sure your memory is actually consistent with be able to recreate simulate the world just right, not any similar. If you're simpler, you may lose part of the predictability, predictability and also ability to predict well. And so that's actually the those two actually coexist, we believe. And it's just the two principles, parsimony and the consistency or self-consistency are actually the two characteristic about how our memory works.

• 9:12Tim Scarfe?

  So we want to have understanding which carves the world up by the joints, which represents the important invariances in the world. And the thesis is, I think, that compression might be necessary for understanding. My possible concern with that is that what we are doing with machine learning is representing extant examples of a long phylogenetic tree of evolution. So to what extent does knowing their representation now help us? Do we also need to know how they evolved and where they might go in the future?

• 9:50Speaker 00

  The the process to acquire knowledge, to gain information about our world, that's a compression. Find what is compressible, what has orders, what phenomenon has orders, has low dimension structures that allow us to predict, to rule out variabilities, to predict world tomorrows or predict world better in that sense. So in a sense that that is ability we believe is really what intelligence is all about. At least the the common intelligence we're talking about, right?

• 10:32Speaker 00

  We can talk about the more higher level intelligence later. And if you look at the the history of life, how life was developed. So where do you come up to believe, right? You know, you know, the, the, the mechanism that we laws that governs the physical law world, we call it physics, right? But what is the mechanism governs the evolution of life? I think it's intelligence, right? Even the process you mentioned that through the evolution and life evolves. Precisely, they, they learn more and more knowledge about the world and they encode them through DNAs to pass it on to next generation.

• 11:14Speaker 00

  And that is a compressing, that's a, that's a process to compress knowledge that learned about the world through our DNAs. But the, the, the mechanism to update it is actually very brutal, very brutal force. And through, you know, random mutation and the natural selection. Yes, it does evolve, it does advance, but at a huge cost of resource, time, and also very unpredictable. Which if you, if you're acute, you probably observe there's some similarity with how current big model evolves, right?

• 11:57Speaker 00

  Many, many groups try without principle, trial and error, empirical. And the lucky ones survive and gets advocated, you know, everywhere and become very, very popular, right? Dominant the practice. So it's, so in the, in the sense that you can make the analogy, right? I think to the people, you know, students ask me at which stage our artificial intelligence is at today.

• 12:23Speaker 00

  Then there's already a, a allergy in, in, in nature, right? We are very much at the early stage of the life form, right? And so hence, that is a compression process, that's a process that also gain knowledge about the world. But of course, later on, we develop, uh, individual animals to develop the brain, develop, you know, neural systems, develop senses, including visual and touch and so on. So we actually start to use a very, very different mechanism to learn, to compress our observations, to learn knowledge and to build memories of Oxford world. And even individuals start to have that ability, rather than just to inherit knowledge from their DNAs.

• 13:06Speaker 00

  So that's a different stages about, and then that part of the knowledge is not no longer encoded in our genetics, in our genes, but also in our brains. And that's actually, that's actually a level of intelligence we talk about most of the time these days, you know, which is common to animals, which is common to, to humans. And the level, the knowledge, the, or the intelligence we talk about, what brain functions.

• 13:32Tim Scarfe?

  Yeah, I mean, I, I, I think we would definitely agree with the statement that intelligence as a system produces artifacts. So Sholley's example is a road building network. It produces roads and the system has adaptivity because it can create new routes where they weren't there before. And then there's the question of, well, there are many ways to compress a thing. So some ways of compression represent the world at a deep abstract level and some don't. So we might argue that L L M's today, even though they do compress the data, they only compress it in a superficially semantic way. And then there's this notion of, well, maybe we agree that intelligence is about the synthesis of new knowledge. So it's the acquisition of new knowledge, but we can only do that if the knowledge

• 14:16Tim Scarfe?

  we already have represents the world at a deep abstract level. So rather than it being random mutations in evolution, it's very, very structured because the processes are physically instantiated, which means rather than just doing something completely random, it's guided by the process which created them.

• 14:36Speaker 00

  There are a lot of confusions about what is the knowledge and also what's the process to gain knowledge, right? For example, many people says, what all this language model is doing to the language, which is By the way, it's very different. Don't forget, our language is the results of compression. Is there is our language is precise, the code we learned to represent the knowledge we learned through our physical senses about the external world.

• 15:15Speaker 00

  Through billions of years of evolution, or billions of years of as as our brain, our our brain evolve. Right? It's a result of that. Our knowledge is that that actually represents knowledge. And hence, the the the language, natural text. We use the language, natural language, to encode other knowledge as common to all people. Now, we're using another model, another compression process to memorize it.

• 15:47Speaker 00

  In the sense that that mechanism, you can argue that mechanism, what are those large language models doing? Are further treat those text as raw signals to further through compression to identify their statistical structures, internal structures. What it's doing is actually not very clear. Is maybe we're just help us to memorizing the the text as they are and regenerate them, right? And it's not going through a process, just like how we acquire our la- how our natural language was developed.

• 16:25Speaker 00

  Is through a very um long and our language are actually grounded with our physical senses. Our world models, as we know as memory, right? Our language is precisely try to describe that. It's abstraction of that world model we have in our brain. It's a small fraction of the knowledge that we're sharing with each other, right? Far smaller than the actual model we have. We have many things in our senses, in our memory. We have, there's no way we can express them in words, right? Hence, you can think, don't forget, right? Our brain, there's small fraction process natural language. But the majority is processing visual senses and also motion senses data.

• 17:09Speaker 00

  So that's actually says how, how, how, what the role of language is doing. Now, people are actually very much confused. The, the process, well, actually now with large language model, we use the process, reprocess the natural language. Or the knowledge, as we know, the human knowledge, we go common to human society. We know this fraction of knowledge is actually very similar to what maybe we're trading our raw data. All right. Hence, you can see that even the mechanism we transformers or whatever architecture we're to do. And we're very common to reprocess them. Uh, for example, the videos, visual data. Or we trading languages as visual data. We tends to confuse, right?

• 17:53Speaker 00

  Um, the, the mechanism, we actually probably be memorizing or extracting knowledge from other senses with what we should be doing with, uh, confused that, that process with, we probably large language model is actually understanding the natural text, right? So there is a very, very fundamental differences, right? Using the same process to compress

• 18:18Speaker 00

  raw data, to extract statistic correlations, to stack low, low dimensional structures. That process works for, for, for, for the senses, right? To build the knowledge. But we're actually applying that to our knowledge, to our natural languages. We pretend that is understanding, that's probably not.

• 18:38Tim Scarfe?

  You cited Max Bennett in your talk. By the way, folks at home, you should watch Professor Mars talk on, on intelligence and some of the ideas in, in your book. And I'll put a link in the video description. But you cited Max Bennett and I read his book, Brief History of Intelligence. And he also had this really interesting idea that, that language is basically a set of pointers. And we're actually sharing simulations and second order. Of simulations and and in a sense, those are just pointers to the simulations. And that's where a lot of the semantic content is. But you also spoke about levels of intelligence. So it's a wonderful idea, isn't it? That we have this phylogenetic information accumulation, which is very slow. So you know, every single physical generation. And then we have this ontogenic accumulation in our lifetimes.

• 19:22Tim Scarfe?

  And then we have social accumulation. So we have a big hard drive in the sky where we kind of store these pointers. And then we also do science, which is very abstract, where we derive. Well, we just hypothesize about things. And what's very interesting about science is, you know, there's always been this division between empiricism and rationalism. So empiricism says, if the idea is on the stage, it's in my mind. And this other idea is that sometimes we just conjure things which are not in the data. So that science thing is very interesting.

• 19:56Speaker 00

  Exactly. So see, you just mentioned the four stages. We sort of elaborate a little bit in our book, right? How people, those four stages actually something in common. The commons exactly try to extract structures from the data and record them. Record that what's predictable and through compression, through denoising, through dimensionality reduction. Capture the correlation in the signals. And I use that for predictivity, for predictions and so on. And that's in common. Hence, for all four stages of of intelligence, the principle of parsimony, the principle of self-consistency or consistency at work.

• 20:39Speaker 00

  However, through different mechanism, right? The different code book, different mechanism for updating, for optimizing the information or even acquiring the information. That's actually very important to know. Now, the the main thing you mentioned, also it's a it's a point of confusion. It's precisely the.

• 21:06Speaker 00

  The animals or humans, even social science at the stage of before science appear, right? Almost you can see almost all the knowledge we gain are through empirical approach. Kind of passive. We observe and we we try, we made some errors. We learn from our mistakes, then we record that, right? For example, Chinese medicine, Indian medicine, for many, many years, it works, right? It's similar, it's very similar to, even to become similar to even how DNA evolves. We search in a sort of a less organized way. And somewhat by chance, some by accidents, and we but we accumulate those knowledge become very, very useful.

• 21:51Speaker 00

  We understand how weather changes, how planets moves in a very empirical way, right? For many, many years. So and we share that. We rewritten them the languages in text and pass those knowledge down to next generation. Similar to what DNA does to to to to to the knowledge they learned through different process. Now, there's a huge distinction. We I quoted, you know, the around the three thousand years ago. We don't know what happened, to be honest, right? We're able to, maybe the whole process to acquire empirical knowledge is just through compression. But suddenly, we be able to do somehow to do abstraction, right? We start to develop knowledge that's far beyond empirical observations.

• 22:36Speaker 00

  Which you think about is very, for example, the the notion of numbers, natural numbers. We count up to five hundred. But suddenly, we realize this process goes to infinity, right? Kids start to do that, right? When they in in middle school, they start to everybody, most people start to get that point. And there's an amazing thing is that about, you know, three thousand years ago, that all the, you know, the Euclid. Formulate the geometry, right? One of the assumption, I don't know if people know this, is very long empirical. Which says two parallel lines never intersect. Why the world lever? It's actually infinite suggesting.

• 23:18Speaker 00

  It's something you never would observe empirically, right? How did we come up with that idea? How, what happened to our brain? We are start to jump from compressing empirical knowledge, identify correlation, to something we are actually formalize, abstract, right? Hence, in my book, I asked in the towards last chapter, I asked, is there a difference? Is compression all there is? Probably not, right? Abstraction is definitely related to compression, but there seems to be something different, something more, right? You know, the Karl Popper, right?

• 23:58Speaker 00

  The famous, you know, scientific philosopher said that the science is the art of oversimplification, right? The ability to abstract, we be able to hypothesize things. And there is a distinction between hallucination and the hypothesizing, I believe. But what is it? All right? And we know through compression, we can memorize, we can learn the data distribution. We can memorize, we can find a good, very good representation. We can even using that representation to regenerate data with the same distribution. We call it a memorization. But is there a difference between memorizing those data distribution than understanding it?

• 24:44Speaker 00

  We can emulating how we conduct logical duration, right? Just like now, we can define, supervise fine tune or through chain of thought to the, and also using the reinforced learning to force the large language model to emulating, memorizing how we solve logical problems, how we solve mathematic problems. But that is, that solution is based on, based upon understanding logic, understanding necessity of logic, mastering the mechanism logic, applied it. Or it's just emulating the process. So we don't know. So we have a lot of question now.

• 25:24Speaker 00

  Is the difference between compression and abstraction? Is the difference between memorizing and understanding, right? And it's kind of similar to when Turing was faced the question, what is computable, what is not computable? We know there is a difference, but how do you crystallize that? Or we also now know is P equal to NP, right? We know there is might be a distinction. Can we formalize that question? If we believe there's no difference, prove it. Or if there is a difference, how can we qualitatively or quantitatively be able to see what's more beyond compression? That will take us to the level of abstraction.

• 26:09Speaker 00

  And hence, I believe this is hence, this is the sort of we call the face transition, right? From empirical knowledge, developing empirical knowledge to scientific knowledge. And what is the distinction? To me, sometimes I call the last stage of, you know, the, the, the, the stage of intelligence as the true artificial intelligence. If people ever bother to read the proposal laid out by the folks in 1956, you'll find that actually that's the level of intelligence they truly mentioned, they truly meant, meant, okay? To, to, to, to, to, to work on. But yet, from all we understand about the practice in the past decade, right?

• 26:50Speaker 00

  We very much are reproducing the kind of mechanism that is at the level of memory, how to form empirical memory. In fact, I believe even the large language model are precisely memorizing

• 27:05Speaker 00

  the large volume of text through how we form empirical knowledge, right? It's using the same mechanism, how we form empirical knowledge. To memorizing the natural language, the knowledge is encoded in natural language. Hence, whether that it's equivalent or equal to understanding, that's a big question mark.

• 27:30Tim Scarfe?

  It is tantalizing, isn't it, thinking about where, when we come up with these new theories that don't seem to come from the data, where do they come from? You could be a Platonist and you could say they are just a gift from God and somehow, or a nativist and somehow they're in our brain. Or maybe we could subscribe to the idea that there is, there's a kind of deductive tree, right? So there is the tree of all possible conceivable knowledge and that represents our cognitive horizon. And if only we could build systems that could acquire that tree very abstractly. And if we could design a compositional system that could creatively explore that tree,

• 28:13Tim Scarfe?

  then somewhere in that tree, we would be able to discover these abstract theorems. But then there's the question of, well, why don't large language models do this now? So we see the ARC challenge, for example, and what we see is that models are very, very bad at doing abstract compositional reasoning, where you need to take abstract things, combine them together to adapt to novelty. They don't do that very well. And in a sense, one kind of optimistic view is that these models are learning lots of factored representations and it is conceivable that something like a large language model could do that. But the other school of thought is that it's just not possible because they don't have

• 28:57Tim Scarfe?

  abstract enough understanding. What do you think?

• 29:00Speaker 00

  I believe at least from our current understanding about what the large language model, at least the current architecture is doing, is precisely using the same mechanism that we extract empirical knowledge from the data, the correlations, the low dimensional structure within the data to memorize in the natural language, right? Which to some extent, representing the knowledge we have, right? Hence. Then the question is then the large language model is precisely using the same process as we acquire our empirical knowledge to process the the the the large volume of the natural language.

• 29:42Speaker 00

  From the mechanism part, I don't think it's actually needs to uh understanding. Now, but in order to make that statement conclusive, you need to know then what is what the extra thing that leads to your question, right? Then what what it means by understanding? What am I being might be able to really have that deductive structure? Right? So this is a big question, to be honest. I I think that's a question truly that scientifically a community needs to answer now to also to address or at least to discuss now. And what do we mean by that? Right? Um. So you know, in a in a in in a sense, we have modern science. There's always two schools of a process that allow us to propel the science to advance. Why is inductive?

• 30:26Speaker 00

  Right? Uh. We do experiment, we observe, and why is deductive? Right? Once we have accumulated enough inductive uh you know, observations, experiments, empirical observation, we start to make hypothesis, assumptions. Right? Axioms, so that then we actually be go through a very rigorous deductive process to derive what's implication of those assumptions, then reach conclusions that is testable. That is actually measurable. Then through the verification, through experiment, we can measure. Then we can either verify or falsify the the original assumptions.

• 31:06Speaker 00

  Right? That's a very powerful process, but that process precisely relies on we have a very rigorous logical deductive system. Without that, right? We cannot overthrive the original assumption, right? So that is very fundamentally deep, um, you know, abstract process. So the way we have that in our brain, we can argue whether or not animal has that or at what point we had this, we reached this uh critical moment that we'd be able to develop that uh ability. Our brain evolved to what? Maybe it's it's a some faith, we let's go through some phase transition in our brain structure. Well, allow us to identify those structures, platonic, whatever you call it, right?

• 31:48Speaker 00

  Structures, and we are able to, not only the original scientist, discover.

• 31:56Speaker 00

  The logical, causal and abstraction. But also other people also gain the ability to understand it. That is actually quite amazing if you think about it, right? Different the human beings through language we communicate. Somehow we all at certain stage develop ability to understand, to be able to learn mathematics. Even this is discovered somewhere else, but I can reach some similar level of understanding. As the original discover, I may also be similar similarly convinced by the proof is rigorous. The logic deduction is something necessary, right? Not just some like natural language has has ambiguity in it, right?

• 32:41Speaker 00

  So this is something up for debate. Is this really that, you know, there's a God there that there also there is some invariant knowledge there truly, you know, ground truth there. Uh, we don't know. Um, but definitely this, the ability, well, how do we allow us to reach that level of ability? I think that is truly the next. We need to understand what are the mechanism?

• 33:06Speaker 00

  Implementable, reproducible.

• 33:10Speaker 00

  Mechanism allows to recreate, to have a system to be to gain that kind of ability. I think that will be the next stage of intelligence. So we'll be able to have artificial system reach the stage of human educated, enlightened human, right? Beyond as our our humans as a just as an animal.

• 33:35Tim Scarfe?

  And you believe in practice that's possible?

• 33:38Speaker 00

  I truly believe that is a part of our brain or function, I think can be reproduced, can be discovered, understood and even reproduced. But for now, uh, what exactly that mechanism is? I don't think we have much clue, right? We know the afterfacts, just like you, your example, we know. The road it build, right, the road building company or network, right? They can build a road, we know, so even the today, we learn, we can learn logic, we can learn mathematics. But what is the mechanism allow us to create a new mathematical theorem, create a new scientific theory?

• 34:21Speaker 00

  Conduct the logic deduction, right? And understand it, that what is our mechanism? We don't understand, we only know the after the facts, right? The logic makes sense, but why it makes sense? That's still, what is the mechanism in our brain? Uh, that we shared, that the all this process, all the deductive process makes sense to all of us? And it's still quite unclear, yeah.

• 34:50Tim Scarfe?

  The reason I ask the question is a lot of cognitive science folks, they say that we understand because we're causally embedded. And what they mean by that is we evolved with the world, which means all of the representations in our brain co-evolved with other things in the world. And it's deeply rooted, so the implication is that intelligence is quite specialized. And you could say, intelligence, oh, it's just the efficient search of the space of Turing machine algorithms. And yeah, that would be correct, but it but it's describing the what, not the how, it's more, it's so. It's almost trivial to say that, it doesn't describe how we would implement intelligence. Whereas if intelligence is the acquisition of knowledge, it must be domain specific, right?

• 35:34Tim Scarfe?

  Because the road building company that we spoke about, they can't build any type of road. They are restricted in terms of the materials that they have access to. And presumably, different types of knowledge are quite different to others. So an intelligent process might be able to acquire knowledge over here, but not over there. How specialized do you think intelligence is?

• 35:57Speaker 00

  Remember, the road building company, that's just a analogy, it's a special cases. That's exactly sort of echo with what we mentioned earlier. The intelligence can has different stages or even different forms, but there are always something in common. Right, even you know, even the early stage, how DNA evolves, how our memory evolves, how human society logic evolves, or how our scientific logic evolves, right? Those are precisely there are different, you can say there are different builds. So some kind of building, uh, building, some are building roads. But the mechanism are common. They're using similar, uh, for example, the concept of compression, discover the, the mechanism that's common behind all this is that to discover what is structured,

• 36:42Speaker 00

  what is not random. And I think that is common through a, you know, under the principle of parsimony or compression or dimension reduction or denoising to discover those structure. That is common, although the mechanism, you know, operation might be different. The domain knowledge distribution that apply to might be different. Some are discrete, some are continuous, some are higher degree, uh, the intrinsic degree are higher, some are lower, some are simpler, so some can be formulated as uh, mathematical, physical equations, some cannot, right? But they still can be learned, can be compressed, can be memorized through other mediums,

• 37:23Speaker 00

  in DNA, in our neural networks, not in, you know, differential equations, right? So those actually, there are things are very common. The mechanism behind purpose are common. We can, even the principles are common, but the realization, right? The physical realization could be different. And then the optimization mechanism could be different. The code book we learn could be different, right? So this is something we need to understand. Um, I, I think once you understand that, a lot of, it can help you understand, we can understand a lot of things around us, right? What is common behind all the intelligent behaviors? And also each stages or each deforms may also has their own uh, domain specific characteristics.

• 38:09Tim Scarfe?

  I know you're a big fan of um, cybernetics, which came from the forties uh, with uh, Norbert Wiener. And it described this cybernetic action loop, which is an agent which senses and acts in, in this closed feedback loop. And. Just, you know, based on what you just said, maybe the mechanism of intelligence is the same, but certainly the action space is different. So, so different embodied agents in this situation would be able to do A, B, and C, and overhear something different. So could it be at least specialized in the action domain?

• 38:42Speaker 00

  Those scientists, those pioneers that, about, those are the pioneers in nineteen forty's, they're interested in intelligence. But the level of intelligence, I think their interest is mostly at the animal level, at the common human level. It's about how our brain works. It's not about DNA, although they make some analogy. But they very much specify how our brain acquire our memory, build a world model quickly through perceptions and, and actions, interacting with it, to predict and make mistakes and learn from that process, right? So I think that's the stage their study. They may not quite get to the advanced stage that we'll talk about in science. They care mostly about how our memory works, the animal, right?

• 39:27Speaker 00

  To build autonomous systems, autonomous machines, so that can emulating our ability at that level, right? So hence it's the, they call, we call the cybernetic programs. By the way, I mean, the point is that a lot of people, there are two things. Of course, cybernetics word was a little bit abused later, right? Become, just like artificial intelligence, actually become very bad word for certain period of time, right? I think it's probably, and also the, many people do its understanding cybernetics in a very narrow minded. They actually think it's just about a control. Actually it's not. If you read the book, right? And by Robert Viner, he actually characterized, it's a book actually characterized all the, at least the necessity,

• 40:07Speaker 00

  necessary characteristic, what the intelligence system, the system at the animal level should have. How do, how do we record information? He actually needs to information theory. How do we correct error, which is the feedback control? How do we improve our decision making? He actually mentioned the game theory, right? And, and so on, so forth. He even discussed the necessity of addressing non-in reality, which actually explain why our brain has waves and so on, so forth. But it's very clear, he's definitely interested, what are the necessary characteristics for a system to be intelligent? Although he may not, he foreshadowed how they should be put together.

• 40:50Speaker 00

  But definitely capture some of the essential characteristics. A autonomous intelligent system should have. Which surprisingly, it's got forgotten by our past decade, the practice of artificial system, building artificial intelligent systems. Right? And don't forget, those are necessary characteristics. That, you know, and at least those pioneers are convinced a intelligent system should have. So this is actually something I think, you know, we should probably learn a little bit lesson from our history.

• 41:24Tim Scarfe?

  Can you sketch out the journey from information theory to your maximum rate coding reduction system?

• 41:30Speaker 00

  This is actually a very interesting question. I'm not actually, honestly, I'm not the information theorists, right? So I was trained as control series, but early when I was a graduate student, I wanted to do communication. I took a lot of communication, random process, information theory course. Although I didn't end up with doing it. So for for for many years, you know, this has become something I never really quite practiced. Until about a few years ago, I was studying what deep networks doing. And also get come across what my intelligences are all about. Realize that maybe intelligence is trying to learn the common mathematical problem

• 42:12Speaker 00

  behind at least the learning knowledge at the level of memory is probably pursuing a low dimensional structure or low dimensional distribution of high dimensional data. Okay? So once that became clear to me, then it becomes a very soul searching process for me. Now, if low dimensionality is the only prior, the only so-called quote unquote inductive bias or assumption we can make. From there, can we deduct, can we deduce everything from there? Now, but even this is some only thing we can use, then what's special about it?

• 42:53Speaker 00

  It's the dimension very low, hence the volume of the data should be very low. Then there's such comes a lot of technical challenges is that then how do you differentiate? If I have a set of data, if there are two models, both are low dimensional, can equally support the data, explain the data, then which one do I choose?

• 43:18Speaker 00

  So the challenge here is that if two models are both low dimensional, then the volumes are all zeros.

• 43:27Speaker 00

  And I have an example in my book, right? It troubles me a lot, right? And we have eight dots on a line. Why is it evenly distributed eight dots? Another one is, you know, four dots clustered, another one. And the interpretation is very ambiguous, if you think about it. What's wrong? I treat all as just eight dots. Each one occurred one eighth, you know, with probability one eighth, right? Nothing wrong with it. Or when should I say they're all lies on a straight line? Right? But a line already has one degree, one dimension higher than dots, right? Then also what's wrong? I say those are eight dots aligned on a plane, right? But they're all zeros, zero dimensions, right?

• 44:10Speaker 00

  So hence there's a question is how do we measure? The volume of the data, right? If you want to compress, you have to have a more generalized notion of volume to measure the space spanned by the data. So that come forced me to come across the concept of entropy. But entropy also is kind of limited because precisely it does not quite differentiate those kind of interpretations, right? If you think about it, they're all zero, one dimensional distribution, then the differential entropy is negative infinity. You are comparing, you know, infinite with another infinite or zero with another zero. So hence that we sort of actually come across maybe a noisy coding, right?

• 44:54Speaker 00

  Is just like Shannon developed after information theory. He actually mentioned that when we actually do coding, we do redistortion, we do noisy coding, right? That actually become a sort of magical sauce, allow us to, now we have a more generalized, much more general measure of volume. For data in arbitrary space, right? And the the the measure, the support can be actually general. We can differentiate one no dimension model against another one, right? And that becomes a measure. We can weight the data to differentiate different models among different models. Now, allow us, if we compress this, the the coding rate, we compress those coding rate

• 45:37Speaker 00

  will allow us to pursue those distributions in a very high dimensional space. In fact, the very popular diffusion de-noising process precisely doing this. Their de-noising process are precisely reducing the entropy so that we pursue a representation that is lower dimensional, lower entropy. In the end, it converge to the distribution of the data. This is the first stage. Now, for memory, it's not just the pursuing, find the word, the distribution is, right? I I use analogy, right? So the find the word, the distribution is, is also about organizing it, right? A lot of people says, oh, you know, learning is about the compression.

• 46:20Speaker 00

  And hence, you know, why don't we do all the way to comograph complexity? No, you don't want to do that. First of all, comograph complexity is not computable. Second of all, we all know, if you really manage to compress your data to comograph complexity, the the codes will be random. Basically, the program itself specify the data will be random. We don't memorize a bunch of random numbers in our brain, right? Hence, actually, our memory in our cortex is highly structured, right? Different type of objects is very well organized in IT cortex. And our spatial understanding is very well organized in hippocampus, right? It's highly structured because the structuriness allow us to access.

• 47:03Speaker 00

  Because we wanted to access the knowledge repeatedly, we want to use it very efficiently. Hence, the knowledge has to be, we use it under very, very different conditions. Hence, I said, you know, our brains very much doing Bayesian inferences. Once we learn that distribution, whether we organize the, we transform the distribution to a very structured and organized way. So the maximal entropy, you can think the maximal rate reduction is precisely reflect that necessity. You do not just by reducing the coding rate to find where the distribution is. You also want to transform it in a new representation such that the rate reduction gets maximized. The reduction gets maximized.

• 47:45Speaker 00

  Hence, the representation subsequently become structured and organized. Then facilitate efficient access. And you can access memory under all types of conditions. Allow us to do all types of condition, prediction, generation, and estimation. Right? That's essence. Hence, reduce entropy, reduce the coding rate, and then maximize that reduction of coding rate. They are actually reflect two related process for building a good memory.

• 48:21Tim Scarfe?

  First of all, the manifold hypothesis comes to mind, which is this idea that all natural data falls on some low dimensional, you know, structure, you know, with a low intrinsic dimension. And the other thing that springs to mind is, I mean, I'm a fan of geometric deep learning, which is this idea of, you know, we should imbue inductive priors in the system which represents symmetries and geometric structures in the world. And I think as a principle, that's deeply embedded in this idea.

• 48:51Speaker 00

  Exactly. If you look at my whole life, I have written four books, right? My early interest in studying computer vision. And the first book is a pursuit division. And from that work, I study multi-view geometry. From that work, I already, in my whole book, all the four books is actually about the one theme. I realized that is about structure in the data, especially all reflecting. The first book of city vision, the last chapter, I realized precisely about what importance of symmetry plays in our perception, right? And we perceive an object, we naturally has a...

• 49:35Speaker 00

  So, human being, we remember our vision.

• 49:40Speaker 00

  We already realized a long time ago, right? These days, people always about the vision is about recreate the 3D. Absolutely not, right? All the people says, we just multiple images recreate a whole point clouds. Mesh, sound dysfunction, lerp, Gaussian splatter, right? We recreate the scene, see, I can see from multiple angles. Is this 3D understanding? Or we create some videos like Sora, look at it, looks good for, you know? Absolutely not, right? This is not the representation or understanding of world model. Right? Our understanding is far beyond getting a whole bunch of, you know,

• 50:23Speaker 00

  but point clouds or Gaussian splatter we can view from different angles. Right? Haven't noticed that when we see something gets really, we get excited because we understand the 3D. We understand the content, we parse it already in our brain. But the machine has no idea what the heck is in there. Okay? It's just a bunch of point clouds. It's a depth map. Okay? We, when we see this change in angle, we saw 3D. We already automatically recognize this is a hand, this is a body, this is a cup, this is a, this is an apple. Right? We do that, we fill in that information with our brain. We think machine, once they can recreate the 3D, they all do understand that. This is completely wrong. Okay?

• 51:04Speaker 00

  Many, many work, they say we are building 3D model by creating something for people to look at. That's completely not the purpose. So look at our vision, our vision model, right? We have hippocampus, we have our ID codes, highly structured. We understand the relationships between our view centric, object centric and allocentric reputation, right? Neuroscientists understand this very well, right? Scientists understand this very well, but not the computer scientists, right? Not the computer vision scientist. Um, some do. See, for example, I give you example, right? In order to do spatial, no, no, we actually did a test with Yann LeCun about a year ago.

• 51:44Speaker 00

  Test all the, you know, top multi-model models, you know, huge models, GPT, you know, that, uh, Gemini, right? To do very simple test. The, the title of the work called Eyes Wide Shut, right?

• 52:00Speaker 00

  And it's a very simple test, just test the given images for the nodes language model or a big model, multi-model, highly trained, highly commercialized models that do they understand the reasoning, spatial reasoning, what is on the left of something? How many objects there is in the space, right? What is behind something? What's on top of something? Very simple spatial, the question requires a little bit, not even very deep spatial understanding, but all the models fails miserably. And the majority of them actually even worse than random guess, right? Only, I think only Gemini and G P T is a little bit about human, a little bit about the random guess, okay? Far below human understanding, so that's, that's the status, right?

• 52:45Speaker 00

  If I see do those, meaning that, you know, those actually 3D understanding is very, very difficult, but human, we do this effortlessly, right? So I can easily point to you, right? Please hand me the, the bottle to your left. If you wanna find, let's say, a shopping center, right? Say, go through the door, turn right, go to the, once you get outside of the building, head south, right? So through this, this, remember, just through this simple sentence, already we switch from view-centric to object-centric to alecentric, right? So our, if we don't have this kind of model, this kind of highly structured 3D models,

• 53:31Speaker 00

  forget about, you know, people talk about embodied AI or world model, right? We cannot conduct this very simple spatial references, interacting. We have the world models not to visualize, we build a 3D model to interact, to manipulate, to influence. Right, we're not building a 3D model just to, oh, this looks like, I can switch, change my view, look at it from this view or that view, right, just visualize, no, turn 360 degree to visualize, no, we don't do that, that's not our purpose, okay? Unfortunately, but that's, you know, it gets distracted for that kind of visualization, it looks cool, but almost to us, if you really work on robotics for, you know,

• 54:15Speaker 00

  navigation, locomotion, manipulation, that's pretty, the usage is pretty limited. I won't necessarily say they're useless, but they are actually pretty limited, yeah.

• 54:26Tim Scarfe?

  We should introduce the coding rate formula, I did have a question about that, which is, there is a, there's an epsilon on there, so there's a bit of a question of, how do we tune that and what does that mean? We should also bring, we've been talking about this a little bit, this concept of an LDR, so a linear discriminative representation. And just more broadly, with these inductive priors, there's always the question of, when we do abstraction to model regularities in the universe, there's always a little bit left over, isn't there? So to what extent can we think of these things as natural?

• 54:58Speaker 00

  Actually, you touched upon about the epsilon, you touched upon a very, very deep question. We actually, it actually took me almost 30 years to understand it, to be honest, right? We did mention that early on, when we do, try to differentiate, measure different volumes. It turns out, loss encoding is necessary, it's not just something that is something hacky, it actually turns out to be necessary and to do loss encoding. In fact, recently we start to realize that noise actually plays very different roles. And yet, it's very confounded, very confusing to many people, even, this is something actually my students recently realized, we actually probably will have some papers about it. I can elucidate this a little bit.

• 55:42Speaker 00

  If you think about the whole diffusion de-noising model, right, people are very popular right now to do, why do we add noise to data, right, and to the whole world? Because we don't know where the diffusion is, right? So there is a phrase everybody knows, all roads to Rome, right? So why is that? Has anyone gave a thought to why all roads to Rome?

• 56:06Speaker 00

  Because, very simple, at some point in history, Rome builds a road to reach the whole world.

• 56:13Speaker 00

  Right, that's a diffusion process.

• 56:16Speaker 00

  Then if you want to know Rome, then you do the denoise, you follow the same way back. You get to where the Rome is. So that's the no damage structure. That's where the knowledge is, right? That's the oasis is, right? So hence, it's a very natural process that we add noise. It's adding noise to the experience precisely we're building the roads. And the denoising brings us back. Remember where we come from and so on. And that's a big epilogue. We have to add the noise to reach the whole earth. There's another, actually, there's another noise, right? Remember, we only have isolated sample. Even we talk about manifold, right?

• 57:00Speaker 00

  But how many points you have on the manifold? How many points you observe them? They're always finite, right? But why do you call it a continuum? Why do you collect dots as lines, planes, surfaces? Wouldn't you do that? Hence, noise plays another role. Within the manifold, even you have finite samples. If you allow loss encoding, if you allow packing spheres in that, things start to connect. You start to connect. Noise is very important to help to connect the dots, right? We all know the phenomena of percolation, right? We see raindrops on the floor. You only see two phases, right? One phase is all the dots are isolated.

• 57:44Speaker 00

  Another phase is all things gets wet. You never see anything in the middle because there's a sharp phase transition. Once the sphere, once the dots, the density gets high enough, they collect everything, right? Maybe that's the phase transition we reach. We realizing a connected plane is a better solution to explain all the data. More parsimonious, more economic. The cost to memorize all the dots versus memorize on the plane start to switch. Maybe abstraction has something to do with that, I don't know. But from a compression point of view, this can already allow us to explain.

• 58:26Speaker 00

  When do we go from zero dimension samples to prefer a low dimensional manifold? And also, how go from that low dimensional manifold to reach the rest of the world? Right? So you can see, even in this process, noise is already playing, this epsilon plays different roles. And there's at some point, they gets collected, right, around the surface. And that's we still try to figure out what happens. But after this big two phases, we already know, right? The role of epsilon actually plays different roles, right? And I think the, definitely in the past many years, our understanding about the subject, how do we compress?

• 59:06Speaker 00

  How do we pursue the low dimensional structure from finite samples? It's a quite, our understanding about this problem has truly advanced dramatically. I'm very happy, honestly, this is a question baffles me when I was a graduate student. You can see even my early work about the loss encoding, loss compression reflected my baffleness about it. And I'm really feel very thrilled that I recently start to understand those things in a more unified, more, a lot only theoretical way, but also even the algorithmic way.

• 59:39Tim Scarfe?

  Yeah, it's so fascinating that we can look out the window. And we ignore so much detail, we don't look at the leaves on the roads, we just find that structure. And that's why when I watched your presentation, I was very intrigued when you said that de-noising, iterative de-noising is a form of compression. I wanted to mention your ICML 2024, so last year, it was in Vienna, right, with Wang. And you found that when you have loss surfaces using this technique, they are dramatically different, they're very smooth, there's no kind of harsh local minima and so on. What's the intuition for that?

• 1:00:15Speaker 00

  In fact, the phenomena, our understanding about those phenomena is actually going back when we, early days, we studied sparsity. You know, when your data lies on very low dimensional sparse surfaces, planes, right? Low dimensional planes, orthogonal planes, or low rank matrix, right? And in there, we learned a very big lesson. The object function to evaluate to those sparsity or low dimensionality, this function are highly nonlinear, non-comebacks.

• 1:00:45Speaker 00

  But yet, you know, traditionally, in our orthodox understanding about the non-convex optimization, they're always hard, right? And the general class is NP-hard, and there's lots of local spires, local minima. You get stuck with local minima. There are some stagnant critical points, a flat surface. So basically, the worst picture is very worse, right? It's a nightmare. But through the study of those low-dimensional structure, sparse structure, that's what I do featured in my previous book, Right High Dimensional, A Low-Dimensional Structure of High-Dimensional Data Analysis. We actually realized that if a lot of non-convex problem, even the optimization problem had non-convex landscape,

• 1:01:27Speaker 00

  if those problem or even those measure arise from nature, very natural resource, those structure actually are very highly regular, highly has symmetry.

• 1:01:42Speaker 00

  The landscape actually are extremely benign, right? Quite contrary to our common understanding about the long linear optimization at all, right? This is a complete 180-degree flip of views. In fact, even the higher dimension helps. The higher the dimension, the better. We call it the blessing of dimensionality. So those regularity, those symmetry will tell us the landscape of this object function are actually beautiful, right? And first of all, they're highly regular. There's no stagnant, there's no flat surface. There's no too many spires, local minima. And even the local minima, they already have very clear geometric or statistical meaning.

• 1:02:25Speaker 00

  And hence, those landscape are very amenable for very simple algorithm to find the optimal solution, such as gradient descent, which almost indirectly explain why even we're doing even more than training neural networks and many more, we're searching low-dimensional distribution in very high-dimensional spaces. But somehow gradient descent always end up with somewhere nice. Okay, yeah, fine, you can run a long time, but somehow you always end up with... Those landscapes are not that hard to traverse, traverse, right? So it actually precisely because those object function are highly irregular. Hence... Now, get back to the rate reduction of the function, right?

• 1:03:10Speaker 00

  If you look at the option function, it's not something arbitrary, right? It's counting the volume of the whole minus the parts, right? It's something extremely objective, right? It's not like a loss function, people come up with randomly, oh, add this term, weighted sum, add different weights. You know, if you use this, you know, sort of empirical penalty or empirical, even some kind of ad hoc. So all the terms are describing physical volumes of the data, right? Hence, you should expect those are the quantity arising nature. Then for our lesson, we realize indeed, actually, that actually, those option function has very benign landscapes.

• 1:03:51Speaker 00

  Uh, even the local minima, not only the global minima, corresponding solutions that are give you orthogonal subspaces. Even the local ones, right? There are a lot of global optimal ones, they have similar geometric structures, right? And there's no other weird, critical points that will slow down the search for those minimas. So that's actually quite interesting. So you can see, this, this revelation allow us to understand, right? Where maybe intelligence is precisely explored and harvesting those things. So there is actually a misunderstanding about, you know, last two years.

• 1:04:31Speaker 00

  When we understand intelligence more and more, there is a very big misunderstanding about intelligence, right? In machine, you, you, you, you, you study, you know, maybe machine learning theory, right? We have a tendency to believe intelligence, especially intelligence that nature, is designed to solve the hardest problem, the worst case. I actually beg to differ. Intelligence is precisely the ability to identify what is easy to address first. What is easy to learn? What is natural to learn first? The only when that has been done and the resource permit is start to get into more and more advanced tasks.

• 1:05:16Speaker 00

  Not everybody needs to learn all the mathematics to survive. Animals don't, right? Nature find what is the easiest things with minimal energy, minimal effort almost to learn the most knowledge so they survive the best, right? Again, this is the principle of parsimony at play. There's another level of resource parsimony at play here, right? Hence, once you realize this, you're realizing understanding intelligence should be really understand what's really most common, right? The low dimensional structure, most easy ones, smooth ones, binar distribution, easy to get, get away with a few samples, fewer samples, right?

• 1:06:01Speaker 00

  And very easy to formulate. In fact, that's what how science progressed. You know, a lot of the physical models, you know, Newton's law, they're very simple. The discover simple ones, then we reach gradually reach to general relativity and then to quantum mechanics. Those are the question gets more and more complicated later, right? So this is hence is the same process. We identify what is the most common first, right? What is the most easiest task first? Hence, we don't want to, many of the machine learning theory try to tends to so derive a bounds for the worst cases. I think that's we probably should think twice, right?

• 1:06:35Tim Scarfe?

  I love that characterization. It's similar to the least action principle in in physics.

• 1:06:40Speaker 00

  Exactly.

• 1:06:41Tim Scarfe?

  In a sense, we solve problems by taking many, many steps in different directions. I think we still leave a little bit of entropy open. We don't do pure hill climbing. But collectively, we acquire these stepping stones and the totality of that process is we solve very complex problems. But I wanted to touch on, you raised a very interesting point, which is we notice that when we have very large deep learning models, they tend to almost self regularize and they they learn better. And there's this phenomenon of double descent and and all of this. Tell me about that.

• 1:07:16Speaker 00

  Fascinating question, actually. You are, this question actually needs to really bring me back to early days when I tried to understand deep learning. When deep network arises, a lot of phenomena we try to understand. I'm I'm one of those, right? Try to understand those phenomenas. You know, some something, there are some good about dropout, something about, you know, thresholding, different thresholding, there's something about normalization. Then I also come to somehow the model are very big.

• 1:07:47Speaker 00

  And parameters a lot. Somehow the deep networks has do not have tendency to overfit. Somehow they still generalize OK, right? And then, of course, people realize there's a sort of a rather like the, you know, the traditional classical bias, viral inspire straight off, but there's time to the double descent. I actually wrote a couple of papers about it and about the normalization, about everything. Around 2000 or late 2019, I really told my student, we should stop not to explain those isolated phenomena. We only see where like the blind mental elephants, each one say a little piece, each theory try to explain a little bit.

• 1:08:32Speaker 00

  I think there should be a total total explanation to this if we get a big picture. All this are just the consequences or implications of that. Suddenly, you know, at that time, we start to touch upon the the the concept of maybe that the the process of deep network are optimizing something. The layer wise realizing optimism, they are optimizing object that promoting parsimony, promoting no dimensionality. Once we realize, actually, I was quite thrilled. So then I told my student from now on, we will no longer write any papers about overfitting. Why?

• 1:09:13Speaker 00

  Because if the neural networks is try to, the operators try to compress, try to realize certain contracting map, compress volume, then you will never overfit, right? Even I over parameterize, it will never overfit. A simple example, if I have data lies on a straight line, a one dimensional curve, whatever. I can embed this one dimensional line in the two dimension, three dimension or a million dimension. But if my operator is always layer wise, each iteration, my operator is always just shrinking my solution towards the line or in all directions.

• 1:09:55Speaker 00

  I would never overfit. Even I overparametrized, embedded the line into billions of dimension. I have billions of parameter, but collectively, all those billions of parameter are all shrinking my solution, pushing the solution, denoise it, compress it towards the line, right? Like a power iteration, just like a PCA, right? Power iteration is regardless of what dimension embedded it, computing the first singular values, right? It's always powerful, always converge with the same speed. You never over... So compression, by nature, if the operator are performing compression or denoising, which means this process will no longer overfit anything.

• 1:10:35Speaker 00

  But if you conduct it right, if you converge, the solution will converge on the structure you desire for.

• 1:10:43Tim Scarfe?

  That raises a natural question. We were interviewing Andrew Wilson from NYU, and he's got this, you know, several papers about implicit biases where you kind of have a combination of, you know, hard biases of symmetries and everything in between. And if what you're saying is true, then why do we need inductive biases at all? Could we not pare back a little bit and just have really big models?

• 1:11:06Speaker 00

  No, I see. See, this is the thing, right? Exactly. So this is the thing.

• 1:11:11Speaker 00

  I was, you know, early on, people don't understand deep networks, and there's a lot of empirical trial and error. People try to, tends to use the phrase inductive bias to either as some kind of magical sauce that either explain the failures or success of you do certain way to the neural networks design, or how do you train the neural networks? To be honest, for a long time, I never understood what the inductive bias is. And maybe some regularization, some people are studying some structures about network, about the data. But nowadays, in my recent work, I said probably from, at least from what I understand, all the inductive bias should be formulated as first principle, right?

• 1:11:55Speaker 00

  At least from, we were able to, for example, deduce all the deep neural architectures, including the recent Ybox crate, or transformer-like, or RedNet-like, ResNet-like architecture, or mixture of expert-like architecture, all from the only inductive bias is assuming your data distribution you are pursuing are low dimensional. Okay? You can already get the form. The the main architecture, a form of operator of each layer. As a red structure, mixture of expert structure and and those operator per layer are precisely conducting denoising, compression or contrasting. Are there additional assumption you can make? Yes, you can.

• 1:12:39Speaker 00

  For example, if I my job is not just to compress the data as it is. I also wanted to induce, I wanted to, for example, in object recognition, I also want to enforce, make all data, I want my classification to be translational environment, which is symmetry. Right? If you allow my task will be environment to certain group action. I want to compress them together. Voila, what do you get? You can, then you still see through compression, then you get a convolution naturally as the structure for the compression operator. So convolution is not what we impose upon, is actually results from the first principle,

• 1:13:20Speaker 00

  the the quote unquote inductive bias assume you want to compress your data. Also, you want your compression to respect translation environment or rotation environment. That's the result. That is the characteristic of the compression operator for you to achieve that task. Right? So there's a lot of, um, so we don't want to build in the the inductive bias while we're searching for the solution. The inductive bias in my understanding is should be the very assumption we make in the very beginning. The rest should be deduction. The rest should have no induction anymore. Otherwise, we're doing trial error. Right?

• 1:14:03Speaker 00

  The inductive, we, so basically when we build a theory, we should have done all the inductive observations, experiment and assumptions already. The good theory should start with the very few inductive bias or assumptions or axioms, then the rest should be deductive. I call that first principle.

• 1:14:26Tim Scarfe?

  We've been speaking about parsimony, which is what to learn. Self-consistency is about how to learn. And we can sketch out a journey, I suppose, from control theory to learning. And and also, this this methodology has some interesting um results, I think, around um, you know, the continual learning problem. So let's sketch that out.

• 1:14:47Speaker 00

  You can see, right, so the the compression um um or the the even the reduction to try to pursue the data distribution. And also transform it, and that's the one way direction. There's almost, there's no theory guarantee. Either your data is sufficient to identify that something. You know, you may start with very, very few samples. There's no way the data is sufficient. I mean, maybe the apple, there are five types. Maybe I only save for four types, right? So and then, but that process goes on, you do compress what you have and the reach a memory, right? And there's no guarantee, even during that process, you may not get stuck. Maybe there's not enough iterations.

• 1:15:32Speaker 00

  You may, so memory you get may not be accurate, may not be correct. Hence, how do we check? How do I further develop, evolve, improve my memory? Or make sure my memory actually be able to authentically predict. This is a word model. The model is accurate, right? So you actually have to decode it. You can think about the the memory formation is an encoding process. Then from my memory, I want to decode, I want to predict what's gonna happen next second from what I observe right now. Or a night, I may want to dream what happened, right? So that's actually the decoding is actually allows us to check if my memory serves to be right, right? How accurate I can predict the next step.

• 1:16:14Speaker 00

  So hence, this actually already form a sort of auto encoding framework. Now, of course, auto encoding, if I have access to both the observation and my memory. Just like our training our big data models, I have control on both ends. I can just force auto encoding back end to end, right? The people like to talk about it. But in our natural, um, in a natural setting, in animal, human setting, we don't have control on both ends, right? We only have control, probably a control of our own brain, what's inside our brain, right? We never really quite have access to measure if or my prediction of the three D world. For example, right, the the the frame of the picture is rectangular.

• 1:17:00Speaker 00

  Do I ever measure it?

• 1:17:03Speaker 00

  Right, you don't have to measure it, but somehow everybody believes, you know, the model is correct. How do we do that? Right, hence, there's actually a self-correcting process. In fact, you know, this is actually probably the idea goes, the idea actually goes back to Nobel winner. Right? And how animal be able to correct its error without, see, cats can capture something very accurately. Even make one single mistake, they can correct that very clearly, right? So somehow they're, they're be able to build a world model very consistent, self-consistent with the world without actually physically measure their errors. Right? So hence, this is idea about you actually loop it back to your brain and close the loop. Right? And allows you to constant predict.

• 1:17:44Speaker 00

  And based on your prediction and your observations, check if there's a difference, the difference between your predictive, prediction and your observation within your brain.

• 1:17:56Speaker 00

  Where if there's error, I using that error to correct. Turns out, um, this is the work with my student. Turns out you cannot do this. Of course, in the, our observation will lose information. Why? Uh, we're introduced noise or lose, lose dimension, lose information. But turns out, as long as the distribution of the world, the data, the definition of the world is low dimensional enough. Even your encoding process, your observation, perception process is nothing. You, this is still doable if the precisely when the distribution of the data outside world has enough structure. It's highly low dimension. And hence, your brain has enough degree of freedom to discern any differences.

• 1:18:38Speaker 00

  So this is actually quite interesting revelation to for us to realize that the low dimensionality is not just a some or technical assumption. It's actually necessary for this kind of closed loop learning. To be possible. And once you be able to close loop, then you actually constantly observe, constantly predict. Hence, you can constantly use your memory to predict and correct it.

• 1:19:03Speaker 00

  Hence, support continuous learning, even lifelong learning, right? Our memory, Rome is not building one day. Our memory is never building one day. We constantly improve it, constantly revise it. And this is the mechanism of intelligence. Hence, this mechanism itself is already generalizable. Hence, you don't need to add the adjective general, general in front of intelligence. There's no point of calling general intelligence. If you implement the intelligence mechanism correctly, it's already generalizable. The knowledge learned by this mechanism at any point of time may not be generalizable. The mechanism does, right? This is very big confusion.

• 1:19:48Speaker 00

  We think if I accumulate enough more knowledge, it's generalizable. No, it's not, will never be. Any scientific theory, by definition, being scientific is falsifiable, which means it's limited, right? Can only explain the world up to certain point or certain accuracy. There's always room for improvement. The scientific activity, our ability to revise our memory, to acquire new memory, that is a generalizable ability, that is intelligence, right? Through natural selection, early days, through our feedback control, feedback correction, through the human history of trial and error, accumulating empirical knowledge,

• 1:20:33Speaker 00

  through scientific discovery, it's all doing this, right? That is common behind intelligence, right? Not the memory accumulated up to certain point. So even we manage to memorize the whole world, the knowledge we have in the whole world, we will no longer be able to apply when we find ourselves in a new environment, in a new situation, observe some phenomena we have never seen before, right? Hence, that's the limitation of, you know, you you try to gain. General intelligence through just accumulating enough knowledge.

• 1:21:04Tim Scarfe?

  We should talk about your CREAT series of architectures. So CREAT stands for Coding Rate Reduction Transformer. And you made some very interesting discoveries. So for example, multi-head self-attention can be derived as a gradient step on rate coding. And also MLPs as specifying specification operators. And and also you were talking about how something like a transformer could be described in a principled way. So there's this interesting thing, isn't there, that we we we designed the even well, we didn't even design them. We we kind of empirically tried of lots of different things and we happened upon the transformer. But something like that can actually come about from a first principles approach.

• 1:21:44Speaker 00

  If you look at the past decade or so, evolution of also this kind of natural selection process for the big models, right? From early days, Alex Lett, Lolet, Alex Lett, VGG, or then ResNet or transformers. By the way, this is just one of those for survivors, right? As I said, just lateral selections, right? Remember, don't people don't forgot, there's a there's a time, there is a very popular areas called autolas, auto ML, right? People tends to do random search for better architectures, right? Somehow, why only a few survived? There must be a reason, right? They must capture certain structures, they must did something right.

• 1:22:30Speaker 00

  Now, from our understanding so far, the ResNet actually capturing the fact that each layer should be doing optimization.

• 1:22:39Speaker 00

  The ResNet precisely reflect the iterative optimization architecture, right? And the MOE precisely capture fact, we're trying to cluster, compress what's similar.

• 1:22:51Speaker 00

  And discern or classify what's different or contrast what's dissimilar, right? And you wanted to develop different experts, right? We call them experts, we call them cluster, we call them group, so be it, right? And the transformer, again, right? Has capture what is the correlation, self-attention head is precisely compute what's? What is the correlation in the data, covariance in the data? What's correlated? And using that to further specify, further classify things to organize the distributions. They must do something, somehow come close to something right, right? So also, it's almost like a belief for us, right? If we believe there's something right, then we should be able to derive, create from first principle,

• 1:23:36Speaker 00

  have a very clear unified understanding. I think we're sort of managed to do that. So for the structure we discover so far, provide rather unified explanation to what they have done. To be honest, the early, even maybe our early, earliest motives try to explain to understand what we have done. But the once we understood it, we realized that we can go much further, right? And realize even the current architectures, there's a lot of room for improvement. Not only we can dramatically simplify them. You can see in the past, after the create, in the past, yes, last year and this year, there's a series of work from my group, right? Really, it's just showcase people, right?

• 1:24:19Speaker 00

  You can actually, you, once you understand what is done with the principle, you can dramatically simplify. You can even throw with the MLP layer. If you only care about the compression, you don't care about the final representation. Or you can make the attention head.

• 1:24:37Speaker 00

  Since we know what is optimizing, it's optimizing the reduction, object function. Then we can.

• 1:24:44Speaker 00

  Find what is the equivalent variation form of that option function, which is much easier to optimize. We end up with a, we call it a task, right? The, the computing, the covariance, the, the self attention step is only linear in the dimension, no longer quadratic, like the current attention is doing. Of course, if you look at the literature, there are other people have found, tried to identify linear complexity, such as, you know, Mamba or I think there's RWVK or something. So empirically, but again, it's through trial and error. But this is, so now we derive this in a math, purely mathematical way. Because we just find the equivalent variation form of the same option function.

• 1:25:29Speaker 00

  They have the same global optimal, but it's just much easier to optimize. This is the trick we do all the time, right? All the tricks, see, in the 200 years plus, years of developing better optimization algorithm, all those ideas can help us now to design better operator, descent operators, or optimization architectures to improve the design of current architectures. Pro, honestly, we have not really started that far, right? There's many acceleration techniques, preconditioning, conjugate gradient, which explore different landscape. Once we understand the landscape, the type, the class of object function better, there's gazillions of ideas.

• 1:26:14Speaker 00

  We can further improve the efficiency. Honestly, we haven't started that far, right? I mean, that's actually got some of my students excited to, to pursue this, realizing how much, how little we have done.

• 1:26:30Speaker 00

  From an optimization perspective, how much room there might still be for improvement. Some of my students are quite excited. So you can see, you know, even within the last couple of years, we already have two or three different.

• 1:26:45Speaker 00

  Generation of architectures that in the past it's almost unthinkable. Because the new generation always come from a different group, right? It's like a random process. Whoever gets lucky, maybe discovers something works. Try hard enough, get something to work.

• 1:26:59Tim Scarfe?

  It's a tantalizing idea, though, that through this principled optimization, there could be, you know, a convergent evolution towards the optimum architecture.

• 1:27:08Speaker 00

  Then the search will no longer be random, will be actually guided, right? There's still, just like back to your earlier suggestion, right? This is becoming intelligent search, it's, you know, guided search. We understand the the structure of the problems now. And hence, we can do science now. We are no longer just to do empirical inductive search process.

• 1:27:27Tim Scarfe?

  Why do OpenAI, they're still using the transformer, even though there are now superior architectures out there? And we should talk about this token statistics transformer. So, as you just said, it's a linear time complexity, which means in principle, this is something which is going to scale dramatically better than the kind of transformers we're using now. So why aren't we using it?

• 1:27:49Speaker 00

  Well, there are tempted to try to scale this up. In fact, even you can think about many of, of course, when you try to scale, the other factor becomes in, right? In terms of whether or not the scalability and so on. It's all related to to to all the design. And indeed, we actually tried something else, you know. For example, things are much more scalable. Also, I also tried to, we also scaled up with all the resource we have. Sometimes, I don't know about company, right? We we are very limited in resource. To verify even our architecture scales, we can only do up to probably a few, you know, couple hundred cars and so on, that's about it with our academic resources. And hopefully that will convince.

• 1:28:34Speaker 00

  But the one thing I think recently we did to simplify the current practice in Dino, right? You know, the meta has to, which is a pre-trained, the state of art, people, everybody talk about word model, visual model, and that's sort of the best model. The meta put a lot of effort, engineering effort, to pre-train the visual representation. Model, uh, which is still the best and they train on gazillions of images. And they were using contrastive learning, but it's a, it's a very remarkable engineering feat. And now people are using it, right? Um, it turns out actually we found that the system can be dramatically simplified. Once we realize what the purpose, what they're actually really trying to do, right?

• 1:29:17Speaker 00

  We have a work called the C Dino, simplified the Dino, a version one, version two, we simply have both version. Their, the architecture is dramatically, so we get rid of, you know, dozens of hyper parameters. Don't have to do and architecture become, you know, extremely, extremely 10 times simpler. And then the performance is better. We managed to scale up to probably a few, few hundred millions of scales. Their apple to apple comparison were dramatically much easier to train, much efficient. Everything is sustainable. I think that has a serious draw attention uh from the meta team and also from the Google team. And it's currently they are, they are, I, I know there are serious effort that they're trying to scale this, the new architectures up.

• 1:29:59Tim Scarfe?

  Yes, um, we interviewed the Dino folks at the time and we've spoken to people like Ishan Misra. There's, I mean, there's a potential tangent there about they're using this kind of um uh, non-contrasting self-supervised learning. And also there's the whole unsupervised thing and how useful those representations are for downstream tasks. Maybe, maybe we could go there, but I should say that Kevin Murphy, I'm, I'm interviewing, interviewing him soon. And I know that he reviewed your, your book very, very carefully. And he asked me to give you this question. He said, code reduction is great, but must be subject to prediction or reconstruction loss in data space. How would you go beyond token prediction, which seems especially weird for images? So that, that's what Kevin asked me to ask you.

• 1:30:42Speaker 00

  So this is actually a great question, right? So in a, in a re-reduction, remember the, the lossiness is actually coded through the, the epsilon ball. We're actually uh, uh, try to capture the, uh, the, the samples, how they connect with one another. Right now, we actually, if we just minimize it. The coding representation uh through this uh uh lossy coding. And the error is kind of controlled by the epsilon, but not enforced, right? So we respect the epsilon um at the through this lossy coding process. Now, to truly ensure, remember, everything could go wrong. And also depends on the number of samples you have.

• 1:31:26Speaker 00

  Maybe the even you choose is wrong because the data does not have that density. So you be not be able to percolate. Hence, the reputation learn can be very, very funky. So now, in order to to to ensure that your your your your learned reputation. Distribution learn uh internally actually authentically reflect. The original distribution up to certain precision, you have to decode, right? There is a constant encoding decoding. Actually, our brain do that all the time, predictive coding and so on, so forth. How? And hence, that encoding decoding and to verify if there's error remained in your prediction, in your reconstruction matters a lot. Only that now, the question is, if we don't have the.

• 1:32:12Speaker 00

  Do I do we necessarily back to our earlier discussion, right? Do we really need to measure that error in the data space, in the original token space? Uh if we have that option, so be it, do that, right? Make the engineering simpler, make the. But if we really want to have a system just like a human to self-learn, just go out to observe, right? Uh with two eyes or with some sensors. Then we have to come up with a way to make sure that our sensing process is accurate enough so that we can do everything internally. We can predict, go back, wrap it back and observe, compare what we predicted and what we observed through same.

• 1:32:57Speaker 00

  Sensing channel, we compare that locally. In theory, I should prove, at least under idealistic cases, this is possible. We can minimize the error whilst we correct error. Hence, the internal representation will, the error in the token original data space will diminish. But under technical condition, under general condition, we still don't know. We actually have a paper prove that for the, when your data distribution is a mixture of subspaces. You can rigorous prove that's possible. And if the dimension of the subspace is low enough compared to the capacity of the perception process. Now, for general distribution, we believe this is true.

• 1:33:39Speaker 00

  This is actually how we'll be able to learn all the low dimensional dynamics structure in the natural data. In the motion, in the, in the predicted world. So I think this is something in the future we can discuss. But end to end works if you have the option to do so. Or if you have, you don't have that option, you have to figure out how to do this autonomously. Under what condition you can do this autonomously and allows you to do autonomously to reduce the error to almost zero.

• 1:34:11Tim Scarfe?

  We spoke about Dino, but another example would be V I T. We interviewed Lucas Bayer in Switzerland earlier this year, so he invented V I T. And if I understand correctly, Crate is now very, very close to V I T, but it's so much more principled. It's explainable and so on. How close are we to knocking V I T off the, off the leaderboard, if you like?

• 1:34:33Speaker 00

  In fact, I think in many of the comparison, we're already very close. If you compare, it's hard to compare apple to apple, but in terms of if you have similar parameters, we're very much on par. And also, by the way, we never really quite put much engineer effort into it. We just want to verify the concept. Indeed, there, and one thing come out of the V I, the Crate is that what we find is not only the architecture designs principled, but then once we did a training, right? The internal structure learned.

• 1:35:05Speaker 00

  Are both semantically, statistically, and geometrically very meaningful. Indeed, each head actually does learns, you know, similar structures or gets, basically each channel, each head be truly become expert of certain type of visual patterns. For example, legs of animals, ears of animals, faces of animals. So we see that very clearly with crates, but we don't observe that in the V I T. Of course, you can see V I T may learn. This is actually the, the, the interesting thing, right? Early days, people, I'm sure, large models, if they have redundancy, they definitely learn things internally. But it's very hard to see which part of that network learn the correct channels, learn the correct operators.

• 1:35:49Speaker 00

  Because it's embedded in a more redundant structure, right? So early days, people call this lottery, you know, lucky lottery or lottery ticket, right? It's somewhere in there, right? Then, then people try to distill, that justify you should distill, you should to actually be able to compress. Even people do this Laura thing, you know, all the post-processing, right? Justify that is necessary. And some people, you find after the post-processing, not only the network becomes smaller, the performance gets better, right? But, and so on, so forth. Now, probably we don't have to do that. At, at least, you know, the, the architecture does what it's supposed, what it's designed to do, right?

• 1:36:31Speaker 00

  And we can actually at least explain what each component is doing something statistically, geometrically very meaningful. And there are also results, if there's enough data, if your optimization is done, training is successful. That's those structure pops up naturally. The, the structure will do what they're designed to do.

• 1:36:53Tim Scarfe?

  And, and final question, many M L engineers and researchers watch the show. Given everything we've spoken about, how can they find out more about your work? And how can they get started building these kind of architectures?

• 1:37:06Speaker 00

  I think most of our architecture are open sourced on GitHub, including Crete, early RedoNet. Their RedoNet may not be as conceptual, but not very practical. Crete are also even tossed, all the codes are available. By the way, they are sort of a kind of academic implement. We don't, we'd never be able to have the resource to scale them up. Most are scaled up to GPT two or ImageNet twenty one, that's sort of we can afford. Dino, simply Dino is the one we scale the most, we exhaust a lot of resource. A little bit higher than that, but still no comparison to all this industrial scale at all. But I do believe that Meta and Google are doing something about Dino, simplify Dino.

• 1:37:51Speaker 00

  And the codes are there. And also, of course, even for the methodology, of course. This is one way, why we bite the bullet to roll the book in the past two years. We believe that although there's a series of papers, but we believe that for people to get a big picture, the most systematic introduction, we put together the books. Also, we open sourced it. And we will post, link all the data, all the code as well. We are also teaching the course. So all the, we actually will have students practice most of the new architectures method. So all those codes will be made publicly available. And so I think that might be a good entrance.

• 1:38:32Speaker 00

  If you want people want to learn the methodology, understand the theoretical chain of evidence, and also even the empirical chain of evidence. And I think the book is, has, is attempt to do that. We are already start organizing, we're not done yet, but we are start already organizing. If you find out chapter seven, we're already doing that in chapter seven to collect the theory seriously to, to all the real world data and the task such as image classification, image segmentation, and the pre-training, and even language, GPT two type scale, language models as well.

• 1:39:09Tim Scarfe?

  Professor Ma, it's been an absolute honor. Thank you so much for joining us today.

• 1:39:12Speaker 00

  Yeah, thank you very much.