Why Scale Will Not Solve AGI | Vishal Misra - The a16z Show

Vishal Misra returns to explain his latest research on how LLMs actually work under the hood. He walks through experiments showing that transformers update their predictions in a precise, mathematically predictable way as they process new information, explains why this still doesn't mean they're conscious, and describes what's actually required for AGI: the ability to keep learning after training and the move from pattern matching to understanding cause and effect. Timestamps 00:00 — Introduction 02:58 — LLM as Giant Matrix 08:24 — What Is In-Context Learning 13:00 — Bayesian Updating as Evidence 19:13 — Bayesian Wind Tunnel Tests 27:22 — Brains Simulate Causality 36:34 — Manifolds and New Representations 42:17 — Simulation as Short Program Read the full transcript here: https://www.a16z.news/s/podcast Resources: Follow Vishal Misra on X: https://x.com/vishalmisra Follow Martin Casado on X: https://x.com/martin_casado Stay Updated: If you enjoyed this episode, be sure to like, subscribe, and share with your friends! Find a16z on X: https://twitter.com/a16z Find a16z on LinkedIn: https://www.linkedin.com/company/a16z Listen to the a16z Show on Spotify: https://open.spotify.com/show/5bC65RDvs3oxnLyqqvkUYX Listen to the a16z Show on Apple Podcasts: https://podcasts.apple.com/us/podcast/a16z-podcast/id842818711 Follow our host: https://x.com/eriktorenberg Please note that the content here is for informational purposes only; should NOT be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security; and is not directed at any investors or potential investors in any a16z fund. a16z and its affiliates may maintain investments in the companies discussed. For more details please see http://a16z.com/disclosures.

• 0:00Speaker 0

  Anthropic makes great product. Clot code is fantastic. Cowork is fantastic. But they are crates of silicon doing matrix multiplication. They don't have consciousness, they don't have an inner monologue. You take an LLM and train it on pre-1916 or 1911 physics and see if it can come up with the theory of relativity. If it does, then we have AGI.

• 0:21Speaker 1

  Just today, by the way, Dario allegedly said that you can't rule out that they're conscious. You can rule out there.

• 0:29Speaker 0

  Come on. To get to what is called AGI. I think there are two things that need to happen. One is

• 0:39Speaker 1

  Michelle, it's great to have you in again. Great to be back. This is one of my favorite topics, which is um how do LLMs actually work? And I think that uh you in my opinion, you've done kind of the best work on this, modeling it out.

• 0:52Speaker 0

  Thank you.

• 0:53Speaker 1

  For those that did not see the original um one, maybe it's probably worth doing just a quick background on kind of what led you to this point, and then we'll just go into the current work that you've been doing.

• 1:04Speaker 0

  Five years ago

• 1:05Speaker 1

  when GPT 3 was

• 1:07Speaker 0

  first released. Uh I got early access to it, and I started playing with it and I was trying to solve a problem related to querying a cricket database.

• 1:19Speaker 1

  Yeah.

• 1:19Speaker 0

  And I got GPD 3 to do in context learning, few short learning, and you know, it was kind of the first, at least uh uh to me it was the first known uh implementation of RAG retrieval augmented generation, which I used to solve this problem of uh querying, getting GPD 3 to translate natural language into something that could be used to query a database that GPD3 had no idea about. I had no access to GPD 3's internals, but I was still able to use it to solve that problem. So it it it worked beautifully. Uh we we deployed uh this uh in production at ESPN in September 21. Wow.

• 2:02Speaker 1

  Wow, you did the first implementation of RAG in 2021.

• 2:07Speaker 0

  No, no, no. In 2020.

• 2:08Speaker 1

  2020.

• 2:09Speaker 0

  2020 I got it working, and by the time you talked to all the lawyers at ESPN and you know, productionized it, it took it took a while. But October 2020, we had well, I had this architecture working. But after I got it to work, I was amazed that it worked. I wanted to understand how it worked. And I looked at you know, the attention is all your date papers and all the other sort of deep learning architecture papers, and I couldn't understand why it worked.

• 2:37Speaker 1

  Yeah.

• 2:38Speaker 0

  So then I started getting sort of uh deep into building a mathematical model.

• 2:43Speaker 1

  Yeah.

• 2:44Speaker 0

  And

• 2:44Speaker 1

  now you published a series of papers. The first one that I read is the one where you had kind of your matrix kind of abstraction. So maybe we'll talk about that and then we'll talk about the more recent work. So perhaps we'll just start with the first one, which is you were trying to you you're trying to describe you're trying to come up with a mathematical model with how LLM works.

• 3:04Speaker 0

  Yeah.

• 3:04Speaker 1

  And you had which was very helpful to me, which was um and at the time you were actually trying to like figure out how in-context learning was working.

• 3:11Speaker 0

  Yes, yeah. And

• 3:11Speaker 1

  you came up with an abstraction for LLMs, which is basically this very, very large matrix, and you use that to describe. So maybe you can kind of walk through that work very

• 3:18Speaker 0

  quickly. Sure. Yeah. So what you do is you you imagine this huge gigantic matrix where every row of the matrix corresponds to a prompt.

• 3:29Speaker 1

  Yeah.

• 3:30Speaker 0

  And the way these LLMs work is given a prompt, they construct a distribution of probabilities of the next token. Next token is next word. So every LLM has a vocabulary, you know, GPD and its variants have a vocabulary of about 50,000 tokens. Yeah. So given a prompt, it'll come up with a distribution of what the next token should be, and then all these models sample from that distribution. Yeah. So that's the posterior distribution. That's a posterior distribution, right? Right? That that's how LLMs work. And so the idea of this matrix is matrixes for every possible combination of tokens, which is a prompt, there's a row. Yeah. And the columns are a distribution over the vocabulary. So if you have like a vocabulary of 50,000 possible tokens, it's a distribution over those 50,000 tokens.

• 4:17Speaker 0

  And by distribution is just the probability. Just the probability, sorry, yeah, just the probability that uh the next token should be this versus that. Yeah. Uh so that that's sort of the idea. And and when you start viewing it that way, it makes things at least uh clearer to uh people like me who want to model it, uh, what what's happening. So concretely, let's say you have an example that uh let's say your prompt is just one word, protein.

• 4:45Speaker 1

  Yeah.

• 4:45Speaker 0

  So if you look at the distribution of the next word, next token after that, uh most of the uh probabilities would be zero. But you'd have non-zero, non-trivial probabilities on let's say two words. One is synthesis, the other is shake. Yeah, right. And now the LLM is going to sample synthesis, uh sample uh this next token and pick synthesis or shake.

• 5:12Speaker 1

  Yeah.

• 5:12Speaker 0

  Or you as a human will give the prompt protein shake or protein synthesis. Now, depending on whether you pick synthesis or shake, the next that row looks very different, right? If you pick protein synthesis, the terms that would have a high probability would be all concerned with biology, right? But if you pick pro protein shake, it'll all be about gyms and exercise and all, you know, bodybuilding stuff. So that synthesis or shake completely changes what comes next. Yeah. So this is an example of uh you can say Bayesian updating. You start with protein, you have a prior that after protein, this is going to happen. As soon as you get new evidence, then the next term is synthesis or shake, you completely update the distribution.

• 6:05Speaker 0

  So now you can imagine that the whole the the entirety of LLMs is this giant matrix where you have every row protein, protein shake, protein synthesis, the cat sat on the you know, humpty dumpty, blah, blah, blah. Now, given uh the vocabulary of uh these LLMs, let's say 50,000 and the context window. So GPD, for instance, uh chat GPD the first version had a context window of 8,000 tokens. Yeah. If you look at all possible combinations of 8,000 tokens and 50,000 uh uh vocabulary, the number of rows in this matrix is more than the number of electrons across all galaxies, right? So uh so there's no way that these LLMs can represent it exactly.

• 6:52Speaker 0

  Now, fortunately this matrix is very sparse. Why? Because you know, an arbitrary combination of these tokens is gibberish. Yeah. We're not never going to use that in natural in real life.

• 7:06Speaker 1

  Yeah.

• 7:06Speaker 0

  Also, the columns are also mainly zero. Yeah. Right. If you have protein, then you won't have lots of, you know, you won't have arbitrary numbers or arbitrary words after that. It's very sparse both in rows and in columns. So i i in kind of an abstract way, what all these LLMs are doing is coming coming up with a compressed representation of this matrix. And when you give a prompt, they try to approximate what the true distribution should have been and try to generate it. That that's what uh in my mind at least at bolts of view. And

• 7:46Speaker 1

  just from my understanding, so if you have a row of uh protein, and then you have one with protein shake. Is protein shake a subset of proteins or is it different?

• 8:00Speaker 0

  It's different. It's a continuation from. I

• 8:03Speaker 1

  see.

• 8:03Speaker 0

  Yeah.

• 8:04Speaker 1

  Right. No, but I'm just saying like the actual the

• 8:06Speaker 0

  actual posterior distribution, is that a subset? You you can say it's a subset, right? Uh if you have protein, then protein shake and protein synthesis are all continuations from protein. So both synthesis and shake have non-zero probabilities. So you can, yeah. You can think of it as somewhat a

• 8:24Speaker 1

  subset.

• 8:24Speaker 0

  Right.

• 8:25Speaker 1

  You you know, you use this approach to describe how in-context learning works. And so maybe first describe what in context

• 8:33Speaker 0

  learning is, and

• 8:34Speaker 1

  then kind of the

• 8:34Speaker 0

  conclusion that

• 8:35Speaker 1

  you

• 8:35Speaker 0

  came from that. So in-context learning is when you uh show the LLM something it has kind of never seen before. You give it a few examples of this is what it wants, uh this is what you're trying to do. Then you give a new problem which is related to the examples that I have shown.

• 8:55Speaker 1

  Yeah.

• 8:56Speaker 0

  And the LLM run learns in real time what it's supposed to do and solves the problem. By

• 9:02Speaker 1

  the way, but the first time I saw this, it absolutely blew my mind. Yeah, actually, I actually use your DSL when I was like first learning about it. So maybe like kind of like a

• 9:13Speaker 0

  DSL thing is just

• 9:15Speaker 1

  uh this works at all.

• 9:17Speaker 0

  It's absolutely, you know, uh mind blowing that it works. And so going back to that cricket problem was you know uh in the mid 90s, uh I was part of a group that had created this uh cricket portal called Crick Info. Yeah. Uh cricket uh is a very star. You know, you think baseball multiply by a thousand, and it's at all kinds of stats. And we had created this uh online searchable database called Statsguru, where you could search for anything, any star related to cricket, and it's been available since 2000.

• 9:52Speaker 1

  Yeah.

• 9:53Speaker 0

  But because you can query for anything, everything was be made available, and how do you make something like that available? To the general public. Well, they're not going to write SQL queries. The next best thing at that time was to create a web form. Unfortunately, everything was crammed into that web form. So as a result, you had like 20 drop-downs, 15 checkboxes, 18 different text fields. It looked like a very complicated, daunting interface. So as a result, even though it could solve or it could answer any query, almost no one used it. A vanishingly small percentage of cricket fans use it because it just looked intimidating. And then ESPN bought that site in 2007. I still know people who run the site, and I always told them, you know, why don't you do something about Stats Guru?

• 10:42Speaker 0

  And in January 2020, uh the editor-in-chief of uh Crick Info, Sambedbal, he's a friend. So he came to New York and we had gone out for drinks. And again, I told him, you know, why don't you do something about Stats Guru? So he looks at me and says, why don't you do something about Stats Guru? He was joking. But uh that idea kind of stayed with me. And when GPT 3 was released, I thought maybe I could use Stats Guru. Uh GPT 3 to create a front end for Stats Guru. And so what I did was uh I designed uh DSL, a domain-specific language, which uh converted queries about cricket stats in natural language into this DSL. No. And to be clear, you created this. It wasn't like part of like any training. No training

• 11:27Speaker 1

  online, yeah, like he could have seen.

• 11:29Speaker 0

  Nothing GPT could have seen. I created it. I thought, okay, this makes sense. Yeah. So I designed that DSL and then I did that few shot learning thing. So I would uh so I created about a database of what I would say of 1500 natural language queries and the DSL corresponding to that query. So when a new query came in, somebody's asking a stats question in English, what I would do is I would go through the natural language queries, do a semantic search, pick the most closely matching top few.

• 12:04Speaker 1

  Yeah.

• 12:04Speaker 0

  Uh and then use that natural language query and its DSL and send that as a prefix. Now GPD3, if you recall, had a uh context window of only 2,000 tokens. Yeah. So you had to be very judicious about which examples that you picked. But you pick that, and then you send the new query, and GPD 3 would complete it in the DSL that I had designed, which until milliseconds ago it had never seen.

• 12:28Speaker 1

  Yeah.

• 12:28Speaker 0

  And I had no access to internals of GPT 3. I had no access to the weights. Yeah. But still it worked. So that that's how. So

• 12:36Speaker 1

  it's not obvious to me given your matrix example of like a prompt and then a distribution, how something like in-context learning

• 12:45Speaker 0

  works. Would work. And so like

• 12:47Speaker 1

  I think your first paper tackled this problem.

• 12:50Speaker 0

  Right.

• 12:51Speaker 1

  Um so maybe you could walk through your understanding of how LLMs do in

• 12:59Speaker 0

  talks in context learning. Yeah. So when you think about what in context learning is, is that as you see evidence. So, you know, uh in the first paper, what I also did was I I took this Cricket DSL example.

• 13:13Speaker 1

  Okay.

• 13:14Speaker 0

  And I uh uh I depicted the next token probabilities of the model as it was shown more and more examples. So the first time you show it this DSL, the natural language and the DSL, the probabilities of the DSL tokens were were extremely low. Because GPD3 had never seen this thing. When it saw the cricket question, in its mind it was trying to continue it with an English answer. So the probabilities that were high were all English words. Yeah. Once it saw my prompt where I had the question and the DSL, the next time I had the question in the next row, the probabilities of the DSL tokens started going up.

• 14:01Speaker 0

  With every example it went up. And finally, when I gave the new query, it was like it had almost 100% probability of getting the right token.

• 14:13Speaker 1

  Yeah.

• 14:13Speaker 0

  So this is an example of in real time the model was updating its posterior probability. It was updating its knowledge that okay, I've seen evidence, this is what I'm supposed to do. Now, this is a colloquial way of saying what Bayesian inferences. Bayesian updating basically is you start with a prior, when you see a new evidence, you update your posterior. That's the mathematical division. But but uh in in English, it's basically you see something, you see new evidence, you update your belief about what's happening.

• 14:47Speaker 1

  Yeah.

• 14:48Speaker 0

  So it was clear to me that LLMs are doing something which resembles Bayesian updating. So in that first paper, I had this matrix formulation, and I showed that you know what it's doing. It looks like Bayesian updating. Yeah. Then we can come to the sort of next series of papers. That's right.

• 15:07Speaker 1

  So okay, so I mean it it's it seemed pretty conclusive to me at that time. And then you went quiet for a while, and then I I still remember the WhatsApp as you said, Martin, I know exactly how these things are working now. Yeah. Well, and then and then, and then listen, you dropped a series of papers that kind of broke the internet. Like you went super viral on Twitter, like I mean, people really noticed. Um so I I want to get to that in just a second. But before that, um I remember when your first paper came out, people would be like, you know, these things are definitely not Bayesian, like, you know, ever you know anything could be considered to be Bayesian, but they're not. Like, why do you think that there was this reaction to like there's something new, they're not Bayesian.

• 15:53Speaker 0

  I mean, I felt like there's almost kind of a

• 15:54Speaker 1

  backlash just because it being characterized BS.

• 15:57Speaker 0

  Yeah, yeah. I I I think uh the this whole world of uh probability and machine learning, that there have been camps of Bayesian and frequentists. Yes. And I don't want to get in the middle of that sort of political battle, but Bayesian has become like almost like people had a reaction to that. It's it's part of that war.

• 16:17Speaker 1

  I see. So it's like the old Bayesian frequentist type battle. Yeah. So the people

• 16:23Speaker 0

  just had, oh no, and you can say anything is Bayesian, right? So I I said, okay, maybe they have a point. Maybe what we are seeing is not really Bayesian. How do we prove that it's Bayesian?

• 16:33Speaker 1

  Right.

• 16:34Speaker 0

  So then first I have to thank you and uh Andres and Horowitz for this. Uh you know, when I when I when I said that I uh in my first paper, I showed these probabilities. Yeah. Uh it was because OpenAI had in its charge uh interface uh this option to displace those probabilities. Then they stopped.

• 17:00Speaker 1

  So

• 17:00Speaker 0

  we could not peer inside what's going what's happening. For some reason they stopped. OpenAI. I'm not going to get into the open and closed joke, but but they stopped.

• 17:10Speaker 1

  So

• 17:10Speaker 0

  then we developed our own interface which could let you look not only at uh the probabilities but also the entropy of the next token.

• 17:18Speaker 1

  Well was this on top of an open source model?

• 17:20Speaker 0

  Yeah, yeah. So you can load any sort of open source model, but you know, being an academia we didn't have access to compute, thanks to your generous uh uh donation, we got uh the clusters to run uh what's called token probe. So you can go to token probe.cs.columbia.edu.

• 17:39Speaker 1

  Is it still running?

• 17:39Speaker 0

  It's still running. It's still running, and people come to it. Uh I use it in my classes uh to get students to do assignments, they write their own DSLs and you know, they say that that it really helps them understand how these LLMs work.

• 17:52Speaker 1

  So I literally my understanding of LLMs came from token probe. So you know, sit there and just look at the distribution as you filled out uh a prompt. It's actually very, very enlightening.

• 18:01Speaker 0

  So for those of you that are listening, um what's

• 18:04Speaker 1

  the URL again?

• 18:05Speaker 0

  Token probe.

• 18:06Speaker 1

  Token

• 18:06Speaker 0

  probe.cs.columbia.edu. Yeah, check it out. It's actually very, very useful way

• 18:12Speaker 1

  to like actually see how the probability distribution gets updated is as you fill out a prompt.

• 18:18Speaker 0

  Right. Yeah. But then I cheated.

• 18:21Speaker 1

  Oh.

• 18:22Speaker 0

  I, you know, it was running. Yeah, but I also had access to the GPUs that were powering it. And then along with colleagues at Columbia, and one of them now is uh is at DeepMind. We started to sort of think about how do you really prove that it's Bayesian. To prove you just

• 18:46Speaker 1

  explain it. Actually, I I actually don't know the answer to this. Yeah. It seemed to me you proved it in the first paper. Like what was missing? Well, in the first paper,

• 18:54Speaker 0

  we showed it. It was empirical. Uh and you could see I see,

• 18:57Speaker 1

  I see. You could see not a mathematical. Because it was not a matter of this to me that's a good idea. Yeah, it was even obvious to

• 19:02Speaker 0

  me. But to convince uh you you could say, you know, people who dismiss it over anything can be Bayesian. I see. We had to show it precisely mathematically. Got it. So then we came up with this idea, you know, my colleagues uh at Namanagarwal and Siddhart Dalal. We uh the series of papers were were written with them. We came up with this idea of a Bayesian wind tunnel. Okay. So what's a wind tunnel? Well, wind tunnel in the aerospace industry is where you test an aircraft in an isolated environment. You don't fly it and you test it against all sorts of uh uh you know uh aerodynamic pressure, then you see what it'll withstand what kind of altitude, pressure, blah, blah, blah. And you don't want to do it up in the air testing.

• 19:47Speaker 1

  Yeah.

• 19:48Speaker 0

  So we said, okay, why don't we create an environment where we take these architectures and we tested Transformers, Mamba, LSTMs, uh, MLPs, all are Architectures. We say, why don't we create, take a blank architecture, give it a task where it's impossible for the architecture to memorize what the solution to that task should be. The space is combinatorially impossible for given the number of parameters, and we took very small models. So it's difficult enough that they cannot memorize it. But it's tractable enough that we know precisely what the the Bayesian posterior should be.

• 20:33Speaker 0

  You can calculate it analytically. So we gave these models a bunch of tasks where again we show that it's impossible to memorize. We train these models and we found that the transformer got the precise Bayesian posterior down to 10 to the power minus three bits accuracy. It was matching the distribution perfectly. So it is actually doing Bayesian in the mathematical sense, given a task where it has to update its belief. LSTMs can do one of the things. So in the papers we have a taxonomy of Bayesian tasks.

• 21:18Speaker 0

  And MLPs fail completely. So is this reflection

• 21:22Speaker 1

  of the data that it's trained on, or is it more reflection of the mechanism?

• 21:29Speaker 0

  It's the mechanism, it's the architecture. The data decides what tasks it learns.

• 21:35Speaker 1

  Right.

• 21:35Speaker 0

  So in the first paper, we had these Bayesian wind tunnels, and we show that you know it's doing the job. We're different tasks. In the second paper, we show why it does it. So we look at the transformers, we look at the gradients, and we show how the gradients actually shape this geometry, which enables this Bayesian updating to happen. Then in the third paper, what we did, we take we took these frontier production LLMs, which have open weights so that we could look inside them. And we did our testing, and we saw that the geometries that we saw in the small uh models persisted in models which are you know hundreds of millions of parameters. The same signature existed.

• 22:21Speaker 0

  The only thing is that uh because you are trained on all sorts of data, it's a little bit dirty or messy.

• 22:29Speaker 1

  Yeah.

• 22:29Speaker 0

  But you can see the same structure. So the the whole idea behind the Bayesian wind tunnel was unlike these uh production LLMs, where you don't know what they have been trained on.

• 22:39Speaker 1

  Right.

• 22:39Speaker 0

  So you cannot mathematically compute the posterior. So again, how do you prove it? I mean, it looks Bayesian, you know, from the first time.

• 22:47Speaker 1

  From the first, it looks Bayesian, but you know, yeah. So

• 22:49Speaker 0

  the wind tunnel sort of solved that problem for us. You said, okay, let's start with a blank architecture, give it a task where we know what the answer is. It cannot memorize it. Let's see what it does. Yeah.

• 23:01Speaker 1

  So do you think this provides any sort of like indication of how humans think, or do you think that these things are totally independent? No, no, it

• 23:08Speaker 0

  it does provide, right? So, you know, human beings also uh update our beliefs as we see new evidence. Okay, right. So we do in some sort of uh in some sense uh Bayesian updating, but we do something more than that. I I'll come to that. But uh these transformers uh or even Mamba do this Bayesian updating. Yeah. And uh but but but the difference with humans is you know, we we'll update our posterior when we see some new evidence. But the way our brains have evolved over hundreds of millions of years, is our optimization objective has been don't die and reproduce, right? That's been sort of the driving force, and our brains have learned to adjust.

• 23:58Speaker 0

  And so when we see some danger, there's some something rustling in that bush, don't go near. We know how to react to that danger, we know how to s uh save ourselves. We internalize that learning, and our brain cells or our synapses remain plastic throughout our lifetime. What happens with LLMs is once the training is done, those weights are frozen. When you're doing an inference, for instance, in-context learning or anything, during that conversation, okay, you're doing Bayesian inference, but then you forget. The next time a new conversation starts with zero context, you don't retain any learning that happened in the previous instance.

• 24:48Speaker 0

  So for instance, with the cricket DSL that I was doing, every invocation of it was fresh. It did not remember the last time I sent a query, what the DC looked like. So that's one difference between uh how humans uh uh use sort of Bayesian uh updating, which is we remain plastic all our lives. Whereas uh LMs are frozen. And there's another uh sort of difference, which uh if you want me to give it away. Coming, yeah, yeah, yeah. Yeah. So the other difference is uh uh well for first, you know, our objective is don't die, reproduce. LLM's objective is predict the next token as accurately as possible, right?

• 25:35Speaker 0

  So all these uh scary stories that uh you you read about that oh the LLM tried to deceive and it tried to prevent itself from being shut down. That's not a function of the architecture. That's a function of the training data. It has been fed, you know, articles on Reddit or SMO or whatever. I mean, just today,

• 25:59Speaker 1

  by the way, Dario allegedly said that uh you can't rule out that they're conscious. You can rule out their costs.

• 26:08Speaker 0

  I

• 26:08Speaker 1

  think I

• 26:09Speaker 0

  mean

• 26:09Speaker 1

  come on.

• 26:10Speaker 0

  As I said, you know, uh anthropic makes great products. Cloud code is fantastic, cowork is fantastic, but they are great of silicon doing matrix multiplication. They don't have consciousness, they don't have an inner monologue, they don't uh they're not driven by the same objective function. Don't die, reproduce, right? They're driven by don't make a mistake on the next token. And that's driven entirely by the training data.

• 26:38Speaker 1

  Right?

• 26:39Speaker 0

  You train the LLM with stories of Asimo or Reddit where you know to survive it's going to do this or that, it'll reproduce that. So it's it it's a reflection, it's not a mind. And and the results just to say it for the tenth time are perfectly Bayesian perfectly, yeah.

• 26:57Speaker 1

  To the to the to the digit.

• 26:59Speaker 0

  To the

• 27:00Speaker 1

  digit.

• 27:00Speaker 0

  Yeah. I mean, I I trained it for 150,000 steps, and uh the accuracy was 10 to the power minus three bits. That's all right. I could have trained it for, you know, this happened in half an hour on the infrastructure that you provided for token property. In the background, I I could use those APUs to train. But uh so thank you again for that. But so now human beings coming back to it, we we are Bayesian,

• 27:24Speaker 1

  yeah,

• 27:24Speaker 0

  but we do something else. You know, when I when I when I throw this pen at you, what will you do?

• 27:30Speaker 1

  Dodge it, dodge it. Yeah.

• 27:31Speaker 0

  Why will you dodge it?

• 27:34Speaker 1

  To avoid being hit.

• 27:36Speaker 0

  Avoid being hit, but your head is not doing a Bayesian calculation of okay, this pen is coming, the probability that it hits me, uh, it'll cause this much pain or all that. What you're essentially doing in your head is you're doing a simulation. You see the uh the pen coming and you know that it'll come and hit me. Your mind simulates and you dodge it, right? So all of deep learning is uh doing correlations. It's not doing causation.

• 28:11Speaker 1

  Yeah.

• 28:12Speaker 0

  Causal models are the ones that are able to do simulations and intervention. So, you know, Judea Pearl has this whole uh causal hierarchy where the first hierarchy in the first hierarchy is association, which is you build these correlation models. Deep learning is beautiful, it's extremely powerful. I mean, you see every day, all these models are like uh amazingly good.

• 28:34Speaker 1

  Yeah,

• 28:35Speaker 0

  they do association. The second is intervention in the hierarchy. Yeah. Deep learning models do not do that. Third is counterfactual. So both intervention and counterfactual, you can imagine it, it it's some sort of simulation. You you build a model of causal model of what's happening, and then you are able to simulate. So our brains do that. The current architectures don't do that. Another example, I think which will make it clear is uh the difference between uh I'll use these technical terms, Shannon entropy and Kulmogrow complexity. Sure. So if you look at the Shannon entropy of the digits of pi, yeah, it's infinite. Sure. It's impossible to predict and learn what digit will come after.

• 29:24Speaker 0

  Yeah. So that's the definition of Shannon entropy. And Shannon entropy it's sort of tries to build a correlation. It tries to learn the correlation, deep learning does the Shannon entropy. Gulmogrowth complexity, on the other hand, is the l is the length of the shortest program which will reproduce uh the string that you that is under question. Yeah. Now the program to get the digits of pi are very small.

• 29:53Speaker 1

  Yeah.

• 29:54Speaker 0

  Thanks to Ramanagement. There are all sorts of really small programs that can reproduce. It exactly. So the Kulmogrove complexity of pi is very small. Shannon entropy is infinite. I think deep learning is still in the Shannon entropy world. It has not crossed over to the Kulmogrove complexity and the causal world.

• 30:16Speaker 1

  Interesting.

• 30:17Speaker 0

  Right. So

• 30:20Speaker 1

  to what extent do you think this provides us research directions to kind of improve the state of the arms? So let me just give you a specific example. You talked about human beings don't actually update, you know, the matrix. They don't kind of update their weights. But right now there's a lot of research on continual learning. Yeah. So does your work provide some guidance of how you might approach those problems? And in particular, I've always had this question, which is we use so much data and so much compute to create these models. Like is it even reasonable to think that you can update the weights and actually have a meaningful impact, you know, with in in real time. I mean, it just seems like you just need so much more data in order to

• 31:04Speaker 0

  do

• 31:04Speaker 1

  that. So

• 31:05Speaker 0

  can you start answering these questions? You can start answering some of these questions. And one of the misconceptions that uh exists today is that scale will solve everything. Scale will not solve everything. You need a different kind of architecture. And this continual learning is a difficult problem. You have to balance the fact that you learn something new against the risk of catastrophic forgetting.

• 31:27Speaker 1

  Right.

• 31:28Speaker 0

  Right? Right. If you update the weights and you forget what was important and what you have already learned, then you are you know, you you're not making progress. Then it'll just be some sort of random chaotic model. So to solve that problem is difficult. That's one aspect of it. So, you know, to get to what uh is called AGI, I think there are two things that need to happen. One is this plasticity, which has to be implemented through container learning. Yeah. Secondly, we have to move from correlation to causation. Yeah. That's uh uh

• 32:04Speaker 1

  I I mean how how how how much is this similar to what Jan McKun talks about with uh causality planning, yeah, you know, predicting uh like how your action would be.

• 32:16Speaker 0

  It is related. You know, he is coming at it from a different angle than the JPER model.

• 32:20Speaker 1

  Right.

• 32:20Speaker 0

  But it is related. The the the other thing is uh, you know, the first time I came on this podcast, I I mentioned this test of AGI. Yeah, the Einstein test.

• 32:29Speaker 1

  I don't remember.

• 32:30Speaker 0

  So I I said, you know, uh uh you take an LLM and train it on pre-1916 or 1911 physics and see if it can come up with the theory of relativity.

• 32:44Speaker 1

  Yeah.

• 32:44Speaker 0

  If it does, then we have AGI. I mean it's a high bar, but you know, we should have high bars. It won't. And this is the same test that I think Demis uh mentioned uh at uh the India AI Summit a couple of weeks ago, it's created a lot of news. But why is that and how is that related to this idea of Shannon versus Quilmogro. So at the time of Einstein, there were a lot of clues. That Newtonian mechanics, there was something missing.

• 33:18Speaker 1

  Yeah.

• 33:18Speaker 0

  Right. Uh people knew that Mercury's orbit didn't make sense. There was something off about it. Then there were these experiments done, uh, the Michelson Morley experiments, where they were trying to figure out uh this uh uh medium called uh the ether through which light travels. And they felt that if you know you bounce light in different directions, uh the speed might change and they they could detect a change in the speed of light. They tried several experiments, they had really precise instruments which could measure the speed and they found nothing. They found that the speed of light did not change at all. Then there were there was a whole issue of black holes.

• 34:07Speaker 0

  Yeah. Then gravitational lensing. So there were a lot of these signs that Newtonian mechanics is not really explaining everything.

• 34:17Speaker 1

  Yeah.

• 34:18Speaker 0

  But until Einstein came up with a new representation of the space-time container.

• 34:24Speaker 1

  Right. We were

• 34:26Speaker 0

  stuck. Yeah. So if you had a model that just looked at correlations and so uh sees all of this, you know, the uh all of these uh pieces of individual evidence and put together, it would not have come up with the beautiful equation that Einstein came up with, you know, uh I'm forgetting exactly what it is, G mu V equals 8 pi Tmu V, some something like that. Where you know uh the the the equation of uh the rel uh with the space-time container, the tensor. So he came up with a new formulation.

• 35:01Speaker 1

  Yeah.

• 35:02Speaker 0

  So he kind of rejected the existing axioms. He came up with a very short cholumograph representation of the world.

• 35:11Speaker 1

  Yeah,

• 35:12Speaker 0

  one equation. From that equation, everything else follows. Right? Whether you're talking about gravitational waves or black holes or mercury or how GPS works. You know, GPS, uh the GPS that we use every day in our phones, it uses the equation of relativity. So do you does this end up becoming

• 35:32Speaker 1

  like um you you almost have to ignore the majority of previous data in order to do it, which LLMs can't because they trained on the majority of previous data. You almost have like this kind of data gravity that's pulling you back. It's like it's like everybody said it's X. There's a little bit of evidence that it's Y, but because everybody said it's X, like the LM will always say it's X.

• 35:57Speaker 0

  It's it'll always say it'll treat that Y as an anomaly. Actually,

• 36:00Speaker 1

  this is actually a very nice way to say it, which is like it's like I just now okay. Now I get your Shannon entropy versus common problems. Like one of them is like the total amount of information there that will always be bound to the total amount of information there, which is what happens right now.

• 36:17Speaker 0

  Yeah,

• 36:18Speaker 1

  where you can actually describe another another motion where you you can describe everything with a shorter description with the new data, which would be a totally different work to which would be like

• 36:32Speaker 0

  you need a new representation, right? Yeah.

• 36:34Speaker 1

  You know, another way that I've always thought about these, and I thought you articulated it well in the last time we talked about it, which is the universe is this very, very complex space. And then you know, somehow humans map it into a manifold that's less complex.

• 36:50Speaker 0

  Yeah.

• 36:51Speaker 1

  And then that gets kind of written down, and then the LM. So that's kind of some some distribution, some, you know, it's still a very large space, but it's it's a bounded space. And the LM learn that manifold, and then they kind of use you know, Bayesian inference to move up and down that manifold. But they're kind of bound to that manifold.

• 37:09Speaker 0

  Yeah.

• 37:10Speaker 1

  And then again, I don't want to put words in your mouth, and then but like what they can't do is generate a new manifold. Right. Which requires understanding the way that the universe works and then coming up with a new representation of the universe.

• 37:21Speaker 0

  And this is what relativity is, right? Yeah, exactly. Einstein had to create a new manifold. Yeah,

• 37:26Speaker 1

  yeah, yeah.

• 37:26Speaker 0

  If you just stuck with the old manifold of the Newtonian physics, then you would see these correlations, but you could not come up with a manifold that explained them. So you need to come up with a new representation. So to me, you know, there are lots of definitions of AGI. Uh you know, Turing tests, we have already passed that. You know, performing economically useful work every day you see, you know, uh LMs are doing that. Do we? I don't know. Well,

• 37:52Speaker 1

  I mean, they are I mean, without human intervention. No,

• 37:55Speaker 0

  no, no. So that that's different. But but still, you know, it's like uh a car can run faster than humans. Yeah. I mean, that's uh

• 38:02Speaker 1

  that's the that's uh very shallow definition. Yeah, so all these definitions do useful. You know,

• 38:08Speaker 0

  maybe you know, in six months you'll have uh cloud or uh what a Gemini do without uh intervention, coding tasks, which are well defined, well scoped. That's possible. But to me, AGI will happen when these two problems get solved. Elasticity, continual learning properly, and building a causal model from you know uh i in a more data efficient manner.

• 38:36Speaker 1

  We are hearing people now talking about you know, seeing general like Donald Knuth, for example, in the last few days, right? You know, had this, you know, this you know, aha moment apparently then kind of made it went viral on X. So do you think that that suggests that we're seeing generality or No, no, no. So that

• 38:55Speaker 0

  actually to me it validates what I've been talking about for a while now.

• 39:01Speaker 1

  How sorry?

• 39:02Speaker 0

  So uh so if you if you read what he did uh with the help of uh you know a colleague, he got the LLMs to solve this particular problem of uh finding Hamiltonian cycles. Odd numbers, we won't get into that. And he got the LLMs to keep solving for one odd number after the other, right? What he also got to do is after it found a solution for a particular value of M, he made the LLM update its memory with exactly what it learnt in solving that problem. So the LLMs tried many different things. You know, something worked, update the m the memory. So that's kind of like hacking together plasticity. Yeah. Right? It's learning what it has done as we went along.

• 39:48Speaker 0

  Again, it's it's a hacked version of it. You're not changing the weights, you're just sort of uh uh uh uh improving the context. Right. Right? But you as you learned and Even after that, so this whole space of Hamiltonian cycles and uh the the associated math is well represented in the manifolds that these LLMs have been trained on.

• 40:12Speaker 1

  Right.

• 40:12Speaker 0

  You just had to have to find the right connection. And LLMs, I know compute, you throw enough compute, they will find the right connection. So was able to find the LLMs uh attempts. And eventually it needed him to put together what he saw into a solution. It definitely helped him get to the solution, but he had to create the new sort of manifold to come to the solution. The LLMs were after a while stuck. I mean, it just up the press uh I think two days ago. But uh eventually he used the solution and he came up with uh the proof.

• 40:58Speaker 1

  Yeah.

• 40:59Speaker 0

  So it's like, you know, it's like Einstein saw all these evidences. Then he thought what will explain uh he came up with a causal model. Yeah. So Canute and his brain is sort of the same

• 41:15Speaker 1

  thing that's in the commodograph.

• 41:18Speaker 0

  Right. And uh the LLMs are extremely efficient at doing the Shannon part of it. It found all the solutions by trying, you know, various things. And

• 41:27Speaker 1

  learning more and more. Clever way to decompose it. I'm wondering, like, do you think this again? I'm I would ask the uh the same question uh again, which is do you think this provides some sort of insight on like the next problem to tackle? Like, is there a mechanism that will get the Kalmagarov complexity or not? Like is this

• 41:45Speaker 0

  it tells us uh which direction

• 41:48Speaker 1

  to pursue but clearly not how to do it. Like not

• 41:50Speaker 0

  how to do it. Uh sort of a theoretical construct. Yeah, for sure. There's no algorithm. There's no uh there haven't been practical implementation of finding the shortest program. We know it exists, you know. You can argue about it, it but uh so that's where uh I think it's my bias, that's where our energy should be focused, not larger models with more tokens. Can you and can

• 42:18Speaker 1

  you can you tie the two things like how does that pair with doing simulation or is it a simulation totally orthogonal?

• 42:25Speaker 0

  No, so simulation

• 42:26Speaker 1

  is uh uh is it related, right? So you think it like basically you do simulation and somehow that is a step towards doing the comma growth complexity?

• 42:38Speaker 0

  The simulator is the is the program that we create. It may not be the perfect program.

• 42:43Speaker 1

  Oh, I see. But in our

• 42:44Speaker 0

  heads, we create this uh simulator that when I'm throwing the pen, you know that it's coming at you, right? And you duck. So so you're not computing the probabilities uh as it goes. But but you have you knew you you build a

• 42:57Speaker 1

  approximately versus we are talking more conceptually.

• 43:00Speaker 0

  Conceptually, but but it's a simple thing. And you think there's the same mechanism? It's the same mechanism. Really? Yeah. You have to build a causal model.

• 43:06Speaker 1

  Yeah, right? I see. For

• 43:08Speaker 0

  most things, right? So you have to move from correlation to causation. I mean, we've heard this term you know, fad infinitum. Yeah. But here it it's making a difference in the way we view intelligence.

• 43:22Speaker 1

  How how is how has the last three papers been received? No, I don't know. Well, uh,

• 43:29Speaker 0

  I mean the archive versions will be.

• 43:33Speaker 1

  Um a lot of great reception. A lot of people read it. I'm just

• 43:36Speaker 0

  wondering like what kind of feedback that you've got. I'm getting good feedback, but I'm an outsider in this field, right?

• 43:42Speaker 1

  Like networking guy. I'm a

• 43:44Speaker 0

  networking guy. Why why is he writing about you know uh learning and machine learning and deep learning and Bayesian? So but but people who have actually taken the time to read those papers, I'm getting really good feedback. Uh there was a recent paper by Google uh research which tried to teach uh LLMs by some sort of RL HF to do Bayesian learning properly. Yeah. And that's going in this direction. I think people are coming around to the view that okay, LLMs are doing Bayesian learning. Um I know that some people also looked at the Beijing Ventanal paper, the archive version, and they reproduced the experiments. That's great. Did they just saw what was written and they they did the training and they saw, yeah, yeah, this is actually happening.

• 44:30Speaker 0

  So that's great. So what's next? Uh what's next is uh, you know, these two parallel uh tracks, I hope to make progress there. Plasticity and cause causal cause. Because

• 44:41Speaker 1

  uh today you've taken an existing mechanism and you've created a formal model how it works. Yeah. And so now you're actually interested in the colour. Yeah. Yeah.

• 44:52Speaker 0

  Yeah.

• 44:52Speaker 1

  And do you think it's an entirely different

• 44:54Speaker 0

  architecture? I think

• 44:55Speaker 1

  or do you think do you think LLMs are like part of the solution? I

• 44:58Speaker 0

  think uh LLMs are definitely part of the solution. I see. But but there has to be something more. And

• 45:02Speaker 1

  other than that,

• 45:03Speaker 0

  so you know, I was not interested in sort of cataloging what all these LLMs can do.

• 45:08Speaker 1

  Yeah.

• 45:08Speaker 0

  I was more interested in why are they and how are they doing it. I think f now uh we have a good grip on the why and how. Yeah. And the next step is to, you know, uh move them to the next level. We we now I I think we have a fairly good understanding of what the limits are.

• 45:26Speaker 1

  Yeah.

• 45:26Speaker 0

  Now

• 45:27Speaker 1

  how do you uh go to the next step? Is there any is there an equivalent kind of theoretical framework for causality that applies here? Like similar to like Bayesian for inference.

• 45:40Speaker 0

  Well the Judea Pearl's whole causal causal hierarchy, I think. I think that's the right one. That's that's a very good one. You know, the whole do calculus uh approach uh uh I I think it's a good way to

• 45:52Speaker 1

  think about it. You know, uh the the sort of uh

• 45:55Speaker 0

  association intervention counterfactuals. Yeah. It takes you from correlation to actually simulation. Yeah.

• 46:03Speaker 1

  In a

• 46:03Speaker 0

  mathematical way.

• 46:05Speaker 1

  That's

• 46:05Speaker 0

  great.

• 46:05Speaker 1

  All right. Well, listen, really appreciate you coming. This is awesome. So we had you here for the first paper where you had the empirical results. Then we had you back when you actually have like the formal proof. And hopefully the next time you come back, you will have a proposal for the mechanism that uh that actually provides the next step. Hopefully. All right, working on it for coming in. Thank you for having

• 46:27Speaker 0

  me.