What's Missing Between LLMs and AGI - Vishal Misra & Martin Casado

Vishal Misra returns to explain his latest research on how LLMs actually work under the hood. He walks through experiments showing that transformers update their predictions in a precise, mathematically predictable way as they process new information, explains why this still doesn't mean they're conscious, and describes what's actually required for AGI: the ability to keep learning after training and the move from pattern matching to understanding cause and effect.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Anthropic makes great products. Plotcode is fantastic. Cowork is fantastic. But they are crazy of silicon doing matrix multiplication. They don't have consciousness. They don't have an inner monologue. You take an LLM and train it on pre 1916 or 1911 physics and see if it can come up with the theory of relativity. If it does, then we have AGI.

• 0:21Martin Casado?

  Just today, by the way, Dario allegedly said that you can't rule out that they're conscious.

• 0:27Speaker 00

  You can rule out their consciousness. Come on. To get to what is called AGI, I think there are two things that need to happen.

• 0:35Erik Torenberg?

  Five years ago, Vishal Misra got GPT-3 to translate natural language into a domain-specific language it had never seen before. It worked. He had no idea why. So he set out to build a mathematical model of how LLMs actually function. The result? A series of papers showing that transformers update their predictions in a precise, mathematically predictable way. In controlled experiments, the models match the theoretically correct answer almost perfectly. But pattern matching is not intelligence. LLMs learn correlation. They don't build models of cause and effect. To get to AGI, Misra argues, we need the ability to keep learning after training and the move from correlation to causation.

• 1:19Erik Torenberg?

  Martin Casado speaks with Vishal Misra, professor and vice dean of computing and AI at Columbia University.

• 1:28Martin Casado?

  Vishal, it's great to have you in again.

• 1:30Speaker 00

  Great to be back.

• 1:31Martin Casado?

  This is one of my favorite topics, which is how do LLMs actually work? And I think that, in my opinion, you've done kind of the best work on this modeling it out.

• 1:39Speaker 00

  Thank you.

• 1:39Martin Casado?

  For those that did not see the original one, maybe it's probably worth doing just a quick background on what led you to this point and then we'll just go into the current work that you've been doing.

• 1:50Speaker 00

  Five years ago, when GPT-3 was first released, I got early access to it. And I started playing with it and I was trying to solve a problem related to querying a cricket database. And I got GPT-3 to do in-context learning, few-shot learning, and it was kind of the...

• 2:12Speaker 00

  First, at least to me, it was the first known implementation of RAG, Retrieval Augmented Generation, which I used to solve this problem of querying, getting G P T three to translate natural language into something that could be used to query a database that G P T three had no idea about. I had no access to G P T three's internals, but I was still able to use it to solve that problem. So it worked beautifully. We deployed this in production at ESPN in September twenty one.

• 2:40Martin Casado?

  Wow. You did the first implementation of RAG in twenty twenty one?

• 2:44Speaker 00

  No, no, no, in twenty twenty.

• 2:45Martin Casado?

  Twenty twenty.

• 2:46Speaker 00

  Twenty twenty, I got it working. And by the time you talked to all the lawyers at ESPN and productionized it, it took a while. But October twenty twenty, we had, well, I had this architecture working. But after I got it to work, I was amazed that it worked. I wanted to understand how it worked. And I looked at the attention of all your deep papers and all the other sort of deep learning architecture papers, and I couldn't understand why it worked.

• 3:13Martin Casado?

  Yeah.

• 3:14Speaker 00

  So then I started getting sort of deep into building a mathematical model.

• 3:19Martin Casado?

  Yeah. And now you've published a series of papers. The first one that I read was the one where you had kind of your matrix kind of abstraction. So maybe we'll talk about that and then we'll talk about the more recent.

• 3:30Speaker 00

  Yeah.

• 3:30Martin Casado?

  Work. So perhaps we'll just start with the first one, which is you're trying to come up with a mathematical model, but how L L M works.

• 3:37Speaker 00

  Yeah.

• 3:37Martin Casado?

  And you have, which is very helpful to me. And at the time, you were actually trying to figure out how in context, learning was working.

• 3:42Speaker 00

  Yes, yeah.

• 3:43Martin Casado?

  And you came up with an abstraction for L L M's, which is basically a very large matrix and you use that to describe. So maybe you can kind of walk through that work very quickly.

• 3:50Speaker 00

  Sure, yeah. So what you do is you imagine this huge gigantic matrix where every row of the matrix corresponds to a prompt.

• 3:59Martin Casado?

  Yeah.

• 4:00Speaker 00

  And the way these L L M's work is given a prompt, they construct a distribution.

• 4:07Speaker 00

  Of probabilities of the next token. Next token is next word. So every L L M has a vocabulary, G P T and its variants have a vocabulary of about fifty thousand tokens. Yeah. So given a prompt, it'll come up with a distribution of what the next token should be. And then all these models sample from that distribution. Yeah, so that's the posterior distribution. That's a posterior distribution, right? That's how L L M's work. And so the idea of this matrix is for every possible combination of tokens, which is a prompt, there's a row. Yeah. And the columns are a distribution over the vocabulary. Yeah. So if you have a vocabulary of fifty thousand possible tokens, it's a distribution over those fifty thousand tokens. And by distribution is just the probability? There's the probability, sorry. Yeah, there's the probability that the next token should be this versus that. Yeah.

• 4:52Speaker 00

  So that's sort of the idea. And when you start viewing it that way, it makes things at least clearer to people like me who want to model it. What's happening? So concretely, let's say you have an example that, let's say your prompt is just one word, protein. Yeah. So if you look at the distribution of the next word, next token after that, most of the probabilities would be zero. But you'd have non-zero, non-trivial probabilities on, let's say, two words. One is synthesis, the other is shake. Yeah. Right? And now the L L M is going to sample this next token and then pick synthesis or shake. Yeah. Or you as a human will give the prompt, protein shake. Yeah.

• 5:37Speaker 00

  Or protein synthesis. Now, depending on whether you pick synthesis or shake, the next, that row looks very different, right? If you pick protein synthesis, the terms that would have a high probability would be all concerned with biology, right? But if you pick protein shake, it'll all be about gyms and exercise and all bodybuilding stuff. So that synthesis or shake completely changes what comes next. Yeah. So this is an example of, you can say, Bayesian updating. You start with protein, you have a prior that after protein, this is going to happen.

• 6:14Speaker 00

  As soon as you get new evidence, then the next term is synthesis or shake, you completely update the distribution. So now you can imagine that the whole, the entirety of L L M's is this giant matrix. Yeah. Where you have every row, protein shake, protein synthesis, the cat sat on the. Yeah. Humpty, dumpty, blah, blah, blah. Now, given the vocabulary of these LLM, let's say fifty thousand and the context window. So G P D, for instance, chat G P D, the first version had a context window of eight thousand tokens.

• 6:49Martin Casado?

  Yeah.

• 6:50Speaker 00

  If you look at all possible combinations of eight thousand tokens and fifty thousand vocabulary, the number of rows in this matrix is more than the number of electrons across all galaxies, right? So there's no way that these L L M scan represent it exactly. Now fortunately, this matrix is very sparse. Why? Because an arbitrary combination of these tokens is gibberish. We're never going to use that in real life. Also, the columns are also mainly zero. And if you have protein, then you won't have lots of, you know, you will have arbitrary numbers or arbitrary words after that. It's very sparse, both in rows and in columns.

• 7:33Speaker 00

  So in kind of an abstract way, what all these L L M's are doing is coming up with a compressed representation of this matrix. And when you give a prompt, they try to approximate what the true distribution should have been and try to generate it. That's what, in my mind at least, it works on, too.

• 7:55Martin Casado?

  And just from my understanding, so if you have a row of protein and then you have one with protein shake. Is protein shake a subset of protein? Or is it different?

• 8:09Speaker 00

  It's different. It's a continuation from.

• 8:12Martin Casado?

  I see.

• 8:12Speaker 00

  Yeah.

• 8:13Martin Casado?

  But no, but I'm saying like the actual posterior distribution, is that a subset?

• 8:17Speaker 00

  You can say it's a subset, right? If you have protein, then protein shake and protein synthesis are all continuations from protein. So both synthesis and shake have non-zero probabilities. So you can, yeah. You can think of it as somewhat a subset, right?

• 8:33Martin Casado?

  You use this approach to describe how in-context learning works. And so maybe first to describe what in-context learning is, and then kind of the conclusion that you came from that.

• 8:44Speaker 00

  So in-context learning is when you. Show the L L M something it has kind of never seen before. You give it a few examples of this is what it wants. This is what you're trying to do. Then you give a new problem which is related to the examples that I have shown.

• 9:02Martin Casado?

  Yeah.

• 9:02Speaker 00

  And the L L M learns in real time what it's supposed to do and solves a problem.

• 9:08Martin Casado?

  By the way, the first time I saw this, it absolutely blew my mind. I actually used your D S L.

• 9:13Speaker 00

  Mm hmm.

• 9:13Martin Casado?

  When I was like first learning about it, so maybe.

• 9:16Speaker 00

  Yeah.

• 9:16Martin Casado?

  The D S L thing was just crazy, this works at all.

• 9:20Speaker 00

  It's absolutely mind blowing that it works. And so going back to that cricket problem was, you know, in the mid nineties, I was part of a group that had created this cricket portal called Cricket Info.

• 9:32Martin Casado?

  Yeah.

• 9:33Speaker 00

  Cricket is a very star rated sport. You think baseball multiplied by a thousand and it's at all kinds of stats. And we had created this online searchable database called Stats Guru, where you could search for anything, any stat related to cricket and has been available since two thousand.

• 9:50Martin Casado?

  Yeah.

• 9:51Speaker 00

  But because you can query for anything, everything was made available. And how do you make something like that available to the general public? Well, they're not going to write SQL queries. The next best thing at that time was to create a web form. Unfortunately, everything was crammed into that web form. So as a result, you had like twenty dropdowns, fifteen checkboxes, eighteen different text fields. It looked like a very complicated, daunting interface. So as a result, even though it could solve or it could answer any query, almost no one used it, a vanishingly small percentage of cricket fans used it because it just looked intimidating. And then ESPN bought that site in two thousand and seven.

• 10:34Speaker 00

  I still know people who run the site and I've always told them, you know, why don't you do something about Stats Guru?

• 10:42Speaker 00

  And in January twenty twenty, uh, the editor in chief of Cricket Info, Sambit Bhargan, he's a friend, so he came to New York and we had gone out for drinks. And again, I told him, you know, why don't you do something about Stats Guru? So he looks at me and says, why don't you do something about Stats Guru? He was joking, but uh, that idea kind of stayed with me. And when G P T three was released, I thought maybe I could use Stats Guru, use G P T three to create a front end for Stats Guru. And so what I did was, uh, I designed a D S L, a domain specific language, which uh, converted queries about cricket stats in natural language into this D S L. Now.

• 11:21Martin Casado?

  And to be clear, you created this, it wasn't like part of like any training data that was online that like G P T could have seen.

• 11:27Speaker 00

  Nothing G P T could have seen. I created it. I thought, okay, this makes sense.

• 11:30Martin Casado?

  Yeah.

• 11:30Speaker 00

  So I designed that D S L and then I did that few short learning thing. So I would, so I created about a database of what I would say of fifteen hundred natural language queries and the D S L corresponding to that query. So when a new query came in, somebody is asking a stats question in English. What I would do is I would go through the natural language queries, do a semantic search, pick the most closely matching top few.

• 12:02Martin Casado?

  Yeah.

• 12:02Speaker 00

  Uh, and then use that natural language query and its D S L and send that as a prefix. Now, G P T three, if you recall, had a, a context window of only two thousand tokens.

• 12:12Martin Casado?

  Yeah.

• 12:13Speaker 00

  So you had to be very judicious about which examples that you picked. But you pick that and then you send the new query and G P T three would complete it in the D S L that I had designed, which until milliseconds ago, it had never seen.

• 12:25Martin Casado?

  Yeah.

• 12:26Speaker 00

  And I had no access to internals of G P T three. I had no access to the weights.

• 12:30Martin Casado?

  Yeah.

• 12:30Speaker 00

  But still it worked. So that, that's how so.

• 12:34Martin Casado?

  So it's not obvious to me, given your matrix example.

• 12:37Speaker 00

  Yeah.

• 12:37Martin Casado?

  Of like a prompt and then a distribution, how something like in context learning

• 12:43Speaker 00

  works.

• 12:43Martin Casado?

  Would work. And so like, I think your first paper tackled this problem.

• 12:48Speaker 00

  Right.

• 12:49Martin Casado?

  Um, and so maybe you could walk through you, your understanding of how L L M's do in touch, in context learning.

• 12:58Speaker 00

  Yeah. So, so when you think about what in context learning is, is that. As you see evidence, so you know, in the first paper, what I also did was I I took this cricket DSL example.

• 13:11Martin Casado?

  Yeah.

• 13:11Speaker 00

  And I uh I depicted the next token probabilities of the model as it was shown more and more examples. So the first time you show it this DSL, the natural language and the DSL, the probabilities of the DSL tokens were extremely low. Because ZP3 had never seen this thing. When it saw the cricket question, in its mind, it was trying to continue it with an English answer. So the probabilities that were high were all English words.

• 13:49Martin Casado?

  Yeah.

• 13:50Speaker 00

  Once it saw my prompt where I had the question and the DSL, the next time I had the question in the next row, the probabilities of the DSL token started going up. With every example, it went up. And finally, when I gave a new query, it was like it had almost hundred percent probability of getting the right token.

• 14:11Martin Casado?

  Yeah.

• 14:11Speaker 00

  So this is an example of in real time, the model was updating its posterior probability. It was updating its knowledge that, okay, I've seen evidence, this is what I'm supposed to do. Now, this is a colloquial way of saying what Bayesian

• 14:28Martin Casado?

  Yeah.

• 14:28Speaker 00

  inference is. Bayesian updating basically is you start with a prior. When you see a new evidence, you update your posterior. That's the mathematical definition. But but in in English, it's basically you see something. You see new evidence, you update your belief about what's happening.

• 14:45Martin Casado?

  Yeah.

• 14:45Speaker 00

  Right? So it was clear to me that LLMs are doing something which resembles Bayesian updating. So in that first paper, I had this matrix formulation and I showed that, you know, what it's doing. It looks like Bayesian updating.

• 15:00Martin Casado?

  Yeah.

• 15:02Speaker 00

  Then we can come to the sort of next series of papers.

• 15:04Martin Casado?

  That's right. So, okay, so I mean, it seemed pretty conclusive to me at that time. And then you went quiet for a while. And then I still remember the WhatsApp text. You said, Martin, I know exactly how these things are working now.

• 15:17Speaker 00

  Yeah.

• 15:18Martin Casado?

  Well, and then, and then, and then, listen, you dropped a series of papers that kind of broke the internet. Like you went super viral on Twitter. Like, I mean, people really noticed.

• 15:26Martin Casado?

  And so I want to get to that in just a second. But before that, I remember when your first paper came out, people would be like, you know, these things are definitely not Bayesian, like, you know, you know, anything could be considered to be Bayesian, but they're not like, why do you think that there was this reaction to like,

• 15:49Martin Casado?

  you know, there's something new, they're not Bayesian. I mean, I felt like there's almost kind of a backlash just because of being characterized as.

• 15:54Speaker 00

  Yeah, I think this whole world of

• 16:00Speaker 00

  probability and machine learning, that there have been camps of Bayesian and frequentists.

• 16:05Martin Casado?

  Yes.

• 16:06Speaker 00

  And I don't want to get in the middle of that sort of political battle, but Bayesian has become like, almost like people had a reaction to that. It's, it's part of that war.

• 16:15Martin Casado?

  I see. So it's like the old Bayesian frequentist type battle.

• 16:20Speaker 00

  Yeah. So the people just had, oh, no, you can say anything is Bayesian, right? So I said, okay, maybe they have a point. Maybe what we are seeing is not really Bayesian. How do we prove that it's Bayesian?

• 16:31Martin Casado?

  Right.

• 16:32Speaker 00

  So then first I have to thank you and Andres and Horowitz for this.

• 16:40Speaker 00

  You know, when I, when I, when I said that I, in my first paper, I showed these probabilities. It was because OpenAI had in its chat interface

• 16:53Speaker 00

  this option to display those probabilities. Then they stopped.

• 16:58Speaker 00

  So we could not peer inside what's going, what's happening. For some reason, they stopped. OpenAI, I'm not going to get into the open and closed. But they stopped. So then we developed our own interface, which could let you look not only at the probability, but also the entropy of the next token.

• 17:16Martin Casado?

  Was this on top of an open source model?

• 17:18Speaker 00

  Yeah, yeah. So you can load any sort of open source model. But, you know, being in academia, we didn't have access to compute. Thanks to your generous donation, we got the clusters to run what's called Token Probe. You can go to tokenprobe.cscolumbia.edu.

• 17:36Martin Casado?

  Is it still running?

• 17:37Speaker 00

  It's still running. It's still running. And people come to it. I use it in my classes to get students to do assignments. They write their own DSLs and, you know, they say that it really helps them understand how these LLMs work.

• 17:50Martin Casado?

  So I literally, my understanding of LLMs came from Token Probe. Sit there and just look at the distribution as you filled out a prompt. It's actually very, very enlightening. So for those of you that are listening, what's the URL again?

• 18:03Speaker 00

  Tokenprobe.cscolumbia.edu.

• 18:07Martin Casado?

  Yeah, check it out. It's actually a very, very useful way to like actually see how the probability distribution gets updated as you fill out a prompt.

• 18:16Speaker 00

  Right. But then I cheated.

• 18:19Martin Casado?

  Oh.

• 18:20Speaker 00

  I, you know, it was running, but I also had access to the GPUs that were powering it. And then, along with colleagues at Columbia, and one of them now is DeepMind, we started to sort of think about how do you really prove that it's Bayesian? To prove that.

• 18:43Martin Casado?

  Can you just explain it? Actually, I actually don't know the answer to this.

• 18:46Speaker 00

  Yeah.

• 18:47Martin Casado?

  It seemed to me you proved it in the first paper. Like, what was missing?

• 18:51Speaker 00

  Well, in the first paper, we showed it. It was empirical. And you could see.

• 18:55Martin Casado?

  I see, I see.

• 18:55Speaker 00

  You could see.

• 18:56Martin Casado?

  Not a mathematical, because it was very obvious to me.

• 18:58Speaker 00

  Yeah, it was even obvious to me. But to convince, you could say, you know, people who dismiss that anything can be Bayesian.

• 19:06Martin Casado?

  I see.

• 19:07Speaker 00

  We had to show it precisely mathematically.

• 19:10Martin Casado?

  Got it.

• 19:11Speaker 00

  So then we came up with this idea. You know, my colleagues, Namanagirwal and Siddharth Dalal, the series of papers were written with them. We came up with this idea of a Bayesian window. Okay, so what's a wind tunnel? Well, wind tunnel in the aerospace industry is where you test an aircraft in an isolated environment. You don't fly it and you test it against all sorts of, you know, aerodynamic pressure. Then you see what little bit stand, what kind of altitude pressure, blah, blah, blah. And you don't want to do it up in the air testing.

• 19:45Martin Casado?

  Yeah.

• 19:46Speaker 00

  So we said, okay, why don't we create an environment where we take these architectures? And we tested transformers, Mamba, LSTMs, MLPs, all architectures. We say, why don't we create, take a blank architecture, given a task where it's impossible for the architecture to memorize what the solution to that task should be.

• 20:13Speaker 00

  The space is combinatorially impossible for, given the number of parameters. And we took very small models. So it's difficult enough that they cannot memorize it. But it's tractable enough that we know precisely what the Bayesian posterior should be. You can calculate it analytically. So we gave these models a bunch of tasks where, again, we show that it's impossible to memorize. We trained these models and we found that the transformer got the precise Bayesian posterior down to ten to the power minus three bits accuracy. It was matching the distribution perfectly.

• 20:54Speaker 00

  So it is actually doing Bayesian in the mathematical sense, given a task where it has to update its belief. Mamba also does it reasonably well. LSTMs can do one of the things. So in the papers, we have a taxonomy of Bayesian task. Transformer does everything, Mamba does most of it, LSTMs do only partially, and MLPs fail completely.

• 21:18Martin Casado?

  So is this a reflection of the data that it's trained on? Or is it more a reflection of the mechanism?

• 21:26Speaker 00

  It's the mechanism, it's the architecture.

• 21:30Speaker 00

  The data decides what task it learns.

• 21:33Martin Casado?

  Right.

• 21:33Speaker 00

  So in the first paper, we had these Bayesian wind tunnels and we showed that, you know, it's doing the job with different tasks. In the second paper, we show why it does it. So we look at the transformers, we look at the gradients and we show how the gradients actually shape this geometry, which enables this Bayesian updating to happen.

• 21:57Speaker 00

  Then in the third paper, what we did, we take, we took these frontier production L L Ms, which have open weights so that we could look inside them. And we did our testing and we saw that the geometries that we saw in the small models persisted in models which are, you know, hundreds of millions of parameters. The same signature existed. The only thing is that because they are trained on all sorts of data, it's a little bit dirty or messy.

• 22:27Martin Casado?

  Yeah.

• 22:27Speaker 00

  But you can see the same structure. So the, the whole idea behind the Bayesian wind tunnel was unlike these production L L Ms, where you don't know what they have been trained on.

• 22:36Martin Casado?

  Right.

• 22:37Speaker 00

  So you cannot mathematically compute the posterior. So again, how do you prove it? I mean, it looks Bayesian, you know.

• 22:43Martin Casado?

  That's not from the first paper.

• 22:45Speaker 00

  From the first, it looks Bayesian, but you know. So the wind tunnel sort of solved that problem for us. It said, okay, let's start with a blank architecture. Give it a task where we know what the answer is. It cannot memorize it. Let's see what it does. And yeah.

• 22:59Martin Casado?

  So do you think this provides any sort of like indication of how humans think? Or do you think that these things are totally independent?

• 23:06Speaker 00

  No, no, it, it does provide, right? So, you know, human beings also update our beliefs as we see new evidence.

• 23:14Martin Casado?

  Right.

• 23:15Speaker 00

  So we do in some sort of, in some sense, Bayesian updating, but we do something more than that. I'll come to that. But these transformers or even Mamba do this Bayesian updating.

• 23:30Martin Casado?

  Yeah.

• 23:31Speaker 00

  And, but, but, but the difference with humans is.

• 23:36Speaker 00

  You know, we will update our posterior when we see some new evidence. But the way our brains have evolved over hundreds of millions of years is our optimization objective has been don't die and reproduce, right? That's been sort of the driving force and our brains have learned to adjust. And so when we see some danger, there's something rustling in that bush, don't go near. We know how to react to that danger. We know how to save ourselves.

• 24:11Speaker 00

  We internalize that learning and our brain cells or our synapses remain plastic throughout our lifetime.

• 24:21Speaker 00

  What happens with L L M's is, once the training is done, those weights are frozen. When you're doing inference, for instance, in context, learning or anything, during that conversation, okay, you're doing Bayesian inference, but then you forget. The next time a new conversation starts with zero context, you don't retain any learning that happened in the previous instance. So for instance, with the cricket D S L that I was doing, every invocation of it was fresh. It did not remember the last time I sent a query, what the D S L looked like. So that's one difference between how humans

• 25:05Speaker 00

  use sort of Bayesian updating, which is we remain plastic all our lives, whereas L L M's are frozen. And there's another sort of difference, which if you want me to get it. Tell me, yeah, yeah, yeah. So the other difference is, well, first, you know, our objective is don't die, reproduce. L L M's objective is predict the next token as accurately as possible, right? So all these scary stories that you you read about that, oh, the L L M tried to deceive and it tried to prevent itself from being shut down, that's not a function of the architecture. That's a function of the training data.

• 25:50Martin Casado?

  That's it.

• 25:50Speaker 00

  It has been fed, you know, articles on Reddit or ASMR or whatever.

• 25:56Martin Casado?

  I mean, just today, by the way, Dario allegedly said that you can't rule out that they're conscious.

• 26:05Speaker 00

  You can rule out their consciousness. I mean, come on. As I said, you know, Anthropic makes great products. Claude Code is fantastic. Coco is fantastic. But they are grains of silicon doing matrix multiplication. They don't have consciousness. They don't have an inner monologue. They don't, they're not driven by the same objective function. Don't reproduce, right? They're driven by, don't make a mistake on the next token. And that's driven entirely by the training data. Right? You train the LLM with stories of ASMR or Reddit where, you know,

• 26:42Speaker 00

  to survive is going to do this or that, it'll reproduce that. So it's, it's, it's a reflection. It's not a mind.

• 26:48Martin Casado?

  And, and the results, just to say it for the 10th time, are perfectly Bayesian.

• 26:54Speaker 00

  Perfectly, yeah.

• 26:55Martin Casado?

  To the, to the, to the digit.

• 26:57Speaker 00

  To the digit. Yeah. I mean, I, I trained it for 150,000 steps. And the accuracy was 10 to the power minus three bits. I could have trained it for, you know, this happened in half an hour. On the infrastructure that you provided for token property. In the background, I could use those APIs to train, but, so thank you again for that. But so, no, human beings, coming back to it, we, we are Bayesian. But we do something else. You know, when I, when I, when I throw this pen at you, what will you do?

• 27:28Martin Casado?

  Dodge it?

• 27:28Speaker 00

  Dodge it. Why will you dodge it?

• 27:32Martin Casado?

  To avoid being hit?

• 27:33Speaker 00

  Avoid being hit, but your head is not doing a Bayesian calculation of, okay, this pen is coming. The probability that it hits me, it'll cause this much pain or all that. What you're essentially doing in your head is you're doing a simulation.

• 27:50Speaker 00

  You see the, the, the, the pen coming and you know that it'll come and hit me. Your mind simulates and you dodge it, right? So all of deep learning is doing correlations.

• 28:07Speaker 00

  It's not doing causation. Yeah. Causal models are the ones that are able to do simulations and interventions. So you know Judea Pearl has this whole causal hierarchy. Yeah. Where the first hierarchy, the first hierarchy is association, which is you build these correlation models. Deep learning is beautiful. It's extremely powerful. I mean, you see every day, all these models are like amazingly good. Yeah. They do association. The second is intervention in the hierarchy. Yeah. Deep learning models do not do that. Third is counterfactual. So both intervention and counterfactual, you can imagine it, it's some sort of simulation.

• 28:49Speaker 00

  You, you build a model of causal model of what's happening and then you are able to simulate. So our brains do that. The current architectures don't do that. Another example, I think, which will make it clear is the difference between, I'll use this technical term, Shannon entropy and Kolmogorov complexity. Sure. So if you look at the Shannon entropy of the digits of pi. Yeah. It's infinite. Sure. It's impossible to predict and learn what digit will come after. So that's the definition of Shannon entropy. And Shannon entropy sort of tries to build a correlation. It tries to learn the correlation. Deep learning does the Shannon entropy.

• 29:35Speaker 00

  Kolmogorov complexity, on the other hand, is the, is the length of the shortest program. Yeah. Which will reproduce the string that you, that is under question. Yeah. Now, the program to get the digits of pi are very small. Yeah. Thanks to Ramanujan, that's, you know, there are all sorts of really small program that can reproduce it exactly. So the Kolmogorov complexity of pi is very small. Shannon entropy is infinite. I think deep learning is still. In the Shannon entropy world, it has not crossed over to the Kolmogroff complexity and the causal world.

• 30:14Martin Casado?

  Wow, interesting.

• 30:15Speaker 00

  Right? So.

• 30:17Martin Casado?

  Do you, so to what extent do you think this provides us research directions to kind of improve the state of the illness? So let me just give you a specific example. You talked about human beings don't actually update, you know, the matrix. They don't kind of update their weights. But right now, there's a lot of research on continual learning.

• 30:37Speaker 00

  Yeah.

• 30:37Martin Casado?

  But it was, you know, so does your work provide some guidance of how you might approach those problems? And in particular, I've always had this question, which is we use so much data and so much compute.

• 30:49Speaker 00

  Yeah.

• 30:49Martin Casado?

  To create these models, like, is it even reasonable to think that you can update the weights and actually have a meaningful impact, you know, in, in real time? I mean, it just seems like you just need so much more data in order to do that. So can you start answering these questions?

• 31:04Speaker 00

  You, you can start answering some of these questions. And, and one of the misconceptions that exists today is that scale will solve everything. Scale will not solve everything. You, you need a different kind of architecture. And this continual learning is a difficult problem. You have to balance the fact that you will learn something new against the risk of catastrophic forgetting.

• 31:25Martin Casado?

  Right.

• 31:26Speaker 00

  Right?

• 31:26Martin Casado?

  Right.

• 31:27Speaker 00

  If you update the weights and you forget what, what was important and what you have already learned, then, then you are, you know, you're not making progress. Then it'll just be some sort of random chaotic model. To, to solve that problem is difficult. That's one aspect of it. So, so, so, you know, to get to what is called AGI, I think there are two things that need to happen. One is this plasticity, which has to be implemented through continual learning.

• 31:55Martin Casado?

  Yeah.

• 31:56Speaker 00

  Secondly, we have to move from correlation to causation.

• 31:59Martin Casado?

  Yeah.

• 31:59Speaker 00

  That's a.

• 32:01Martin Casado?

  I, I, how, how much is this similar to what Jan Leckun talks about with the?

• 32:07Speaker 00

  So Jan Leckun.

• 32:08Martin Casado?

  Causality planning.

• 32:10Speaker 00

  Yeah.

• 32:10Martin Casado?

  You know, predicting how, like how your action would.

• 32:13Speaker 00

  It, it is, it is related. You know, he is coming at it from a different angle than the J P R model.

• 32:17Martin Casado?

  Right.

• 32:18Speaker 00

  But it is related. The, the, the other thing is, you know, the first time I came on this podcast, I, I mentioned this test of AGI.

• 32:25Martin Casado?

  Yeah.

• 32:26Speaker 00

  The Einstein test.

• 32:27Martin Casado?

  I don't remember.

• 32:28Speaker 00

  So I said, you know, uh, uh, you take an L L M. And train it on pre nineteen sixteen or nineteen eleven physics. And see if it can come up with the theory of relativity.

• 32:42Martin Casado?

  Yeah.

• 32:42Speaker 00

  If it does, then we have A G I. I mean, it's a high bar, but you know, we should have high bars. It won't. And this is the same test that I think Demis uh. Mentioned at the India A I summit a couple of weeks ago. It's created a lot of news. But why? Why is that? And how is that related to this idea of Shannon versus Kolmogro? So. At the time of Einstein, there were a lot of clues. That Newtonian mechanics, there was something missing.

• 33:16Martin Casado?

  Yeah.

• 33:16Speaker 00

  Right? Uh, people knew that Mercury's orbit didn't make sense. There was something off about it. Then there were these experiments done, the Michelson-Morley experiments. Where they were trying to figure out uh. Uh, this uh, medium called the ether through which light travels. And they felt that if, you know, you bounce light in different directions. Uh, the speed might change and they they could detect a change in the speed of light.

• 33:51Speaker 00

  They tried several experiments. They had really precise instruments which could measure the speed. And they found nothing. They found that the speed of light did not change at all. Then there were, there was a whole issue of black holes.

• 34:05Martin Casado?

  Yeah.

• 34:05Speaker 00

  Then gravitational lensing. So there were a lot of these signs that Newtonian mechanics.

• 34:13Speaker 00

  Is not really explaining everything.

• 34:15Martin Casado?

  Yeah.

• 34:16Speaker 00

  But until Einstein came up with a new representation of the spacetime continuum.

• 34:23Martin Casado?

  Right.

• 34:23Speaker 00

  You were stuck.

• 34:24Martin Casado?

  Yeah.

• 34:24Speaker 00

  So if you had a model that just looked at correlations and sees all of this, you know,

• 34:32Speaker 00

  all of these pieces of individual evidence and put together, it would not have come up with the beautiful equation that Einstein came up with, you know.

• 34:44Speaker 00

  I'm forgetting exactly what it is, G mu v equals eight pi t mu v, something like that. Where, you know, the equation of the space time continuum, the tensor. So he came up with a new formulation.

• 34:59Martin Casado?

  Yeah.

• 35:00Speaker 00

  So he kind of rejected the existing axioms. He came up with a very short Kolmogorov representation of the world. One equation, from that equation, everything else follows. Right? Whether you're talking about gravitational waves or black holes or mercury or how GPS works. You know, GPS, the GPS that we use every day on our phones, it uses the equation of relativity.

• 35:27Martin Casado?

  So does this end up becoming like,

• 35:33Martin Casado?

  you, you, you almost have to ignore the majority of previous data in order to do it, which L L M's can't because they trained on the majority of previous data. It's like you almost have like this kind of data gravity that's pulling you back. It's like, it's like, everybody said it's X. There's a little bit of evidence that it's Y, but because everybody said it's X, like the L M will always say it's X.

• 35:54Speaker 00

  It's, it'll always say X. It'll treat that Y as an anomaly.

• 35:58Martin Casado?

  Actually, this is a very nice way to say it, which is like, it's like, I just now, okay, now I get your Shannon entropy versus Kolmogorov. Like one of them is like, the total amount of information there, that will always be bound to the total amount of information there, which is what happens right now.

• 36:15Speaker 00

  Yeah.

• 36:16Martin Casado?

  Where you can actually describe.

• 36:19Martin Casado?

  And another, another motion, you can describe everything with a shorter description with the new data, which would be a totally different motion, which would be like.

• 36:29Speaker 00

  Yeah, you need a new representation, right?

• 36:31Martin Casado?

  Yeah. You know, another way that I've always thought about these, I thought you articulated it well in the last time we talked about it, which is the universe is this very, very complex space. And then, you know, somehow humans map it into a manifold. That's less complex.

• 36:48Speaker 00

  Yeah.

• 36:49Martin Casado?

  And then that gets kind of written down. And then the L M, so that's kind of some, some distribution, some, you know, it's still a very large space, but it's a bounded space. And the L M learn that manifold. And then they kind of use, you know, Bayesian inference to move up and down that manifold. But they're kind of bound to that manifold.

• 37:07Speaker 00

  Yeah.

• 37:08Martin Casado?

  And then, again, I don't want to put words in your mouth and then, but like, what they can't do is generate a new manifold.

• 37:13Speaker 00

  New amount, yeah.

• 37:14Martin Casado?

  Which requires understanding the way that the universe works and then coming up with a new representation of the universe.

• 37:19Speaker 00

  And this is what relativity is, right?

• 37:21Martin Casado?

  Yeah, exactly.

• 37:21Speaker 00

  Einstein had to create a new manifold.

• 37:23Martin Casado?

  Yeah, yeah, yeah.

• 37:24Speaker 00

  If you're just stuck with the old manifold of the Newtonian physics.

• 37:27Martin Casado?

  Right.

• 37:28Speaker 00

  Then you would see these correlations, but you could not come up with a manifold that explained them. So you need to come up with a new representation. So to me, you know, there are lots of definitions of AGI. You know, Turing test, we have already passed that. You know, performing economically useful work every day, you see, you know, L M are doing that.

• 37:49Martin Casado?

  Do we? I don't know.

• 37:50Speaker 00

  Well, I mean, they are.

• 37:51Martin Casado?

  I mean, without human intervention?

• 37:53Speaker 00

  No, no, so that's different.

• 37:54Martin Casado?

  Okay.

• 37:54Speaker 00

  But still, you know, it's like a car can run faster than humans, right?

• 37:59Martin Casado?

  I mean, that's a, that's a, that's a, that's a very shallow definition.

• 38:03Speaker 00

  Yeah, so all these definitions.

• 38:04Martin Casado?

  Cars do useful work.

• 38:06Speaker 00

  Maybe, you know, in six months, you'll have Claude or what a Gemini do without intervention. Coring tasks, which are well defined, well scoped. That's possible. But to me, AGI will happen when these two problems get solved. Elasticity, continual learning properly and building a causal model from, you know.

• 38:31Speaker 00

  In a more data efficient manner.

• 38:34Martin Casado?

  We are hearing people now talking about, you know, seeing general, like Donald Knuth, for example, in the last few days, right? You know, had this, you know, this, you know, aha moment apparently that kind of made, went viral on X. So do you think that that suggests that we're seeing generality?

• 38:52Speaker 00

  No, no, no. So that actually, to me, validates what I've been talking about for a while now.

• 38:59Martin Casado?

  How so?

• 39:00Speaker 00

  So if you read what he did with the help of, you know, colleague, he got the L L Ms to solve this particular problem of finding Hamiltonian cycles, odd numbers, we won't get into that. And he got the L L Ms to keep solving for one odd number after the other, right? What he also got to do is after it found a solution for a particular value of M, he made the L L M update its memory with exactly what it learnt in solving that problem. So the L L Ms tried many different things. You know, something worked, update the memory. So that's kind of like hacking together plasticity.

• 39:43Martin Casado?

  Yeah.

• 39:43Speaker 00

  Right? It's learning what it has done as we went along. Again, it's, it's a hacked version of it. You're not changing the weights, you're just sort of improving the context.

• 39:54Martin Casado?

  Right.

• 39:55Speaker 00

  But as you learned, and even after that, so this whole space of Hamiltonian cycles and the associated math is well represented in the manifolds that these L L Ms have been trained on.

• 40:10Martin Casado?

  Right.

• 40:10Speaker 00

  You just had to find the right connection. And L L Ms, I know, compute, you throw enough compute, they will find the right connection. So Knuth was able to find the L L Ms attempts and eventually it needed him.

• 40:29Speaker 00

  To put together what he saw into a solution. It definitely helped him get to the solution. But he had to create the new sort of manifold to come to the solution. The L L M's were after a while stuck. Right, he, he, you, you read what he has written, I mean, they just hot up the press, I think two days ago.

• 40:50Martin Casado?

  Two days ago, yeah.

• 40:50Speaker 00

  Two days ago. But eventually he used the solution and he came up with the proof.

• 40:55Martin Casado?

  Yeah.

• 40:56Speaker 00

  Right, so it's like, you know, it's like Einstein saw all these evidences. Then he thought, what will explain?

• 41:07Speaker 00

  He came up with a causal model.

• 41:09Martin Casado?

  Yeah.

• 41:10Speaker 00

  So Knut and his brain i- is sort of the.

• 41:12Martin Casado?

  That's in the Kolmogroff.

• 41:14Speaker 00

  Kolmogroff, yeah, yeah, yeah.

• 41:15Martin Casado?

  Is the human.

• 41:16Speaker 00

  Right. And the L L M's are extremely efficient at doing the Shannon part of it. It, it, it found all the solutions by trying, you know, various things and, and learning more and more.

• 41:26Martin Casado?

  Clever way to decompose it. I'm wondering, like, do you think this, again, I'm going to ask the same question again, which is, do you think this provides some sort of insight on like the next problem to tackle? Like.

• 41:35Speaker 00

  Yeah.

• 41:35Martin Casado?

  Like, like, is there a mechanism that will get the Kolmogroff complexity?

• 41:41Martin Casado?

  Or not, like, is this?

• 41:43Speaker 00

  It, it, it tells us which direction.

• 41:46Martin Casado?

  But not, but clearly not how to do it, like.

• 41:48Speaker 00

  Not how to do it, but even Kolmogroff complexity has largely remained a sort of a theoretical construct.

• 41:54Martin Casado?

  Yeah, for sure. It, it, there's no algorithm.

• 41:57Speaker 00

  There's no, they haven't been practical implementations of finding the shortest program.

• 42:02Martin Casado?

  Yeah.

• 42:02Speaker 00

  We know it exists, you know, you can argue about it, but so, so that's where I think.

• 42:09Martin Casado?

  Yeah.

• 42:10Speaker 00

  It's my bias, that's where our energy should be focused, not larger models with more tokens.

• 42:15Martin Casado?

  Can you, and can you, can you tie the two things, like, how does that pair with doing simulation, or is that simulation totally orthogonal?

• 42:23Speaker 00

  No, simulation is is a trilogy, right?

• 42:26Martin Casado?

  So you think it like basically you do simulation? And somehow that is a step towards doing the Kama graph complexity?

• 42:34Speaker 00

  It it's the simulator is the is the program that we create. It may not be the perfect program.

• 42:41Martin Casado?

  Oh, I see.

• 42:42Speaker 00

  But in our heads, we create this simulator that when I'm throwing the pen, you know that it's coming at you, right? And you duck. So so you're not computing the probabilities as it goes. But but you have, you know, you you.

• 42:54Martin Casado?

  That's a very physical thing versus we are talking more conceptually.

• 42:57Speaker 00

  Conceptually, but but it's.

• 42:59Martin Casado?

  And you think those are the same mechanism?

• 43:00Speaker 00

  Is the same mechanism.

• 43:01Martin Casado?

  Really?

• 43:01Speaker 00

  Yeah, you you have to build a causal model.

• 43:04Martin Casado?

  Yeah, I see.

• 43:05Speaker 00

  For most things, right? So you have to move from correlation to correlation. I mean, we've heard this term.

• 43:10Martin Casado?

  Yeah.

• 43:11Speaker 00

  You know.

• 43:13Speaker 00

  Ad infinitum. But here it it's making a difference in the way we view intelligence.

• 43:20Martin Casado?

  How how how has the last three papers been received?

• 43:24Speaker 00

  No, I don't know.

• 43:26Speaker 00

  Well.

• 43:26Martin Casado?

  I mean, I mean.

• 43:27Speaker 00

  The archive versions.

• 43:28Martin Casado?

  Let me tell you. I mean.

• 43:32Martin Casado?

  Lot of great reception, a lot of people read it. I'm just wondering like what kind of feedback that you've got.

• 43:36Speaker 00

  I'm getting good feedback, but I'm an outsider in this field, right?

• 43:39Martin Casado?

  Right, that's right.

• 43:40Speaker 00

  Like.

• 43:40Martin Casado?

  This networking guy.

• 43:41Speaker 00

  I'm a networking guy. Why is he writing about, you know, learning and machine learning and deep learning and Bayesian? But but people who have actually taken the time to read those papers, I'm getting really good feedback. There was a recent paper by Google Research.

• 43:58Martin Casado?

  Yeah.

• 43:58Speaker 00

  Which tried to teach L L M's by some sort of R L H F to do Bayesian learning properly.

• 44:04Martin Casado?

  Yeah.

• 44:04Speaker 00

  And that's going in this direction. I think people are coming around to the view that, OK, L L M's are doing Bayesian learning. I know that some people also looked at the Bayesian ventanal paper, the archive version, and they reproduced the experiments.

• 44:19Martin Casado?

  That's great.

• 44:19Speaker 00

  They just saw what was written and they they did the trading and they saw, yeah, yeah, this is actually happening. So.

• 44:24Martin Casado?

  That's great.

• 44:25Speaker 00

  So what's next? What's next is, you know, these two parallel tracks, I hope to make progress there. Plasticity and causal.

• 44:38Martin Casado?

  Because today you've taken an existing mechanism.

• 44:42Speaker 00

  Yeah.

• 44:42Martin Casado?

  And you've created a formal model of how it works.

• 44:44Speaker 00

  Yeah.

• 44:45Martin Casado?

  And so now you're actually interested in improving.

• 44:48Speaker 00

  In improving, yeah.

• 44:48Martin Casado?

  Creating a new mechanism.

• 44:49Speaker 00

  Yeah, yeah.

• 44:50Martin Casado?

  And do you think it's an entirely different architecture?

• 44:53Speaker 00

  I.

• 44:53Martin Casado?

  Or do you think, do you think L L M's are like part of the solution?

• 44:55Speaker 00

  I think L L M's are definitely part of the solution.

• 44:58Martin Casado?

  I see.

• 44:58Speaker 00

  But, but there has to be something more.

• 45:00Martin Casado?

  Another.

• 45:01Speaker 00

  So, you know, I was not interested in sort of cataloging what all these.

• 45:05Martin Casado?

  Yeah.

• 45:05Speaker 00

  L L M's can do.

• 45:06Martin Casado?

  Yeah.

• 45:06Speaker 00

  I was more interested in why are they and how are they doing it?

• 45:09Martin Casado?

  Yeah.

• 45:10Speaker 00

  I think now we have a good grip on the why and how.

• 45:16Martin Casado?

  Yeah.

• 45:16Speaker 00

  And the next step is to, you know, move them to the next level. We, we now, I, I think we have a fairly good understanding of what the limits are.

• 45:23Martin Casado?

  Yeah.

• 45:24Speaker 00

  Now, how do you go to the next step?

• 45:28Martin Casado?

  Is there any, is there an equivalent kind of theoretical framework for causality that applies here? Like similar to like Bayesian for inference?

• 45:38Speaker 00

  Well, the Judea Paul's whole causal, causal hierarchy, I think.

• 45:41Martin Casado?

  That's the right one.

• 45:42Speaker 00

  That's, that's a very good one, you know, the whole do calculus approach. I, I, I think it's a good way to think about it. You know, the, the sort of association, intervention, counterfactuals.

• 45:56Martin Casado?

  Yeah.

• 45:57Speaker 00

  It takes you from correlation to actless simulation. In a mathematical way.

• 46:03Martin Casado?

  That's great. All right. Well, listen, really appreciate you coming. This is awesome. So we had you here for the first paper where you had the empirical results.

• 46:09Speaker 00

  Mm hmm.

• 46:10Martin Casado?

  And then we had you back when you actually have like the formal proof. And hopefully the next time you come back, you will have a proposal for the mechanism that, that actually provides the next step.

• 46:21Speaker 00

  Hopefully, yeah.

• 46:22Martin Casado?

  All right, cool.

• 46:23Speaker 00

  We're looking on it.

• 46:23Martin Casado?

  For coming in.

• 46:24Speaker 00

  Thank you for having me.

• 46:29Erik Torenberg?

  Thanks for listening to this episode of the A16Z Podcast. If you liked this episode, be sure to like, comment, subscribe, leave us a rating or review, and share it with your friends and family. For more episodes, go to YouTube, Apple Podcasts, and Spotify. Follow us on X at A16Z and subscribe to our Substack at a16z.substack.com. Thanks again for listening, and I'll see you in the next episode.

• 46:54Erik Torenberg?

  As a reminder, the content here is for informational purposes only, should not be taken as legal, business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any A16Z fund. Please note that A16Z and its affiliates may also maintain investments in the companies discussed in this podcast. For more details, including a link to our investments, please see a16z.com/disclosures.