Will LLMs Get Us To AGI?

LLMs have made tremendous progress in modeling human language. But can they go beyond that to make new discoveries and move the needle on novel scientific progress? We sat down with distinguished Columbia CS professor Vishal Misra to discuss this, plus why chain-of-thought reasoning works so well, and what real AGI would look like. Timecodes: 0:00 Intro 0:32 How LLMs and humans reason through manifolds 4:15 Token prediction, entropy & confidence 8:05 Chain-of-thought reasoning and entropy reduction 10:20 Vishal’s background 14:10 Inventing RAG 17:30 The rise of LLMs and the question of plateau 21:00 The Matrix Model / how prompts map to token distributions 28:10 Why LLMs can’t recursively self-improve 34:02 Defining AGI 38:25 Future architectures & multimodal intelligence 42:00 Modeling vs prompt engineering 47:20 What would prove AGI has arrived? 50:01 Closing thoughts Resources: Follow Dr. Misra on X: https://x.com/vishalmisra Follow Martin on X: https://x.com/martin_casado Stay Updated: If you enjoyed this episode, be sure to like, subscribe, and share with your friends! Find a16z on X: https://x.com/a16z Find a16z on LinkedIn: https://www.linkedin.com/company/a16z Listen to the a16z Podcast on Spotify: https://open.spotify.com/show/5bC65RDvs3oxnLyqqvkUYX Listen to the a16z Podcast on Apple Podcasts: https://podcasts.apple.com/us/podcast/a16z-podcast/id842818711 Follow our host: https://x.com/eriktorenberg Please note that the content here is for informational purposes only; should NOT be taken as legal, business, tax, or investment advice or be used to evaluate any investment or security; and is not directed at any investors or potential investors in any a16z fund. a16z and its affiliates may maintain investments in the companies discussed. For more details please see a16z.com/disclosures.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Any LLM that was trained on pre-1915 physics would never have come up with a theory of relativity. Einstein had to sort of reject the Newtonian physics and come up with this space-time continuum. He completely rewrote the rules. AGI will be when we're able to create new science, new results, new math. When an AGI comes up with a theory of relativity, it has to go beyond what it has been trained on to come up with new paradigms, new science. That's my definition of AGI.

• 0:31Speaker 01

  Vishal Misra was trying to fix a broken cricket stats page and accidentally helped spark one of AI's biggest breakthroughs. On this episode of the A16Z podcast, I talk with Vishal and A16Z's Martine Tesado about how that moment led to retrieval augmentation generation and how Vishal's formal models explain what large language models can and can't do. We discuss why LLMs might be hitting their limits, what real reasoning looks like, and what it would take to go beyond them. Let's get into it.

• 1:00Speaker 01

  Martine, I know you wanted to have Vishal on. What do you find so remarkable about him and his contributions that inspired this?

• 1:06Martin Casado?

  Vishal and I actually have very similar backgrounds. We both come from networking. He's a much more accomplished networking guy than I am.

• 1:11Speaker 01

  That's a high bar given to you.

• 1:13Martin Casado?

  And so we actually view the world in an information theoretic way. It is actually part of networking. And with all this AI stuff, there's so much work trying to create models that can help us understand how these LLMs work. And in my experience over the last three years, the ones that have most impacted my understanding, and I think have been the most predictive, are the ones that Vishal has come up with. He did a previous one that we're going to talk about called Matrix, is it?

• 1:41Speaker 00

  Beyond the Black Box, but yeah.

• 1:43Martin Casado?

  Beyond the Black Box. Actually, we should put this in the notes for this, but the single best talk I've ever seen on trying to understand how LLMs work is one that Vishal did at MIT, which Hari Balakrishnan pointed me to, and I watched that. So he did that work, and then he's doing more recent work that's actually trying to scope out not only how LLMs reason, but like it has some reflections on humans reason too. And so I just think he's doing some of the more profound work and trying to understand and come up with models, formal models for how LLMs reason.

• 2:13Erik Torenberg?

  On that note, you said his most recent work helped you change how humans think. Why don't you flesh that out a little bit? How did it sort of?

• 2:20Martin Casado?

  Well, okay, so can I just try to take a rough sketch at it? And then you just tell me how wrong I am.

• 2:23Erik Torenberg?

  Right ahead.

• 2:24Martin Casado?

  You're trying to describe how LLMs work. And one thing that you found is that they reduce a very, very complex, multi-dimensional space into basically a geometric manifold that's a reduced state space. So it's reduced degrees of freedom. But you can actually predict where in the manifold the reasoning can move to, roughly. So you've reduced the dimensionality of the problem to a geometric manifold. And then you can actually formally specify kind of how far you can reason within that manifold. So and the articulation is that we, or one of the intuitions is that we as humans do the same thing.

• 3:08Martin Casado?

  Is we take this very complex, heavy-tailed stochastic universe and we reduce it to kind of this geometric manifold. And then when we reason, we just move along that manifold.

• 3:18Speaker 00

  Yeah, I think you captured it accurately. That's kind of the spirit of the work.

• 3:22Martin Casado?

  Yeah. Wait, can I just hear it in your words? Because I'm a VC, so.

• 3:28Speaker 00

  You're a VC with an H-index of what, 60?

• 3:31Martin Casado?

  True.

• 3:32Speaker 00

  Yeah, so ultimately, what all these LLMs are doing, whether the early LLMs or the LLMs that we have today with all sorts of post-training RLHF, whatever you do, at the end of the day, what they do is they create a distribution for the next token. So given a prompt, these LLMs create a distribution for the next token or the next word. And then they pick something from that distribution using some, some kind of algorithm to predict the next token, pick it, and then keep going. Now, what happens because of the way we train these LLMs, the architecture of the transformers and the loss function, the way you put it is right.

• 4:17Speaker 00

  It sort of reduces the world into these Bayesian manifolds.

• 4:21Martin Casado?

  Yeah.

• 4:22Speaker 00

  And as long as the LLM is going in, sort of traversing through these manifolds, it is confident. And it can produce something which makes sense. The moment it sort of veers away from the manifold, then it starts hallucinating and starts spouting nonsense. Confident nonsense, but nonsense. Yeah. So it creates these manifolds and the trick is the distribution that is produced. You can measure the entropy of the distribution.

• 4:53Speaker 00

  Right, but entropy the way Shannon just said. Share an entropy. Shannon entropy, not thermodynamic entropy. So suppose you have a vocabulary of, let's say, fifty thousand different tokens. And you have a distribution, next token distribution over these fifty thousand tokens. So let's say the cat sat on the, right? If that is a prompt, then the distribution will have a high probability for mat. Yeah. Or hat or table. And a very low probability of, let's say, ship or whale or something like that, right? Yeah. So because of the way it's trained, it it has these distributions. Now, their distributions can be low entropy or high entropy. Yeah.

• 5:37Speaker 00

  A high entropy distribution means that there are many different ways that the L L M can go. Yeah. With a high enough probability for all those paths. Yeah. Low entropy means that there are only a small set of choices for the next token. And the prompts also you can categorize into two kinds of prompts. One prompt is is you can say high information entropy. Yep. And one prompt is low information entropy. Yep. So the way these manifolds work, the L L M start paying attention to prompts that have high information entropy. Yeah. And low prediction entropy.

• 6:22Speaker 00

  So what do I mean by that? So so when I say I'm going out for dinner. Yep. Right? So when I say I'm going out for dinner, that phrase, the L L M's have been trained. They've seen it a lot and there are many different directions I can go with it. I can say I'm going for dinner tonight, I'm going for dinner to McDonald's or I'm going to dinner blah, blah, blah. There are many different. But when I say I'm going to dinner with Martin Casado, you know, the L L M, now this is information rich. This is sort of a rare phrase. And now the sort of realm of possibilities reduces because Martin is only going to take me to Michelin star restaurants.

• 7:04Martin Casado?

  Yep, yep, yep.

• 7:05Speaker 00

  I'm not going to go to McDonald's. You get what I'm saying. The moment you add more context, you make the prompt information rich, the prediction entropy reduces.

• 7:16Martin Casado?

  Yep, yep, yep, yep.

• 7:18Speaker 00

  And another example that I often.

• 7:21Martin Casado?

  But just quickly, what so what? But what is your takeaway? What is your implication on that? Which is of course, as you're in. So, yeah, so you're so sorry. I forgot how you described it, but so the more precise you are, the more tokens you are, I presume the less options you have for the next token. Is that correct or not correct?

• 7:44Speaker 00

  Yeah, yeah, essentially.

• 7:45Martin Casado?

  So you're redo, you're reducing it, you're reducing it to a very specific state space when it comes to confidence in an answer. And this is kind of a manifold that you can go on. And then, I mean, do you? Do you have kind of a conclusion of what that means for systems or what that means for reasoning? Or is it just a nice way to articulate the bounds of L L Ms?

• 8:10Speaker 00

  No, there is something. I don't know. I don't know if I should say profound, but there's something about it which tells what these L L Ms can or cannot do. Right? So one of the examples that I often tell is suppose I ask you, what is 769 times 1025?

• 8:31Speaker 00

  You have no idea. You can have some vague idea given the two numbers, right? And so in your mind, the next token distribution of the answer is going to be diffuse, right? You don't know. You have maybe a vague guess. If you are mathematically very good, maybe your guess is more precise, but it's just going to be diffuse and it's not going to be the correct answer. But if I, if you say, can I write it down and do it? The way we have learned multiplication tables, now you know exactly what to do next step, right? You write seven hundred and sixty nine and then one zero two five and then you know exactly. So at each stage of that process, your prediction entropy is very low.

• 9:17Speaker 00

  You know exactly what to do because you have been taught this algorithm. And by invoking this algorithm saying, okay, I'm not going to just guess the answer, but I'm going to do it step by step. Then your prediction entropy reduces and you can arrive at an answer which you're confident of and which is correct. And the LLMs are pretty much the same way. That's why chain of thought works. What happens with chain of thought is you ask the LLM to do something chain of thought. It starts breaking the problem into small steps. These steps it has seen in the past. It has been trained on, maybe with some different numbers, but the concept it has been trained on.

• 10:01Speaker 00

  And once it breaks it down, then it's confident. Okay, now I need to do A, B, C, D, and then I arrive at this answer, whatever it is.

• 10:11Erik Torenberg?

  Let's zoom back. I want to get into LLMs, but first of all, maybe you can give more context on your background and how that informs your work here.

• 10:21Speaker 00

  Okay. So yeah, as Martin said, my background is very similar to his. We, you know, we come from doing networking. So my PhD thesis, my sort of early work at Columbia has all been in networking. But there's another side of me, another hat that I wear, which is both an entrepreneur and a cricket fan.

• 10:43Erik Torenberg?

  I was gonna say, don't you own a cricket team or something?

• 10:46Speaker 00

  I am a minority owner at your, for your local cricket in the San Francisco Unicorns.

• 10:51Martin Casado?

  Yeah, that's right. Very proud to have you.

• 10:55Speaker 00

  So, but, uh, so say in the nineties, I was one of the people who started this portal called Cricket Info.

• 11:06Speaker 00

  And Cricket Info, at one point it was the most popular website in the world. It had more hits than Yahoo. That was before India came on.

• 11:15Martin Casado?

  That's remarkable.

• 11:15Speaker 00

  And so, you know, uh, we built, cricket is a very start rich sport. You think baseball multiplied by a thousand. And we had built this free searchable stats database on cricket called Stats Guru. And this has been available on Cricket Info since 2000. But because you can search for anything, everything was made available on Stats Guru. And you know, you can't expect people to write SQL queries to query everything. So how do you, how did we do it? Well, it was a web form. You know, where you could formulate your query using that form that, and in the backend that, that was translated into SQL query, got the results and got it back.

• 12:01Speaker 00

  But as a result, that because you could do everything, everything was made available. The web form had like 25 different checkboxes, 15 text fields, 18 different dropdowns. The interface was a mess. It was very daunting. So.

• 12:18Speaker 00

  And ESPN acquired Cricket Info in the mid 2006, I think. But they still kept the same interface. And that has always sort of nagged me. And so I still know the people.

• 12:32Martin Casado?

  Wait, wait. What nagged you? Is that Cricket Info did not have informal language? It had a web form for doing queries?

• 12:38Speaker 00

  That web form was terrible.

• 12:43Speaker 00

  Because that only the real nerds use.

• 12:45Martin Casado?

  Of all the things in the world to bother you, the fact that an old website was a web form. I appreciate, I appreciate your commitment to aesthetic.

• 12:57Speaker 00

  So I'm still friendly with the people who run ESwent and for the editor in chief, whenever he comes to New York, you know, we meet up, we go out for a drink. And so he was here in 2000. So now the story shifts to how LLMs and me sort of met. So January 2000, right before the pandemic, he was here. And I again said, why don't you do something about StatsGuru? And he looks at me and says, why don't you do something about StatsGuru? He was kind of joking, but he thought maybe, you know, I had some ways to fix the interface. So anyway, then the pandemic hit, the world stopped. But

• 13:35Speaker 00

  in July of 2020, the first version of GPT-3 was released. And I saw someone use GPT-3 to write a SQL query for their own database using natural language. And I thought, can I use this to fix StatsGuru? So I got early access to GPT-3, you know, getting access those days was difficult, but somehow I got it. But soon I realized that, you know, no, I cannot really do it. Because StatsGuru, the backend databases were so complex. And if you remember, GPT-3

• 14:14Speaker 00

  had only a 2048 token context window. There was no way in hell I could fix the complexities of that database and that context window. And GPT-3 also did not do instruction following at that time. But then in trying to solve this problem, I accidentally invented what's now called RAG. Where based on the natural language query, I created a database of natural language queries and structure, the structured queries. I created a DSL, which then translated into a REST call to StatsGuru.

• 14:56Speaker 00

  So based on the new query, I would look through my set of natural language queries. I had about fifteen hundred examples and I would pick the six or seven most relevant ones. And then that and the structured query I would send as a prefix and the new query then G P T three magically completed it and the accuracy was very high. So that had been running in production since September twenty twenty one. You know, about fifteen months before chat G P D came. And you know, the whole revolution in some sense started and RAG became very popular. I didn't call it RAG, but this is something sort of I accidentally did in trying to solve that problem for quick info. Now, once I

• 15:39Speaker 00

  once I built it, you know, I was thrilled that this worked, but I had no idea why it worked. You know, I stared at that, I stared at that what transformer architecture diagram. I read those papers, but I couldn't understand how or why it worked. So then I started in this journey of developing a mathematical model, trying to understand how it worked.

• 16:04Speaker 00

  So that's been sort of my

• 16:07Speaker 00

  journey through this world of A I and L L M's because I was trying to solve this cricket problem.

• 16:13Erik Torenberg?

  Amazing. And so maybe reflecting back since the release of G P T three, what has most surprised you about how L L M's have have developed?

• 16:22Speaker 00

  So what has most surprised me, the pace of development. So G P T three was, you know, it was a nice parlor trick and you had to jump through hoops to get it to do something useful. But starting with the, you know, chat G P D was an advance over G P T three. And then you had all these things like chain of thought, instruction following. G P T four really made it polished. And you know, the pace of development has really surprised me. Now, you know, when I started working with G P T three, I could sort of see what its limitations were, what I could make it do, what I couldn't make it do. But I never thought of it as, you know, what it, what these L L M's have become for me now and what what have become

• 17:07Speaker 00

  from billions of people around the world. We treat these models as our coworkers. Almost like an intern that, you know, you're constantly chatting with them, brainstorming, making them do all sorts of work, which we couldn't imagine, you know. Just when ChatGPT was released, it was nice. It could write poems, it could write limericks. It could answer some hallucinated questions. But the capabilities that have emerged now, that pace has been very surprising to me.

• 17:40Erik Torenberg?

  Do you see progress plateauing? Or how do you, either now or in the near future? How do you see it going?

• 17:48Speaker 00

  I, yes. In some sense, progress is plateauing. It's like the iPhone. You know, when the iPhone came out, wow, what is this thing? And the early iterations, you know, constantly we were amazed by new capabilities. But the last, you know, seven, eight, nine years, it's maybe the camera got a little bit better or, you know, one thing changed here or memory is more. But there has been no fundamental advance in what it's capable of. You can sort of see a similar thing happening with these LLMs. And this is not true for just one, one company and one model, right? You look at what OpenAI is coming up with

• 18:32Speaker 00

  or what Anthropic, Google,

• 18:37Speaker 00

  or all these open source Chinese model or Mistral. The capabilities of LLMs has not fundamentally changed. They've become better, right? They've improved, but they have not crossed into a different realm.

• 18:53Martin Casado?

  So this is something that I really appreciate about your work. And so the thing that really struck me is as soon as these things showed up, you actually got busy trying to have a formal model of what they're capable of, which was in stark contrast to what everybody else was doing. Everybody else was like AGI. These things are gonna, you know, recursively self-improve. Like, or, or, or they'll say, oh, these are just stochastic parrots, which doesn't mean anything. So everybody had rhetoric. And sometimes this rhetoric was fanciful. And sometimes this rhetoric was almost reductionist. Like, oh, it's just a database, which is clearly not true. And the thing that really struck me about your work is you're like, no, let's figure out exactly what's going on.

• 19:38Martin Casado?

  Let's come up with a formal model. And once we have a formal model, we could reason about what that means. And then, you know, in my reading of your work, I kind of break it up in two pieces. There's the first one where you basically, you came up with this, you know, matrix abstraction. I think it's worth you talking through. And then you took in-context learning as an example and you mapped it to Bayesian reasoning, which to me was incredibly powerful because at the time, nobody knew why in-context learning worked. So I think it'd be great for you to discuss that because again, I think it was the first real kind of formal effect on like, like, how are these things working? And then the more recent work that you're working on now is a kind of more generalized

• 20:20Martin Casado?

  version of, of, of what is the state space that these models output when it comes to, comes to confidence, which is the manifold that we're talking about before. So I would, I think it'd be great if you just described your matrix model and then how you use that to just, to, to provide some bounds what in-context learning is doing, what's happening.

• 20:45Speaker 00

  Okay, so, so yeah, let's start with that matrix abstraction. So, so the idea behind the matrix is you have this gigantic matrix where every row corresponds to a prompt.

• 21:00Speaker 00

  And then the number of columns of this matrix is the vocabulary of the LLM, the number of tokens it has that it can emit. So for every prompt, this matrix contains the distribution over this vocabulary.

• 21:16Martin Casado?

  Yep.

• 21:17Speaker 00

  So when you say the cat sat on the, you know, the column that corresponds to mat will have a high probability. Most of them will be zero, but you know, reasonable uh continuations will have a non-zero probability. And so you can imagine that there's this gigantic matrix. Now, the size of this matrix is, you know, if you just take just the old uh first generation GPT-3 model, which had a context window of 2,000 tokens and a vocabulary of 50,000

• 21:53Speaker 00

  Next tokens or 50,000 tokens. Then the size of it, the number of rows in this matrix is more than the number of atoms across all galaxies that we know of. So clearly we cannot represent it exactly. Now fortunately, a lot of these rows are, do not appear in real life, right? An arbitrary collection of tokens, you are not going to use that as a prompt. Similarly, you saw so a lot of these rows are absent and a lot of the column values are also zero, right? When you say the cat sat on the, it's unlikely to be followed by the token corresponding

• 22:37Speaker 00

  to let's say numbers or you know, an arbitrary collection of tokens. There will be only a very small subset of tokens that can follow a particular prompt. So this matrix is very, very sparse. But even after that sparsity and even after removing the sort of gibberish prompts, the size of this matrix is too much for these models to represent, even with a trailing parameters. So what in an abstract sense, what is happening is the models get trained on certain, you know, data from the training set and certain some, a subset, a small subset of these rows,

• 23:20Speaker 00

  you have reasonable values for the next token distribution. Whenever you give the prompt something new, right? Then it'll try to interpolate with what it has learned and what's there in the new prompt and come up with a new distribution. But it's basically, so it's more than a stochastic parrot. It is sort of Bayesian on this, on this subset of the matrix that it has been trained on. So, so when I say, you know, I'm going out for dinner with Martin tonight.

• 24:00Speaker 00

  Now, I'm reasonably sure that it has never encountered that phrase in its training data, right? But it has encountered variants of this phrase.

• 24:12Speaker 00

  And given that I'm going out with Martin, it it can produce a Bayesian posterior. It uses that evidence that Martin is the one that I'm going for dinner with. And it'll produce a next token distribution that will focus on the likely places that we are going.

• 24:30Speaker 00

  So so this matrix, because it's represented in a compressed way, yet the models respond to everything, every prompt. How do they do it? Well, they they go back to what they've been trained on, interpolate there and use the prompt as sort of some evidence to compute a new distribution.

• 24:52Martin Casado?

  Right, so right, so the the the context of the prompt impacts the posterior distribution.

• 24:59Speaker 00

  Exactly, yeah.

• 25:01Martin Casado?

  Right, and you mapped to Bayesian learning where the the the context is the new evidence.

• 25:10Speaker 00

  New evidence, exactly. So so I'll give you, so so for instance, the cricket example that I spoke about earlier.

• 25:17Martin Casado?

  Yeah.

• 25:17Speaker 00

  So I I created my own DSL.

• 25:19Martin Casado?

  Yep.

• 25:20Speaker 00

  Which, you know, mapped a natural language query in cricket to this DSL, which then I can translate into a SQL query or a REST API, whatever. But getting the DSL is important. Now, these LLMs have never seen that DSL. I designed it.

• 25:35Martin Casado?

  Yeah.

• 25:36Speaker 00

  Right? But yet, after showing a few examples, it learned it.

• 25:42Speaker 00

  How did it learn it?

• 25:43Martin Casado?

  And this is this is in the prompt. You didn't, no training, no post.

• 25:46Speaker 00

  It's in the prompt.

• 25:47Martin Casado?

  Hundred percent in the prompt, right? So like it's the the weights are varied, yeah, yeah.

• 25:51Speaker 00

  Yeah, yeah, this was this was happening in October 2020. Right? I had no access to internals of OpenAI. I could just, you know, access their API. OpenAI had no access to internal structure of Stats Guru or the DSL that I cooked up in my head. Yet, after showing it only a few examples, it learned it right away. So that's an example where it has seen DSLs or structures in the past.

• 26:20Speaker 00

  And now using this evidence that I show, okay, this is what my DSL looks like. Now, a new natural language query, it is able to create the right posterior distribution for the tokens. That map to the example that I've seen. Now, the other beautiful thing about this is,

• 26:40Speaker 00

  this is an example of few-shot learning or in-context learning, right? But when I give that prompt along with these examples to this LLM, I'm not saying to the LLM, "Okay, this is an example of few-shot learning, so learn from these examples." Right? You just pass this to the LLM as a prompt, and it processes it exactly the way it would process any other prompt, which is not an example of in-context learning. So that really means that the underlying mechanism is the same. Right? Whether you give a set of examples and then ask it to complete a task like in-context learning,

• 27:21Speaker 00

  or just give it some prompt for continuation, and "I'm going out for dinner with Martin tonight." There's no in-context learning there. But the process with which it's generating or doing this inferencing is exactly the same. And that's what I have been trying to model and come up with a formal model of.

• 27:42Martin Casado?

  What I've found very impressive is you've used this basic model to show a number of things, right? To describe in-context learning and to map to basic learning. But you did it for another one where you kind of, you've sketched out this almost glib argument on Twitter, on X, where you made this,

• 28:04Martin Casado?

  you made a rough argument for why recursive self-improvement can't happen without additional information. And so maybe just walk through very quickly how like this same model, you can just very quickly show that a model can never recursively self-improve.

• 28:22Speaker 00

  So, you know, another phrase that

• 28:29Speaker 00

  we've been using recently is, you know, the output of the LLM is the inductive closure of what it has been trained on.

• 28:35Martin Casado?

  Yeah.

• 28:37Speaker 00

  So when you say that it can recursively self-improve.

• 28:44Speaker 00

  It could mean one of two things. So let's get back to the.

• 28:47Martin Casado?

  Well, actually, you know what's kind of interesting is like often the most people agree that if you have one LLM and you just feed the output into the input, like it's not going to do anything. But then often people will say, well, what if you have two LLM, you have no external information, but you have two LLMs talking to each other. Maybe they can improve each other and then you can have like, you know, a takeoff scenario. But again, you even address this, even in the case of like N number of LLMs using kind of the matrix model to show that like you just aren't getting any information.

• 29:18Speaker 00

  Yeah.

• 29:18Martin Casado?

  Entry.

• 29:19Speaker 00

  Yeah. So you can represent the sort of information contained in these models. And let's go back to that matrix analogy that have the matrix abstraction. So like I said, you know, these models are represent a subset of the rows. Right?

• 29:38Martin Casado?

  Yeah.

• 29:38Speaker 00

  So a subset of the rows are represented, but some of these rows are able to help fill out some of the missing rows. For instance, you know, if the model knows how to do multiplication during the step by step, then every row that is corresponding to let's say 769 times 125 or whatever, all those.

• 30:01Martin Casado?

  It can fill out the answer. Yeah.

• 30:02Speaker 00

  It can fill out the answer because it has those algorithms sort of embedded in them. You just need to unroll them.

• 30:09Martin Casado?

  Yeah.

• 30:10Speaker 00

  So it can sort of self-improve up to a point. But beyond the point, these models can only

• 30:19Speaker 00

  sort of generate what they have been trained on. So let me give you. I'll give you three examples.

• 30:25Martin Casado?

  Yeah.

• 30:26Speaker 00

  So any model, any LLM that was trained on

• 30:33Speaker 00

  pre 1915 physics would never have come up with a theory of relativity.

• 30:40Speaker 00

  Einstein had to sort of reject the Newtonian physics and come up with this spacetime continuum. He completely rewrote the rules, right? So that is an example of, you know, AGI.

• 30:53Speaker 00

  Where you are generating or generating new knowledge. It's not simply.

• 30:58Martin Casado?

  About the universe, right? It's like it's not computing something. It's actually discovering something fundamental about the universe.

• 31:04Speaker 00

  Fundamental and for that you have to go outside your training set. Similarly, you know, any L L M that was trained on it would not have come up with quantum mechanics. Right, that that's way particle duality or this whole probabilistic notion. Or that, you know, energy is not continuous, but it is quantized. You had to reject Newtonian physics.

• 31:23Martin Casado?

  Yeah.

• 31:24Speaker 00

  Or Gettle's incompleteness theorem.

• 31:27Martin Casado?

  Yeah.

• 31:27Speaker 00

  He had to go outside the axioms to say that, okay, it is incomplete. So those are examples where you're creating new science or fundamentally new results. That kind of self-improvement is not possible with these architectures. They can refine these, they can fill out these roles. Where the answer already exists. Another example, you know, which has received a lot of press these days is these I M O results. International Math Olympiad.

• 31:57Martin Casado?

  Yeah.

• 31:57Speaker 00

  You know, whether it's a human solving it or the L L M solving it. They are not inventing new kinds of math.

• 32:07Speaker 00

  They are able to connect known results in a sequence of steps to come up with the answer.

• 32:16Speaker 00

  So even the L L M, what they are doing is they are exploring all sorts of solutions. In some of these solutions, they they start going on this path where their next token entropy is low.

• 32:30Speaker 00

  So that's where I say they are in that Bayesian manifold.

• 32:33Martin Casado?

  Yeah, yeah.

• 32:34Speaker 00

  Where you have this entropy collapse. And by doing those steps, you arrive at the answer. But you're not inventing new math. You're not inventing new axioms or new branches of mathematics. You're sort of using what you've been trained on to arrive at that answer. So those things L L M's can do, you know, they'll get better at it of connecting the known dots.

• 32:58Martin Casado?

  Yeah.

• 32:59Speaker 00

  But creating new dots, I think we need. An architectural advance.

• 33:06Erik Torenberg?

  Yeah. So Martin was talking earlier about how the discourse was either stochastic parrots or AGI recurses. How are you, how do you conceive of the AGI discourse or even the concept? What does it mean to the extent that it's useful? How do you think about that?

• 33:25Speaker 00

  So the way I think about it, the way we have tried to formulate in our papers, it's beyond a stochastic parrot, but it's not AGI. It's doing Bayesian reasoning over what it has been trained on. It's a lot more sophisticated than just a stochastic parrot.

• 33:42Erik Torenberg?

  How do you define AGI?

• 33:46Speaker 00

  Okay, so AGI, so how do I define AGI? So the way I would say that LLMs currently navigate through this known Bayesian manifold, AGI will create new manifolds. So right now, these models navigate, they do not create. AGI will be when we are able to create new science, new results, new math. When an AGI comes up with a theory of relativity, I mean, it's an extremely high bar, but you get what I'm saying. It has to go beyond what it has been trained on to come up with new paradigms, new science.

• 34:29Speaker 00

  That's my definition of AGI.

• 34:31Martin Casado?

  Vishal, can you, do you think that based on the work you've done, can you bound the amount of data, computer or data or compute that would be needed in order for it to evolve? So one of the problems, if you just take LLMs as they exist, is there was so much data used to create them. To create a new manifold will need a lot more data just because of the basic mechanisms, right? Otherwise, it'll just kind of like, you know, get kind of consumed into the existing set of data. Like, have you found any bounds of what would be needed to actually evolve the manifold in a useful way? Or do you think we just need a new architecture?

• 35:14Speaker 00

  I personally think that we need a new architecture. The more data that we have, the more compute we have, we'll get maybe smoother manifolds. So it's like a map.

• 35:24Martin Casado?

  Yeah, because I mean, there's this view that people have, they're like, well. Vishal, this is all good and well, but I could just take an LLM and I can give it eyes and I can give it ears and I could put it in the world and it'll gain information and based on that information, it'll improve itself and therefore it can learn new things. But the counterpoint that I've always just intuitively thought to that is the amount of data used to train these things is so large, how much can you actually evolve that manifold given incremental, I mean, almost none at all, right? There has to be some other way to generate new manifolds that aren't evolving the existing one.

• 36:03Speaker 00

  I completely agree. There has to be a new sort of architectural leap that is needed to go from the current, you know, just throwing more data and more compute, you know, it's going to plateau. It's, you know, the iPhone 15, 16, 17.

• 36:19Erik Torenberg?

  And are there any research directions that are promising in your mind that might help us, you know, go beyond LLM limitations?

• 36:26Speaker 00

  So, I mean, again, I love LLMs. They are fantastic and they are going to increase productivity like nobody's business, but I don't think they are the answer. So, you know, Yad Likkun famously says that LLMs are a distraction on the road to AGI.

• 36:46Martin Casado?

  They're a dead end, they're a dead end, AGI.

• 36:47Speaker 00

  I don't think, I'm not quite in that camp, but I think we need a new, new architecture to sit on top of LLMs to reach AGI. You know, a very basic thing, you know, what Martin just said, you give them eyes and you give them ears, you make them multimodal, of course, they'll become more powerful. But you need a little bit more than that. You know, the way human brains learns with very few examples, that's not the way transformers learn. And, you know, I'm not saying that we need to create an Einstein or a Gator, but there has to be an architectural leap that is able to create these manifolds and just throwing new data will not do it. It'll just smoothen out the already existing manifolds.

• 37:31Martin Casado?

  Is that something, so is your goal to actually help, like, think through new architectures or are you primarily focused on putting formal bounds on existing architectures?

• 37:44Speaker 00

  A bit of both. I mean, the former goal is the more ambitious one that everybody is seizing and yeah, I think about that constantly.

• 37:54Erik Torenberg?

  Are there any new even like sort of hints at a new architect or like have we started to make any progress on new architectures? Or is it?

• 38:09Speaker 00

  You know, Jan has been pushing at this JPA architecture, energy-based architectures, they seem promising. The way I have been sort of thinking about it is,

• 38:26Speaker 00

  you know, there's this set of benchmarks or the ARC Prize.

• 38:32Erik Torenberg?

  Yeah.

• 38:33Speaker 00

  Right, that Mike Kanup and Francois Chalais have. And if you understand why the LLMs are failing on this test, maybe you can sort of reverse engineer a new architecture that will help you succeed in that, right? And I agree with a lot of what several people say that, you know, language is great, but language is not the answer. You know, when I'm looking at catching a ball that is coming to me, I'm mentally doing that simulation in my head. I'm not translating it to language to figure out where it'll land. I do that simulation in my head.

• 39:14Speaker 00

  So, you know, one of the new architectures, architectural things is, how do we do, how do we get these models to do approximate simulations? To test out that idea and whether to proceed or not. So, so yeah, you know, another thing that I've always wondered about is, did we develop as humans, did we develop language because we were intelligent? Or because we developed language, we accelerated our intelligence?

• 39:49Speaker 00

  So I don't know which side of the camp you fall on that question.

• 39:53Martin Casado?

  Well, I mean, what's interesting is like you have these anecdotal examples of humans developing languages de novo that have been recorded, right? Like it's either the Guatemalan or Nicaraguan sign language, right? Where there is these students that develop their own language without being taught. And so that would suggest that language just follows intelligence. The problem is, is they're all anecdotal, right? Like who knows if somebody didn't teach them sign language? Like, nobody really knows, there is no controls. So this is all these observational studies and there's so few of them. You have to wonder if it's just kind of sloppy observation. And so I think that the question is still outstanding.

• 40:33Speaker 00

  Yeah. So

• 40:37Speaker 00

  I mean, language definitely accelerated our intelligence. There's no question about that.

• 40:42Martin Casado?

  Yeah.

• 40:42Speaker 00

  But which followed which, we don't know.

• 40:45Martin Casado?

  But I view it as a networking problem naturally, which is once you have languages, you can communicate.

• 40:50Speaker 00

  You can communicate, you can store, you can replicate, yeah?

• 40:53Martin Casado?

  Yeah.

• 40:53Speaker 00

  Exactly.

• 40:54Martin Casado?

  Exactly, right. Cool. Again, this is kind of a wonky question, but

• 41:02Martin Casado?

  you know, I think one thing that you've brought to the discourse, and for those that are listening to this, I really think that you should look up Vishal's work and read it. I just think it'll give you a really, really, especially if you have a systems background, like a networking or systems background, give you a really, really good understanding of kind of the bounds on these. But like the toolkit that you draw from is like information theory and like more formal. Have you found that the AI community is receptive to this? Or is it like two different cultures, two different planets trying to communicate and not a lot of common ground? Like, how have you found like bringing like the networking view of the world to the AI realm?

• 41:40Speaker 00

  Some of them are receptive to it, definitely. But

• 41:46Speaker 00

  you know, these large conferences and their reviewing process, it's so random. And the kind of questions they ask, you know, I'm a modeling person. I like to bottle things. And, you know, I submitted one version of this work to one very famous

• 42:05Speaker 00

  machine learning or AI conference and the reviewer said, okay, this is a model, so what?

• 42:12Speaker 00

  So that is.

• 42:19Martin Casado?

  That's absolutely remarkable. So like you've actually taken a system that nobody understands. We have no models for it. You actually provided some model that we can use to analyze it. And that alone wasn't sufficient.

• 42:30Speaker 00

  They're asking, so where are the large scale experiments to prove this?

• 42:35Martin Casado?

  I do, listen, I honestly, I mean, I find there's so much empiricism in like the current, you know, AI community, exactly because we don't understand the systems. You know, it kind of reminds me, I feel like, I feel like systems went the other way, right? It's like, we had all of these models, but then we didn't understand how the systems worked. And then we just like actually did measurement. It feels like ML and the AI system is the opposite, which is like, we know we don't understand them. And so we just measured them, but now we're trying to like come up with the models.

• 43:05Speaker 00

  Yeah, exactly. So it was so easy in some sense to build these artifacts and then just measure them that people have been going on, trying to do that. And you know, one term I really dislike is prompt engineering.

• 43:23Martin Casado?

  Why?

• 43:23Speaker 00

  You know, engineering used to mean sending a man to the moon or providing five nights reliability. Prompt engineering is prompt twiddling.

• 43:32Martin Casado?

  Yeah.

• 43:33Speaker 00

  You fiddle with a prompt and the model changes and the inference, the output changes. And you know, you have like hundreds of papers, just, just, you know, doing one experiment on the other, changing a prompt this way, that way, and writing their observations. And as a result, you know, lots of these papers are being written, are being submitted for review. Reviewers get busy looking at all this kind of empirical work. And my personal taste is to first try to understand, model it.

• 44:05Martin Casado?

  Yeah.

• 44:06Speaker 00

  And then you can do the other thing.

• 44:08Martin Casado?

  So like a truth theory guy.

• 44:12Martin Casado?

  I don't know about this bit twiddling.

• 44:15Erik Torenberg?

  Let me ask one more LM question, which is, are there any benchmarks or real world tasks that if they, if they occurred, you'd sort of reevaluate and say, hey, maybe LLMs are, you know, closer to the path to AGI than I thought?

• 44:31Speaker 00

  If there were any real world tasks.

• 44:40Speaker 00

  Good question.

• 44:47Speaker 00

  You know, which for LLMs or these models, the one domain where you have the most training data is probably coding.

• 45:03Speaker 00

  And coding is where you can also have the most structure. And yet, anyone who has used these tools, whether it's Curse or whatever, or Cloud Code,

• 45:20Speaker 00

  LLMs continue to hallucinate, continue to generate unreasonable code. You know, you have to, you have to constantly babysit these models. So the day an LLM can create a large software project without any babysitting is the day I'll be a little bit convinced that it's towards easy. But again, I don't think it'll be able to create new science. If it does, that's when I'll be convinced.

• 45:54Martin Casado?

  I, you know, I think that you can almost take a definitional approach to answer this question, Vishal. Like, the problem with these types of questions is, is if you have billions of dollars and you can collect whatever data you want, you can make a model do anything you want, right? And so like, you know what I'm saying? Like, at some level, you've got this entire capital structure machinery behind these models. So you're like, oh, it can be good at science. Well, sure, you put a billion dollars in solving materials science and collect all this data, you'll be good at material science or, or whatever it is. And so, but there is a definitional answer, which is. And I'm going to draw from your work, which is there is a manifold that's in there based on the data it's been training on.

• 46:35Martin Casado?

  And then the question is, is if it ever produces something that's off, like a new manifold. So considering the existing traded data, if it ever does that, if it does something that's outside of that distribution, then clearly we're on a path to learning new things. And if not, then everything is just a computational step from what's already known.

• 46:52Speaker 00

  Yeah.

• 46:53Martin Casado?

  And then I guess, I guess the counter, I guess the counter to that would be maybe all humans do is work on their own manifold and Einstein, you know, was lucky or something, I guess, would be the counter to that, but I.

• 47:07Speaker 00

  Well, you know, there's several mini Einstein examples and yeah, it's creating this new manifold. I didn't want to use that definition answer. I thought it might sound too, too wonky, too mathematical, but essentially if LLMs really created this new manifold, then I would be convinced. But so far, they have just gotten better at navigating the existing manifold, the existing training set.

• 47:34Martin Casado?

  Which is hugely powerful and is going to change the world.

• 47:36Speaker 00

  Which is hugely powerful. I'm not denying that. I think they are extremely, extremely good at what they can do, but there's a limit to what they can do.

• 47:43Martin Casado?

  So I have one quick question. What's next for you? I mean, you've, you've tackled in context learning, you've got a model for LLMs, and I've got a generalized model for like, you know, like their solution space. What are you thinking about tackling next?

• 47:56Speaker 00

  In terms of modeling or?

• 48:00Martin Casado?

  Academically, an LLM.

• 48:03Speaker 00

  Academically, academically, I'm, you know, I'm thinking of this, what is the architectural leap that is needed?

• 48:13Martin Casado?

  Oh, that's exciting.

• 48:14Speaker 00

  To create this new manifold and how do we use, you know, multimodal data?

• 48:20Martin Casado?

  Awesome.

• 48:21Speaker 00

  To expand.

• 48:23Martin Casado?

  When you figure that out, come back and talk to us.

• 48:25Speaker 00

  That's right. We'd love that. So I mean, you know, even with LLMs, you know, in the paper, we say that you can improve the inference by following this low or minimum entropy path. So that's a very sort of small step that we are taking, you know, we are building and training models that'll do inference based on the entropy path.

• 48:50Martin Casado?

  Yeah.

• 48:51Speaker 00

  By the way, is model probe still up? Token Probe. Yeah, yeah. Token Probe is still up and you can see actually the, you know, Token Probe is a software that we built and thanks to Martin and A16Z's generosity, it's running on your servers and anyone can go and test. And what we have done there is we actually show the entropy.

• 49:14Martin Casado?

  Yeah. It is so enlightening. I recommend anybody listening to this who's interested. Actually, check out Token Probe. It literally shows you the confidence. Yeah, as you go along. It's remarkable.

• 49:24Speaker 00

  So in context learning, you know, you create your new DSL and you give it to the prompt and you can see the confidence rising with each new example, the entropy reducing. And that sort of is a validation of the model. You can see it sort of unfolding and right in front of your eyes. The Token Probe is still running. Thanks. Thanks again. Michelle, thanks so much for coming on the podcast. It was a great conversation. Of course. It was great fun. Thank you. Thank you so much again.

• 49:54Speaker 01

  Thanks for listening to this episode of the A16Z Podcast. If you liked this episode, be sure to like, comment, subscribe, leave us a rating or review, and share it with your friends and family. For more episodes, go to YouTube, Apple Podcasts, and Spotify. Follow us on X at A16Z and subscribe to our Substack at A16Z.substack.com. Thanks again for listening, and I'll see you in the next episode. As a reminder, the content here is for informational purposes only, should not be taken as legal, business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any A16Z fund. Please note that A16Z and its affiliates may also maintain investments in the companies discussed in this podcast. For more details, including

• 50:39Speaker 01

  a link to our investments, please see A16Z.com/disclosures.