Andrej Karpathy — AGI is still a decade away

The Andrej Karpathy https://x.com/karpathy episode. During this interview, Andrej explains why reinforcement learning is terrible (but everything else is much worse), why AGI will just blend into the previous ~2.5 centuries of 2% GDP growth, why self driving took so long to crack, and what he sees as the future of education. It was a pleasure chatting with him. Watch on YouTube https://youtu.be/lXUZvyajciY; read the transcript https://dwarkesh.substack.com/p/andrej-karpathy. Sponsors * Labelbox https://labelbox.com/dwarkesh helps you get data that is more detailed, more accurate, and higher signal than you could get by default, no matter your domain or training paradigm. Reach out today at labelbox.com/dwarkesh https://labelbox.com/dwarkesh * Mercury https://mercury.com helps you run your business better. It’s the banking platform we use for the podcast — we love that we can see our accounts, cash flows, AR, and AP all in one place. Apply online in minutes at mercury.com https://mercury.com * Google’s Veo 3.1 https://blog.google/technology/ai/veo-updates-flow/ update is a notable improvement to an already great model. Veo 3.1’s generations are more coherent and the audio is even higher-quality. If you have a Google AI Pro or Ultra plan, you can try it in Gemini today by visiting https://gemini.google Timestamps (00:00:00) – AGI is still a decade away (00:29:45) – LLM cognitive deficits (00:40:05) – RL is terrible (00:49:38) – How do humans learn? (01:06:25) – AGI will blend into 2% GDP growth (01:17:36) – ASI (01:32:50) – Evolution of intelligence & culture (01:42:55) - Why self driving took so long (01:56:20) - Future of education This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.dwarkesh.com https://www.dwarkesh.com?utm_medium=podcast&utm_campaign=CTA_1

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Dwarkesh Patel?

  Today, I'm speaking with Andrej Karpathy. Andre, why do you say that this will be the decade of agents and not the year of agents?

• 0:07Speaker 01

  Well, first of all, thank you for having me here. I'm excited to be here. So the quote that you've just mentioned, it's the decade of agents. That's actually a reaction to an existing, pre-existing quote, I should say, where I think a lot of, some of the labs, I'm not actually sure who said this, but they were alluding to this being the year of agents with respect to LLMs and how they were going to evolve. And I think I was triggered by that because I feel like there's some overpredictions going on in the industry. And in my mind, this is really a lot more accurately described as the decade of agents. And we have some very early agents that are actually like extremely impressive and that I use daily. You know, Claude and Codex and so on, but I still feel like there's so much work to be done. And so I think my, like, my reaction is like, we'll be working with these things for a decade. They're going to get better and it's going to be wonderful.

• 0:52Speaker 01

  But I think I was just reacting to the timelines, I suppose, of the, of the implication.

• 0:58Dwarkesh Patel?

  And what do you think will take a decade to accomplish? What are the bottlenecks?

• 1:02Speaker 01

  Well, actually make it work. So in my mind, I mean, when you're talking about an agent, I guess, or what the labs have in mind and what maybe I have in mind as well, is it's, you should think of it almost like an employee or like an intern that you would hire to work with you. So for example, you work with some employees here. When would you prefer to have an agent like Claude or Codex do that work? Like currently, of course, they can't. What would it take for them to be able to do that? Why don't you do it today?

• 1:24Dwarkesh Patel?

  Yeah.

• 1:24Speaker 01

  And the reason you don't do it today is because they just don't work. So like, they don't have enough intelligence, they're not multimodal enough, they can't do computer use and all this kind of stuff. And they don't do a lot of the things that you've alluded to earlier. You know, they don't have continual learning. You can't just tell them something and they'll remember it.

• 1:38Dwarkesh Patel?

  Yeah.

• 1:38Speaker 01

  And they're just cognitively lacking and it's just not working. And I just think that it will take about a decade to work through all of those issues.

• 1:44Dwarkesh Patel?

  Interesting. So as a professional podcaster and a viewer of AI from afar, it's easy to identify for me, like, oh, here's what's lacking. Continual learning is lacking or multimodality is lacking. But I don't really have a good way of trying to put a timeline on it. Like, if somebody's like, how long will continual learning take? There's no prior I have about like this is a project that should take five years, 10 years, 50 years. Why a decade? Why not one year? Why not 50 years?

• 2:16Speaker 01

  Yeah, I guess this is where you get into like a bit of, I guess, my own intuition a little bit. And also just kind of doing a bit of an extrapolation of with respect to my own experience in the field, right? So I guess I've been in AI for almost two decades. I mean, it's gonna be maybe 15 years or so, not that long. You had Richard Sutton here, who was all around, of course, for much longer. But I do have about 15 years of experience of people making predictions of seeing how they actually turned out. And also, I was in the industry for a while and I was in research and I've worked in the industry for a while. So I guess I kind of have just a general intuition that I have left from that. And I feel like the problems are tractable, they're surmountable.

• 2:52Dwarkesh Patel?

  Yeah.

• 2:53Speaker 01

  But they're still difficult. And if I just average it out, it just kind of feels like a decade, I guess, to me.

• 2:58Dwarkesh Patel?

  This is actually quite interesting. I want to like hear not only the history, but what people in the room felt was about to happen at various different breakthrough moments.

• 3:09Speaker 01

  Right.

• 3:09Dwarkesh Patel?

  What were the ways in which their feelings were either overly pessimistic or overly optimistic?

• 3:14Speaker 01

  Yeah.

• 3:14Dwarkesh Patel?

  Yeah, maybe should we just go through each of them one by one?

• 3:16Speaker 01

  Oh, yeah. I mean, that's a giant question because, of course, you're talking about 15 years of stuff that happened. I mean, AI is actually like so wonderful because there have been a number of, I would say, seismic shifts.

• 3:24Dwarkesh Patel?

  Yeah.

• 3:24Speaker 01

  That were like the entire field has sort of like suddenly looked a different way, right? And I guess I've maybe lived through two or three of those.

• 3:30Dwarkesh Patel?

  Yeah.

• 3:31Speaker 01

  And I still think there will continue to be some because they come with some kind of like almost surprising irregularity. Well, when my career began, of course, like when I started to work on deep learning, when I became interested in deep learning, this was just kind of like by chance of being right next to Jeff Hinton at the University of Toronto. And Jeff Hinton, of course, is kind of like the godfather figure of AI. And he was training all these neural networks and I thought it was incredible and interesting. But this was not like the main thing that everyone in AI was doing by far.

• 3:54Dwarkesh Patel?

  Yeah.

• 3:55Speaker 01

  This was a niche little subject on the side. That's kind of maybe like the first like dramatic sort of seismic shift that came with the AlexNet and so on. I would say like Alex and it sort of reoriented everyone and everyone started to train neural networks. But it was still like very like per task, per specific task. So maybe I have an image classifier or I have a neural machine translator or something like that. And people became very slowly actually interested in basically kind of agents, I would say. And people started to think, okay, well, maybe we have a checkmark next to the visual cortex or something like that. But what about the other parts of the brain and how can we get an actual like full agent or in full entity that can actually interact in the world? And I would say the Atari sort of deep reinforcement learning shift in two thousand thirteen or so was part of that early effort of agents in my mind.

• 4:38Speaker 01

  Because it was an attempt to try to get agents that not just perceive the world, but also take actions and interact and get rewards from environments. And at the time, this was Atari games, right? And I kind of feel like that was a misstep, actually. And it was a misstep that actually even the early OpenAI that I was a part of, of course, kind of adopted. Because at that time, the zeitgeist was reinforcement learning environments, games, game playing, beat games, get lots of different types of games. And OpenAI was doing a lot of that. So that was maybe like another like prominent part of, I would say, AI where maybe for two or three or four years, everyone was doing reinforcement learning on games. And basically, that was a little bit of a misstep. And what I was trying to do at OpenAI actually is like, I was always a little bit suspicious of games as being like this thing that would actually lead to AGI.

• 5:23Speaker 01

  Because in my mind, you want something like an accountant or like something that's actually interacting with the real world. And I just didn't see how games kind of like add up to it. And so my project at OpenAI, for example, was within the scope of the Universe Project on an on an agent that was using keyboard and mouse to operate webpages. And I really wanted to have something that like interacts with, you know, the actual digital world that can do knowledge work. And it just so turns out that this was extremely early, way too early. So early that we shouldn't have been working on that, you know? Because if you're just stumbling your way around and keyboard mashing and mouse clicking and trying to get rewards in these environments, your reward is too sparse and you just won't learn and you're going to burn a forest computing. And you're never actually going to get something off the ground.

• 6:07Speaker 01

  And so what you're missing is this power of representation in the neural network. And so, for example, today, people are training those computer-using agents, but they're doing it on top of a large language model. And so you actually have to get the language model first, you have to get the representations first. And you have to do that by all the pre-training and all the LLM stuff. So I kind of feel like maybe loosely speaking, it was like people keep maybe trying to get the full thing too early a few times where people like really try to go after agents too early, I would say, and that was Atari and Universe and even my own experience. And you actually have to do some things first before you sort of get to those agents. And maybe now the agents are a lot more competent, but maybe we're still missing sort of some parts of that stack. But I would say maybe those are like the three like major buckets of what people were

• 6:50Speaker 01

  doing, training neural nets per tasks, trying to the first round of agents, and then maybe the L L M's and actually seeking the representation power of the neural networks before you tack on everything else on top.

• 7:02Dwarkesh Patel?

  Interesting. Yeah, I guess if I were to steel man the sort of sudden perspective would be that humans actually can just take on everything at once, or even animals can take on everything at once, right? Animals are maybe a better example because they don't even have the scaffold of language. They just get thrown out into the world and they just have to make sense of everything without any labels.

• 7:20Speaker 01

  Right.

• 7:21Dwarkesh Patel?

  And the vision for AGI then should just be something which like just looks at sensory data, looks at the computer screen, and it just like figures out what's going on from scratch. I mean, if a human was put in a similar situation, that'd be trained from scratch. But I mean, this is like a human growing up or animal growing up. So why shouldn't that be the vision for AI rather than like this thing where we're doing millions of years of training?

• 7:41Speaker 01

  I think that's a really good question, and I think, I mean, so Sutton was on your podcast and I saw the podcast and I had a write up about that podcast almost that gets into a little bit of how I see things. And I kind of feel like I'm very careful to make analogies to animals because they came about by very different optimization processes. Animals are evolved and they actually come with a huge amount of hardware that's built in. And when, for example, my example in the post was the zebra, a zebra gets born and a few minutes later is running around and following its mother. That's an extremely complicated thing to do. That's not reinforcement learning. That's something that's baked in and evolution obviously has some way of encoding the weights of our neural nets and ATCGs. And I have no idea how that works, but it apparently works.

• 8:24Speaker 01

  So I kind of feel like brains just were came from a very different process. And I I'm very hesitant to take inspiration from it because we're not actually running that process. So in my post, I kind of said, we're not actually building animals. We're building ghosts.

• 8:38Dwarkesh Patel?

  Yeah.

• 8:38Speaker 01

  Or spirits, or whatever people want to call it. Because we're not, we're not doing training by evolution. We're doing training by basically imitation of humans.

• 8:48Dwarkesh Patel?

  Yeah.

• 8:49Speaker 01

  And the data that they've put on the internet. And so you end up with these like sort of ethereal spirit entities because they're fully digital. And they're kind of like mimicking humans. And it's a different kind of intelligence. Like, if you imagine a space of intelligences, we're starting off at a different point almost. We're not, we're not really building animals. But I think it's also possible to make them a bit more animal like over time. And I think we should be doing that. And so I kind of feel like, sorry, just, I guess, one more point is, I do feel like Sutton basically has a very, like, his framework is like, we want to build animals.

• 9:16Dwarkesh Patel?

  Yeah.

• 9:16Speaker 01

  And I actually think that would be wonderful. If we can get that to work, that would be amazing. If there was a single like algorithm that you can just, you know, run on the internet and it learns everything, that would be incredible. I almost suspect that, I'm not actually sure that it exists. And that's certainly actually not what animals do. Because animals have this outer loop of evolution.

• 9:34Dwarkesh Patel?

  Right.

• 9:35Speaker 01

  And a lot of what looks like learning is actually a lot more maturation of the brain. And I think there's actually, actually very little reinforcement learning for animals. And I think a lot of the reinforcement learning is actually like more like motor tasks. It's not intelligence tasks. So I actually kind of think humans don't actually like really use RL, roughly speaking is what I would say.

• 9:52Dwarkesh Patel?

  Can you repeat the last sentence? A lot of that intelligence is not motor tasks. It's what, sorry?

• 9:55Speaker 01

  A lot of the reinforcement learning, in my perspective, would be things that are a lot more like motor like, like, uh, simple, kind of like tasks, throwing a hoop, or something like that. But I don't think that humans use reinforcement learning for a lot of intelligence tasks, like problem solving and so on.

• 10:10Dwarkesh Patel?

  Interesting.

• 10:11Speaker 01

  That doesn't mean we don't have, we, we shouldn't do that for research, but I just feel like that's what animals do or don't.

• 10:17Dwarkesh Patel?

  I'm going to take a second to digest that because there's a lot of different ideas. Maybe it was one clarifying question I could ask to understand the perspective. So. I think you suggest that, look, evolution is doing the kind of thing that pre-training does in the sense of building something which can then understand the world. The difference, I guess, is that evolution has to be titrated, in the case of humans, through three gigabytes of DNA. And so that's very unlike the weights of a model. I mean, literally, the weights of the model are a brain, which obviously is not encoded in the sperm and the egg, or does not exist in the sperm and the egg. So it has to be grown. And also, the information for every single synapse in the brain simply cannot exist

• 11:02Dwarkesh Patel?

  in the three gigabytes that exist in the DNA. Evolution seems closer to finding the algorithm, which then does the lifetime learning. Now, maybe the lifetime learning is not analogous to RL, to your point. Is that compatible with the thing you were saying, or would you disagree with that?

• 11:17Speaker 01

  I think so. I would agree with you that there's some miraculous compression going on, because obviously, the weights of the neural net are not stored in ATCGs. There's some kind of a dramatic compression, and there's some kind of like learning algorithms encoded that take over and do some of the learning online. So I definitely agree with you on that. Basically, I would say I'm a lot more practically minded. I don't come at it from the perspective of, like, let's build animals. I come from the perspective of, like, let's build useful things. So I have a hard hat on, and I'm just observing that, look, we're not going to do evolution, because I don't know how to do that. But it does turn out we can build these ghost, spirit-like entities by imitating internet documents. This works. And it's actually kind of like, it's a way to bring you up to something that has a lot of sort of built-in knowledge and intelligence in some way, similar to maybe what evolution has done.

• 12:01Speaker 01

  So that's why I kind of call pre-training this kind of crappy evolution. It's like the practically possible version with our technology and what we have available to us to get to a starting point where we can actually do things like reinforcement learning and so on.

• 12:14Dwarkesh Patel?

  Just to steelman the other perspective, because after doing this on an interview and thinking about it a bit, he has an important point here. Evolution does not give us the knowledge, really, right? It gives us the algorithm to find the knowledge. And that seems different from pre-training. So if perhaps the perspective is that pre-training helps build the kind of entity which can learn better. It teaches meta-learning and therefore it is similar to like finding an algorithm. But if it's like evolution gives us knowledge, pre-training gives us knowledge, that analogy seems to break down.

• 12:42Speaker 01

  So it's subtle and I think you're right to push back on it. But basically, the thing that pre-training is doing, so you're basically getting the next token predictor on over the internet and you're training that into a neural net. It's doing two things, actually, that are kind of like unrelated. Number one, it's picking up all this knowledge, as I call it. Number two, it's actually becoming intelligent. By observing the algorithmic patterns in the internet, it actually kind of like boots up all these like little circuits and algorithms inside the neural net to do things like in-context learning and all this kind of stuff. And actually, you don't actually need or want the knowledge. I actually think that's probably actually holding back the neural networks overall because it's actually like getting them to rely on the knowledge a little too much sometimes. For example, I kind of feel like agents, one thing they're not very good at is going off the data manifold of what exists on the internet.

• 13:25Speaker 01

  If they had less knowledge or less memory, actually, maybe they would be better. And so what I think we have to do kind of going forward, and this would be part of the research paradigms, is actually think we need to start, we need to figure out ways to remove some of the knowledge and to keep what I call this cognitive core. Is this like intelligent entity that is kind of stripped from knowledge but contains the algorithms and contains the magic, you know, of intelligence and problem solving and the strategies of it and all this kind of stuff.

• 13:50Dwarkesh Patel?

  There's so much interesting stuff there. Okay, so let's start with in-context learning. This is an obvious point, but I think it's worth just like saying it explicitly and meditating on it. The situation in which these models seem the most intelligent, in which they are like, I talk to them and I'm like, wow, there's really something on the other end that's responding to me, thinking about things. If it like makes a mistake, it's like, oh wait, that's actually the wrong way to think about it, I'm backing up. All that is happening in context. That's where I feel like the real intelligence you can like visibly see. And that in-context learning process is developed by gradient descent on pre-training, right? Like it spontaneously meta-learns in-context learning. But the in-context learning itself is not gradient descent.

• 14:31Dwarkesh Patel?

  In the same way that our lifetime intelligence as humans to be able to do things is conditioned by evolution, but our actual learning during our lifetime is like happening through some other process.

• 14:42Speaker 01

  I actually don't fully agree with that, but you should continue your thought.

• 14:44Dwarkesh Patel?

  Okay. Actually, then I'm very curious to understand how that analogy breaks down.

• 14:48Speaker 01

  I think I'm hesitant to say that in-context learning is not doing gradient descent because, I mean, it's not doing explicit gradient descent, but I still think that... So in-context learning, basically, it's pattern completion within a token window, right? And it just turns out that there's a huge amount of patterns on the internet. And so you're right, the model kind of like learns to complete the pattern.

• 15:05Dwarkesh Patel?

  Yeah.

• 15:06Speaker 01

  And that's inside the weights. The weights of the neural network are trying to discover patterns and complete the pattern. And there's some kind of an adaptation that happens inside the neural network, right? Which is kind of magical and just falls out from internet, just because there's a lot of patterns. I will say that there have been some papers that I thought were interesting that actually look at the mechanisms behind in-context learning. And I do think it's possible that in-context learning actually runs a small gradient descent loop internally in the layers of the neural network. And so I recall one paper in particular where they were doing linear regression, actually, using in-context learning. So basically, your inputs into the neural network are X, Y pairs, X, Y, X, Y, X, Y, that happen to be on the line.

• 15:44Dwarkesh Patel?

  Yeah.

• 15:44Speaker 01

  And then you do X and you expect the Y.

• 15:46Dwarkesh Patel?

  Yeah.

• 15:47Speaker 01

  And the neural network, when you train it in this way, actually does do linear regression. And normally, when you would run linear regression, you have a small gradient descent optimizer that basically looks at X, Y, looks at an error, calculates the gradient of the weights, and does the update a few times. It just turns out that when they looked at the weights of that in-context learning algorithm, they actually found some analogies to gradient descent mechanics. In fact, I think even the paper was stronger because they actually hardcoded the weights of a neural network to do gradient descent through attention and all the internals of the neural network. So I guess that's just my only pushback is that who knows how in-context learning works, but I actually think that it's probably doing a little bit of some kind of funky gradient descent internally, and that I think that that's possible.

• 16:32Speaker 01

  So I guess I was only pushing back on you're saying it's not doing in-context learning. Who knows what it's doing, but it's probably maybe doing something similar to it, but we don't know.

• 16:39Dwarkesh Patel?

  So then it's worth thinking about, okay, if both of them are implementing... Gradient descent, sorry, if in-context learning and pre-training are both implementing something like gradient descent.

• 16:48Speaker 01

  Right.

• 16:49Dwarkesh Patel?

  Why does it feel like in-context learning, actually, we're getting to this continual learning, real intelligence-like thing, whereas you don't get the analogous feeling just from pre-training, at least you could argue that. And so if it's the same algorithm, what could be different? Well, one way you can think about it is, how much information does the model store per information it receives from training? And if you look at pre-training, if, I think, if you look at Llama 3, for example, I think it's trained on 15 trillion tokens, and if you look at a 70B model, that would be the equivalent of 0.07 bits per token in that it sees in pre-training, in terms of the information in the weights of the model compared to the tokens it reads. Whereas if you look at the KV cache and how it grows per additional token in in-context

• 17:34Dwarkesh Patel?

  learning, it's like 320 kilobytes.

• 17:37Speaker 01

  Yeah.

• 17:37Dwarkesh Patel?

  So that's a 35 million-fold difference in how much information per token is assimilated by the model.

• 17:44Speaker 01

  Yeah.

• 17:45Dwarkesh Patel?

  I wonder if that's relevant at all.

• 17:46Speaker 01

  I think I kind of agree. I mean, the way I usually put this is that anything that happens during the training of the neural network, the knowledge is only kind of like a hazy recollection of what happened in the training time. And that's because the compression is dramatic. You're taking 15 trillion tokens and you're compressing it to just your final neural network with a few billion parameters. So obviously, it's a massive amount of compression going on. So I kind of refer to it as like a hazy recollection of the internet documents, whereas anything that happens in the context window of the neural network, you're plugging all the tokens and it's building up all this KV cache representation, is very directly accessible to the neural net. So I compare the KV cache and the stuff that happens at test time to more like a working memory. Like, all the stuff that's in the context window is very directly accessible to the neural net.

• 18:27Speaker 01

  So there's always like these almost surprising analogies between LLMs and humans, and I find them kind of surprising because we're not trying to build a human brain, of course, just directly, we're just finding that this works and we're doing it. But I do think that Anything that's in the weights, it's kind of like a hazy recollection of what you read a year ago. Anything that you give it as a context at test time is directly in the working memory. And I think that's a very powerful analogy to things for things. So when you, for example, go to an LLM and you ask it about some book and what happened in it, like Nick Lane's book or something like that. The LLM will often give you some stuff, which is roughly correct. But if you give it the full chapter and ask it questions, you're going to get much better results because it's now loaded in the working memory of the model. So I basically agree with your very long way of saying that I kind of agree, and that's why.

• 19:12Dwarkesh Patel?

  Stepping back, what is it the part about human intelligence that we like have most fail to replicate with these models?

• 19:18Speaker 01

  I almost feel like just a lot of it still. So maybe one way to think about it, I don't know if this is the best way, but I almost kind of feel like, again, making these analogies, imperfect as they are, we've stumbled by with the transformer neural network, which I think is extremely powerful, very general. You can train transformers on audio or video or text or whatever you want, and it just learns patterns and they're very powerful and it works really well. That to me almost indicates that this is kind of like some piece of cortical tissue. It's something like that because the cortex is famously very plastic as well. You can rewire, you know, plastic brains. And there was the slightly gruesome experiments with rewiring like visual cortex of the auditory

• 20:01Speaker 01

  cortex and this animal like learned five, et cetera. So I think that this is kind of like cortical tissue. I think when we're doing reasoning and planning inside the neural networks, so basically doing a reasoning traces for thinking models, that's kind of like the prefrontal cortex. And then I think we, maybe those are like little check marks, but I still think there's many brain parts and nuclei that are not explored. So maybe, for example, there's a basic ganglia doing a bit of reinforcement learning when we fine tune the models on reinforcement learning. But you know, whereas like the hippocampus, not obvious what that would be. Some parts are probably not important, maybe the cerebellum is like not important to cognition, its thoughts of, so maybe we can skip some of it. But I still think that's, for example, the amygdala, all the emotions and instincts.

• 20:44Speaker 01

  And there's probably like a bunch of other nuclei in the brain that are very ancient that I don't think we've like really replicated. I didn't actually know that we should be pursuing, you know, the building of an analog of human brain. I'm again, an engineer mostly at heart, but. I still feel like maybe another way to answer the question is you're not going to hire this thing as an intern. And it's missing a lot of it's because it comes with a lot of these cognitive deficits that we all intuitively feel when we talk to the models. And so it's just like not fully there yet. You can look at it as like not all the brain parts are checked off yet.

• 21:16Dwarkesh Patel?

  This is maybe relevant to the question of thinking about how fast these issues will be solved. So sometimes people will say about continual learning, look, actually, you could already, you could easily replicate this capability. Just as in context, learning emerged spontaneously as a result of pre-training. Continual learning over longer horizons will emerge spontaneously if the model is incentivized to recollect information over longer horizons or horizons longer than one session. So if there's some like outer loop RL, which has many sessions within that outer loop. Then like this continual learning where it uses like it fine tunes itself or it writes to an external memory or something will just sort of like emerge spontaneously.

• 22:00Dwarkesh Patel?

  Do you think, do you think things are plausible? I just, I don't have really a prior over like how plausible is that? How likely is that to happen?

• 22:06Speaker 01

  I don't know that I fully resonate with that because I feel like these models, when you boot them up and they have zero tokens in the window. They're always like restarting from scratch where they were. So I don't actually know in that worldview what it looks like. Because again, maybe making some analogies to humans, just because I think it's roughly concrete and kind of interesting to think through. I feel like when I'm awake, I'm building up a context window of stuff that's happening during the day. But I feel like when I go to sleep, something magical happens where I don't actually think that that context window stays around. I think there's some process of distillation into weights of my brain. And this happens during sleep and all this kind of stuff. We don't have an equivalent of that in large language models. And that's to me more adjacent to when you talk about continual learning and so on as absent.

• 22:49Speaker 01

  These models don't really have this distillation phase of taking what happened, analyzing it, obsessively thinking through it.

• 22:58Speaker 01

  Basically doing some kind of a synthetic data generation process and distilling it back into the weights. And maybe having a, you know, specific neural net per person, maybe it's a LoRa, it's not a full. Yeah, it's not a full weight neural network. That's it's just small, some of the small sparse subset of the weights are changed. But basically, we do want to create ways of creating these individuals that have very long contexts. It's not only remaining in the context window because the context windows grow very, very long. Like, maybe we have some very elaborate sparse attention over it.

• 23:28Dwarkesh Patel?

  Hmm.

• 23:29Speaker 01

  But I still think that humans obviously have some process for distilling some of that knowledge into the weights. We're missing it. And I do also think that humans have some kind of a very elaborate sparse attention scheme.

• 23:40Dwarkesh Patel?

  Hmm.

• 23:41Speaker 01

  Which I think we're starting to see some early hints of. So DeepSeek V3.2 just came out, and I saw that they have like a sparse attention as an example. And this is one way to have very, very long context windows. So I almost feel like we are redoing a lot of the cognitive tricks that evolution came up with through a very different process. But we're, I think, going to converge on a similar architecture cognitively.

• 24:01Dwarkesh Patel?

  Interesting. In 10 years, do you think it'll still be something like a transformer, but with a much more modified attention and more sparse MLPs and so forth?

• 24:10Speaker 01

  Well, the way I like to think about it is, okay, let's translation invariance in time, right? So 10 years ago, where were we? 2015, we had convolutional neural networks primarily. Residual networks just came out. So remarkably similar, I guess, but quite a bit different still. I mean, transformer was not around. You know, all the, all these sort of like more modern tweaks on the transformer were not around. So maybe some of the things that we can bet on, I think, in 10 years by translational sort of equivalence is we're still training giant neural networks with forward, backward pass and update through gradient descent. But maybe it looks a little bit different. And it's just, everything is much bigger. Actually, recently, I also went back all the way to 1989, which was kind of a fun

• 24:54Speaker 01

  exercise for me a few years ago, because I was reproducing Yann Le Cun's 1989 convolutional network, which was the first neural network I'm aware of trained via gradient descent, like modern neural network trained gradient descent on digit recognition.

• 25:08Dwarkesh Patel?

  Hmm.

• 25:08Speaker 01

  And I was just interested in, okay, how can I modernize this? How much of this is algorithms? How much of this is data? How much of this progress is compute and systems? And I was able to very quickly, like, half the learning rate, just knowing by time travel by 33 years. So if I time travel by algorithms to thirty-three years, I could adjust what Yann LeCun did in nineteen eighty-nine, and I could basically half the learning, half the error. But to get further gains, I had to add a lot more data. I had to like ten X the training set. And then I had to actually add more computational optimizations. Had to basically train for much longer with dropout and other regularization techniques. And so it's almost like all these things have to improve simultaneously. So, you know, we're probably gonna have a lot more data. We're probably gonna have a lot better hardware. Probably gonna have a lot better kernels and software. We're probably gonna have better algorithms.

• 25:53Speaker 01

  And all of those, it's almost like no one of them is winning too much. All of them are surprisingly equal. And this has kind of been the trend for a while. So I guess to answer maybe your question, I expect differences algorithmically to what's happening today. But I do also expect that some of the things that have stuck around for a very long time will probably still be there. It's probably still giant neural network trained with gradient descent. That would be my guess.

• 26:16Dwarkesh Patel?

  It's surprising that all of those things together only halved, um, uh, half the error.

• 26:23Speaker 01

  Yeah.

• 26:24Dwarkesh Patel?

  Which is, so like thirty years of progress is, uh, maybe, maybe half is a lot. Cause like if you half the error, that actually means that.

• 26:29Speaker 01

  Half as well, yeah.

• 26:30Dwarkesh Patel?

  Yeah, yeah, okay.

• 26:31Speaker 01

  But it's, I guess what was shocking to me is everything needs to improve across the board.

• 26:35Dwarkesh Patel?

  Yeah.

• 26:35Speaker 01

  Architecture, optimize a loss function, and also has improved across the board forever. So I kind of expect all those changes to be alive and well.

• 26:42Dwarkesh Patel?

  Well, yeah, actually, I was about to ask you a very similar question about NanoChat. Cause since you just put it up recently, every single sort of step in the, you know, process of building a chatbot is like freshening your RAM.

• 26:53Speaker 01

  Yeah.

• 26:54Dwarkesh Patel?

  And I'm curious if you had similar thoughts about like, oh, there was no one thing that was relevant to going from GPT two to NanoChat. What are sort of like surprising takeaways from the experience?

• 27:07Speaker 01

  Building NanoChat?

• 27:08Dwarkesh Patel?

  Yeah.

• 27:08Speaker 01

  So NanoChat is a kind of a repository I released, was it yesterday or the day before? I can't remember.

• 27:14Dwarkesh Patel?

  We can see this leap variation that went into the.

• 27:18Speaker 01

  Well, it's just trying to be a, it's trying to be the simplest complete repository that covers the whole pipeline end to end of building a chat GPT clone. And so, you know, you have all of the steps, not just any individual step, which is a bunch. I worked on all the individual steps sort of in the past and really small pieces of code that could have show you how that's done in algorithmic sense.

• 27:40Speaker 01

  In like simple code, but this kind of handles all the entire pipeline. I, I think in terms of learning, it's not, it's not so much, I don't know that I actually found something that I learned from, from it necessarily. I kind of already had in my mind as like how you build it. And this is just a process of mechanically building it and making it clean enough and so that people can actually learn from it and that they find it useful.

• 28:03Dwarkesh Patel?

  Yeah. What is the best way for somebody to learn from it? Is it just like delete all the code and try to re-implement from scratch, try to add modifications to it?

• 28:10Speaker 01

  Yeah, I think that's a, that's a great question. I would probably say, so basically it's about 1,000 lines of code. That takes you through the entire pipeline. I would probably put it on the right monitor. Like if you have two monitors, you put it on the, on the right. And you want to build it from scratch, you build it from start. You're not allowed to copy paste. You're allowed to reference. You're not allowed to copy paste. Maybe that's how I would do it. But I also think the repository by itself, it is like a pretty large beast. I mean, it's, you know, it's, it's. When you write this code, you don't go from top to bottom. You go from chunks and you grow the chunks and that information is absent, like you wouldn't know where to start. And so I think it's not just a final repository that's needed. It's like the building of the repository, which is a complicated chunk growing process.

• 28:49Dwarkesh Patel?

  Right.

• 28:50Speaker 01

  So that part is not there yet. I, I would love to actually like add that probably later this week or something in some way like either it's a, it's probably a video or something like that, but, but maybe roughly speaking, that's what I would try to do is build the stuff yourself. But don't allow yourself copy paste.

• 29:05Dwarkesh Patel?

  Yeah.

• 29:06Speaker 01

  I do think that there's two types of knowledge almost like there's the high level surface knowledge. But the thing is that when you actually build something from scratch, you're forced to come to terms with what you don't actually understand and you don't know that you don't understand it.

• 29:17Dwarkesh Patel?

  Interesting.

• 29:17Speaker 01

  And it always leads to a deeper understanding. And it's like just the only way to, to build is like, if I can't build it, I don't understand it. Is that a Feynman quote, I believe, or something along those lines? A hundred percent, I've always believed this very strongly. Because there's all these like micro things that are just not properly arranged and you don't really have the knowledge. You just think you have the knowledge. So don't write blog posts, don't do slides, don't do any of that. Like build the code, arrange it, get it to work. It's the only way to go, otherwise you're missing knowledge.

• 29:44Dwarkesh Patel?

  You tweeted out that coding models were actually of very little help to you in assembling this repository. And I'm curious why that was.

• 29:52Speaker 01

  Yeah. So the repository, I guess I built it over a period of a bit more than a month. And I would say there's like three major classes of how people interact with code right now. Some people completely reject all of LLMs and they are just writing by scratch. I think this is probably not the right thing to do anymore. The intermediate part, which is where I am, is you still write a lot of things from scratch, but you use the autocomplete. That's basically available now from these models. So when you start writing out a little bit piece of it, it will, it will all complete for you and you can just tap through. And most of the time it's correct. Sometimes it's not and you edit it. But you're still very much the sort of architect of what you're writing. And then there's the, you know, vibe coding, you know, hi, please implement this or that, you know, enter and then let the model do it.

• 30:36Speaker 01

  And that's the agents. I do feel like the agents work in very specific settings and I would use them in specific settings. But again, these are all tools available to you and you have to like learn what they, what they're good at and what they're not good at and when to use them. So the agents are actually pretty good, for example, if you're doing boilerplate stuff. Boilerplate code, that's like just, you know, just copy paste stuff. They're very good at that. They're very good at stuff that occurs very often in the internet. Because there's lots of examples of it in the training sets of these models. So, so there's like features of things that where the models will do very well. I would say NanoChat is not an example of this because it's a fairly unique repository. There's not that much code, I think, in a way that I've structured it. And, and it's not boilerplate code.

• 31:20Speaker 01

  It's like actually like intellectually intense code almost. And everything has to be very precisely arranged. And the models are always trying to, they kept trying to, I mean, they have so many cognitive deficits, right? So one example, they keep trying to, they keep misunderstanding the code.

• 31:34Speaker 01

  Because they have too much memory from all the typical ways of doing things on the internet that I just wasn't adopting. So the models, for example, I mean, I don't know if I want to get into the full details, but they keep thinking I'm writing normal code and I'm not.

• 31:48Dwarkesh Patel?

  Maybe one example.

• 31:49Speaker 01

  Maybe one example is, so the way to synchronize, so we have eight GPUs that are all doing forward and backwards. The way to synchronize gradients between them is to use a distributed data parallel container of PyTorch, which automatically does all the, as you're doing the backward, it will start communicating and synchronizing gradients. I didn't use DDP because I didn't want to use it because it's not necessary. So I threw it out. And I basically wrote my own synchronization routine that's inside the step of the optimizer. And so the models were trying to get me to use the DDP container.

• 32:17Dwarkesh Patel?

  Yeah.

• 32:17Speaker 01

  And they were very concerned about, okay, this gets way too technical, but I wasn't using that container because I don't need it and I have a custom implementation of something like it.

• 32:25Dwarkesh Patel?

  And they just couldn't internalize that you had your own.

• 32:27Speaker 01

  Yeah, they couldn't get past that. And then they kept trying to like mess up the style, like they're way too over defensive. They make all these try catch statements. They keep trying to make a production codebase. And I have a bunch of assumptions in my code and it's okay. And it's just like, I don't need all this extra stuff in there. And so I just kind of feel like they're bloating the codebase, they're bloating the complexity, they keep misunderstanding, they're using deprecated APIs a bunch of times. So it's total mess. And it's just, it's just not net useful. I can go in, I can clean it up, but it's not net useful. I also feel like it's kind of annoying to have to like type out what I want in English because it's just too much typing. Like if I just navigate to the part of the code that I want and I go where I know the code has to appear and I start typing out the first three letters, autocomplete gets

• 33:12Speaker 01

  it and just gives you the code. And so I think it's, this is a very high information bandwidth to specify what you want. If you point to the code where you want it and you type out the first few pieces.

• 33:21Dwarkesh Patel?

  Yeah.

• 33:21Speaker 01

  And the model will complete it. So I guess what I mean is. I think these models are good in certain parts of the stack. Actually, use the models a little bit in. There are two examples where I actually use the models that I think are illustrative. One was when I generated the report, that's actually more boilerplatey. So actually vibe coded partially some of that stuff. That was fine because it's not like mission critical stuff and it works fine. And then the other part is when I was rewriting the tokenizer in Rust. I'm actually not as good at Rust because I'm fairly new to Rust. So I was doing, there's a bit of vibe coding going on in when I was writing some of the Rust code. But I had Python implementation that I fully understand and I'm just making sure I'm making more efficient version of it and I have tests. So I feel safer doing that stuff.

• 34:05Speaker 01

  And so basically they lower or like the increased accessibility to languages or paradigms that you might not be as familiar with. So I think they're very helpful there as well.

• 34:15Dwarkesh Patel?

  Yeah.

• 34:16Speaker 01

  Because there's a ton of Rust code out there. The models are actually pretty good at it. I happen to not know that much about it. So the models are very useful there.

• 34:21Dwarkesh Patel?

  Right. The reason I think this question is so interesting is because the main story people have about AI exploding and getting to superintelligence pretty rapidly is AI automating, AI engineering and AI research. So they'll look at the fact that you can have Claude code and make entire application, crowd applications from scratch and be like, if you had this capability inside of OpenAI and DeepMind and everything, well, just imagine the level of like, just, you know, a thousand of you or a million of you in parallel finding little architectural tweaks. And so it's quite interesting to hear you say that this is the thing they're sort of asymmetrically worse at. And it's like quite relevant to forecasting whether the AI 2027 type explosion is likely to happen anytime soon.

• 35:04Speaker 01

  I think that's a good way of putting it. And I think you're getting at some of my, like why my timelines are a bit longer. You're right. I think, yeah, they're not very good at code that hasn't never been written before. Maybe is like one way to put it, which is like what we're trying to achieve when we're building these models.

• 35:18Dwarkesh Patel?

  Very naive question, but the architectural tweaks that you're adding to NanoChat, they're in a paper somewhere, right? They might even be in a repo somewhere. So it's, is it, is it surprising that they aren't able to integrate that into whenever you're like, add rope embeddings or something, they do that in the wrong way?

• 35:41Speaker 01

  It's tough. I think they kind of know, but they don't fully know. And they don't know how to fully integrate it into the repo and your style and your code and your place and some of the custom things that you're doing. And how fits with all the assumptions of the repository and all this kind of stuff. So I think they do have some knowledge, but they haven't gotten to the place where they can actually integrate it, make sense of it and so on. I do think that a lot of the stuff, by the way, continues to improve. So I think currently probably state of the art model that I go to is the GPT-5 Pro. And that's a very, very powerful model. So if I actually have 20 minutes, I will copy paste my entire repo and I go to GPT-5 Pro, the oracle for like some questions. And often it's not too bad and surprisingly good compared to what existed a year ago. But I do think that overall the models are, they're not there.

• 36:26Speaker 01

  And I kind of feel like the industry, it's, it's, it's over, it's making too big of a jump. And it's trying to pretend like this is amazing and it's not, it's slop. And I think they're not coming to terms with it. And maybe they're trying to fundraise or something like that. I'm not sure what's going on, but it's, we're at this intermediate stage. The models are amazing. They still need a lot of work. For now, autocomplete is my sweet spot. But sometimes for some types of code, I will go to a null agent.

• 36:52Dwarkesh Patel?

  Yeah. Actually, this is also, here's another reason why this is really interesting.

• 36:57Dwarkesh Patel?

  Through the history of programming, there's been many productivity improvements, compilers, linting, better programming languages, et cetera, which have increased program productivity, but have not led to an explosion. So that's like one, that sounds very much like autocomplete tab. And this other category is just like automation of the programmer.

• 37:18Speaker 01

  Yeah.

• 37:19Dwarkesh Patel?

  And it's just interesting you're seeing more in the category of the historical analogies of like, you know, better compilers or something.

• 37:25Speaker 01

  And maybe because this gets at one other kind of thought about is like, I do feel like I have a hard time differentiating where AI begins and stops. Because I do see AI as fundamentally an extension of computing in some, in some pretty fundamental way. And I feel like I see a continuum of this kind of like recursive self-improvement or like of speeding up programmers all the way from the beginning. Like even like, I would say like code editors.

• 37:48Speaker 01

  Syntax highlighting.

• 37:49Dwarkesh Patel?

  Yeah.

• 37:50Speaker 01

  Syntax or like checking even of the of the types like data type checking.

• 37:56Speaker 01

  All these kinds of tools that we've built for each for each other, even search engines. Like, why aren't search engines part of AI? Like, I don't know, like ranking is kind of AI, right? At some point, Google was like, even early on, they were thinking of themselves as an AI company doing Google Search Engine, which I think is totally fair. And so I kind of see it as a lot more of a continuum than I think other people do. And I don't, it's hard for me to draw the line. And I kind of feel like, OK, we're now getting a much better autocomplete. And now we're also getting some agents, which are kind of like these loopy things, but they kind of go off rails sometimes.

• 38:24Speaker 01

  And what's going on is that the human is progressively doing a bit less and less of the low level stuff. For example, we're not writing the assembly code because we have compilers.

• 38:31Dwarkesh Patel?

  Yeah.

• 38:32Speaker 01

  Right, like compilers will take my high level language in C and write the assembly code. So we're abstracting ourselves very, very slowly. And there's this, what I call autonomy slider of like more and more stuff is automated, of the stuff that can be automated at any point in time. And we're doing a bit less and less and raising ourselves in the layer abstraction over the automation.

• 38:49Dwarkesh Patel?

  One of the big problems with RL is that it's incredibly information sparse. Labelbox can help you with this by increasing the amount of information that your agent gets to learn from with every single episode. For example, one of their customers wanted to train a coding agent. So Labelbox augmented an IDE with a bunch of extra data collection tools and staffed a team of expert software engineers from their aligner network to generate trajectories that were optimized for training. Now, obviously, these engineers evaluated these interactions on a pass fail basis, but they also rated every single response on a bunch of different dimensions, like readability and performance. And they wrote down their thought processes for every single rating that they gave. So you're basically showing every single step an engineer takes

• 39:34Dwarkesh Patel?

  and every single thought that they have while they're doing their job. And this is just something you could never get from usage data alone. And so Labelbox packaged up all these evaluations and included all the agent trajectories and the corrective human edits for the customer to train on. This is just one example. So go check out how Labelbox can get you high quality frontier data across domains, modalities, and training paradigms. Reach out at labelbox.com/thoarkesh.

• 40:05Dwarkesh Patel?

  Let's talk about ARL a bit. You tweeted us some very interesting things about this. Conceptually, how should we think about the way that humans are able to build a rich world model just from interacting with our environment? And in ways that seems almost irrespective of the final reward at the end of the episode? If somebody, you know, somebody's starting to start a business, and at the end of 10 years, she finds out whether the business succeeded or failed, we say that she's earned a bunch of wisdom and experience. But it's not because like, the log probes of every single thing that happened over the last 10 years are upweighted or downweighted. Something much more deliberate and rich is happening. What is the ML analogy? And how does that compare to what we're doing with other ones right now?

• 40:46Speaker 01

  Yeah, maybe the way I would put it is, humans don't use reinforcement learning, is maybe what I've, as I've said at all. I think they do something different, which is, yeah, you experience. So reinforcement learning is a lot worse than I think the average person thinks. Reinforcement learning is terrible. It just so happens that everything that we had before is much worse. Because previously, we're just imitating people, so it has all these issues. So in reinforcement learning, say you're working with, you're solving a math problem. This is very simple. You're given a math problem, and you're trying to find a solution. Now, in reinforcement learning, you will try lots of things in parallel first. So you're given a problem, you try hundreds of different attempts. And these attempts can be complex, right?

• 41:30Speaker 01

  They can be like, oh, let me try this, let me try that. This didn't work, that didn't work, etc. And then maybe you get an answer. And now you check the back of the book, and you see, okay, the correct answer is this. And then you can see that, okay, this one, this one, and that one got the correct answer, but these other 97 of them didn't. So literally what reinforcement learning does is it goes to the ones that worked really well. And every single thing you did along the way, every single token gets upweighted of like, do more of this. The problem with that is, I mean, people will say that your estimator has high variance, but what, I mean, it's just noisy. It's noisy. So basically, It kind of almost assumes that every single little piece of the solution that you made, that righted right answer was correct thing to do, which is not true. Like you may have gone down the wrong alleys until you arrived at the right solution. Every single one of those incorrect things you did, as long as you got to the correct

• 42:15Speaker 01

  solution, will be upweighted as do more of this. It's terrible. It's noise. You've done all this work only to find a single, at the end, you get a single number of like, oh, you did correct. And, and based on that, you weigh that entire trajectory as like upweight or downweight. And so you're, the way I like to put it is you're sucking supervision through a straw, because you've done all this work, that could be a minutes of rollout, and you're, you're like sucking the bits of supervision of the final reward signal through a straw and you're like putting it, you're like, you're basically like, um, yeah, you're broadcasting that across the entire trajectory and using that to upweight or downweight that trajectory. It's crazy. A human would never do this. Number one, a human would never do hundreds of rollouts. Number two, when a person sort of finds a solution, they will have a pretty complicated

• 42:58Speaker 01

  process of review of like, okay, I think these parts that I did well, these parts I did not do that well, I should probably do this or that. And they think through things. There's nothing in current LLMs that does this. There's no equivalent of it. But I do see papers popping out that are trying to do this because it's obvious to everyone in the field. So I kind of see as like, the first imitation learning actually, by the way, was extremely surprising and miraculous and amazing that we can fine tune by imitation on humans. And that was incredible because in the beginning, all we had was base models. Base models are autocomplete. And it wasn't obvious to me at the time, and I had to learn this. And the paper that like blew my mind was InstructGPT, because it pointed out that, hey, you can take the pre-trained model, which is autocomplete. And if you just fine tune it on text that looks like conversations, the model will

• 43:43Speaker 01

  very rapidly adapt to become very conversational, and it keeps all the knowledge from pre-training. And this blew my mind because I didn't understand that it's just like stylistically can adjust so quickly and become an assistant to a user, through just a few loops of fine tuning on that kind of data. It was very miraculous to me that, that that worked. So incredible. And that was like two years, three years of work. And now came RL. And RL allows you to do a bit better than just imitation learning, right? Because you can have these reward functions and you can hill climb on the reward functions. And so some problems have just correct answers. You can hill climb on that without getting expert trajectories to imitate. So that's amazing. And the model can also discover solutions that the human might never come up with. So this is incredible. And yet, it's so stupid.

• 44:27Speaker 01

  So I think we need more. And so I saw a paper from Google yesterday that tried to have this reflect and review page idea in mind. What was the memory bank paper or something? I don't know. I've actually seen a few papers along these lines. So I expect there to be some kind of a major update to how we do algorithms for LLMs coming in that realm. And then I think we need three or four or five more. Something like that.

• 44:53Dwarkesh Patel?

  But you're so good at coming up with the evocative phrases. Sucking supervision through a straw is like so good. Why hasn't? So you're saying like your problem with outcome based reward is that you have this huge trajectory and then at the end, you're trying to learn every single possible thing about what you should do and what you should learn about the world from that one final bit. Why hasn't, given the fact that this is obvious, why hasn't process based supervision as an alternative been a successful way to make models more capable? What has been preventing us from using this alternative paradigm?

• 45:28Speaker 01

  So process based supervision just refers to the fact that we're not going to have a reward function only at the very end of after you've made 10 minutes of work. I'm not going to tell you you did well or not well. I'm going to tell you at every single step of the way how well you're doing. And this is basically the reason we don't have that is not tricky. It's tricky how you do that properly. Because you have partial solutions and you don't know how to assign credit. So when you get the right answer, it's just an equality match to the answer. Very simple to implement. If you're doing basically process supervision, how do you assign in an automatable way? Partial credit assignment. It's not obvious how you do it. Lots of labs, I think, are trying to do it with these LLM judges. So basically, you get LLMs to try to do it. So you prompt an LLM, hey, look at a partial solution of a student. How well do you think they're doing if the answer is this? And they try to tune the prompt.

• 46:11Speaker 01

  The reason that I think this is kind of tricky is quite subtle. And it's the fact that anytime you use an LLM to assign a reward, those LLMs are giant things with billions of parameters and they're gamable. And if you're reinforcement learning with respect to them, you will find adversarial examples for your LLM judges, almost guaranteed. You can't do this for too long. You do maybe 10 steps or 20 steps, maybe it will work. But you can't do 100 or 1,000 because it's not obvious. Because I know, I understand it's not obvious. But basically, the model will find little cracks. It will find all these like spurious things in the nooks and crannies of the giant model and find a way to cheat it. So one example that's prominently in my mind is, I think this, I think this was probably public. But basically, if you're using an LLM judge for a reward,

• 46:55Speaker 01

  so you just give it a solution from a student and ask it if the student will or not. We were training with reinforcement learning against that reward function. And it worked really well. And then suddenly, the reward became extremely large. Like it was massive jump and it did perfect. And you're looking at it like, wow, this, this means the student is perfect. And all these problems, it's fully solved math. But actually, what's happening is that when you look at the completions that you're getting from the model, they are complete nonsense. They start out okay, and then they change to da, da, da, da, da, da. So it's just like, oh, okay, let's take 2 plus 3 and we do this and this and then da, da, da, da, da, da, da. And you're looking at it, it's like, this is crazy. How is it getting a reward of 1 or 100%? And you look at the LLM judge and it turns out that da, da, da, da, da is an adversarial examples for the model and it assigns 100% probability to it.

• 47:38Speaker 01

  And it's just because this is an out of sample example to the LLM. It's never seen you during training and you're in pure generalization land.

• 47:45Dwarkesh Patel?

  Right.

• 47:45Speaker 01

  It's never seen you during training. And in the pure generalization land, you can find these examples that, that break it.

• 47:51Dwarkesh Patel?

  You're basically training the LLM to be a prompt injection model.

• 47:56Speaker 01

  Not even that. Prompt injection is way too fancy. You're, you're finding adversarial examples, as they're called. These are nonsensical solutions that are obviously wrong, but the model thinks are amazing.

• 48:06Dwarkesh Patel?

  So do they say you think this is the bottleneck to making RL more functional? Then that will require making LLMs better judges if you want to do this in an automated way. And then so is it just going to be like some sort of GAN-like approach? Or you had to train models to be more robust to?

• 48:21Speaker 01

  I think the labs are probably doing all that like, okay, so the obvious thing is like the the the should not get 100% reward. Okay, well, take the the the, put in the training set of the LLM judge and say, this is not 100%, this is 0%. You can do this, but every time you do this, you get a new LLM and it still has adversarial examples. There's infinity adversarial examples. And I think probably if you iterate this a few times, it'll probably be harder and harder to find adversarial examples. But I'm not 100% sure because this thing has a trillion parameters or whatnot. So I bet you the the labs are trying. I don't actually, I still think, I still think we need other ideas.

• 48:56Dwarkesh Patel?

  Interesting. Do you have some shape of what the other idea could be?

• 49:01Speaker 01

  So like this, this idea of like every review, review solution encompass synthetic examples such that when you train on them, you get, you get better and like meta learn it in some way. And I think there's some papers that I'm starting to see pop out. I only am at a stage of like reading abstracts because a lot of these papers, you know, they're just ideas. Someone has to actually like make it work on a frontier LLM lab scale in full generality. Because when you see these papers, they pop up and it's just like a little bit of noisy, you know? It's cool ideas, but I haven't actually seen anyone convincingly show that this is possible. That's that the LLM labs are fairly closed, so who knows what they're doing now, but yeah.

• 49:37Dwarkesh Patel?

  So I guess I could, I, I, I see a very, not easy, but like I, I can conceptualize how you would be able to train on synthetic examples or synthetic problems that you have made for yourself. But there seems to be another thing humans do, maybe sleep is this, maybe daydreaming is this, which is not necessarily come up with fake problems, but just like reflect.

• 49:57Speaker 01

  Yeah.

• 49:58Dwarkesh Patel?

  And I'm not sure what the ML analogy for, you know, daydreaming or sleeping, but just like, just reflecting. I haven't come up with a new problem. I mean, obviously the very basic analogy would just be like fine tuning on reflection bits. But I feel like in practice, that probably wouldn't work that well. So I don't know if you have. Some take on what the analogy of like this thing is.

• 50:16Speaker 01

  Yeah, I do think that we're missing some aspects there. So as an example, when you're reading a book.

• 50:22Speaker 01

  I almost feel like currently when L L M's are reading a book, what that means is we stretch out the sequence of text and the model is predicting the next token and it's getting some knowledge from that. That's not really what humans do, right? So when you're reading a book, I almost don't even feel like the book is like exposition I'm supposed to be attending to and training on. The book is a is a set of prompts for me to do synthetic data generation. Or for you to get into a book club and talk about it with your friends. And it's by manipulating that information that you actually gain that knowledge. And I think we have no equivalent of that, again with L L M's. They don't really do that, but I'd love to see during pre-training some kind of a stage that thinks through the material and tries to reconcile it with what it already knows and thinks through for like some amount of time and gets that to work. And so there's no equivalence of any of this. This is all research.

• 51:05Speaker 01

  There's some subtle, very subtle, that I think are very hard to understand, reasons why it's not trivial. So if I can just describe one. Why can't we just synthetically generate and train on it? Well, because every synthetic example, like if I just give synthetic generation of the model thinking about a book. You look at it and you're like, this looks great. Why can't I train on it? Well, you could try, but the model will actually get much worse if you continue trying. And that's because all of the samples you get from models are silently collapsed. They're silently, this is not obvious, if you look at any individual example of it. They occupy a very tiny manifold of the possible space of sort of thoughts about content. So the L L M's, when they come off, they're what we call collapsed. They have a collapsed data distribution. If you sample, one easy way to see it is go to Chat G P T and ask it, tell me a joke.

• 51:49Speaker 01

  It only has like three jokes. It's not giving you the whole breadth of possible jokes. It's giving you like, it knows like three jokes. They're silently collapsed. So basically, you're not getting the richness and the diversity and the entropy from these models as you would get from humans. So humans are a lot more sort of noisier, but at least they're not biased. They're not, in a statistical sense, they're not silently collapsed. They maintain a huge amount of entropy. So how do you get synthetic data generation to work despite the collapse and while maintaining the entropy is a research problem?

• 52:19Dwarkesh Patel?

  Just to make sure I understood, the reason that the collapse is relevant to synthetic data generation is because you want to be able to come up with synthetic problems or reflections which are not already in your data distribution?

• 52:31Speaker 01

  I guess what I'm saying is.

• 52:34Speaker 01

  Say we have a chapter of a book and I ask an alum to think about it.

• 52:38Speaker 01

  It will give you something that looks very reasonable. But if I ask it 10 times, you'll notice that all of them are the same.

• 52:43Dwarkesh Patel?

  You can't just keep scaling, scaling quote unquote reflection on the same amount of, you know, prompt information and then get returns from that.

• 52:52Speaker 01

  Yeah, yeah, yeah. So any individual sample will look okay, but the distribution of it is quite terrible. And it's quite terrible in such a way that if you continue training on too much of your own stuff, you actually collapse. I actually think that there's no like fundamental solutions to this possibly. And I also think humans collapse over time. I think this is, again, these analogies are surprisingly good, but humans collapse during the course of their lives. This is why children have completely, you know, they haven't overfit yet. And they will say stuff that will shock you because it's kind of, you can see where they're coming from, but it's just not the thing people say. And because they're not yet collapsed. But we're collapsed, we end up revisiting the same thoughts, we end up, you know, saying more and more of the same stuff. And the learning rates go down and the collapse continues to get worse.

• 53:35Speaker 01

  And then everything deteriorates.

• 53:38Dwarkesh Patel?

  Have you seen this super interesting paper that dreaming is a way of preventing this kind of overfitting and collapse? That the reason dreaming is evolutionary adaptive is to put you in weird situations that are like very unlike your day to day reality so that to prevent this kind of overfitting.

• 53:56Speaker 01

  It's an interesting idea. I mean, I do think that when you're generating things in your head and then you're attending to it, you're kind of like training on your own samples. You're training on your synthetic data. And if you do it for too long, you go off rails and you collapse way too much. So you always have to like seek entropy in your life. So talking to other people is a great source of entropy and things like that. So maybe the brain has also built some internal mechanisms for increasing the amount of entropy

• 54:22Speaker 01

  in that process. But yeah, maybe that's an interesting idea.

• 54:25Dwarkesh Patel?

  This is a very ill-formed thought, so I'll just put it out and let you react to it. The best learners that we are aware of, which are children, are extremely Bad at recollecting information. In fact, at the very earliest stages of childhood, you will forget everything. You're just an amnesiac about everything that happens before a certain year date. But you're extremely good at picking up new languages and learning from the world. And maybe there's some element of being able to see the forest for the trees. Whereas if you compare it to the opposite end of the spectrum, you have LLM pre-training, which these models will literally regurgitate word for word, what is the next thing in a Wikipedia page. But their ability to learn abstract concepts really quickly, the way a child can, is much more limited. And then adults are somewhere in between,

• 55:09Dwarkesh Patel?

  where they don't have the flexibility of childhood learning. But they can, adults can memorize facts and information in a way that is harder for kids. And I don't know if there's something interesting about that.

• 55:19Speaker 01

  I think there's something very interesting about that. Yeah, 100%. I do think that humans actually, they do have a lot more of an element compared to LLMs, of seeing the forest for the trees. And we're not actually that good at memorization, which is actually a feature. Because we're not that good at memorization, we actually are forced to find the patterns in a more original sense. I think LLMs, in comparison, are extremely good at memorization. They will recite passages from all these training sources. You can give them completely nonsensical data. Like, you can hash some amount of text or something like that. You get a completely random sequence. If you train on it, even just, I think, a single iteration or two, it can suddenly regurgitate the entire thing. It will memorize it. There's no way a person can read a single sequence of random numbers

• 56:04Speaker 01

  and recite it to you. And that's a feature, not a bug, almost, because it forces you to only learn the generalizable components. Whereas LLMs are distracted by all the memory that they have of the pre-training documents. And it's probably very distracting to them in a certain sense. So that's why, when I talk about the cognitive core, I actually want to remove the memory, which is what we talked about. I'd love to have less memory so that they have to look things up. And they only maintain the algorithms for thought and the idea of an experiment and all this cognitive glue of acting.

• 56:37Dwarkesh Patel?

  And this is also relevant to preventing model collapse?

• 56:42Speaker 01

  Let me think.

• 56:46Speaker 01

  I'm not sure. I think it's almost like a separate axis. It's almost like the models are way too good at our memorization and somehow we should we should remove that. And I think people people are much worse, but it's a good thing.

• 56:58Dwarkesh Patel?

  What is a solution to model collapse? I mean, you could, so there's very naive things you could attempt is just like

• 57:04Dwarkesh Patel?

  the distribution over logits should be wider or something like there's many naive things you could try. What ends up being the problem with the naive approaches?

• 57:12Speaker 01

  Yeah, I think that's a great question. I mean, you can imagine having a regularization for entropy and things like that. I guess they just don't work as well empirically because right now, like the models are collapsed. But I will say most of the tasks that we want of them don't actually demand the diversity. It's probably the the answer of what's going on. And so it's just that the model, the frontier labs are trying to make the models useful. And I kind of just feel like the diversity of the outputs is not so much. Number one, it's much harder to work with and evaluate and all this kind of stuff. But maybe it's not what's actually capturing most of the value.

• 57:43Dwarkesh Patel?

  In fact, it's actively penalized, right? If you if you're like super creative in RL, it's like not good.

• 57:48Speaker 01

  Yeah, or like maybe if you're doing a lot of writing help from LLMs and stuff like that, I think it's probably bad because the models will give you these like silently all the same stuff, you know. So they're not, they won't explore lots of different ways of answering a question, right? But I kind of feel like maybe this diversity is just not as big of a yeah, maybe like, yeah, not as many applications needed, so the models don't have it. But then it's actually a problem with synthetic generation time, etc. So we're actually shooting ourselves in the foot by not allowing this entropy to maintain in the model. And I think possibly the labs should try harder.

• 58:18Dwarkesh Patel?

  And then I think you hinted that it's a it's a very fundamental problem. It won't be easy to solve. And yeah, what's your intuition for that?

• 58:25Speaker 01

  I don't actually know if it's super fundamental. I don't actually know if I intended to to say that. I do think that

• 58:33Speaker 01

  I haven't done these experiments, but I do think that you could probably regularize the entropy to be to be higher. So you're encouraging the model to give you more and more solutions. But you don't want it to start deviating too much from the training data. It's going to start making up its own language. It's going to start using words that are extremely rare. So it's going to drift too much from the distribution. So I think controlling the distribution is just like a tricky, it's just like someone just has to. It's probably not trivial in that sense.

• 58:59Dwarkesh Patel?

  How many bits should the optimal core of intelligence end up being if you just had to make a guess? The thing we put on the von Neumann probes.

• 59:08Speaker 01

  Yeah, yeah, yeah.

• 59:09Dwarkesh Patel?

  How big does it have to be?

• 59:11Speaker 01

  So it's really interesting in the history of the field because at one point, everything was very scaling pilled in terms of like, oh, we're gonna make much bigger models, trillions of parameter models. And actually, what the models have done in size is they've gone up and now they've actually kind of like, actually even come down. Their models are smaller. And even then, I actually think they memorized way too much. So I think I had a prediction a while back that I almost feel like we can get cognitive cores that are very good at even like a billion, billion parameters. It should be all very like, like, if you talk to a billion parameter model, I think in 20 years, you can actually have a very productive conversation. It thinks, and it's a lot more like a human, but if you ask it some factual question, it might have to look it up. But it knows that it doesn't know and it might have to look it up and it will just do all the reasonable things.

• 59:55Dwarkesh Patel?

  That's actually surprising that you think it will take a billion, because already we have a billion parameter models or a couple billion parameter models that are like very intelligent.

• 1:00:02Speaker 01

  Well, certainly our models are like a trillion parameters, right? But they remember so much stuff, like.

• 1:00:07Dwarkesh Patel?

  Yeah, but I'm surprised that in 10 years, given the pace, okay, we have GPT OSS 20B. That's way better than GPT four original, which was a trillion plus parameters. So given that trend, I'm actually surprised you think in 10 years, the cognitive core is still a billion parameters. I would, oh yeah, I'm surprised you're not like, it's gonna be like tens of millions or millions.

• 1:00:31Speaker 01

  No, because I basically think that the training data is, so here's the issue, the training data is the internet, which is really terrible. So there's a huge amount of gains to be made because the internet is terrible. Like, if you actually, and even the internet, when you and I think of the internet, you're thinking of like, oh, Wall Street Journal, or that's not what this is. When you're actually looking at a pre-training data set in the frontier lab and you look at a random internet document, it's total garbage. Like, I don't even know how this works at all. It's some like stock ticker symbols. It's a huge amount of slop and garbage from like all the corners of the internet. It's not like your Wall Street Journal article that's extremely rare. So I almost feel like because the internet is so terrible, we actually have to sort of almost like build really big models to compress all that. Most of that compression is memory work instead of like cognitive work.

• 1:01:15Dwarkesh Patel?

  Interesting.

• 1:01:15Speaker 01

  But what we really want is the cognitive part actually delete the memory.

• 1:01:17Dwarkesh Patel?

  Right.

• 1:01:18Speaker 01

  And then, so I guess what I'm saying is like, we need. Intelligent models to help us refine even the pre-training set to just narrow it down to the cognitive components. And then I think get away with a much smaller model because it's a much better dataset and you could train it on it. But probably it's not trained directly on it. It's probably distilled for a much better model still.

• 1:01:35Dwarkesh Patel?

  Right. But why is a distilled version still a billion is I guess the thing I'm curious about?

• 1:01:40Speaker 01

  I just feel like distillation work extremely well. So almost every small model, if you have a small model, it's almost certainly distilled. Why would you train on?

• 1:01:47Dwarkesh Patel?

  Right, no, no, but why is the distillation not in 10 years not getting below 1 billion?

• 1:01:51Speaker 01

  Oh, you think it should be smaller than a billion? I mean, come on, right? I don't know. At some point, it should take at least a billion knobs to do something interesting. You're thinking it should be even smaller?

• 1:02:02Dwarkesh Patel?

  Yeah, I mean, just like if you look at the trend over the last few years, just finding low-hanging fruit and going from like trillion plus models that are like literally two orders of magnitude smaller in a matter of two years and having better performance.

• 1:02:14Speaker 01

  Yeah, yeah.

• 1:02:15Dwarkesh Patel?

  It makes me think the sort of like core of intelligence might be even way, way smaller, like plenty of room at the bottom to pair our first Feynman.

• 1:02:23Speaker 01

  I mean, I almost feel like I'm already contrarian by talking about a billion in the parameter cognitive core and you're outdoing me. I think, yeah, maybe we could get a little bit smaller. I mean, I still think that there should be enough, yeah, maybe it can be smaller. I do think that practically speaking, you want the model to have some knowledge. You don't want it to be looking up everything.

• 1:02:39Dwarkesh Patel?

  Yeah.

• 1:02:40Speaker 01

  Because then you can't like think in your head, you're looking up way too much stuff all the time. So I do think it needs to be some basic curriculum needs to be there for knowledge, but it doesn't have esoteric knowledge, you know?

• 1:02:49Dwarkesh Patel?

  Yeah. So we're discussing what like plausibly could be the cognitive core. There's a separate question, which is what will actually be the size of frontier models over time? And I'm curious to have prediction. So we had increasing scale up to maybe 4.5 and now we're seeing decreasing slash plateauing scale. There's many reasons that could be going on, but do you have a prediction about going forward? Will scale, will the biggest models be bigger? Will they be smaller? Will they be the same?

• 1:03:14Speaker 01

  Yeah, I don't know that I have a super strong prediction. I do think that the labs are just being practical. They have a flops budget and a cost budget. And it just turns out that pre-training is not where you want to put most of your flops or your cost. So that's why the models have gotten smaller because they are a bit smaller, the pre-training stage is smaller, et cetera, but they make it up in reinforced learning and all this kind of stuff, mid-training and all this kind of stuff that follows. So they're just being practical in terms of all the stages and how you get the most bang for the buck. So I guess like forecasting the trend I think is quite hard. I do still expect that there's so much low-hanging fruit. That's my basic, that's my basic expectation.

• 1:03:48Speaker 01

  And so I have a very wide distribution here.

• 1:03:52Dwarkesh Patel?

  Do you expect the low hanging fruit to be similar in kind to the kinds of things that have been happening over the last two to five years? Like just in terms of like, if I look at Nano Chat versus Nano GPT and then the architectural tweaks you made, is that basically like the flavor of things you continue to keep happening? Or is there, you're not expecting any giant paradigm shift?

• 1:04:12Speaker 01

  I expect the data sets to get much, much better because when you look at the average data sets, they're extremely terrible. Like so bad that I don't even know how anything works, to be honest. Like, look at the average example in the training set. Like factual mistakes, errors, nonsensical things. Somehow, when you do it at scale, the noise washes away and you're left with some of the signal. So data sets will improve a ton. It's just everything gets better. So our hardware, our older kernels, all the kernels for running the hardware and maximizing what you get with the hardware. You know, so Nvidia is slowly tuning the actual hardware itself, tensor cores and so on. All that needs to happen and will continue to happen. All the kernels will get better and utilize the chip to the max extent. All the algorithms will probably improve over optimization, architecture and

• 1:04:55Speaker 01

  just all of the modeling components of how everything is done and what the algorithms are that we're even training with. So I do, I do kind of expect like a just very just everything. Nothing dominates, everything plus 20%.

• 1:05:08Dwarkesh Patel?

  Right, interesting.

• 1:05:09Speaker 01

  This is like roughly what I've seen.

• 1:05:10Dwarkesh Patel?

  Okay, this is my general manager, Max.

• 1:05:13Speaker 02

  Good to be here, here every day.

• 1:05:14Dwarkesh Patel?

  And you have been here since you were onboarded about six months ago.

• 1:05:17Speaker 02

  Eight months ago?

• 1:05:18Dwarkesh Patel?

  Oh, right. Time passes so fast. But when I onboarded you, I was in France. And so we basically didn't get the chance to talk at all, almost.

• 1:05:27Speaker 02

  And you basically just gave me one login.

• 1:05:30Dwarkesh Patel?

  I gave you access to my Mercury platform, which is the banking platform that I was using at the time to run the podcast.

• 1:05:36Speaker 02

  And so I logged into Mercury, assuming that that would just be the first of many steps. But I realized that was how you were running the entire business, even down to a lot of our editors, our international contractors. And so you had just figured out how to set up these recurring payments to set up basic payroll.

• 1:05:50Dwarkesh Patel?

  I mean, Mercury made the experience of all of these things I was doing before so seamless that it didn't even occur to me until you pointed it out that this is not the natural way to set up. Payroll or invoicing or any of these other things.

• 1:06:00Speaker 02

  Yeah, I was surprised, but I was like, it's worked so far.

• 1:06:04Dwarkesh Patel?

  That's right.

• 1:06:04Speaker 02

  So maybe I'll trust it. And then now I can't think of doing anything else.

• 1:06:07Dwarkesh Patel?

  All right, you heard him. Visit mercury.com to apply online in minutes. Cool. Thanks, Max.

• 1:06:13Speaker 02

  Thanks for having me.

• 1:06:14Dwarkesh Patel?

  Dude, you're great at this.

• 1:06:15Speaker 02

  I'm so nervous, but thank you.

• 1:06:17Dwarkesh Patel?

  Mercury is a financial technology company, not a bank. Banking services provided through Choice Financial Group, Callman A and Evolve Bank and Trust, members FDIC. People have proposed different ways of charting how much progress you've made towards full AGI. Because if you can come up with some line, then you can see where that line intersects with AGI and where that would happen on the X axis. And so people have proposed, oh, it's like the education level, like we had a high schooler and then then they went to college with RL and they're going to get a PhDs.

• 1:06:45Speaker 01

  I don't like that one.

• 1:06:46Dwarkesh Patel?

  Or then they'll propose Horizon Link. So maybe they can do tasks that take a minute. They can do those autonomously, then they can autonomously do tasks that take an hour, a human an hour, a human a week, et cetera. How do you think about what is the relevant Y axis here? What is the, how should we think about how AI is making progress?

• 1:07:06Speaker 01

  So I guess I have two answers to that. Number one, I'm almost tempted to reject the question entirely because again, like I see this as an extension of computing. Have we talked about like how to chart progress in computing or how do you chart progress in computing since 1970s or whatever? What is the X axis? So I kind of feel like the whole question is kind of like funny from that perspective a little bit. But I will say, I guess, like when people talk about AI and the original AGI and how we spoke about it when we, when opening, I started. AGI was a system you can go to that can do any task that is economically valuable, any economically valuable task at human performance or better. Okay, so that was the definition and I was pretty happy with that at the time. And I kind of feel like I've stuck to that definition forever. And then people have made up all kinds of other definitions, but I like, I feel like I like that definition.

• 1:07:52Speaker 01

  Now, number one, the first concession that people make all the time is they just take out all the physical stuff. Because we're just talking about digital knowledge work. I feel like that's a pretty major concession compared to the original definition, which was like any task a human can do. I can lift things, et cetera. Like AI can't do that, obviously. So, okay, but we'll take it. What fraction of the economy are we taking away by saying, oh, only knowledge work? I don't actually know the numbers. I feel like it's about 10 to 20% if I had to guess is only knowledge work. Like, someone could work from home and perform tasks, something like that. I still think it's a really large market. Like, yeah, what is the size of the economy and what is 10, 20%? Like, we're still talking about a few trillion dollars of, even in the US, of market share almost, or like work.

• 1:08:37Speaker 01

  So it's still a very massive bucket. So, but I guess, like, going back to the definition, I guess what I would be looking for is, to what extent is that definition true? So are there jobs or lots of tasks? If we think of tasks as, you know, not jobs, but tasks, kind of difficult. Because the problem is, like, society will refactor based on the tasks that make up jobs compared to what's based on what's automatable or not. But today, what jobs are replaceable by AI? So a good example recently was Jeff Hinton's prediction that radiologists would not be a job anymore. And this turned out to be very wrong in a bunch of ways, right? So radiologists are alive and well and growing, even though computer vision is really, really good at recognizing all the different things that they have to recognize in images.

• 1:09:18Speaker 01

  And it's just messy, complicated job with a lot of surfaces and dealing with patients and all this kind of stuff in the context of it.

• 1:09:26Speaker 01

  So I guess, I don't actually know that by that definition, AI has made a huge amount of dent yet. But some of the, some of the jobs maybe that I would be looking for have some features that I think make it very amenable to automation earlier than later. As an example, call center employees often come up, and I think rightly so. Because call center employees have a number of simplifying properties with respect to what's automatable today.

• 1:09:49Speaker 01

  Their jobs are pretty simple. It's a sequence of tasks and every task looks similar. Like, you take a phone call with a person, it's 10 minutes of interaction or whatever it is, probably a bit longer, in my experience, a lot longer.

• 1:10:01Speaker 01

  And you complete some task in some scheme and you change some database entries around or something like that. So you keep repeating something over and over again, and that's your job. So basically, you do want to bring in the task horizon, how long it takes to perform a task. And then you want to also remove context, like, you're not dealing with different parts of services of companies or other customers. It's just the database, you and a person you're serving. And so it's more closed, it's more understandable, and it's purely digital. So I would be looking for those things, but even there, I'm not actually looking at full automation yet. I'm looking for an autonomy slider, and I almost expect that we are not gonna Instantly replace people. We're going to be swapping in AIs that do eighty percent of the volume. They delegate twenty percent of the volume to humans. And humans are supervising teams of five AIs doing the call center work that's more rote.

• 1:10:47Speaker 01

  So I would be looking for new interfaces or new companies that provide some kind of a layer that allows you to manage some of these AIs that are not yet perfect.

• 1:10:57Dwarkesh Patel?

  Yeah.

• 1:10:58Speaker 01

  And then I would expect that across the economy. And a lot of jobs are a lot harder than call center employee.

• 1:11:03Dwarkesh Patel?

  I wonder with radiologists.

• 1:11:06Dwarkesh Patel?

  I'm totally speculating. I have no idea how, what the actual workflow of a radiologist involves. But one analogy that might be applicable is

• 1:11:15Dwarkesh Patel?

  when Waymo was their first being ruled out, there'd be a person sitting in the front seat. And you just had to have them there to make sure that if something went really wrong, they're there to monitor. And I think even today, people are still watching to make sure things are going well. RoboTaxi, who was just deployed, actually still has a person inside it. And we could be in a similar situation where if you automate ninety-nine percent of a job, that last one percent the human has to do is incredibly valuable because it's bottlenecking everything else. And if it had, if it was the case with like with radiologists where the person sitting in the front of the Uber or the front of the Waymo has to be specially trained for years in order to be able to provide the last one percent, their wages should go up tremendously because they're like the one, the one thing bottlenecking why deployment. So radiologists, I think their wages have gone up for similar reasons.

• 1:11:57Dwarkesh Patel?

  If you're like the last bottleneck, you should, you're like, and you're not fungible, which like, you know, a Waymo driver might be fungible with other things.

• 1:12:05Dwarkesh Patel?

  So you might see this thing where like your wages go like whoop, and then to get a ninety percent and then like, just like that. And when the last one percent is gone.

• 1:12:11Speaker 01

  I see.

• 1:12:12Dwarkesh Patel?

  And I wonder if we're seeing similar things with radiology or salaries of call center workers or anything like that.

• 1:12:17Speaker 01

  Yeah, I think that's an interesting question. I don't think we're currently seeing that with radiology or, and I don't have like, in my understanding, but I think radiology is not a good example, basically. I don't know why Jeff Hinton picked on radiology because I think it's an extremely messy, messy, complicated profession.

• 1:12:35Dwarkesh Patel?

  Yeah.

• 1:12:36Speaker 01

  So I would be a lot more interested in what's happening with call center employees today, for example, because I would expect a lot of the rote stuff to be automatable today. And I don't have a first level access to it, but maybe I would be looking for trends of what's happening with the call center employees. Maybe some of the things I would also expect is maybe they are swapping in AI, but then I would still wait for a year or two because I would potentially expect them to pull, pull back and actually rehire some of the people.

• 1:13:00Dwarkesh Patel?

  I think there's been evidence that that's already been happening in the generally in like companies that have been adopting AI, which I think is quite surprising. And I also find what was really surprising, okay, AGI, right? Like a thing which would do everything and okay, we'll take out physical work. So thing which should be able to do all knowledge work. And what you would have naively anticipated that the way this progression would happen is like, you would take a little task that a consultant is doing, you take that out of the bucket. You take a little task that an accountant is doing, you take that out of the bucket. And then you're just doing this across all knowledge work. But instead, if we do believe we're on the path of AGI with the current paradigm, the progression is very much not like that, at least it just does not seem like consultants and accountants and whatever are getting like huge product improvement.

• 1:13:45Dwarkesh Patel?

  It's very much like programmers are like getting more and more chills of the way of their work. If you do look at the revenues of these companies, discounting just like normal chat revenue, which I think is like, I don't know, that's similar to like Google or something. Just looking at API revenues, it's like dominated by coding, right? So this thing which is general quote unquote, which should be able to do any knowledge work, is just overwhelmingly doing only coding. It's a surprising way that you would expect like the AGI to be deployed.

• 1:14:14Speaker 01

  So I think there's, there's an interesting point here because I do believe coding is like the perfect first thing for, for, for these LLMs and agents. And that's because coding has always fundamentally worked around text. It's computer terminals and text and everything is based around text. And LLMs, the way they're trained on the internet love text. And so they're perfect text processors and there's all this data out there and it's just perfect fit. And also we have a lot of infrastructure pre-built for handling code and text. So for example, we have a Visual Studio code or you know, your favorite IDE showing you code. And then agent can plug into that. So for example, if an agent has a diff where it made some change, we suddenly have all

• 1:14:57Speaker 01

  this code already that shows all the differences to a codebase using a diff. So we've, it's almost like we've pre-built a lot of the, a lot of the infrastructure for code. Now contrast that with some of the things that, that don't enjoy that at all. So as an example, like there's people trying to build automation, not for coding, but for example for slides, like I saw a company doing slides. That's much, much harder. And the reason it's much, much harder is because slides are not text. Slides are little graphics and they're arranged spatially and there's visual components to it and. And slides don't have this pre-built infrastructure. Like, for example, if an agent is to make a different change to your slides, how does a thing show you the diff? How do you see the diff? There's no, there's nothing that shows diffs for slides.

• 1:15:38Dwarkesh Patel?

  Mm hmm.

• 1:15:39Speaker 01

  Someone has to build it. So it's just some of these things are not amenable to AIs as they are, which is text processors. And code surprisingly is.

• 1:15:48Dwarkesh Patel?

  I, I, actually, I'm not sure if that alone explains it because I personally have tried to get LLMs to be useful in domains which are just pure language in, language out. Like rewriting transcripts, like coming up with clips based on transcripts, et cetera. And you might say, well, I didn't, it's very plausible that like, I didn't do every single possible thing I could do. I put a bunch of, you know, good examples in context, but maybe I should have done like some kind of fine tuning, whatever. So our mutual friend, Annie Matushak, told me that he actually tried 50 billion things to try to get models to be good at writing spaced repetition prompts. Again, very much language in, language out tasks, the kind of thing that should be dead center in the repertoire of these LLMs.

• 1:16:33Speaker 01

  I see.

• 1:16:33Dwarkesh Patel?

  And he tried in-context learning, obviously, with a few shot examples. He tried, I think, he told me like a bunch of things, like supervised fine tuning and like, you know, retrieval, whatever.

• 1:16:44Speaker 01

  Yeah.

• 1:16:45Dwarkesh Patel?

  And he just could not get them to make cards to a satisfaction. So I find it striking that even in language out domains, it's actually very hard to get a lot of economic value out of these models.

• 1:16:55Speaker 01

  I see.

• 1:16:56Dwarkesh Patel?

  Separate from coding. And I don't know what, what explains that.

• 1:16:58Speaker 01

  Yeah, I think, I think that makes sense. I mean, I would say, yeah, I'm not saying that anything text is trivial, right?

• 1:17:05Dwarkesh Patel?

  Yeah.

• 1:17:06Speaker 01

  I do think that code is like, it's pretty structured. Text is maybe a lot more flowery and there's a lot more like, like entropy in text, I would say. I don't know how else to put it. And also, I mean, code is hard. And so people sort of feel quite empowered by LLMs, even from like, simple, a simple kind of knowledge. I basically, I don't actually know that I have a very good answer. I mean, obviously, like text makes it much, much easier, maybe, is maybe why I put it. But it doesn't mean that all text is trivial.

• 1:17:35Dwarkesh Patel?

  Mm hmm. How do you think about superintelligence? Do you expect it to feel qualitatively different from normal?

• 1:17:44Dwarkesh Patel?

  Humans or human companies?

• 1:17:46Speaker 01

  I guess I think I see it as like a progression of automation in society, right? And again, like extrapolating the trend of computing. I just I feel like there will be a gradual automation of a lot of things. And superintelligence will be sort of like the extrapolation of that. So I do think we expect more and more autonomous entities over time that are doing a lot of the digital work and then eventually even the physical work, probably some amount of time later. But basically, I see it as just automation, roughly speaking.

• 1:18:11Dwarkesh Patel?

  I guess automation includes the things humans can already do and superintelligence supplies things humans.

• 1:18:16Speaker 01

  Well, but some of the things that people do is invent new things, which I would just put into the automation, if that makes sense.

• 1:18:21Dwarkesh Patel?

  Yeah, but I guess maybe less abstractly and more sort of like qualitatively. Do you expect something to feel like, okay, this because this thing can either think so fast or has so many copies or the copies can merge back into themselves? Or is quote unquote much smarter? Any number of advantages an AI might have. It will qualitative, the civilization in which these AIs exist will just feel qualitatively different from human civilization.

• 1:18:51Speaker 01

  I mean, it is fundamentally automation, but I mean, it will be like extremely foreign. I do I do think it will look really strange because, like you mentioned, we can run all of this on a computer cluster, et cetera, and much faster and all this thing. I mean, maybe some of the scenarios, for example, that I start to get like nervous about with respect with respect to when the world looks like that is this kind of like gradual loss of control and understanding of what's happening. And I think that's actually the most likely outcome, probably, is that there will be a gradual loss of understanding of and we'll gradually layer all this stuff everywhere and there'll be fewer and fewer people who understand it. And then there will be a sort of this like scenario of gradual loss of control and understanding of what's happening. That to me seems most likely outcome of how all the stuff will go down.

• 1:19:32Dwarkesh Patel?

  Let me probe on that a bit. It's not clear to me that loss of control and loss of understanding are the same things. A board of directors at like, whatever, TSMC, Intel, name a random company. They're just like prestigious 80 year olds. They have very little understanding and maybe they don't practically actually have control, but. Actually, maybe a better example is the President of the United States. The President has a lot of fucking power. I'm not trying to make a good statement about the current operant, but maybe I am. But like, the actual level of understanding is very different from the level of control.

• 1:20:07Speaker 01

  Yeah, I think that's fair. That's a good pushback. I think like, I guess I expect the loss of both.

• 1:20:16Dwarkesh Patel?

  How come? I mean, the loss of understanding is obvious, but why a loss of control?

• 1:20:20Speaker 01

  So, so we're really far into a territory of, I don't know what this looks like, but if I was to write sci-fi novels, they would look along the lines of not even a single like entity or something like that, that just sort of like takes over everything, but actually like multiple competing entities that gradually become more and more autonomous. And some of them go rogue and the others like fight them off and all this kind of stuff. And it's like this, this hot pot of completely autonomous activity that we've delegated to. I, I kind of feel like it would have that flavor.

• 1:20:53Dwarkesh Patel?

  It is not the fact that they are smarter than us that is resulting in a loss of control.

• 1:20:57Speaker 01

  No, not necessarily.

• 1:20:57Dwarkesh Patel?

  It is the fact that they are competing with each other. And whatever arises out of that competition that leads to the loss of control.

• 1:21:07Speaker 01

  I mean, I basically expect there to be, I mean, a lot of these things, I mean, they will be tools to people and the people could, some of the population is like, they're acting on behalf of people or something like that. So maybe those people are in control, but maybe it's a loss of control overall for society in the sense that of like outcomes we want or something like that. Where you have entities acting on behalf of individuals that are still kind of roughly seen as out of control.

• 1:21:30Dwarkesh Patel?

  Yeah, yeah. This is a question I should have asked earlier. So we were talking about how currently it feels like when you're doing AI engineering or AI research, these models are more like in the category of compiler rather than in the category of a replacement. At some point, if you have quote unquote AGI, it should be able to do what you do. And do you feel like having a million copies of you in parallel results in some huge speed up of AI progress? Basically, if that does happen, would you, do you expect to see an intelligence explosion? Or even once we have a true AI, I'm not talking about LLMs today, but real AGI.

• 1:22:02Speaker 01

  Like what I mean is, I do, but it's business as usual because we're in an intelligence explosion already and have been for decades. And when you look at GDP, it's basically the GDP curve that is an exponential weighted sum over so many aspects of the industry. Everything is gradually being automated, has been for hundreds of years. Industry revolution is automation and some of the physical components and tool building and all this kind of stuff. Compilers are early software automation, et cetera. So I kind of feel like we've been recursively self-improving and exploding for a long time. Maybe another way to see it is, I mean, Earth was a pretty, I mean, if you don't look at the biomechanics and so on, it was a pretty boring place, I think, and looked very similar if you just look from space. And Earth is spinning and then like we're in the middle of this like firecracker event.

• 1:22:46Dwarkesh Patel?

  Right.

• 1:22:47Speaker 01

  But we're seeing it in slow motion, but I definitely feel like this is, this has already happened for a very long time. And again, like I don't see AI as like a distinct technology with respect to what has already been happening for a long time.

• 1:23:00Dwarkesh Patel?

  Is it, you think it's going to continue with this hyper exponential trend?

• 1:23:03Speaker 01

  And that's why, like, this is, this was very interesting to me because I was, I was trying to find AI in the GDP for a while. I thought that GDP should go up, but then I looked at some of the other technologies that I thought were very transformative. Like maybe computers or mobile phones or et cetera, you can't find them in GDP. GDP is the same exponential. And it's just that even, for example, the early iPhone didn't have the App Store and it didn't have a lot of the bells and whistles that the modern iPhone has. And so even though we think of 2008 was it when iPhone came out as like a major seismic change, it's actually not. Everything is like so spread out and so slowly diffuses that everything ends up being averaged up into the same exponential. And it's the exact same thing with computers. You can't find them in the GDP is like, oh, we have computers now. That's not what happened because it's such a slow progression. And with AI, we're going to see the exact same thing. It's just more automation.

• 1:23:48Speaker 01

  It allows us to write different kinds of programs that we couldn't write before, but AI is still fundamentally a program. And it's a new kind of computer and a new kind of, kind of computing system. But it has all these problems, it's going to diffuse over, over time, and it's still going to add up to the same exponential. And we're still going to have an exponential that's going to get extremely. Vertical and it's going to be very foreign to live in that kind of an environment.

• 1:24:10Dwarkesh Patel?

  Are you saying that like what will happen is so if you go if you look at the trend before the industrial revolution to currently you have a hyper exponential where you go from like 0% growth to then 10,000 years ago 0.02% growth. And then currently we're at 2% growth. So that's a hyper exponential and you're saying if you're charting AI on there, then it's like AI takes you to 20% growth or 200% growth. Or you could be saying if you look at the last 300 years, what you've been seeing is you have technology after technology, computers, electrification, steam, steam engines, railways, et cetera. But the rate of growth is the exact same. It's 2%. So are you saying the rate of growth will?

• 1:24:46Speaker 01

  Obviously, I expect this, the rate of growth has also stayed roughly constant, right?

• 1:24:51Dwarkesh Patel?

  For only the last 200, 300 years, but over the course of human history, it's like exploded, right? It's like gone from like 0% basically to like faster, faster, faster industrial explosion, 2%.

• 1:25:01Speaker 01

  Basically, I guess what I'm saying is for a while I tried to find AI or look for AI in like the GDP curve and I kind of convinced myself that this is false. And that even when people talk about recursive self-improvement and labs and stuff like that, I even don't, this is business as usual. Of course, it's going to recursively self-improve and it's been recursively self-improving. Like LLMs allow the engineers to work much more efficiently to build the next round of LLM. And a lot more of the components are being automated and tuned and et cetera. So all the engineers having access to Google search is sort of part of it. All the engineers having an ID, all of them having autocomplete or having Claude code, et cetera. It's all just part of the same speed up of the whole thing. So it's just so smooth.

• 1:25:42Dwarkesh Patel?

  But just to clarify, you're saying that the rate of growth will not change. Like, you know, the intelligence explosion will show up as like, it just enabled us to continue staying on the 2% growth trajectory. Just that the internet helped us stay on the 2% growth trajectory.

• 1:25:53Speaker 01

  Yeah, my expectation is that it stays the same pattern.

• 1:25:56Dwarkesh Patel?

  Yeah, I mean, just to throw the opposite argument against you. My expectation is that it like blows up because I think. True AGI, and I'm not talking about LLM coding bots, I'm talking about like actual, this is like a replacement of a human in a server is qualitatively different from these other productivity improving technologies because it's labor itself, right? I think we live in a very labor constrained world. Like if you talk to any startup founder, any person, you can just be like, okay, what do you need more of? You just like need really talented people. And if you just have billions of extra people who are inventing stuff, integrating themselves, making companies bottoms start to finish, that feels qualitatively different from just

• 1:26:40Dwarkesh Patel?

  like a single technology. It's just sort of like just asking if you like, if you had 10 billion extra people on the planet.

• 1:26:44Speaker 01

  I mean, maybe a counterpoint. I mean, number one, I'm actually pretty, um, pretty willing to be convinced one way or another on this point. But I will say, for example, computing is labor. Computing was labor. Computers, like a lot of jobs disappeared because computers are automating a bunch of digital information processing that you now don't need a human for. And so computers are labor and that has played out. And, you know, self-driving as an example is also like computers doing labor. So like, I guess that's already been playing out. So it's still business as usual.

• 1:27:13Dwarkesh Patel?

  Yeah. I guess you have a machine which is spitting out more things like that at potentially faster pace. And so we historically have, we have examples of the growth regime changing where like you went from, you know, 0.2% growth to 2% growth. So it seems very plausible to me that like a machine which is then spitting out the next self-driving car and the next internet and whatever.

• 1:27:34Speaker 01

  I mean, I kind of, yeah, I, I see where it's coming from. At the same time, I do feel like people make this assumption of like, okay, we have God in a box and now it can do everything. And it's just, it just won't look like that. It's gonna be, it's gonna be able to do some of the things. It's gonna fail at some other things. It's gonna be gradually put into society and basically end up with the same pattern is my prediction.

• 1:27:53Dwarkesh Patel?

  Yeah.

• 1:27:53Speaker 01

  Because, because this assumption of suddenly having a completely intelligent, fully flexible, fully general human in a box and we can dispense it with arbitrary problems in society. I, I, I don't think that we will have this like discrete change. And, um, and so I think we'll arrive at the same. At the same kind of gradual diffusion of this across the industry.

• 1:28:14Dwarkesh Patel?

  Hmm. I, I, I think what often ends up being misleading in these conversations is people that I don't like to use the word intelligence in this context. Because intelligence implies you think like, oh, super, uh, super intelligence will be sitting, there'll be a single super intelligence sitting in a server and it'll like divine how to come up with new technologies and inventions that causes this explosion. And that's not what I'm imagining when I'm imagining 20% growth. I'm imagining that there's billions of, you know, basically like very smart, human-like minds potentially, or that's all that's required.

• 1:28:46Dwarkesh Patel?

  But the fact that there's hundreds of millions of them, billions of them, each individually making new products, figuring out how to integrate themselves into the economy, just the way if like a highly experienced, smart immigrant came to the country, you wouldn't need to like figure out how to integrate them in the economy. They, they figure it out, they could start a company, they could like, uh, make inventions, you know, or like just increase productivity in the world. And we have examples, even in the current regime of places that have had 10, 20% economic growth. You know, if you just have a lot of people and less capital in comparison to the people, you could have Hong Kong or Shenzhen or whatever, just had decades of 10% plus growth.

• 1:29:23Dwarkesh Patel?

  And I think it's just like, there's a lot of really smart people who are ready to like make use of the resources and do this like period of catch up because we've had this discontinuity. And I think AI might be similar.

• 1:29:34Speaker 01

  So I think, um, I, I think I understand, but I still think that you're presupposing some discrete jump, there were just some unlock that we're waiting to claim. And suddenly we're going to have geniuses in data centers. And I, I still think you're presupposing some discrete jump that I think has basically no historical precedent that I can't find in any of the statistics and that I think probably won't happen.

• 1:29:53Dwarkesh Patel?

  I mean, the industrial revolution is such a jump, right? You went from like 0% growth or 0.2% growth to 2% growth. I'm just saying like you'll see another jump like that.

• 1:30:00Speaker 01

  I, I, I'm a little bit suspicious. I would have to look at it. I'm a little bit suspicious and I would have to take a look. For example, like maybe the, some of the logs are not very good from before the industrial revolution or something like that. Uh, so I'm a little bit suspicious of it, but um, yeah, maybe you're right. I don't, I don't have strong opinions. Maybe you're saying that this was a singular event that was extremely magical and you're saying that maybe there's going to be another event that's going to be just like that, extremely magical. It will break paradigm and so on.

• 1:30:24Dwarkesh Patel?

  I actually don't think the, I mean, the crucial thing with the industrial revolution was that it was not magical, right? Like. If you just zoomed in, what you would see in 1770 or 1870 is not that there was some key invention.

• 1:30:39Speaker 01

  Yeah, exactly.

• 1:30:39Dwarkesh Patel?

  But at the same time, you did move the economy to a regime where the progress was much faster and the exponential 10xed. And I expect a similar thing from AI where it's not like there's going to be a single moment where we made the crucial invention.

• 1:30:53Speaker 01

  There's still some overhang that's being unlocked, like maybe there's a new energy source. There's some unlock, in this case, some kind of a cognitive capacity. And there's an overhang of cognitive work to do.

• 1:31:03Dwarkesh Patel?

  That's right.

• 1:31:03Speaker 01

  And you're expecting that overhang to be filled by this new technology when it crosses the threshold.

• 1:31:07Dwarkesh Patel?

  Yeah, and I mean, maybe one way to think about it is through history, a lot of growth, I mean, growth comes because people come up with ideas and then people are like out there doing stuff to execute those ideas and make valuable output. And through most of this time, population isn't exploding, that has been driving growth. For the last 50 years, people have argued that growth has stagnated. Population in frontier countries is also stagnated. I think we go back on the hyper exponential growth in population and output. Right, it's our exponential growth in population that causes hyper exponential growth in output.

• 1:31:38Speaker 01

  Yeah, I mean, yeah, it's really hard to tell.

• 1:31:41Dwarkesh Patel?

  Yeah.

• 1:31:41Speaker 01

  I understand that viewpoint.

• 1:31:43Dwarkesh Patel?

  Yeah.

• 1:31:43Speaker 01

  I don't intuitively feel that viewpoint.

• 1:31:45Dwarkesh Patel?

  So we just got access to Google's VO 3.1. And it's been really cool to play around with. The first thing we did was run a bunch of prompts through both VO3 and 3.1 to see what's changed in the new version. So here's VO3. Hi, I'm Max and I got stuck in a local minimum again.

• 1:32:04Speaker 02

  It's okay, Max. We've all been there. Took me three epochs to get out.

• 1:32:09Dwarkesh Patel?

  And here is VO 3.1. Hi, I'm Max and I got stuck in a local minimum again.

• 1:32:14Speaker 03

  It's okay, Max. We've all been there. Took me three epochs to get out.

• 1:32:18Dwarkesh Patel?

  3.1's output is just consistently more coherent and the audio is noticeably higher quality. We've been using VO for a while now, actually. We released an essay earlier this year about AI firms, fully animated by VO2. And it's been amazing to see how fast these models are improving. This update makes VO even more useful in terms of animating our ideas and our explainers. You can try VO right now in the Gemini app with Pro and Ultra subscriptions. You can also access it through the Gemini API or through Google Flow. You recommended Nick Lane's book to me and then on that basis, I also found it super interesting and I interviewed him. And so I actually have some questions about sort of thinking about intelligence in evolutionary history. Now that you, over the last 20

• 1:33:02Dwarkesh Patel?

  years of doing AI research, you maybe have a more tangible sense of what intelligence is, what it takes to develop it. Are you more or less surprised as a result that evolution just sort of spontaneously stumbled upon it?

• 1:33:17Speaker 01

  I love Nick Lane's books, by the way. So yeah, I was just listening to his podcast on the way up here. With respect to intelligence and its evolution, I do came it came fairly, I mean, it's very, very recent, right? I am surprised that it evolved. Yeah. I find it fascinating to think about all the worlds out there. Like, say, there's a thousand planets like Earth and what they look like. I think Nick Lane was here talking about some of the early parts, right? Like, okay, he expects basically very similar life forms, roughly speaking, and bacteria-like things in most of them. And then there's a few breaks in there. I would expect that the evolution of intelligence intuitively feels to me like it should be fairly rare event. And there have been animals for, I guess maybe you should base it on how long something has existed. So for example, if bacteria have

• 1:34:01Speaker 01

  been around for 2 billion years and nothing happened, then going to your carrier is probably pretty hard because bacteria actually came up quite early in Earth's evolution or history. And so I guess, how long have we had animals? Maybe a couple hundred million years, like multicellular animals that like run, run, crawl, et cetera, which is maybe 10% of Earth's lifespan or something like that. So maybe on that timescale is actually not too tricky. I still feel like it's still surprising to me, I think intuitively, that it developed. I would maybe expect just a lot of like animal-like life forms doing animal-like things. The fact that you can get something that creates culture and knowledge and accumulates it is surprising to me.

• 1:34:42Dwarkesh Patel?

  Okay, so there's actually a couple of interesting follow-ups. If you buy this sun perspective that actually the crux of intelligence is animal intelligence, what the quote he said is if you got to the squirrel, you'd be most of the way to AGI. Then we got to squirrel intelligence, I guess, right after the Cambrian explosion 600 million years ago. It seems like what instigated that was the oxygenation event 600 million years ago. But immediately, the sort of like intelligence algorithm was there to like make the squirrel intelligence, right? So it's suggestive that animal intelligence was like that. As soon as you had the oxygen in the environment, you had the Curiot, you could just like get the algorithm. Maybe there was like sort of an accident that evolution stumbled upon it so fast, but

• 1:35:27Dwarkesh Patel?

  I don't know if that suggests it's like actually quite, at the end, going to be quite simple.

• 1:35:31Speaker 01

  Yes, basically, it's so hard to tell, right? With any of this stuff. I guess you can base it a little bit on how long something has existed, or how long it feels like something has been bottlenecked. So Nick Lane is very good about describing this like very apparent bottleneck in bacteria in archaea. For 2 billion years, nothing happened, like extreme diversity of chemical, of biochemistry, and yet nothing that grows to become animals 2 billion years. I don't know that we've seen exactly that kind of an equivalent with animals and intelligence, to your point, right? But I guess maybe we could also look at it with respect to how many times we think evolution, certain intelligence has like individually sprung up.

• 1:36:07Dwarkesh Patel?

  That's a really good, that's a really good thing to investigate.

• 1:36:09Speaker 01

  Maybe one thought on that is, I almost feel like, well, there's the hominid intelligence, and there's, I would say, like the bird intelligence, right? Like ravens, etc., are extremely clever, but they actually, their brain, brain parts are actually quite distinct, and we don't have that much existence. So maybe that's a slight event of, there's a slight indication of maybe intelligence bringing up a few times. And so in that case, you'd maybe expect it more frequently or something like that.

• 1:36:33Dwarkesh Patel?

  Yeah. A former guest, Gurn, and also Carl Shulman, have made a really interesting point about that, which is, their perspective is

• 1:36:42Dwarkesh Patel?

  That the scalable algorithm which humans have and primates have arose in birds as well. And maybe other times as well. But in humans found a evolutionary niche which rewarded marginal increases in intelligence.

• 1:36:58Dwarkesh Patel?

  And also had a scalable brain algorithm that could achieve those increases in intelligence. And so, for example, if a bird had a bigger brain, it would just like collapse out of the air. So it's very smart for the size of its brain, but it's like, it's not in a niche which rewards the brain getting bigger.

• 1:37:15Speaker 01

  Yeah.

• 1:37:15Dwarkesh Patel?

  Maybe similar with some really smart.

• 1:37:17Speaker 01

  Dolphins, etc.

• 1:37:18Dwarkesh Patel?

  Exactly, yeah. Whereas humans, you know, like we have hands that like reward being able to learn how to do tool use. We can externalize digestion, more energy to the brain. And that kicks off the flywheel.

• 1:37:28Speaker 01

  Oh yeah, and just stuff to work with. I mean, I'm guessing it would be harder to, if I was a dolphin. I mean, how do you do, you can't have fire, for example, and stuff like that. I mean, they're probably like the universe of things you can do in water, like inside water is probably lower than what you can do on land.

• 1:37:43Dwarkesh Patel?

  Yeah.

• 1:37:43Speaker 01

  Just chemically.

• 1:37:44Dwarkesh Patel?

  Right.

• 1:37:44Speaker 01

  Yeah, I do, I do agree with this, with this viewpoint of these niches and what's, what's being incentivized. I still find it kind of miraculous that, I don't, I would have maybe expected things to get stuck on like animals with bigger muscles, you know?

• 1:37:57Dwarkesh Patel?

  Yeah.

• 1:37:58Speaker 01

  Like going through intelligence is actually a really fascinating breaking point.

• 1:38:02Dwarkesh Patel?

  The way it burned pretty is, the reason it was so hard is, is a very tight line between being in a situation where something is so important to learn that it's not just worth distilling the exact right circuits.

• 1:38:15Speaker 01

  Yeah.

• 1:38:16Dwarkesh Patel?

  Directly back into your DNA versus it's not important enough to learn at all.

• 1:38:20Speaker 01

  Yeah.

• 1:38:21Dwarkesh Patel?

  It has to be something which is like, you have to incentivize building the algorithm to learn in lifetime.

• 1:38:28Speaker 01

  Yeah, exactly. You have to incentivize some kind of adaptability. You actually want something that, you actually want environments that are unpredictable. So evolution can't bake your algorithms into your weights. A lot of, a lot of animals are basically pre-baked in this sense. And so humans have to figure it out at test time when they get born. And so maybe there was, you actually want these kinds of environments that actually change really rapidly or something like that where you can't foresee what will work well. And so you actually put all that intelligence, you create intelligence to figure it out at test time.

• 1:38:56Dwarkesh Patel?

  So Quentin Pope had this interesting blog post where he's saying, the reasoning doesn't expect a sharp takeoff is.

• 1:39:03Dwarkesh Patel?

  So humans had the sharp takeoff where 50,000 years ago, we seem to have had the cognitive architectures that we have today. And 10,000 years ago, agricultural revolution, modernity, dot, dot, dot. What was happening in that 50,000 years? Well, you had to build this sort of like cultural scaffold where you can accumulate knowledge over generations. This is an ability that exists for free in the way we do AI training. Where if you retrain a model, it can still, I mean, in many cases, they're literally distilled, but they can be trained on each other. They can be trained on the same pre-training corpus. They don't literally have to start from scratch. So there's a sense in which the thing which it took humans a long time to get this cultural loop going just comes for free with the way we do LLM training.

• 1:39:47Speaker 01

  Yes and no, because LLMs don't really have the equivalent of culture. And maybe we're giving them way too much and incentivizing not to create it or something like that. But I guess like the mention of culture and of written record and of like passing down notes between each other. I don't think there's an equivalent of that with LLMs right now. So LLMs don't really have culture right now. And it's kind of like one of the, I think, impediments, I would say.

• 1:40:08Dwarkesh Patel?

  Can you give me some sense of what LLM culture might look like?

• 1:40:11Speaker 01

  So in the simplest case, it would be a giant scratch pad that the LLM can edit. And as it's reading stuff or as it's helping out with work, it's editing the scratch pad for itself. Why can't an LLM write a book for the other LLMs? That would be cool. Like, why can't other LLMs read this LLM's book and be inspired by it or shocked by it or something like that? There's no equivalence for any of this stuff.

• 1:40:31Dwarkesh Patel?

  Interesting. When would you expect that kind of thing to start happening? And more general question about like multi-agent systems and a sort of like independent AI civilization and culture.

• 1:40:42Speaker 01

  I think there's two powerful ideas in the realm of multi-agent that have both not been like really claimed or so on. The first one I would say is culture and LLMs basically a growing repertoire of knowledge for their own purposes. The second one looks a lot more like the powerful idea of self-play. In my mind, it's extremely powerful. So evolution actually is a lot of. Competition, basically, driving intelligence and evolution. And in AlphaGo, more algorithmically, like AlphaGo is playing against itself and that's how it learns to get really good at Go. And there's no equivalent of self-playing LLMs, but I would expect that to also exist, but no one has done it yet. Like, why can't an LLM, for example, create a bunch of problems that another LLM is learning to solve?

• 1:41:25Speaker 01

  And then the LLM is always trying to, like, serve more and more difficult problems, stuff like that, you know? So like, I think there's a bunch of ways to actually organize it, and I think it's a realm of research. But I think, I haven't seen anything that convincingly, like, claims both of those, like, multi-agent improvements. I still think we're mostly in the realm of a single individual agent, but I think, I also think that will change. And in the realm of culture also, I would bucket also organizations. And we haven't seen anything like that convincingly either.

• 1:41:54Speaker 01

  So that's why we're still early.

• 1:41:55Dwarkesh Patel?

  And can you identify the key bottleneck that's preventing this kind of collaboration between LLMs?

• 1:42:01Speaker 01

  Maybe like the way I would put it is somehow remarkably, again, some of these analogies work and they shouldn't, but somehow remarkably they do. A lot of the smaller models, or the dumber, like, the smaller models somehow remarkably resemble like a kindergarten student, or then like a elementary school student, or high school student, et cetera. And somehow, we still haven't graduated enough where the stuff can take over. Like, it's still mostly, like, my cloth code or codex, they still kind of feel like this elementary grade student. I know that they can take PhD quizzes, but they still cognitively feel like a kindergarten or an elementary school student. So I don't think they can create culture because they're still kids, you know? Like, they're savant kids. They have upside, they have perfect memory of all this stuff, et cetera, and they can

• 1:42:45Speaker 01

  convincingly create all kinds of slop that looks really good. But I still think they don't really know what they're doing, and they don't really have the cognition across all these little checkboxes that we still have to collect.

• 1:42:54Dwarkesh Patel?

  Yeah. So you've talked about how you were at Tesla, leading self-driving from 2017 to 2022, and then you firsthand saw this progress from, we went from cool demos to now thousands of cars out there actually autonomously doing drives. Why did that take a decade? Like, what was happening through that time?

• 1:43:14Speaker 01

  Yeah. So I would say, one thing I would almost instantly also push back on is, this is not even near done.

• 1:43:21Speaker 01

  So in a bunch of ways that I'm gonna get to. I do think that self-driving is very interesting because it's definitely like where I get a lot of my intuitions because I spent five years on it. And it has this entire history where actually the first demos of self-driving go all the way to 9080s. You can see a demo from CMU in 1986, there's a truck that's driving itself on roads.

• 1:43:43Speaker 01

  But okay, fast forward, I think when I was joining Tesla, I had I had a very early demo of a Waymo. And it basically gave me a perfect drive in 2014 or something like that. So perfect Waymo drive a decade ago.

• 1:43:58Speaker 01

  Took us around Palo Alto and so on because I had a friend who worked there.

• 1:44:02Speaker 01

  And I thought it was like very close and then still took a long time. And I do think that some there's for some kinds of tasks and jobs and so on. There's a very large demo to product gap where the demo is very easy, but the products are very hard. And it's especially the case in cases like self-driving where the the cost of failure is too high. Right? Many many industries, tasks and jobs maybe don't have that property. But when you do have that property, that definitely increases the timelines. I do think that, for example, in software engineering, I do actually think that that property does exist. I think for a lot of vibe coding, it doesn't. But I think if you're writing actual production-grade code, I think that property should exist. Because any kind of mistake actually leads to a security vulnerability or something like that.

• 1:44:43Speaker 01

  And millions and hundreds of millions of people's personal social security numbers, etc. get leaked or something like that. And so I do think that it is a case that in software, people should be careful. Kind of like in self-driving. Like in self-driving, if you if things go wrong, you might get injury in. I guess there's worse outcomes, but I guess in in software, I almost feel like it's almost unbounded how terrible some things could be.

• 1:45:08Speaker 01

  So I do think that they share that property. And then I think basically what takes the long amount of time and the way to think about it is that. It's a march of nines and every single nine is a constant amount of work. So every single nine is the same amount of work. So when you get a demo and something works 90% of the time, that's just, uh, that's just the first nine. And then you need a second nine, a third nine, fourth nine, fifth nine. And while I was at Tesla for, was it five years or so? I think we went through maybe three nines, two nines. I don't know what it is, you know? But like multiple nines of iteration, there's still more nines to go. And so that's why these things take, it takes so long.

• 1:45:42Speaker 01

  And so it's definitely formative for me, like seeing something that was a demo. I'm very unimpressed by demos. So whenever I see demos of anything, I'm extremely unimpressed by that. It works better if you can, um, if it's a demo that someone cooked up and is just showing you it's worst. If you can interact with it, it's a bit better. But even then you're not done. You need actual product. It's going to face all these challenges in, when it comes in contact with reality and all these different pockets of behavior that need patching. And so I think we're going to see all this stuff play out. It's a march of nines. Each nine is constant. Demos are encouraging, still a huge amount of work to do. Uh, I do think it is a, um, kind of a critical safety domain, unless you're doing vibe coding, which is all nice and fun and so on. And, uh, so that's why I think this also enforced my timelines from that perspective.

• 1:46:26Dwarkesh Patel?

  Hmm. That's, that's very interesting to hear you say that the sort of safety guarantees you need from software are actually not dissimilar to self-driving. Because what people will often say is that self-driving took so long because the cost of failure is so high. Like a human makes a mistake on average every 400,000 miles or every seven years. And if you had to release a coding agent that couldn't make a mistake for at least seven years, it would be much harder to deploy. But I guess your point is that if you made a catastrophic coding mistake, like breaking some important system every seven years.

• 1:46:59Speaker 01

  Very easy to do.

• 1:47:00Dwarkesh Patel?

  And in fact, in terms of sort of wall clock time, it would be much less than seven years because you're like constantly outputting code like that, right? So it's like per tokens, or in terms of tokens, it would be seven years. But in terms of wall clock time, it would be pretty close.

• 1:47:12Speaker 01

  It's a much harder problem. I mean, self-driving is just one of thousands of things that people do. It's almost like a single vertical, I suppose. Whereas when we're talking about general software engineering, it's even more, there's more surface area.

• 1:47:22Dwarkesh Patel?

  There's another, uh,

• 1:47:25Dwarkesh Patel?

  Objection people make to that analogy, which is that with self-driving, what took a big fraction of that time was solving the problem of building basic, having basic perception that's robust and building representations and having a model that has some common sense so it can generalize to when it sees something that's slightly out of distribution. If somebody's waving down the road this way, you don't need to train for it. The thing will have some understanding of how to respond to something like that. And these are things we're getting for free with LLMs or VLMs today. So we don't have to solve these very basic representation problems. And so now deploying AIs across different domains will sort of be like deploying a self-driving car with current models to a different city, which is hard, but not like a 10-year-long task.

• 1:48:10Speaker 01

  Yeah, basically, I'm not 100% sure if I fully agree with that. I don't know that we're, how much we're getting for free. And I still think there's like a lot of gaps in understanding in what we are getting. And we're definitely getting more generalizable intelligence in a single entity, whereas self-driving is a very special purpose task that requires, in some sense, building a special purpose task is maybe even harder in a certain sense because it doesn't like fall out from a more general thing that you're doing at scale, if that makes sense. So, but I still think that the analogy doesn't, I still don't know if it fully resonates because like the LLMs are still pretty fallible and I still think that they have a lot of gaps and that it still needs to be filled in. And I don't think that we're getting like magical generalization completely out of the box, sort of in a certain sense. And the other aspect that I want to also actually return to when I was in the, in

• 1:48:55Speaker 01

  the beginning was self-driving cars are nowhere near done still. So even though, so the deployments still are pretty minimal, right? So even Waymo and so on has very few cars and they're doing that roughly speaking because they're not economical, right? Because they've built something that, that lives in the future. And so they, they had to like pull back future, but they had to make it uneconomical. So they have all these like, you know, there's all these costs, not just marginal costs for those cars and their operation and maintenance, but also the capex of the entire thing. So making it economical is still going to be a slog, I think, for them. And then also, I think when you look at these cars and there's no one driving, I also think it's a little bit deceiving because there are actually very elaborate teleoperation centers of people actually kind of like in a loop with these cars.

• 1:49:40Speaker 01

  And I don't have the, I don't have the full extent of it, but I think there's more human in the loop than you might expect. And there's people somewhere out there, basically beaming in from the sky. And I don't actually know they're fully in the loop with the driving. I think some of the times they are, but they're certainly involved and there are people. And in some sense, we haven't actually removed the person. We've like moved them to somewhere where you can't see them. I still think there will be some work, as you mentioned, going from environment to environment. And so I think like there's still challenges to, to make self-driving real. But I, I do agree that it's definitely across the threshold where it kind of feels real, unless it's like really teleoperated. For example, Waymo can't go to all the different parts of the city. My suspicion is it's like parts of city where you don't get good signal.

• 1:50:21Speaker 01

  Anyway, so basically I don't actually know anything about the stack. I mean, I'm just making up, making up stuff.

• 1:50:26Dwarkesh Patel?

  You let self-driving for five years at Tesla.

• 1:50:29Speaker 01

  Sorry, I don't know anything about the specifics of Waymo. I feel like talk about them. Actually, by the way, I love Waymo and I take it all the time.

• 1:50:34Dwarkesh Patel?

  Yeah.

• 1:50:34Speaker 01

  So I don't want to say like,

• 1:50:35Dwarkesh Patel?

  Sure.

• 1:50:36Speaker 01

  I just think that people again are sometimes a little bit too naive about some of the progress. And I still think there's a huge amount of work. And I think Tesla took, in my mind, a lot more scalable approach.

• 1:50:45Dwarkesh Patel?

  Yeah.

• 1:50:45Speaker 01

  And I think the team is doing extremely well and it's gonna, and I, I'm kind of like on the record for predicting how this thing will go, which is like Waymo had like early start because you can package up so many sensors. But I do think Tesla is taking the more scalable strategy and it's gonna look a lot more like that. So I think this will have to still play out and hasn't. But basically like, I don't want to talk about self-driving as something that took a decade because it didn't take, it didn't take yet.

• 1:51:10Speaker 01

  If that makes sense.

• 1:51:11Dwarkesh Patel?

  Because one, it's the, the start is at 1980, not 10 years ago. And then two, the end is not here yet.

• 1:51:16Speaker 01

  Yeah, the end is not, not near yet. Because when we're talking about self-driving, usually in my mind, it's self-driving at scale.

• 1:51:21Dwarkesh Patel?

  Yeah.

• 1:51:22Speaker 01

  People don't have to get a driver's license, et cetera.

• 1:51:24Dwarkesh Patel?

  I'm curious to bounce to other ways in which the analogy. Might be different. And the reason I'm especially curious about this is because I think the question of how fast AI is deployed, how valuable it is when it's early on is like potentially the most important question in the world right now, right? Like if you're trying to model what the year 20 or 30 looks like, this is the question you want to have some understanding of. So another thing you might think is, one, you have this latency requirement with self-driving, where you have, I have no idea what the actual models are, but I assume like tens of millions of parameters or something, which is not the necessary constraint for knowledge work with LLMs. Or maybe it might be with the computer use and stuff, but anyways.

• 1:52:05Dwarkesh Patel?

  The other big one is, maybe more importantly, on this capex question, yes, there is additional cost to serving up an additional copy of a model, but the sort of OpEx of a session is quite low and you can amortize the cost of AI into the training run itself, depending on how inference scaling goes and stuff. But it's certainly not as much as like building a whole new car to serve another instance of a model. So it just, the economics of deploying more widely are much more favorable.

• 1:52:40Speaker 01

  I think that's right. I think if you're sticking in the realm of bits, bits are like a million times easier than anything that touches the physical world. I definitely grant that. Bits are completely changeable, arbitrarily reshuffleable at a very rapid speed. So you would expect a lot more faster adaptation also in the industry and so on. And then, what was the first one?

• 1:53:01Dwarkesh Patel?

  The latency requirements.

• 1:53:02Speaker 01

  Oh, the latency requirements.

• 1:53:03Dwarkesh Patel?

  What are the implications for model size?

• 1:53:04Speaker 01

  I think that's roughly right. I mean, I also think that if we are talking about knowledge work at scale, there will be some latency requirements, practically speaking, because we, you know, we're going to have to make, create a huge amount of compute instead of that.

• 1:53:17Speaker 01

  And then I think like the last aspect that I very briefly want to also talk about is like all the, all the rest of it. Just all the rest of it. So. What does society think about it? What is the legal, how is it working legally? How is it working insurance wise? Who's really like, what is the, what are those layers of it and aspects of it? What happens with, what is the equivalent of people putting a cone on a Waymo?

• 1:53:38Dwarkesh Patel?

  Yeah.

• 1:53:39Speaker 01

  You know, there's going to be equivalents of all that. And so I, I do think that, I almost feel like self-driving is a very nice analogy that you can borrow things from. Yeah, what is the equivalent of a cone on a car? What is the equivalent of a teleoperating worker who's like hidden away? And almost like all the aspects of it.

• 1:53:56Dwarkesh Patel?

  Yeah. Do you have any opinions on whether this implies that the current AI build out, which would like 10x the amount of available compute in the world in a year or two? And maybe like hundred, more than a hundred exit by the end of the decade. If the use of AI will be lower than some people nighly predict, does that mean that we're overbuilding compute? Or do you, is that a separate question?

• 1:54:18Speaker 01

  Kind of like what happened with railroads and all this kind of stuff.

• 1:54:20Dwarkesh Patel?

  With what, sorry?

• 1:54:21Speaker 01

  Was it railroads? Oh, sorry.

• 1:54:22Dwarkesh Patel?

  Yeah, that's right.

• 1:54:23Speaker 01

  Yeah. There was, there is like historical precedent, or was it with telecommunications industry, right? Like pre-paving the internet that only came like a decade later, you know? And creating like a whole bubble in the telecommunications industry in the late nineties kind of thing. Yeah. So I don't know. I mean, I, I understand I'm sounding very pessimistic here. I'm only doing that, I'm actually optimistic. I think this will work. I think it's tractable. I'm only sounding pessimistic because when I go on my Twitter timeline, I see all this stuff that makes no sense to me. And, and I think there's a lot of reasons for why that exists. And I think a lot of it is, I think honestly, just fundraising. It's just incentive structures. A lot of it may be fundraising. A lot of it is just attention, you know, converting attention to money on the internet, you know, stuff like that.

• 1:55:09Speaker 01

  So I think there's, there's a lot of that going on and I think I'm only reacting to that. But I'm still like overall very bullish on technology. I think we're going to work through all this stuff. And I think there's been a rapid amount of progress. I don't actually know that there's overbuilding. I think that there's going to be, we're going to be able to gull up what in my understanding is being built. Because I do think that, for example, Claude Code or OpenAI Codex and stuff like that, they didn't even exist a year ago, right? Is that right? I think it's roughly right. This is miraculous technology that didn't exist. I think there's going to be a huge amount of demand as there, as we see the demand in ChatGPT already and so on. So, yeah, I don't actually know that there's overbuilding. But I guess I'm just reacting to like some of the very fast timelines that people continue to say incorrectly.

• 1:55:53Speaker 01

  And I've heard many, many times over the course of my 15 years in AI, where very reputable people keep getting this wrong all the time. And I think I want us to be properly calibrated. And I think some of this also, it does have like geopolitical ramifications and things like that when like some of these questions. And I think I don't want people to make mistakes on that, on that sphere of things. So I do want us to be grounded in reality of what technology is and isn't. So.

• 1:56:19Dwarkesh Patel?

  Let's talk about education and Eureka and stuff. One thing you could do is start another AI lab and try to solve those problems. Yeah, you're curious what you're up to now.

• 1:56:32Speaker 01

  Yeah.

• 1:56:33Dwarkesh Patel?

  And then, yeah, why not AI research itself?

• 1:56:35Speaker 01

  I guess maybe like the way I would put it is I feel some amount of like determinism around the things that AI labs are doing. And I feel like I could help out there, but I don't know that I would like uniquely, I don't know that I would like uniquely improve it. But I think like my personal big fear is that a lot of this stuff happens on the side of humanity and that humanity gets disempowered by it. And I, I kind of like, I care not just about all the Dyson spheres that we're going to build and that AI is going to build in a fully autonomous way. I care about what happens to humans.

• 1:57:08Dwarkesh Patel?

  Yeah.

• 1:57:09Speaker 01

  And I want humans to be well off in this future. And I feel like that's where I can a lot more uniquely add value than like an incremental improvement in the frontier lab. And so I guess I'm most afraid of something maybe like depicted in movies like Wally or Idiocracy or something like that, where humanity is sort of on the side of this stuff. And I want humans to be much, much better in this future. And so I guess to me, this is kind of like through education that you can actually achieve this.

• 1:57:36Dwarkesh Patel?

  And so what are you working on there?

• 1:57:38Speaker 01

  Oh, yeah. So Eureka is trying to build, I think maybe the easiest way I can describe it is we're trying to build the Starfleet Academy. I don't know if you watch Star Trek.

• 1:57:46Dwarkesh Patel?

  I haven't, but yeah.

• 1:57:47Speaker 01

  Okay, Starfleet Academy is this like elite institution for frontier technology, building spaceships and graduating cadets to be like in the pilots of these spaces. So I just imagine like an elite institution for technical knowledge and and basically a kind of school that's very up to date and very like premier institution.

• 1:58:08Dwarkesh Patel?

  A category of questions I have for you is just explaining how one teaches technical or scientific content well, because you are one of the world masters at it. And then I'm curious both about how you think about it for content you've already put out there on YouTube.

• 1:58:23Speaker 01

  Yeah.

• 1:58:24Dwarkesh Patel?

  But also, to the extent it's any different, how you think about it for Eureka?

• 1:58:27Speaker 01

  Yeah. Well, with respect to Eureka, I think like one thing that is very fascinating to me about education is like, I do think education will pretty fundamentally change with AIs on the side. And I think it has to be rewired and changed to some extent. I still think that we're pretty early. I think there's going to be a lot of people who are going to try to do the obvious things, which is like, oh, have an LLM and ask it questions and get, you know, do all the basic things that you would do via prompting right now. I think it's helpful, but it still feels to me a bit slop, like slop. I'd like to do it properly, and I think the capability is not there for what I would want. What I'd want is like an actual tutor experience. Maybe a prominent example in my mind is I was recently learning Korean, some language learning. And I went through a phase where I was learning Korean by myself on the internet. I went through a phase where I was actually

• 1:59:13Speaker 01

  part of a small class in Korea, taking Korean with a bunch of other people, which was really funny. But we had a teacher and like 10 people or so taking Korean. And then I switched to a one-on-one tutor. And I guess what was fascinating to me is I think I had a really good tutor. But I mean, just thinking through like what this tutor was doing for me and how incredible that experience was, and how high the bar is for like what I actually want to build eventually. Because, I mean, she was extremely, so she instantly from a very short conversation understood like where I am as a student, what I know and don't know. And she was able to like probe exactly like the kinds of questions or things to understand my world model. No LLM will do that for you, hundred percent, right now, not even close, right? But a tutor will do that if they're good.

• 1:59:58Speaker 01

  Once she understands, she actually like really served me all the things that I needed at my current sliver of capability. I need to be always appropriately challenged. I can't be faced with something too hard or too trivial. And a tutor is really good at serving you just the right stuff. And so basically, I felt like I was the only constraint to learning, like my own. I was the only constraint. I was always given the perfect information. I'm the only constraint. And I felt good because I'm the only impediment that exists. It's not that I can't find knowledge or that it's not properly explained or et cetera. Like, it's just my ability to memorize and so on. And this is what I want for people.

• 2:00:30Dwarkesh Patel?

  How do you automate that?

• 2:00:31Speaker 01

  So, very good question. At the current capability, you don't. But I do think that with, as, and that's why I think it's not actually the right, right time to actually build this kind of an AI tutor. I still think it's a useful product and lots of people will build it. But I still feel like the bar is so high and the capability is not there. But, I mean, even today, I would say ChatGPT needs an extremely valuable educational product. But I think for me, it was so fascinating to see how high the bar is. And when I was with her, I almost felt like, there's no way I can build this.

• 2:01:04Dwarkesh Patel?

  But you are building it, right?

• 2:01:05Speaker 01

  Anyone who's had a really good tutor is like, how are you gonna build this? So I guess I'm waiting for that capability. I do think that in a lot of ways in the industry, for example, I did some AI consulting for computer vision. A lot of my times, the value that I brought to the company was telling them not to use AI. It wasn't like I was the AI expert and they described the problem and I said, don't use AI. This was my value add. And I feel like it's in the same, in education right now, where I kind of feel like, for what I have in mind, it's not yet the time, but the time will come. But for now, I'm building something that looks maybe a bit more conventional, that has a physical and digital component and so on. But I think there's obvious, there's obvious, it's obvious how this should look like in the future.

• 2:01:46Dwarkesh Patel?

  Do they send you a willing to say, what is the thing you hope will be released this year or next year?

• 2:01:52Speaker 01

  Well, so I'm building the first course and I want to have a really, really good course. State of the art, obvious state of the art destination you go to learn AI in this case. Because that's just what I'm familiar with, so I think it's a really good first product to get to be really good. And so that's what I'm building. And NanoChat, which you briefly mentioned, is a capstone project of LLM 101N, which is a class that I'm building. So.

• 2:02:14Speaker 01

  That's a really big piece of it, but now I have to build out a lot of the intermediates. And then I have to actually like hire a small team of, you know, TAs and so on, actually like build the entire course. And maybe one more thing that I would say is like, many times when people think about education, they think about sort of like the more, what I would say is like kind of a softer component of like diffusing knowledge or like, but I actually have something very hard and technical in mind. And so in my mind, education is kind of like the very difficult, technical like process of building ramps to knowledge. So in my mind, NanoChat is a ramp to knowledge because it's a very simple, it's like the super simplified, full stack thing. If you give this artifact to someone and they like look through it, they're learning a ton of stuff. And so it's giving you a lot of what I call eureka's per second, which is like understanding per second.

• 2:02:59Speaker 01

  That's what I want, lots of eureka's per second. And so to me, this is a technical problem of how do we build these ramps to knowledge? And so I almost think of eureka as almost like a, it's not like maybe that different, maybe through some of the frontier labs or some of the work that's going to be going on, because I want to figure out how to build these ramps very efficiently so that people are never stuck. And everything is always not too hard or not too trivial. And you have just the right material to actually progress.

• 2:03:27Dwarkesh Patel?

  Yeah, so you're imagining the short term, that instead of a tutor being able to like probe your understanding, if you have enough self-awareness to be able to probe yourself, you're never going to be stuck. You can like find the right answer between talking to the TA or talking to an L and looking at the reference implementation. It sounds like automation or AI is actually not even like, so far, it's actually the big alpha here is your ability to explain AI codified in the source material of the class, right? That's like fundamentally what the course is.

• 2:04:02Speaker 01

  I mean, I think you always have to be calibrated to what capability exists in the industry. And I think a lot of people are going to pursue like, oh, just ask ChatGPT, et cetera. But I think like right now, for example, if you go to ChatGPT and you say, oh, teach me AI, there's no way, it's gonna give you some slop, right? Like when I, AI is never gonna write NanoChat right now, but NanoChat is a really useful, I think, intermediate point. So I still, I'm collaborating with AI to create all this material. So AI is still fundamentally very helpful. Earlier on, I built a CS two thirty one at Stanford, which was one of the earlier, actually, sorry, I think it was the first deep learning class at Stanford, which became very popular. And the difference in building out 231N and L101N now is quite stark.

• 2:04:44Speaker 01

  Because I feel really empowered by the LLMs as they exist right now, but I'm very much in the loop. So they're helping me build little materials. I go much faster. They're doing a lot of the boring stuff, etc. So I feel like I'm developing the course much faster and those LLM infused in it. But it's not yet at a place where I can creatively create the content. I'm still there to do that. So like, I think the trickiness is always calibrating yourself to what exists.

• 2:05:06Dwarkesh Patel?

  And so when you imagine what is available through Eureka in a couple of years. It seems like the big bottleneck is going to be finding Karpathy in field after field who can convert their understanding into these ramps, right?

• 2:05:20Speaker 01

  So I think it would change over time. So I think right now it would be hiring faculty to help work hand in hand with AI and a team of people probably to build state of the art courses. And then I think over time it can, maybe some of the TAs can actually become AIs. Because some of the TAs like, okay, you just take all the course materials. And then I think you could serve a very good like automated TA for the student when they have more basic questions or something like that, right? But I think you'll need faculty for the overall architecture of a course and making sure that it fits. And so I kind of see a progression of how this will evolve and maybe at some future point. You know, I'm not even that useful and AI is doing most of the design much better than I could. But I still think that that's going to take some time to play out.

• 2:06:01Dwarkesh Patel?

  But are you imagining that like people who have expertise in other fields are then contributing courses? Or do you feel like it's actually quite essential to the vision that you, given your understanding of how you want to teach, are the one designing the content? Like, I don't know, Sal Khan is like narrating all the videos on Khan Academy. Are you imagining something like that or?

• 2:06:22Speaker 01

  Oh, no, I will hire faculty, I think, because there are domains in which I'm not an expert. And I think that's the only way to offer the state of the art experience for the student ultimately, so.

• 2:06:33Speaker 01

  Yeah, I do expect that I would hire faculty, but I will probably stick around in AI for some time.

• 2:06:37Dwarkesh Patel?

  Yeah.

• 2:06:37Speaker 01

  But I do have something, I think, more conventional in mind for the current capability, I think, than what people would probably anticipate. And when I'm building Starfield Academy, I do probably imagine a physical institution and maybe a tier below that, a digital offering. That is not the state, not the state of the art experience you would get when someone comes in physically.

• 2:06:57Dwarkesh Patel?

  Yeah.

• 2:06:57Speaker 01

  Full time and we work through material from start to end and make sure you understand it. That's the physical offering.

• 2:07:02Dwarkesh Patel?

  Yeah.

• 2:07:03Speaker 01

  The digital offering is, yeah, a bunch of stuff on the internet and maybe some LLM assistant and it's a bit more gimmicky and a tier below, but at least it's accessible to like eight billion people, so.

• 2:07:12Dwarkesh Patel?

  Yeah, I, I, I think you're basically inventing college from first principles for the tools that are available today and then just like for, just like selecting for people who have the motivation and the interest of actually really engaging with material.

• 2:07:28Speaker 01

  Yeah, and I think there's gonna have to be a lot of, not just education, but also re-education. And I would love to help out there because I think the jobs will probably change quite a bit. And so, for example, today, a lot of people are trying to upskill in AI specifically, so I think it's a really good course to teach in this, in this respect.

• 2:07:45Speaker 01

  And, yeah, I think the motivation wise, before AGI, motivation is very simple to solve because people want to make money and this is how you make money in the industry today. I think post AGI is a lot more interesting, possibly because, yeah, if everything is automated and there's nothing to do for anyone, why would anyone go to a school, et cetera?

• 2:08:05Speaker 01

  So I think, I guess, like I often say that pre AGI, education is useful, post AGI, education is fun. And in a similar way, as people, for example, people go to gym today.

• 2:08:16Dwarkesh Patel?

  Yeah.

• 2:08:17Speaker 01

  But we don't need their physical strength to manipulate heavy objects because we have machines to do that.

• 2:08:22Dwarkesh Patel?

  Yeah.

• 2:08:23Speaker 01

  They still go to gym, why do they go to gym? Well, because it's fun, it's healthy, it's, and it's, and you look hot when you have a six pack, I don't know. I guess like, so it's, I guess what I'm saying is it's attractive for people to do that.

• 2:08:35Dwarkesh Patel?

  Right.

• 2:08:35Speaker 01

  In a certain, like, very deep psychological evolutionary sense for humanity.

• 2:08:39Dwarkesh Patel?

  Yeah.

• 2:08:40Speaker 01

  And so I kind of think that education will kind of play out in the same way, like you'll go to school, like you go to gym. And you'll, and I think that right now, I think not that many people learn because learning is hard. You bounce from material because and some people overcome that barrier, but for most people it's hard.

• 2:08:55Dwarkesh Patel?

  Yeah.

• 2:08:56Speaker 01

  But I do think that we should, it's a technical problem to solve. It's a technical problem to do what my tutor did for me when I was learning Korean. I think it's tractable and billable and someone should build it. And I think it's gonna make learning anything like trivial and desirable and people will do it for fun. Because it's trivial. If I had a tutor like that for any arbitrary piece of like knowledge, I think it's gonna be so much easier to to learn anything. And people will do it and they'll do it for the same reasons they go to gym.

• 2:09:20Dwarkesh Patel?

  I mean, that sounds different from using this, so post-AI, you're using this to um basically as entertainment or as like a self-betterment, but it sounded like you had a vision also that this education is relevant to keeping humanity in control of AI.

• 2:09:38Speaker 01

  I see.

• 2:09:39Dwarkesh Patel?

  And they sound different and I'm curious, is it like it's entertaining for some people, but then empowerment for some others? How do you think about that?

• 2:09:44Speaker 01

  I think this, um, so I do definitely feel like people will be, um, I do think like eventually it's a bit of a losing game. If that makes sense. I do think that it is in long term.

• 2:09:54Dwarkesh Patel?

  Yeah.

• 2:09:54Speaker 01

  Long term, which I think is longer than I think maybe most people in history, it's a losing game. I I do think that people can go so far and that we barely scratched the surface of much a person can can go. And that's just because people are bouncing off of material that's too easy or too hard. And they, and and I I actually kind of feel that people will be able to go much further. Like anyone speaks five languages because why not? Because it's so trivial. Anyone knows, you know, all the basic curriculum of undergrad, et cetera.

• 2:10:21Dwarkesh Patel?

  Now that I'm understanding the vision, that that's very interesting. Like, I think it actually has a perfect analog in gym culture. I don't think a hundred years ago, anybody would be like ripped. Like, nobody would have, you know, be able to like just spontaneously bench two plates or three plates or something. It's actually very common now. And you're because this idea of systematically training and lifting weights in the gym or systematically training to be able to run a marathon, which is a capability spontaneously you would not have or most humans would not have. And you're imagining similar things for learning across very many different domains, much more intensely, deeply faster.

• 2:10:56Speaker 01

  Yeah, exactly. And I kind of feel like I am betting a little bit implicitly on some of the timelessness of human nature.

• 2:11:01Dwarkesh Patel?

  Yeah.

• 2:11:01Speaker 01

  And I think um, I think it will be desirable to be to to do all these things.

• 2:11:07Dwarkesh Patel?

  Yeah.

• 2:11:09Speaker 01

  And I think people will look up to it as they have for millennia.

• 2:11:13Dwarkesh Patel?

  Yeah.

• 2:11:13Speaker 01

  Because and I think this will continue to be true. And actually, also, maybe there's some evidence of that historically. Because if you look at, for example, aristocrats, or you look at maybe ancient Greece or something like that. Whenever you had little pocket environments that were post AGI in a certain sense. I do feel like people have spent a lot of their time flourishing in a certain way.

• 2:11:29Dwarkesh Patel?

  Right.

• 2:11:29Speaker 01

  Either physically or cognitively. And so I think I feel OK about the prospects of that. And I think if this is false and I'm wrong and we end up in like, you know, Wally or Idiocracy future, then I think it's very. I don't even care if there's like Dyson spheres. This is terrible outcome.

• 2:11:48Dwarkesh Patel?

  Yeah.

• 2:11:48Speaker 01

  Like, I actually really do care about humanity. Like, everyone has to just be superhuman in a certain sense.

• 2:11:54Dwarkesh Patel?

  I guess it's still a world in which that is not enabling us to. It's like the culture world, right? Like, you're not fundamentally going to be able to like transform the trajectory of.

• 2:12:05Speaker 01

  Yeah.

• 2:12:06Dwarkesh Patel?

  Technology or.

• 2:12:08Speaker 01

  Yeah.

• 2:12:08Dwarkesh Patel?

  Influence decisions by your own labor or cognition alone. Maybe you can influence decisions because the AI is like for approval. But you're not like, it's not because I've like, I can, because I've invented something or I like come up with a new design. I'm like really influencing the future.

• 2:12:23Speaker 01

  Um, yeah, maybe. I don't actually think that. I think there will be transitionary period where we are going to be able to be in the loop and, you know, advance things if we actually understand a lot of stuff. I do think that long term that probably goes away, right? But maybe it's going to even become a sport. Like, right now, you have powerlifters who go extreme on this direction. So what is powerlifting in a cognitive era?

• 2:12:44Dwarkesh Patel?

  Yeah.

• 2:12:44Speaker 01

  Maybe it's people who are really trying to make Olympics out of knowing stuff.

• 2:12:48Dwarkesh Patel?

  Yeah.

• 2:12:48Speaker 01

  Uh, like, and if you have a perfect AI tutor, maybe you can get extremely far.

• 2:12:54Dwarkesh Patel?

  Yeah.

• 2:12:55Speaker 01

  I almost feel like we're just barely, the geniuses of today are barely touching the surface of what a human mind can do, I think.

• 2:13:01Dwarkesh Patel?

  Yeah, I love this vision. I also. It's like, I feel like the person you have like most product market fit with is like me because like my job involves having to learn different subjects every week. And I am, I am like very excited if you can.

• 2:13:19Speaker 01

  I'm similar for that matter. I mean, I, you know, a lot of people, for example, hate school and want to get out of it. I was, I was actually, I really liked school. I loved learning things, et cetera. I wanted to stay in school. I stayed all the way until PhD. And then they wouldn't let me stay longer, so I went to the industry. But I mean, basically, it's roughly speaking, I love, I love learning, even for the sake of learning. But I also love learning because it's a form of empowerment and being useful and productive.

• 2:13:41Dwarkesh Patel?

  I, I think you also made a point that was subtle, so just to spell it out. I think what's happened so far with online courses is that why haven't they already enabled us to enable every single human to know everything? And I think they're just so motivation laden because there's not obvious onramps. And it's like so easy to get stuck. And if you had instead this, this thing, basically like a really good human tutor, it would just be such an unlock from a motivation perspective.

• 2:14:13Speaker 01

  I think so.

• 2:14:13Dwarkesh Patel?

  Yeah.

• 2:14:14Speaker 01

  Because it feels bad to bounce from material. It feels bad. You get negative reward from sinking amount of time in something and this doesn't pan out. Or like being completely bored because what you're getting is too easy or too hard. So I think, yeah, I think when you actually do it properly, learning feels good.

• 2:14:29Dwarkesh Patel?

  Yeah.

• 2:14:30Speaker 01

  And I think it's a technical problem to get there. And I think for a while, it's going to be AI plus human collab. And at some point, maybe it's just AI.

• 2:14:38Dwarkesh Patel?

  Can I ask some questions about teaching well? If you had to like sort of like give advice to another educator in another field that you're curious about, to make the kinds of YouTube tutorials you've made. Maybe it may be especially interesting to talk about domains where you can't just like, you can't test somebody's technical understanding by having them code something up or something. What advice would you give them?

• 2:15:00Speaker 01

  So I think that's a pretty broad topic. I do feel like there's basically, I almost feel like there are 10, 20 tips and tricks that I kind of semi-consciously probably do, but.

• 2:15:10Speaker 01

  I guess like on a high level, I always try to, I think a lot of this comes from my physics background. I really, really did enjoy my physics background. I have a whole rant when I think how everyone should learn physics in early school education, because I think early school education is not about criminaling knowledge or memory for tasks later in the industry. It's about booting up a brain. And I think physics uniquely boots up the brain the best. Because some of the things that they get you to do in your brain during physics is extremely valuable later. The idea of building models and abstractions and understanding that there are, there's a first order of approximation that describes most of the system, but then there's a second order, third order, first order terms that may or may not be present. And the idea that you're observing like a very noisy system, but actually there's like these fundamental frequencies that you can abstract away. Like when a physicist walks

• 2:15:55Speaker 01

  into the class and they say, oh, assume there's a spherical cow and dot, dot, dot. And everyone laughs at that, but actually it's brilliant. It's brilliant thinking. That's very generalizable across the industry, because yeah, cow is, can be approximated as a sphere, I guess, in a bunch of ways. There's a really good book, for example, Scale. It's basically from a physicist talking about biology. And maybe this is also a book I would recommend reading. But you can actually get a lot of really interesting approximations and chart scaling laws of animals. And you can look at their heartbeats and things like that, and they actually line up and with the size of the animal and things like that. You can talk about an animal as a volume, and you can actually derive a lot of, you can talk about the heat dissipation of that, because your, your heat dissipation grows as the

• 2:16:36Speaker 01

  surface area, which is growing a square, but your heat creation or generation is growing as a cube. And so I just feel like physicists have all the right cognitive tools to approach problem solving in the world. So I think because of that training, I always tried to find the first order terms or the second order terms of everything. When I'm observing a system or a thing, I have a tangle of a web of ideas or knowledge in my world, in my mind. And I'm trying to find what is the, what is the thing that actually matters? What is the first order component? How can I simplify it? How can I have a simple thing that actually shows that thing, right? Shows it in action. And then I can tackle on the other terms. Maybe, maybe an example from my, from one of my repos that I think illustrates it well is called MicroGrad. I don't know if you're familiar with this, but. So micrograd is 100 lines of code that shows backpropagation.

• 2:17:19Speaker 01

  It can, you can create neural networks out of simple operations like plus and times, et cetera, Lego blocks of neural networks. And you build up a computational graph and you do a forward pass and a backward pass to get the gradients. Now, this is at the heart of all neural network learning. So micrograd is a 100 lines of pretty interpretable Python code. And it can do forward and backward arbitrary neural networks, but not efficiently. So micrograd, these 100 lines of Python are everything you need to understand how neural networks train. Everything else is just efficiency.

• 2:17:47Dwarkesh Patel?

  Yeah.

• 2:17:48Speaker 01

  Everything else is efficiency. And there's a huge amount of work to do efficiency, you know, you need your tensors, you lay them out, you stride them, you make sure your kernels are orchestrating memory movement correctly, et cetera. It's all just efficiency, roughly speaking.

• 2:17:58Dwarkesh Patel?

  Yeah.

• 2:17:58Speaker 01

  But the core intellectual sort of piece of neural network training is micrograd. 100 lines can easily understand it. You're chaining, it's recursive application of chain rule to derive the gradient, which allows you to optimize any arbitrary differential function. So it's a, I love finding these like, you know, the smaller the terms and serving them in a, on a platter and discovering them. And I feel like education is like the most intellectually interesting thing because you have a tangle of understanding and you're trying to lay it out in a way that creates a ramp, where everything only depends on the thing before it. And I find that this like, you know, untangling of knowledge is just so intellectually interesting as a cognitive task.

• 2:18:36Dwarkesh Patel?

  Yeah.

• 2:18:36Speaker 01

  And so I love doing it personally, but I just find I have fascination with trying to lay things out in a certain way. And maybe that, that helps me.

• 2:18:42Dwarkesh Patel?

  It, it also just makes a learning experience so much more motivated. You, your tutorial on the transformer begins with bigrams, literally like a lookup table from, here's the word right now. Or here's the previous word, here's the next word. And it's literally just a lookup table.

• 2:18:59Speaker 01

  Yes, the essence of it, yeah.

• 2:19:00Dwarkesh Patel?

  I mean, it's such a brilliant way, like, okay, start with a lookup table and then go to a transformer. And then each piece is motivated, why would you add that? Why would you add the next thing? You couldn't memorize this sort of attention formula, but it's like having an understanding of why this is, every single piece is relevant, what it, problem it solves.

• 2:19:15Speaker 01

  Yeah, yeah. Yeah, you're presenting the pain before you present a solution. And how clever is that? And you want to take the student through that progression. So there's a lot of like other small things like that, that I think make it, make it nice and engaging, interesting. And, and you know, always prompting the student, there's, there's a lot of small things like that that I think are, you know, important and a lot of good educators will do. Like, how would you solve this? Like, I'm not going to present a solution before you're going to guess. That would be wasteful. That would be, that's, that's a little bit of a. I don't want to swear, but like it's a it's a it's a dick move towards you to present you with the solution before I give you a shot to try to um.

• 2:19:49Dwarkesh Patel?

  Right.

• 2:19:49Speaker 01

  Uh to come up with it yourself.

• 2:19:50Dwarkesh Patel?

  Yeah, and and because because if you try to come with yourself, you you I guess you get a better understanding of like. What is the action space?

• 2:19:58Speaker 01

  Yeah.

• 2:19:59Dwarkesh Patel?

  And then what is the sort of like objective that like, why does only this action fulfill that objective, right?

• 2:20:04Speaker 01

  Yeah, well, you have a chance to like try yourself and you you have an appreciation when I give you the solution. And uh you it maximizes the amount of knowledge per new fact added.

• 2:20:12Dwarkesh Patel?

  That's right, yeah, yeah. Why do you think by default, people who are genuine experts in their field are often bad at explaining it to somebody ramping up?

• 2:20:25Speaker 01

  Well, it's the curse of knowledge and expertise.

• 2:20:27Dwarkesh Patel?

  Yeah.

• 2:20:27Speaker 01

  This is a real phenomenon, and I actually suffered from it myself as much as I try to not not suffer from it, but you take certain things for granted and you can't put yourself in the shoes of new of people who are just starting out. And this is pervasive and happens to me as well. One thing that I actually think is extremely helpful as an example, someone was trying to show me a paper in biology recently. And I just had instantly so many terrible questions. So what I did was I used chat G P T to ask the questions with the with the paper in the context window. And then uh it worked through some of the simple things. And then I actually shared the thread to the person who shared it, uh who actually like wrote that paper, like worked on that work. And I almost feel like it was like a like if they can see the dumb questions I had, it might help them explain it better in the future or something like that because um.

• 2:21:11Speaker 01

  So for example, for my material, I would love if people shared their dumb conversations with chat G P T about the stuff that I've created. Because it really helps me put myself again in the shoes of someone who's starting out.

• 2:21:21Dwarkesh Patel?

  Another trick like that that I just works astoundingly well.

• 2:21:27Dwarkesh Patel?

  If somebody writes a paper or a blog post or an announcement, it is in a hundred percent of cases true. That just the narration or the transcription of how they would explain it to you over lunch is way more, not only understandable, but actually also more accurate and scientific. In the sense that people have a bias to explain things in the most abstract, jargon-filled way possible and to clear their throat for four paragraphs before they explain the central idea. But there's something about communicating one-on-one with a person, which compels you to just say the thing.

• 2:22:09Speaker 01

  To say the thing.

• 2:22:09Dwarkesh Patel?

  Yeah.

• 2:22:10Speaker 01

  Actually, I saw that tweet. I thought it was really good. I shared it with a bunch of people, actually. I think it was really good. And I noticed this many, many times. Maybe the most prominent example is, I remember back in my PhD days doing research, et cetera. You read someone's paper, right, and you work to understand what it's doing, et cetera. And then you catch them, you're having beers at the conference later, and you ask them, so like this paper, like, so what were you doing? Like, what is the paper about? And they will just tell you these like three sentences that like perfectly capture the essence of that paper and totally give you the idea, and you didn't have to read the paper.

• 2:22:39Dwarkesh Patel?

  Yeah.

• 2:22:40Speaker 01

  And like, it's only when you're sitting at the table with a beer or something like that and like, oh yeah, the paper is just, oh, you take this idea, you take that idea, and you try this experiment, and you try out this thing. And they have a way of just putting it conversationally.

• 2:22:50Dwarkesh Patel?

  Right.

• 2:22:51Speaker 01

  And just like perfectly, like, why isn't that the abstract?

• 2:22:53Dwarkesh Patel?

  Exactly.

• 2:22:57Dwarkesh Patel?

  This is coming from the perspective of how somebody who's trying to explain an idea should formulate it better. What is your advice as a student to other students? Where if you don't have a carpathy who is doing the exposition of an idea, if you're reading a paper from somebody or reading a book, what strategies do you employ to learn material you're interested in, in fields you're not an expert in?

• 2:23:21Speaker 01

  I don't actually know that I have like unique tips and tricks, to be honest.

• 2:23:27Speaker 01

  Basically, it's a, it's kind of a painful process. But, you know, like, redraft one. I think like one thing that has always helped me quite a bit is,

• 2:23:38Speaker 01

  I had a small tweet about this, actually. So like, learning things on demand is pretty nice, learning depth-wise. I do feel like you need a bit of alternation of learning depth-wise on demand. You're trying to achieve a certain project that you're gonna get a reward from. And learning breath-wise, which is just, oh, let's do whatever one-on-one. And here's all the things you might need, which is a lot of school. There's a lot of breath-wise learning, like, oh, trust me, you'll need this later. You know, that kind of stuff. Like, okay, I trust you, I'll learn it because I guess I need it.

• 2:24:02Dwarkesh Patel?

  Yeah.

• 2:24:02Speaker 01

  But I love the kind of learning where you actually get a reward out of doing something and you're learning on demand. The other thing that I've found is extremely helpful is, um, maybe this is an aspect where education is a bit more selfless, because explaining things to people is a beautiful way to learn something more deeply. This happens to me all the time. I think it probably happens to other people too, because I realize if I don't really understand something, I can't explain it, you know? And I'm trying and I'm like, actually, actually, I don't understand this. And it's so annoying to come to terms with that. And then you can go back and make sure you understood it. And so it fills these gaps of your understanding. It forces you to come to terms with them and to reconcile them. I love to re-explain and things like that. And I think people should be doing that more as well. I think that forces you to manipulate the knowledge and make sure that you, you know

• 2:24:47Speaker 01

  what you're talking about when you're explaining it.

• 2:24:49Dwarkesh Patel?

  Oh, yeah. I think that's an excellent note to close on.

• 2:24:51Speaker 01

  Yeah.

• 2:24:51Dwarkesh Patel?

  Andre, that was great.

• 2:24:52Speaker 01

  Yeah, thank you.

• 2:24:53Dwarkesh Patel?

  Thanks.

• 2:24:54Speaker 01

  Take your time.

• 2:24:55Dwarkesh Patel?

  Hey, everybody. I hope you enjoyed that episode. If you did, the most helpful thing you can do is just share it with other people who you think might enjoy it. It's also helpful if you leave a rating or a comment on whatever platform you're listening on. If you're interested in sponsoring the podcast, you can reach out at dwarkesh.com/adandise. Otherwise, I'll see you on the next one.