Rich Sutton and Khurram Javed: Why AI Models Stop Learning, and How to Start It Again

Rich Sutton, who helped pioneer reinforcement learning and wrote the seminal AI essay The Bitter Lesson, has now cofounded Oak Lab with his former student Khurram Javed. Their goal: to build agents that continuously learn from their own experience rather than from us. Rich doesn't think he holds a radical view: "I'm not weird. The field is weird." He says all learning is continual, and the field is the one that needed a new name for it. Rich and Khurram argue synthetic data is "a big mistake." Their "big world hypothesis" is that the world is massively more complex than any agent or simulator, so approximations have to be updated continuously rather than frozen at deployment. Rich calls LLMs an unanticipated scientific breakthrough, but says they represent roughly a quarter of intelligence. He says catastrophic forgetting is "totally curable" with the ideas behind their continual backprop algorithm. Khurram explains why the frontier labs can't follow: they sit in a local minimum where a new paradigm gets worse before it gets better. Their target, five to ten years out, is a trillion-parameter mind that keeps learning, stays coherent, and runs on 20 watts. Hosted by Sonya Huang and Alfred Lin, Sequoia Capital 00:00 Introduction 02:10 An AI winter, a cancer diagnosis, and the move to Alberta 07:07 Writing "The Bitter Lesson," and what people get wrong 09:53 Are LLMs a positive or a negative example of it? 11:03 Synthetic data is "just a big mistake," and the Big World Hypothesis 18:01 AlphaGo, human priors, and why prior knowledge and learning should be friends 22:37 "Their weights never change": do LLM assistants actually learn? 26:09 Babies, squirrels, and why no animal learns by supervised learning 32:02 Rockets, imagination, and where paradigm shifts come from 36:42 The Alberta Plan and its 12 steps 38:53 Catastrophic forgetting and the cure 43:43 Oak's biggest ambition: a self-maintaining mind 47:56 Why the big labs are stuck in a local minimum 49:13 If everything goes right: LLMs, many minds, and hiring

• 0:00Speaker 00

  People think I have a radical point of view sometimes. They start questions saying how what I'm thinking is so different from everyone else. But I don't see it that way at all. I see it as like, I'm thinking the ordinary way. It's just everyone else that's thinking a bit weird.

• 0:17Speaker 00

  And I mean that, like, you know, it's just the recent times people are thinking weird. Before there was all this AI craziness,

• 0:25Speaker 00

  you talk about, you wouldn't have to say continual learning because it wouldn't make any sense to talk about learning that wasn't continual. All learning is continual. We always act and we learn. That's just the normal way of thinking. I'm not weird. The field is weird. The field, they need to call it continual learning. It's just learning.

• 0:59Speaker 01

  We are honored to have the great Rich Sutton with us here today. Rich, you invented reinforcement learning. You wrote the seminal textbook. You had the key students in the field, folks like Dave Silver. You wrote the essay, The Bitter Lesson, that I believe is the bible of the field. And you have just been one of the greats in propelling the field forward. So thank you for taking the time to join us today. Rich is joined by Khurram Javed, his co-founder and former student from the University of Alberta. The two of you have set off to found Oak Lab. I'm very excited to talk to you about that today. So for today's session, we're going to start talking about The Bitter Lesson, the state of the world as we know it today,

• 1:39Speaker 01

  whether LLMs will get us there or not. And then we're going to transition to start talking about your resource agenda and your plan for Oak. Rich, maybe take us back. I was going to start with The Bitter Lesson, but I actually want to start earlier than that. Decades ago, you decided to dedicate your career to reinforcement learning, to deep reinforcement learning in particular. And you established the University of Alberta as a bastion of that, back when I think the field was very much in its infancy. What gave you the conviction to do that?

• 2:10Speaker 00

  What else are you going to do? We're trying to figure out the mind and learning is a central part of the mind. And having a goal is a central part of the mind, central part of intelligence. Yeah, so I was just doubling down on what I was always thinking.

• 2:26Speaker 01

  Do people think you were crazy at the time?

• 2:31Speaker 00

  It was a winter. It was an AI winter.

• 2:34Speaker 01

  What year was this?

• 2:35Speaker 00

  It was in 2003.

• 2:37Speaker 01

  Okay.

• 2:38Speaker 00

  And it's kind of crazy, actually, the truth, because I was like really sick. I was dying. I was actually dying of cancer in 2003. And but I wasn't quite dead, you know, I've been trying for a number of years. And I wasn't dead, I was in another, another remission. And so, so I said, well, I'm not dying. I haven't succeeded in dying. So I might as well, you know, it's going on long enough, I might as well just try to get another job. And so I, so I went to Alberta and started teaching there. And then in the end, I didn't die. It's kind of amazing. It's because, you know, it's like that. I'm joking about it now, but it was quite serious.

• 3:19Speaker 00

  And it's an even more poignant question. Why, why did I continue to work on this research stuff when I was, you know, I only had a few months? I would always keep reminded what, I think it's Benjamin Franklin is supposed to have said, that, you know, if you ever wonder why someone is doing something, it's almost always one of two things. It's either habit or vanity. Okay, so I think it was, I think that's probably true. Maybe it was a habit to just kept doing what I always was doing. Or maybe it was vanity. I don't know. I think it was more like habit because I was, I was dying.

• 3:56Speaker 01

  Wow, divine intervention.

• 3:57Speaker 00

  Yeah, it's always been easy for me to keep be very determined. And I'm going to go even longer on this answer.

• 4:06Speaker 01

  Please do.

• 4:08Speaker 00

  People think I have a radical point of view sometimes. They, they start questions saying how, how what I'm thinking is so different from everyone else. But I don't see it that way at all. I see it as like, I'm thinking the ordinary way. It's just everyone else that's thinking a bit weird.

• 4:26Speaker 00

  And I mean that, like, you know, it's just the recent times people are thinking weird. If you look back, what would people thought about the mind for, you know, even just a decade, you'll find the kind of thoughts that, you know, learning is important, you've got to have a goal. And, you know, perception is important. We have, we are, we are low level, we are low level beings. We are generating actions and receiving data at a fast speed, and yet we have to think at higher levels. And, you know, go back a few, before there was all this AI craziness,

• 5:01Speaker 00

  you talk about, you wouldn't have to say continual learning because it wouldn't make any sense to talk about learning that wasn't continual. All learning is continual. You know, it's not a special phase. We always act and we learn. That's just the normal way of thinking. I'm not weird. The field is weird. The field, they need to call it continual learning. It's just learning.

• 5:22Speaker 01

  I'm not weird. Everybody else is. That's a good, good motto to live by.

• 5:26Speaker 02

  We're going to have to send out a next post about that. We're very happy that you, you lived on, the field is happy that you lived on, and thank you for pushing the frontier of AI. Thank you for pushing the frontier of AI. I'm sure really happy and thank you for all of that and push. And you've like been able to sort of educate a lot of students who push the frontier as well. How'd you pick them? How'd you, over the last 20, 30 years?

• 5:51Speaker 00

  Oh, you are giving me opportunities to be, to be humble. Like, I like to be humble and point out how all these great decisions are, are just happen. And it's that, that's the way I feel about students. I don't feel that I choose them very well. I've just, sometimes I'm lucky, sometimes unlucky. I don't feel I'm particularly good at picking my students.

• 6:14Speaker 00

  I'm looking at Khurram now. I think sometimes you end up with really great ones. David picked, David Silver picked me.

• 6:20Speaker 03

  Yeah.

• 6:20Speaker 00

  How is it that I got you, Khurram?

• 6:23Speaker 03

  Yeah, that was also, so I finished my masters, not with you, and I was planning to join industry. And then we were collaborating on a project, which also just started organically. Like, there was something I worked on that Rich was in a meeting, then they mentioned that I worked on it. So I got pulled into it. We started collaborating. It went really well. Like, I felt so happy with that collaboration. Rich also felt really good about it. And then six months down the road, we had made some progress and it just made sense to convert that into a thesis proposal. So no point that I apply, at no point that I ask, should you be my PhD advisor? We worked together. Then we decided this would be a pretty good thesis. And then, then after that, I applied for PhD.

• 7:08Speaker 01

  Life works in unexpected ways. Take us to 2019. You wrote The Bitter Lesson, which has become the, the modern tome. 2019 was a funny time to be writing that piece because ImageNet was 2009, AlphaGo was 2015. What caused you in 2019 to reflect and to write that? Because it was before the current kind of scaling paradigm around large language models had taken off. But it was after deep learning had really proven itself.

• 7:36Speaker 00

  Well, it was a long time coming. You know, as The Bitter Lesson expresses, it's something that you can observe for a long time, for many decades. And it's definitely at least as much due to the round of symbolic AI, which I lived through. It's all about not getting distracted by trying to put in your human knowledge and just paying attention to what the problem needs and how you can scale with computation. I know I, I wrote versions of it at least a year before, and I gave talks. I gave a talk a year before.

• 8:12Speaker 00

  And it wasn't a particular response to the moment. It was a particular response to my, my long experience. Different people trying to think in different ways about how you can make smart systems.

• 8:29Speaker 01

  What is the essence of The Bitter Lesson?

• 8:32Speaker 00

  Essence of The Bitter Lesson.

• 8:33Speaker 01

  You know, it may be the phrase that I hear used the most in my meetings these days is bitter lesson pill, is it not bitter lesson pill?

• 8:40Speaker 00

  Yeah.

• 8:40Speaker 01

  I would imagine, given the popularity of the phrase, it's probably been tortured and misused in different ways that you didn't originally intend it. So what do you think, what is the essence of it? And where do you think people go wrong in their, in their attempt to understand it?

• 8:53Speaker 00

  Yeah, you're making me think about X now. And I, I recently made a post where I tried to do the, the bitter lesson in 26 words. It goes something like, don't be distracted by human knowledge as AI traditionally has been many times. Instead, focus on learning methods that will scale with computation, you know, like search and like learning. So it's really all about focusing on algorithms and improvements. It's not, it's not saying you don't need fancy algorithms. You need fancy algorithms.

• 9:35Speaker 00

  But you want fancy algorithms that will scale with, scale with computation.

• 9:40Speaker 02

  Rather than scaling with data?

• 9:41Speaker 00

  Rather than scaling with human input.

• 9:45Speaker 00

  Yeah, and then the question, if I can anticipate,

• 9:51Speaker 00

  yeah, what about large language models?

• 9:52Speaker 01

  Yeah.

• 9:53Speaker 00

  Are they?

• 9:54Speaker 02

  Consistent or inconsistent with your essay?

• 9:57Speaker 00

  And I thought about this, and I think there's another X post about it. But the conclusion is that it's both a positive example and a negative example of the bitter lesson. First, large language models enabled enormous scaling with computation. And you could just drink in the internet and scale so much. So it was a way of getting a much more capable system just by methods of scale. Then, after that, as you go on further, it eventually gets limited by that information. The internet is finite, and it's hard to get more examples.

• 10:39Speaker 00

  And the world is big, and the world is massively bigger than everything we stored on the internet. And so in the end, it seems like it could be, I guess that would be a positive example of when, you know, we relied too much on human knowledge, and it eventually holds us back.

• 11:00Speaker 01

  Can I just push on this a little bit?

• 11:02Speaker 00

  Yeah.

• 11:03Speaker 01

  It seems like a lot of what the foundation model labs are working on right now is synthetic data generation in order to kind of get us beyond the fossil fuel that is the existing human internet. Is synthetic data generation kind of as part of this LLM scaling paradigm? Is that a general method that leverages computation?

• 11:24Speaker 00

  No, that's just a big mistake.

• 11:27Speaker 01

  Why?

• 11:28Speaker 00

  Well, it's such a big, maybe it's the next big lesson. It's been floating around Alberta for five or 10 years.

• 11:40Speaker 01

  Okay.

• 11:40Speaker 00

  And we call it the big world perspective or the big world hypothesis. Khurram, who eventually wrote it up as a paper, there's a little paper called the big world hypothesis.

• 11:50Speaker 03

  The big world is that the world is infinitely big. There are infinitely many things to learn. And you can have people generating these synthetic data sets, but there would always be more things to learn. And because of that, if you could just learn from experience, you could remove the humans from the loop, then you would have systems that can do everything. Because, you know, the world is big, there are many tasks that we want them to do, and they would be able to do anything by learning from their own experience. Going back to the synthetic data question too, who decides what's a good synthetic data and what's the bad synthetic data? Because I can write a program that can output a lot of synthetic data, which would hurt programs. Right now, I would say humans decide, and that's the bottleneck where,

• 12:35Speaker 03

  okay, you can have humans deciding how to generate these data sets, but you need human experts who know what's a good data set and what's a bad data set for that approach to scale. So it is bottlenecked by humans.

• 12:45Speaker 01

  Doesn't my loss curve decide? Like, how much better did I get with this data set versus that data set?

• 12:50Speaker 03

  Right, but if all the engineers, OpenAI, Anthropic, or all the big Neo labs, that engineers went on vacation, who would generate the synthetic data? That's the question. It doesn't come from agents' experience. It's not something that the agent is generating itself. Some human has to decide what is the right synthetic data to generate, and that requires human expertise. So for example, if you want a system to do something very challenging from a physics point of view, maybe you want a drone that flies with echolocation, like a bat, for example. What's the right synthetic data for that? I think you would need to hire domain experts to go figure out what is the right data and generate it, and then maybe you would be able to learn from that. But the domain expert has to exist first.

• 13:35Speaker 03

  So we are bottlenecked by human expertise at that point.

• 13:37Speaker 01

  But you can have infinite synthetic worlds. The existing world is finite.

• 13:42Speaker 03

  But let's go back to the echolocation thing, right? That's what I want. I want a drone that can localize itself and move with echolocation. That's my goal. The robot, that's a robot, is generating its own experience. So it could totally learn from its own experience, but it wouldn't be able to. It doesn't matter how much synthetic data you generate. It doesn't matter if you generate synthetic data that captures 50 different universes. It would not allow you to do that task without humans figuring it out first.

• 14:12Speaker 00

  At first, just say, the synthetic data is wrong. I mean, it won't be correct. It'll be a synthetic world, it won't be the real world, and it will matter. The world is incredibly complex. If you write a little program, because this can be a little program that will generate the synthetic data, it'll be a very, it'll be a small world.

• 14:36Speaker 00

  So, for example, what's important to me is what's going on in your mind right now. Okay, and you're saying, why don't I get some synthetic data to tell me what's going on in other people's minds? No, there's no way we can have synthetic data for other people's minds, and other people's minds matter to us. You know, I talked to you guys about investing today, so I care what's going on in your minds. And how could I get synthetic data on such a thing?

• 15:05Speaker 00

  Really, you can't even get synthetic data on anything. You can't get synthetic data on how the drone is going to interact with its environment in the physical world, and the friction, and where in the motors of this robot. The world is infinitely complex, and any simulation of it is like microscopic. The big world hypothesis, let's say what it is, is that the world is massively more complex than your mind, than any agent. And this is obvious because the world contains many other agents. So because the world is massively complex,

• 15:46Speaker 00

  there's no way you can do anything like, anything that might claim to be optimal or perfect. You're going to be imperfect, and you have to have approximations. And those approximations will be severe. And so because of that, that is ultimately the reason why we have to continue learning. If you want to think of it as a reason, we have to continue learning because we'll encounter some particular part of this immense world, and we'll have to learn an approximation that's tuned to the part of the world we're in, not to all the other parts that we're not in.

• 16:19Speaker 01

  Yeah. I'm going to push on this one more time. And sorry, I'm being argumentative for the sake of being argumentative, but I'm trying to understand. My understanding is that the newest cohort of self-driving car companies, many of them were primarily trained in sim, and then they, you know, have to do some sort of post-training, I guess, to make sure they work in the real world, but that it's been a very effective pipeline.

• 16:43Speaker 03

  Yeah, so I think the important question to ask here is, how many engineers were involved in building that simulation? And are we ready to say that the only problem worth solving are those where we can hire a large team of engineers to first make a simulation? And I'm sure they had to do multiple iterations where they made the simulation, they learned in it, they realized there was a sim to real gap that was not acceptable. Then they fixed it. So there is this human in the loop fixing the simulation. Like, they're getting feedback from the real world, humans, and then they're fixing the simulation. Why can't we just remove the human and let the agent do it itself?

• 17:18Speaker 00

  And then when it actually drives, again, something unexpected will happen.

• 17:22Speaker 01

  And that's when you actually want to learn from experience. So your point is there's just so much more data that's going to come from experience than there possibly can be from humans curating, creating data.

• 17:34Speaker 03

  Yeah, and I think there is a, there is obviously value in learning from simulation. And there is a way of doing it. The agent can learn a model from its own experience. And when the agent learns it, it's very, it's much better because if the model is incorrect, it can fix it by continuing learning. If the humans are making the simulator, then the model only gets updated when the humans figure out that something is wrong. So, yes, planning is important. The agents should learn from simulators, but simulators they make themselves.

• 18:01Speaker 01

  Okay, I want to move to another part of the bitter lesson, removing human knowledge. From your essay, quote, "Seeking an improvement that makes a difference in the shorter term, researchers seek to leverage their human knowledge of the domain, but the only thing that matters in the long run is the leveraging of computation." And if I, you know, your former student, Dave, with AlphaGo and AlphaZero, for me, that was an example of a triumph of removing human priors. Did that result surprise you? I guess why or why not?

• 18:31Speaker 00

  Of course, it made me very happy. It made me, you know, feel vindicated.

• 18:36Speaker 00

  You know, it could have gone either way. It wasn't that, because prior knowledge can help. You know, there's nothing wrong with prior knowledge. You know, and I say this right at the very beginning of the bitter lesson. I say, there's no reason why there has to be a conflict between prior knowledge and then learning knowledge. You know, you can put some prior in there and then start learning. There's no reason in principle why these have to be opposed. In fact, they're all about knowledge. You know, life is gaining knowledge and having knowledge. And why are these, how somehow, you know, nature and nurture became enemies? But really, you know, prior learning is what you already, and then you learn more, and it's, they should be friends.

• 19:20Speaker 00

  But as I say at the beginning of the bitter lesson, in practice, they have been enemies. In practice, people who had an affection for existing human knowledge ended up, you know, wanting that to win. And so they wanted to minimize or dismiss learning. And so now, I'm sure your sense of me is that I'm someone who loves learning and wants to dismiss prior knowledge.

• 19:47Speaker 00

  But you know, I'm really someone who's interested in the mind. The mind is you have prior knowledge and then you get more. And then, once you've gotten more, then that becomes your prior knowledge as you get more and more and more. And this, these two things work together. I end up appearing to be someone who's interested in learning primarily because all the rest of the world is talking about, all you need is enough knowledge. You don't need to learn. You know, large language models are, oh, we're going to put all this knowledge into the system. And the large language model will not learn when it runs. You know, it's talking to people, it's interacting. It is absolutely, the weights never change. So, you know, I am not the weird one.

• 20:32Speaker 00

  It's you guys that are the weird one that think that that's possible, that you could possibly, you know, they claim they can take a PhD level experience and expertise out of something that doesn't learn at all anymore. You know, so, you know, I'm not the weird one.

• 20:47Speaker 02

  So your recommendations, let the algorithms run for a much, much longer period of time before feeding.

• 20:52Speaker 00

  Continually learn.

• 20:54Speaker 02

  Before you feed it data, prior data.

• 20:58Speaker 00

  So.

• 20:59Speaker 02

  Drip or drip the prior data along the way, prior knowledge.

• 21:04Speaker 00

  So both are important, but in the long run, you've got to gain and structure the gaining of new knowledge. And that's, that's what all that matters in the long run. And as you are doing this, yeah, there'll be some that you got previously.

• 21:23Speaker 00

  Like, how it work if we, you know, look into the future, when we have intelligent robots, will we have them all learn from scratch? Or will we like copy them and ask them to keep learning from wherever they are?

• 21:39Speaker 00

  I mean, they'll be digital and it'll be easy to copy them. And so instead of having like this huge thing where we're spending zillions of dollars to retrain them from the internet, we'll just copy the agent and keep learning from there. And so in some sense, the prior knowledge will be, should be dismissed because you're just going to copy it from the previous robot.

• 22:02Speaker 02

  So why don't you just describe for us what you think a machine or a computer that learns from experience looks like?

• 22:09Speaker 00

  Well, it could look like a robot. It could, but it also could live entirely on the internet. You could like, for example, routing of packets through the internet and do that in a way that's sensitive to experience and becomes better over time. Or you can interact, be the user interface that's interacting with people, like on your phone or on your computer, and it becomes better over time. You know, like an intelligent assistant, you know, has to become better over time, has to know what you want.

• 22:37Speaker 01

  Would your contention be that the current paradigm of, you know, the popular LLM-based assistance, would your contention be that these are not experiential learners or continual learners? And if so, what is the fundamental gap?

• 22:52Speaker 00

  Are you serious?

• 22:55Speaker 00

  I mean, obviously.

• 22:56Speaker 01

  They have, they learn memories about me. They're, you know, they're doing some in-context learning.

• 23:03Speaker 00

  Their weights never change.

• 23:05Speaker 01

  And by the way, is a small number of the weights changing sufficient? Or do you need all the weights to be changing?

• 23:10Speaker 00

  Well, so I think of all the structuring and generation of new concepts that went into creating the large language models. All that is the weight learning.

• 23:22Speaker 00

  And you want to continue, be able to continue doing that. You don't want that to happen just once.

• 23:27Speaker 02

  Is another way of saying is we do too much pre-training and post-training before we launch the models? They don't learn after that.

• 23:35Speaker 03

  The only point that the biggest agreement is we don't let them learn after that.

• 23:38Speaker 02

  Yeah, we don't let them learn.

• 23:39Speaker 03

  We can do as much pre-training as we want, that's okay. Post-training is fine. But then when I'm using the model, I cannot, it stops learning. You can give it more context, you can change the state of the model by giving it more context. And so it had already learned that if the state is different, if the state says something new, then it will use that to make the next prediction. But the model is on learning.

• 24:01Speaker 01

  Cursors, Tab, other complete model. It is, you know, it does get updated based on.

• 24:07Speaker 02

  Those models, those weights change.

• 24:08Speaker 03

  Those weights change. Those are two examples, like Cursors, Tab, and I think the Composer, they were also updating.

• 24:13Speaker 01

  Yep.

• 24:13Speaker 03

  Those are two examples of continual learning. But it can be much better. So the way they do it, as far as I understand, is a lot of people are using Tab. They collect all this data, so coming from millions of users or thousands of users. And then they do one update of the policy from this batch data. So this could work, but imagine I want to teach this model something specific. I don't want to fight with 100,000 other people about what they want to create, teach their models. I want to teach my model something very specific and I want to do it to my version of the model. Like, I don't care about the shared knowledge that the model has coming from other people. And so it's a very inefficient way of doing it.

• 24:51Speaker 01

  It seems like the way that this is currently done is that there's fundamental skills maybe that are learned in the weights that are common to everybody. And then there's personalization that happens in the form of context, right?

• 25:04Speaker 03

  Yeah.

• 25:05Speaker 01

  Is that not the right mental model for how learning should work? Like, should all the contexts live in the weights themselves?

• 25:11Speaker 03

  So context can be in the state too, could be both. But you still need to be able to update the weights. So if I give you an example, some really good use studies are with human disabilities. When human go through something that changes their mind or some sensors, you can see them adapt. So for example, we have proprioception, we have internal sensors that tell us where the, how the body is positioned, and we use this for walking. There are cases where people lose this ability completely, and then they can't walk at all because that is literally the foundation of their walking policies. It is ingrained in the brain. But then over the course of two, three years, they can learn to walk again by looking at their feet. So visual feedback through that.

• 25:52Speaker 03

  So brain is insanely plastic in the sense that it can learn a lot of things, something that has been true for 20 years. When it stops being true, it can go and update that and get rid of that. And that is the capability, I think, that's extremely useful, we would want in our systems.

• 26:09Speaker 01

  What is there for us to learn from how human babies or animals learn? And how much inspiration do you take from that?

• 26:16Speaker 00

  Well, we take a lot of inspiration. We don't take it as a requirement that the AI has to behave like the natural system, like babies or people or animals. But it's a source of inspiration, inspiration but not constraint from animal.

• 26:34Speaker 01

  Be consistent with the bitter lesson.

• 26:39Speaker 01

  Where do you think we should most seek to draw inspiration from the way that biological learning works? That is not present in today's systems.

• 26:48Speaker 00

  I feel like I'm just giving opinions now, but they're just obvious opinions. So I think it's apparent that no animal learns by supervised learning.

• 27:00Speaker 00

  Because we don't get examples of how our muscles should twitch. And that's our output.

• 27:07Speaker 01

  But all of school is supervised learning.

• 27:09Speaker 00

  I know, absolutely not. But even if it was, school is like a tiny fraction of what we learn. Like, we learn to see and we learn to walk and we learn how the world works. But yeah, even in school, you know, no one tells us how we should twitch our muscles.

• 27:31Speaker 01

  The knowledge skills I acquire were from supervised learning in school.

• 27:35Speaker 00

  I don't want to say that learning from others, transmission from others is not important. It's like extremely important and language is extremely important. But what are we, what are we missing? You know, there is, there is no supervised learning. There's no targets that are given to us. You know, you hear the right answer is, you know, where is, what's the capital of France? And we know the answer is Paris. Okay, but no one tells me how I should pronounce Paris. You say the answer is Paris and I listen to you and I hear your words. And, you know, I will make some other muscle motions to produce the answer, Paris.

• 28:20Speaker 00

  It's not literally supervised learning.

• 28:24Speaker 00

  Anyway, yeah, so I think it's really true. I mean, well, anyway, the first thing is the school is irrelevant. Like, you know, squirrels don't go to school and learn that. Animals don't learn that way. It's, and school is a very special thing that even we didn't have up until, you know, I don't know, a few hundred years ago.

• 28:46Speaker 01

  But isn't it something we've evolved to?

• 28:47Speaker 00

  It's not part of the essence of intelligence. And it's a distraction to think of that as your primary example of learning is this thing which we didn't do.

• 28:58Speaker 00

  As animals.

• 28:59Speaker 02

  I wish he had been around to tell my parents that. Forced me to have good go to school and

• 29:06Speaker 02

  deal with all the structure.

• 29:08Speaker 01

  The thing is like squirrels are wonderful at jumping off trees, but squirrels can't prove math theorems. And if I want to learn how to prove a math theorem, I go to school.

• 29:14Speaker 00

  Yeah, uh, they also don't have

• 29:19Speaker 00

  DVDs and and iPods, you know, there are a lot of things they can do things that we can't do. But math theorems.

• 29:31Speaker 00

  Yeah, and they don't play chess. You know, it's sort of like more of X paradox. There are there are these advanced things that we think of as really intelligent. But they're sort of easier for computers to do as opposed to all these regular things that are hard, like moving and seeing with attention and everything. I think supervised learning is a is a good thing. You know, just mentions, I like to think look for obvious things. No one tells us how to twitch our muscles by giving us examples because they couldn't possibly because we have had to twitch our muscles. We've had to figure that out.

• 30:06Speaker 01

  Yeah.

• 30:06Speaker 03

  And their answer would be wrong, right? So if I move my mouth and my tongue and my vocal cords exactly the same way that Rich does to pronounce Paris, I'm sure a very different sound would come out. So in some sense, which or no one knows the right way of producing a sound with my body. Only I know that.

• 30:24Speaker 01

  Yeah. It seems to me that many of the most, I guess, the most raw, like sensory motor capabilities, especially related to movement in the physical world. I agree with you that that seems something that's inherently learned from experience. It seems to me, though, that there are higher levels of abstraction that bring us closer to, you know, what makes humans great. And much of that doesn't live in this low level of sensory motor learning. Does your world model, I guess, span sensory motor learning all the way up?

• 30:56Speaker 00

  Yeah, that's the ambition, absolutely. And squirrels, by the way, can do some enormously abstract things.

• 31:03Speaker 01

  What's the coolest thing a squirrel can do?

• 31:05Speaker 00

  Well, it can always get into your bird feeder. No matter what obstacles you put in the way, you know, it can find new ways to jump and climb and and do lots of things.

• 31:16Speaker 02

  You calculate trajectories pretty well. Animals are pretty good at understanding the physical world without the mental calculations that we think we are doing when we think about launching ourselves into space.

• 31:29Speaker 03

  Breaking a fall, they can do it in real time, the right way to prevent injuries.

• 31:33Speaker 01

  Okay, fair enough.

• 31:34Speaker 00

  I think it's just a question of degree between. And I like to think that animals, other animals are are very close to humans. I think it's hubristic to try to emphasize what we do differently, you know, how we're different from animals. It's better to see the commonalities. And I think we are just a question of degree. It's degree and of course, society and culture give us big advantages. Language gives us big advantages.

• 32:00Speaker 02

  Can I just push on some of this?

• 32:02Speaker 00

  Yeah, good.

• 32:02Speaker 02

  Because I want to back up Sonia. So I believe animals and children learn from experience and do incredible things learning from experience. And when my son was two or three or four, I'm like, wow, this is really interesting that they're my son can learn these things without nobody really teaching them how to do these things. But at the same time, what Sonia's saying is like, what makes human uniquely human? To be able to go to outer space, build a rocket. Those are not things that are learned 100% from experience because before you launch the rocket, you actually have to abstract thinking through it in a way that is not

• 32:46Speaker 02

  learned from quote unquote, experience. Because you don't know if it's going to work or not. You have to imagine it. How do you we teach a machine to imagine things that were not available before? That's probably the thing that we're trying to like push on because that we're not quite understanding that.

• 33:04Speaker 00

  We're absolutely going to agree with you there. You have to be able to plan. You have to be able to imagine.

• 33:09Speaker 02

  Yeah.

• 33:10Speaker 03

  Would you say that humans a thousand years ago, before they had done all most of the things that we're talking about, were they as intelligent? If, for example, someone from that era was exposed to this new culture, would they be able to get the same skills and and start doing useful things?

• 33:27Speaker 02

  Even over the last 10,000 years, I don't think the human brain has evolved that much.

• 33:31Speaker 00

  Fundamentally the same machine.

• 33:33Speaker 02

  Fundamentally the same machine, but we've built up 10,000 years of knowledge.

• 33:37Speaker 03

  Yes.

• 33:37Speaker 02

  And I get to learn 10,000 years of knowledge by going to school through supervised learning.

• 33:42Speaker 00

  Right.

• 33:42Speaker 02

  And I get all that much, much faster than trying to learn through experience.

• 33:47Speaker 03

  So right. So I think you're like totally right. So we would want our systems to learn from experience and part of their experience would be getting exposed to our culture and then learning from about our culture. They should learn from that. That's all good. But let's talk about when someone goes and does a paradigm shifting things. So everyone gives the example of Einstein, but I think there are many examples. Learning is that too, like looking at learning thing versus programming things. So when these paradigm shifts happen, I would say it's a human who has accumulated all this knowledge. And then from their experience, they're building new abstraction, they're planning with them, and then they're discovering new knowledge. And that skill of of coming up with new abstraction and then learning what models

• 34:28Speaker 03

  and planning with them, that problem is that skill is totally missing in our current systems. And you can expose this at the edge of human knowledge, but you can also study this problem at the sensory motor stream level.

• 34:41Speaker 00

  So we're not arguing with the with the principle. We need to form abstractions so we can reason at a high level. You guys are coming close to doing that thing that I said we should never do, which is argue is prior knowledge important or gaining knowledge important? You know, that's what you guys just said. You said it's you're still going to have to learn things. And you're saying, oh, I can get things from my culture and from prior knowledge. But these should not fight for each other.

• 35:10Speaker 02

  On the exact thing around paradigm shifts, how do we create a machine that understands when to shift the paradigm?

• 35:16Speaker 03

  Yeah, I think through its experience, right? So it would have to, through its own experience, it can't rely on human knowledge because we're assuming the humans see one paradigm and we want a different way of looking at things. And so through its experience, it has to find something that is better. Maybe it's generalizes better and makes better predictions. Maybe it's better in some other ways, but it has to be through its own experience.

• 35:40Speaker 00

  The big challenge that we don't see in our field, the ability we don't see in our field yet is the ability to learn a model and then plan with a model. We can do the the math things and we can do AlphaGo because the games, we know the model, we know how the moves work. And in math, we know what the operators are. You know, we know lean will take us from one state of knowledge to the state of the proof to the next state. But if we have to learn the models, there are no I'm going to say it, it's probably maybe a weird example, but a counter example, but I can see there's no instances of learning the model and then planning with the model in our field.

• 36:23Speaker 03

  At least not with

• 36:26Speaker 03

  like self-discovered abstractions. So there are people who say, I'm just going to learn a model of what happens in the next second or next millisecond, but that's not how our models work. Our models are more abstract. Our models are

• 36:41Speaker 03

  quite different.

• 36:42Speaker 01

  So one of the things I like about what you're doing here is you're not just sitting around pontificating or lamenting the state of the world as it is. You're very action oriented. That's why you started a company. So let's let's start talking about that a bit. In 2022, Rich, you laid out a very specific 12-point plan, the Alberta Plan for AI Research. Maybe tell us about that.

• 37:04Speaker 00

  So the Alberta Plan came about because we did have general ideas, but we also needed to convert them into smaller chunks. And so the 12 steps are the attempt to crystallize particular chunks. There's a very important early step, step two, which is continual deep learning. And we think that one is like almost the most important because it unlocks everything else. If you could do continual deep learning, you could then continually update your model of the world. And then if you knew how to do the abstraction, right? And like the second half of the steps are all about how to get the abstractions right.

• 37:47Speaker 00

  So and not only my abstractions are right, what I mean, but I don't mean get the right abstractions because no one can say what the right abstractions are. That depends on the world that you're in. Your agent would have to learn the correct abstractions for whatever world it's in. And so, you know, if you, maybe those are the two key things. You have to find the right abstractions and then you have to do, able to do continual deep learning.

• 38:10Speaker 03

  I think that a lot of the people in the field realize that we need models, we need to plan with them, but the abstractions tell us what the model should be conditioned on. So what should you, what should the model predict? What are you going to do? And then something is going to happen. And more importantly, where would that come from? So I really like the example of elite athletes. If you ask elite athletes about how they do certain things, they would have weird niche terminologies for doing very specific things. They were like, you know, I do this thing. And they would have a name for it if they communicate. Sometimes they don't even have a name for it if they're just doing it alone. So how did they come up with those abstractions? That's in some sense, a crucial thing that's missing that the later half of the plan answers.

• 38:53Speaker 01

  Can we talk about the continual deep learning part? Is it an algorithmic gap that exists today? Or is it a, just a practical deployment, infrastructure, data privacy gap? Because if I wanted to do, call it naive updating of weights based on user interaction, I can do that today. Right? And so what in your opinion is the biggest thing that we're missing to kind of get to continual deep learning?

• 39:18Speaker 03

  Yeah, so it's absolutely an algorithmic gap. You, you can do the naive thing, but then you'll see all sorts of problems. For example, if you say, I'm going to take one sample and then I'm going to update my whole model with that one sample, you will run into this problem that now all of the previous knowledge in the model, it's impacted negatively. And the way currently we, we get around this is exactly what Cursor does. They don't use one example. They use a large batch coming from a lot of users. So in use cases where you can have that, you can do continual learning. But most use cases, you don't have that. Most use cases, you have a single stream of data. And then if you apply it to the naive thing, it just completely destroys your prior knowledge in a very destructive way.

• 40:00Speaker 00

  Catastrophic forgetting.

• 40:02Speaker 01

  Yeah.

• 40:02Speaker 03

  That is so, yeah.

• 40:04Speaker 00

  But it's totally curable. Have the right algorithm.

• 40:07Speaker 01

  What's the cure?

• 40:07Speaker 02

  Yeah, exactly.

• 40:09Speaker 00

  What is it? Well, you know, first you need to do what we call step size optimization. And it means every weight in your network has to have a separate step size. So some will move fast, some will move slow. And you will, we, you will have to meta learn these step sizes for each weight. Most of your network will be have, have weights that have tiny step sizes. So then when you train on a new example, they don't get destroyed. It happens just to the right places. And then secondly, you have to use some form of generate and test, which is in feature space. So you come up with new features or new units and, and

• 40:52Speaker 00

  without following gradients, because gradients is a very slow process. You only move in a direction if you know it's the helpful one. And that's always going to be very slow and doesn't give you a path to grow more and more complex and, and to have sustained learning. You need to have something that just proposes a bunch of new units and, and, and then goes from there. I guess so. There is a specific thing I can say that make it at least concrete, which is to say, we have this algorithm called continual backprop. We used, published in, in Nature a couple of years ago, and it is exactly like backprop, but every, but you also plant new seeds of units that are newly initialized with random

• 41:36Speaker 00

  weights. Backprop only has random weights at the beginning of time. And then as you go on, all that randomness, all that variety from the randomness gets used up. And with continual backprop, you keep injecting a bit of randomness, a bit of generate and test, a bit of generate, and then the, the operation of backprop is the tester. So you need, you need that. And, and if you put those together really well, I think you'll have a new generation of massively superior continual deep learning. And that's what we hope to do in the next couple of years.

• 42:10Speaker 01

  Wonderful. Do you think that these algorithms can be applied to the current state of affairs with people scaling L L M's and trying to get them to do continual learning without catastrophic forgetting?

• 42:22Speaker 03

  Yeah, absolutely. I think it's, so I don't think that you could take an existing model and say, I'm going to just start updating it with these algorithms, because these algorithms meta learn how to learn. So really you have to say, I'm going to learn from scratch. So let's say learn a new foundation model, but I'm going to learn with this, these new algorithms. These new algorithms, in addition to learning the knowledge, they're also going to learn how to learn future things. So they're learning two things at the same time. And then, then I think you would be able to learn new things without catastrophic forgetting.

• 42:57Speaker 02

  Is that the most radical thing and that you're trying to do in your company in terms of from the current state of affairs to try to do these two things at the same time?

• 43:06Speaker 00

  Most radical thing is, I think that's.

• 43:09Speaker 01

  This goes back to, I'm not crazy, everyone else is crazy.

• 43:12Speaker 02

  Yeah.

• 43:13Speaker 03

  That's perhaps not totally radical. There was a point in like 2016 to 2018 where a lot of people were exploring these ideas quite a bit. They were doing it in a much more limited setting. So they would say, we have a distribution of problems. And then in this specific case, we'll do it. Whereas we want to do it from a single stream of experience. So our method should be more generally applicable. So I think many people have explored this. No one has explored this in the general setting where the resulting algorithm would be applicable everywhere.

• 43:43Speaker 02

  So what would be the most radical thing that your company, your new company is trying to do that other people are not doing?

• 43:49Speaker 00

  What's the most ambitious thing? Remember, I don't think I'm weird. So I don't want to say it's radical.

• 43:55Speaker 02

  What's the most ambitious?

• 43:56Speaker 00

  What's ambitious thing, I think, is to try to have the full spectrum of knowledge, both about the tiny things and about the big things, you know, like thinking about how you take an airplane from one city to another. That's a, a very big, you know, it's, it's more, it's like your, your space flight example, but it's just kind of more common sense to think about. Cause we all, many of us take airplanes and all of us use abstractions in all kinds of our life. And even, even the squirrels use abstractions. So to have that spectrum of, of knowledge from the small to the big and to treat it in a uniform

• 44:37Speaker 00

  way and to be able to have it self maintaining. You know, the big question is always, you have your knowledge base system and what keeps the knowledge in it correct? Well, what keeps the knowledge correct in a large language model is, well, people did a lot of post-training and they, they, they made it sure it was correct. And then they freeze it after that. So that's what keeps it correct. But really our minds, we are, we're always changing things. And yet something keeps it organized and coherent and, and, and settling back into a good place rather than drifting off into crazy land. That is, I think, our, our

• 45:18Speaker 00

  biggest ambition, to have a mind that is self-consistent and, and can keep training itself and make it coherent.

• 45:26Speaker 01

  I love that. Can I ask? It almost seems that it's, it's such an ambitious vision and the idea that all these things can be unified into a single mind is so ambitious.

• 45:37Speaker 00

  It's within reach. I think it's within reach. It's here, it's 2026.

• 45:41Speaker 01

  Yeah.

• 45:42Speaker 00

  And our computers are so fast. You know, is it, is it so ambitious that it's out of reach? Or do we have already inklings of how all the steps can be done? And we, I, I think we have a vision and inklings. I don't think it's, I don't think it's inappropriate.

• 46:02Speaker 02

  Your vision involves a trillion parameter model with 20 watts. That seems pretty ambitious.

• 46:10Speaker 03

  That is ambitious. In some sense, with current technology, I would say is also impossible. Like just storing a trillion parameters in memory would probably use more than 20 watts of energy with current memory technologies. But we are really thinking of, okay, things are getting better. Competition is getting cheaper. It is getting more energy efficient. So where would be, would be in five to 10 years. And I think five to 10 years with the right algorithms and we can totally be in a world where this would be possible.

• 46:43Speaker 00

  So five to 10 years is two orders of magnitude of Moore's law. It's a standard improvement. If we double every 18 months, 10 years, we'd be two orders of magnitude. And so for Kerm's statement to be plausible, then today you should be able to do it for, for what, 20 watts, two orders of magnitude, 2,000 watts. If you can do it with 2,000 watts today.

• 47:11Speaker 02

  Yep.

• 47:11Speaker 00

  Then in 10 years, you'll be able to do it for 20 watts.

• 47:14Speaker 02

  Think you can do it for 2,000 watts? You have lots of people at Research Labs that have access to way more than that.

• 47:21Speaker 03

  Yeah, I think we can. Can be more efficient than that even now with the right algorithm.

• 47:28Speaker 01

  If we can be more efficient than that, then why aren't we? It's not like people just want to spend all their money, spend all our money.

• 47:34Speaker 00

  Sometimes it seems like they.

• 47:38Speaker 00

  Sometimes it seems like they want to.

• 47:39Speaker 03

  Yeah.

• 47:40Speaker 00

  Doesn't it? I think that's how they show they're they're real men by using lots of energy.

• 47:46Speaker 03

  At least when I look at different research groups, I don't even see anyone believing in it's possible. And I think if you don't believe in it, you're just not going to work on the technical problems and work through them.

• 47:56Speaker 02

  Is it that it's not possible or it's that there's so much waste in the system? Like one, which one is it? Is there someone who knows how to do it efficiently?

• 48:05Speaker 03

  Yeah.

• 48:06Speaker 02

  And then there's 10 times the number of people in the same lab doing all these other things. And so nine out of 10 people are wasting.

• 48:16Speaker 03

  In some sense, the way I think about it is that we are stuck in a local minnow. So if we want to move towards this new kind of algorithms, it is almost impossible that things will not get worse before they get better. So when we start exploring this new directions, you're not going to get state of the art performance from day one. But it is because it is a different paradigm. But that path leads to similar performance at a higher energy. And these big labs, they are so locked into a product that they it is not possible for them to pursue a path where things get worse first.

• 48:49Speaker 02

  Because their current paradigm allows them to keep scaling. And this new paradigm, they have to take a bet and they.

• 48:55Speaker 03

  And they have to figure out some some technical things that are difficult that we have thought about it for many years. We know people who have thought about these things for many years. And when I talk to them, it makes sense that it's doable. But you need to think about those challenges for a long period of time.

• 49:13Speaker 02

  So if everything goes right with Oak, what happens with the company? What do you, what kind of company are you building?

• 49:20Speaker 00

  If everything goes right, we implement the architecture. We can have genuine continual learning and we can form abstractions so that we can do planning and reasoning and and we so sort of like true intelligence. And then, you know, it's hard to imagine just exactly what will happen by then. But I think.

• 49:41Speaker 02

  Humans will become irrelevant.

• 49:43Speaker 00

  I don't think that's true at all.

• 49:46Speaker 03

  We don't either.

• 49:47Speaker 00

  I think the world becomes exciting and even more exciting and interesting and for humans. But in particular, I think there are the, you have to wonder about the large language models. They might be at risk. When this eventually happens, you know, I'm sure they'll get a good run. They've already had a good run, you know, they've been very successful. And let me say just for for clarity that large language models are an amazing scientific breakthrough. A breakthrough in the skillful use of language by neural networks is totally unanticipated. You know, it was always a holdout for symbolic methods in language. And they have totally changed how that's thought about now.

• 50:31Speaker 00

  Yeah, it's a big breakthrough. So it's frustrating to me that we have to, you don't just celebrate that we've made this great progress in the subset of the problem of AI and enjoy that. Instead, it has to pretend to be all of AI, all of intelligence is not fluid capable use of language. There's so much more. It's an important part, you know, it's like 20% or a quarter of intelligence. There's more.

• 50:59Speaker 01

  Yeah.

• 51:00Speaker 00

  We're not done.

• 51:01Speaker 01

  Yeah. If everything goes right, are you imagining that there's a single mind that can do everything from learn how to swing from tree branches to make a spaceship? To, you know, all these various things we've talked about today, is it a single mind? And is it a single set of weights that can that can do all these things? Or is it?

• 51:22Speaker 00

  It's a single design.

• 51:23Speaker 01

  Okay.

• 51:24Speaker 00

  The many different, there'll be many minds.

• 51:27Speaker 01

  Okay, so it's a single design that reacts to different environments.

• 51:30Speaker 03

  And different versions of that mind would learn different things because they have different experience. This sort of goes back to the big world hypothesis that there are infinitely many things to learn. So one system cannot learn infinitely many things. And like, I think Rich already mentioned this, but if you have two of these systems, if you have two of the largest systems in the world, then it is trivial that they cannot model each other because they are equally complex. So a single system would never be able to get to a point where it can learn everything. It would always be multiple systems that are learning from their own experience.

• 52:03Speaker 00

  You guys are hiring, what kind of people are you looking for?

• 52:06Speaker 03

  We are hiring. The initial team, most of it, we already have in our mind. So these are people who have thought about these ideas in the past, in many, for a long time. And we are going to take a slightly different approach because this is a different paradigm. It doesn't make sense to become large very quickly because in some sense, everyone we hire has to come to see what we see and not everyone sees that. So we're going to start small, slowly grow to maybe a handful or two or three and then go from there.

• 52:40Speaker 00

  We want to be super aligned.

• 52:42Speaker 03

  Want to be super aligned.

• 52:43Speaker 00

  So that we can be very productive, working together and scaling the progress. Very, very cool.

• 52:52Speaker 01

  Wonderful. I love this conversation. Thank you for taking the time to share what you're up to. You are, you know, an unusually deep thinker about where reinforcement learning and algorithmic design will go. And it was a true pleasure to get to explore it together with you today. So thank you.

• 53:10Speaker 00

  Thank you very much.

• 53:11Speaker 03

  Thank you.

• 53:11Speaker 00

  It's our pleasure.