ReflectionAI Founder Ioannis Antonoglou: From AlphaGo to AGI

Ioannis Antonoglou, founding engineer at DeepMind and co-founder of ReflectionAI, has seen the triumphs of reinforcement learning firsthand. From AlphaGo to AlphaZero and MuZero, Ioannis has built the most powerful agents in the world. Ioannis breaks down key moments in AlphaGo's game against Lee Sodol (Moves 37 and 78), the importance of self-play and the impact of scale, reliability, planning and in-context learning as core factors that will unlock the next level of progress in AI. Hosted by: Stephanie Zhan and Sonya Huang, Sequoia Capital Mentioned in this episode: PPO: Proximal Policy Optimization algorithm developed by DeepMind in game environments. Also used by OpenAI for RLHF in ChatGPT. MuJoCo: Open source physics engine used to develop PPO Monte Carlo Tree Search: Heuristic search algorithm used in AlphaGo as well as video compression for YouTube and the self-driving system at Tesla AlphaZero: The DeepMind model that taught itself from scratch how to master the games of chess, shogi and Go MuZero: The DeepMind follow up to AlphaZero that mastered games without knowing the rules and able to plan winning strategies in unknown environments AlphaChem: Chemical Synthesis Planning with Tree Search and Deep Neural Network Policies DQN: Deep Q-Network, Introduced in 2013 paper, Playing Atari with Deep Reinforcement Learning AlphaFold: DeepMind model for predicting protein structures for which Demis Hassabis, John Jumper and David Baker won the 2024 Nobel Prize in Chemistry

• 0:00Speaker 00

  Go is a complex game. There was always a bit of worry about whether AlphaGo was truly as good as we believed. So we actually had the conviction that deep reinforcement learning is the answer, based on everything that we could measure and everything we could see. But the thing about these systems is that they're not like classic computers, where you just know that they always produce the same answer. They're like stochastic. They are creative. And they have some blind spots. They hallucinate, similarly to how model LLMs hallucinate. So you need to really push them and just see exactly where they break. And the only way you could actually do that is by having the best humans playing against them.

• 0:58Speaker 01

  Today, we're excited to welcome Yannis Antonoglou, a researcher and an engineer who has contributed to some of the most significant breakthroughs in AI. As a founding engineer at DeepMind, Yannis played a crucial role in developing AlphaGo, which made history by defeating Go world champion Lee Sodol.

• 1:16Speaker 01

  He later co-led the development of MuZero, which pushed the boundaries even further by mastering multiple games autonomously.

• 1:24Speaker 01

  Now, as he embarks on his latest venture with Reflection, he's focused on building the next generation of AI agents. We're excited to talk to Yannis about the breakthrough moments in AI history that he's witnessed firsthand. From AlphaGo's famous move 37 to his perspective today on what's next for the combination of reinforcement learning and large language models on the way to AGI.

• 1:51Speaker 02

  Yannis, thank you so much for joining us today.

• 1:53Speaker 00

  Thank you so much for having me.

• 1:55Speaker 02

  Yannis, you have an incredible background, having worked at DeepMind as a founding engineer for over a decade, starting with some of the most notable projects that have really defined the industry. DeepMind, quite notably, created this notion of building AI within games to start. Can you share a little bit more about why DeepMind chose to start with games at the time?

• 2:18Speaker 00

  Yeah, so DeepMind was the first company to truly embrace the concept of artificial general intelligence, or AGI. From the outset, they had grand ambitions, aiming to build systems that would match or exceed human intelligence. So the big question was, and still is, how do you build AGI? And more importantly, how do you measure intelligence in a way that allows for meaningful research and performance improvements? So the idea of using video games as a testing ground came naturally to DeepMind founders, it was Demis Hassabis and Shane Legg. Because Demis had a background in the gaming industry, and Shane's PhD thesis defined AGI as a system that could learn to complete any task. Video games provided a controlled, yet complex, environment where these ideas could be explored and tested.

• 3:02Speaker 02

  And to what extent, you mentioned games are, they provide a very controlled environment. To what extent are games representative or not of the real world? Like, if you have a result in games, do you think that generalizes naturally to the real world or not?

• 3:14Speaker 00

  So, I mean, I guess games have indeed been viable for developing AI, and you actually have like a few examples of that. So you can see that PPO, for example, which is currently being used in RLHF, was developed using OpenAI, GIM, and Mchoco and Atari. And similarly, you have like MCTS, which was developed, which stands for Monte Carlo Research, and was developed through board games like Pac, Am, and Go. But at the same time, games have like a number of limitations. So the real world is messy, is unbounded, and it's a much tougher nut to crack than even the most complex games. So even though it just like gives you an interesting testbed to develop new ideas,

• 3:55Speaker 00

  it's definitely limiting, and it doesn't really capture all the complexity of the real world.

• 4:01Speaker 02

  Okay, interesting though. So about a lot of the techniques and algorithms that you've developed in a game environment, DPO, et cetera, these are used in the real world?

• 4:10Speaker 00

  Yeah, so PPO is actually like exactly what chess PD used for RLHF. And so MCTS, it's used in Muzeo and Muzeo has been used in like in the real world in things like, you know, compression, video compression for YouTube.

• 4:28Speaker 00

  In it was part of the self-driving system at Tesla at some time.

• 4:35Speaker 00

  And it was also like used for developing a pilot like that was completely controlled by an AI. So yeah, I mean, you can see methods like that being used in the world to solve their problems.

• 4:49Speaker 02

  So interesting. Yannis, I remember back in 2017 when AlphaGo the movie came out and it featured the incredible game of AlphaGo against Lee Sodol. Can you take us back to that moment in time and maybe the years leading up to it as you're building AlphaGo? How was AlphaGo specifically chosen as the game to focus on?

• 5:11Speaker 00

  So I think like games have always been a benchmark for AI research. So like before Go, you had chess and chess was like a major milestone with like IBM's Deep Blue defeat in Gargasparov in the late 90s. And I mean, even though you know chess and Go are completely different games and Go is definitely a different beast. There is a like games have always been acted as testbeds for like the development, especially board games for the development of like new AI methods. Actually, even going back to the earliest days of AI research, Turing and Shannon, they both worked on their own versions of like chessboards. So now the thing about like Go is that it's a much harder problem than like chess.

• 5:56Speaker 00

  And the reason for that is because it's almost closely possible to define an evaluation method, a heuristic. So in chess, you can just like take a look at the board, you can count the number of pawns that like each side has, you can see what the ranks of these pawns are. And then you can just like make some, you can draw some conclusions for like who is winning and why. But like in Go, there's, there's nothing like that, like it's mostly human intuition. And if you ask like a Go, you know, professional player, like how they know whether, whether a position is a good one or a bad one. They would say that like, you know, after having played the game for so long, they can just like feel it in their gut that like this is a better position than the other one. So now. It's actually a question of how do you encode the feeling in your gut into like an AI system, right?

• 6:42Speaker 00

  So this is exactly the reason why Solving Go was considered the holy grail of AI research for a long time. And it was a challenge that seemed almost impossible, but at the same time was like within reach. People felt that like they could actually get it cracked. And this is exactly what AlphaGo did back in 2016. And it kind of like showcased two new methods, which is like deep learning and reinforcement learning. Because back in 2015 and 2016, like now we kind of think of deep learning and reinforcement learning as mature technologies. But like back then, we're kind of like literally like making the, they're taking their first steps. And they were kind of like the new kid in the block. And most people were kind of like really skeptical about them.

• 7:23Speaker 00

  Like everyone thought that deep learning was a, was another AI fad that would just like won't last the test of time. So yeah, I mean, AlphaGo was chosen because it was like clear to show days that you actually have like the most, the most performant agent in the world. You could actually evaluate it, you can have it play with other humans. And at the same time, it was within reach, given like the latest developments in deep learning and reinforcement learning.

• 7:50Speaker 03

  I remember reading that there's more configurations of, of the Go board than, than atoms in the universe. By many words in I2, and that blew me away because I mean, I grew up playing Go and it felt like such a, you know, it's a very simple in terms of the rules, but I see why it was the holy grail. Maybe, can you explain how AlphaGo worked technically? Maybe explain it to me like I'm a fifth grader because that is, that is effectively my, my level of sophistication, understanding these things. But how did it work? And you mentioned that both reinforcement learning and deep learning were involved. I'd love to peel that back a little bit.

• 8:24Speaker 00

  Yeah, absolutely. So AlphaGo.

• 8:27Speaker 00

  Has two deep neural networks. So like a neural network is a function that like takes something as an input and produces something as an output. And it's literally like a black box. We don't really know exactly how it does it. Just like know that you can actually, if you train it on enough data, it will just like learn the mapping. If we learn the function like from input to the output space. So AlphaGo actually had access to two deep neural networks, the policy network and the value network. And the policy network suggested the most promising move. So it will just take a look at a current board position. And I'll just like say, okay, you know, based on the current position, this is the list of moves that I would recommend you just like consider playing. And it also had access to the value network, who would just take a look at like a board position.

• 9:11Speaker 00

  And just like give you a winning probability, like what are your chances of actually winning the game starting from this position? This is exactly the gut feeling, like it had like its own gut feeling on like whether the position is a good one or a bad one. So once you have access to these two networks, then you can actually like play in your imagination a number of games. You can consider like the most promising moves, then you can consider your opponent's most promising moves. And then you can just like evaluate its moves, like the value network. And then, you know, you can use a method called min max. What that says is that I want to win the game, but I also like know that my opponent wants to win the game.

• 9:48Speaker 00

  So I want to just like pick a move that will maximize my chances of winning, knowing that like my opponent will try to maximize their chances of winning. So if you actually like do that and simulate a bunch of moves, then you can just like get the optimal action. And you know, the way to just like do this imagination, this planning, this search in the most efficient way is by using a a tree sets method called multijet tree sets, so MCTS. So whenever people talk about MCTS, they literally just like mean this heuristic of how do I You know, how do I choose which futures to consider so that like I can make informed decisions? The role for reinforcement learning and deep learning in building AlphaGo was that

• 10:34Speaker 00

  AlphaGo first of all was a success of reinforcement learning and deep learning. Because like this is exactly the two methods that powered AlphaGo. And the policy network was initially trained on a large set of human games. So you had like many games played by human professionals. And you just like consider every position and you consider the move they took at this position. And then you have like a deep neural network that tries to predict this move. Then, once you have the policy network, you need to somehow find a way to just like obtain a value network. So we did it in two ways. First, we just took the policy network and we had it play against itself. And we used reinforcement learning to to improve it, to improve the blank strength of the model.

• 11:19Speaker 00

  So we used a technique called policy gradient. So what policy gradient does is that it just like looks at the game. And then it looks at the outcome. This is the simplest version, kind of like of policy gradient. It looks at the outcome of the game. And for all the moves that led to a win, they'll just like say, great, you know, just increase the probability of choosing this move. And for all the moves that led to a loss, it says, great, now decrease the probability of like this move being selected in the future. And if you do that like, you know, for many games and for long enough, then you just like get an improved policy. Now, once you have this improved policy, you can just generate a new data set of games where like the policy plays against itself. And then you have like a huge amount of games where for each position, you know who the final winner was.

• 12:03Speaker 00

  So then you can take this network, you can take another network, a value network, and have it predict the outcome of a game based on the current position. So what the network will learn is that if I start at this position and I play under my current policy, on average, this is the player who wins, like it's either a black player or they're a white player. So this is the first version of like a value network. And you can just like use it within AlphaGo by combining it with the policy network.

• 12:32Speaker 02

  And what were some of the biggest challenges in building this and how did you overcome them?

• 12:38Speaker 00

  Yeah, so AlphaGo was not just a research challenge, but was mostly, I'd say, an engineering marvel. It was the early versions ran on 1,200 CPUs and 176 GPUs. And the version that played against listed old used 48 TPUs. So like TPUs were like the first accelerator, custom accelerators. And these were like, uh, these accelerators were like really primitive back then. Because literally it was like the first version, right? Like now the later accelerators are much, much better and much more stable. So the system had to be highly optimized to minimize latency, maximize throughput. We had to build large-scale infrastructure for training these networks. And it was a massive endeavor. It just required a lot of coordinated effort from many talented individuals

• 13:23Speaker 00

  working on different aspects of the project. But, uh, you know, I just like walked you through a number of steps to just like obtain the policy network and the value network. And each of these steps had to just be implemented at the, at the limits of like what was available and what was possible back then in terms of scale. And it had to be implemented in a way where people could just like think everything. They could just like try the research ideas fast and get results, results fast. So, yeah, lots of people, uh, scale in, you know, at levels that hadn't been implemented before and kind of like working at the forefront of what was capable, what was possible back then.

• 14:04Speaker 02

  I love your highlight of it being a research marvel and an engineering marvel. And I remember you sharing one time that part of the reason this project came about also was because Google had TPUs that they needed to, they needed a test customer for. And that was the spark, this AlphaGo project. So that's pretty incredible.

• 14:21Speaker 03

  How much conviction did the DeepMind team have that this is going to work? You mentioned that, you know, at the time, deep learning, reinforcement learning were still relatively novel, but DeepMind was very much founded with that belief. But did you guys think that you were going to be able to have kind of these superhuman level results, beating the top Go player in the world? Like, was it a crazy idea and maybe it'll work? Or did the team have conviction like this is going to work?

• 14:45Speaker 00

  Yeah, so I'd say I think the team had a cautious optimism. So one of AlphaGo's lead developers, Ajah Huang, he is a strong amateur Go player and he had been working on Go for like a decade before AlphaGo happened. And we also had like a lead report of computer game, of computer players. And you could see that AlphaGo was significantly stronger than anything that had come before. But Go is a complex game and there's, there was always a bit of worry about whether AlphaGo was truly as good as we believed. So we actually had the conviction that deep reinforcement learning is the answer, based on everything that we could measure and everything we could see. But that's, uh, the thing about these systems is that, you know, they're,

• 15:29Speaker 00

  they're not like classic computers where you just like know that they always produce the same answer. They're like stochastic, uh, they are creative. Uh, so, and they, they, they all have like, they, they, they have like some blind spots. They hallucinate, like similarly to how like model L L M scholars mate. So you, you, you need to just like really push them and just like see exactly where they break. And the only way you could actually do that is by having like the best humans playing against them.

• 15:57Speaker 02

  Um, move thirty seven, can you tell us what that was? It was such a monumental move and I think everyone watching it at the time, uh, it was, and at least it all maybe primarily was confused by that move. Um, what was going on in your head when that happened?

• 16:12Speaker 00

  So yeah, I mean, uh, move thirty seven, uh, yeah, uh, in game two against Lissabon was literally, it was a spectacular moment in the sense that it kind of showcased to the world that AlphaGo has creativity. And it demonstrated that AI could come up with strategies that even top human players hadn't considered. So at first, like I still remember that, like, we thought that AlphaGo made an error. So that's, uh, it actually like hallucinated, it did something that like it didn't need to do. But then it turned out to be a brilliant and a conventional move that underscored that the system had a deep understanding of the game. The, the system actually had like, uh, Creativity, it's good thing of things that like people hadn't thought of before.

• 16:54Speaker 03

  I want to take us to another key move in the game. I think it was in game four. At this point, I was rooting for Lee because I was like, oh, poor guy needs to win a game. Move seventy-eight, I think, I think AlphaGo made a mistake and Lee Sodol noticed it. I guess, what was the weakness there that Lee found during the game?

• 17:11Speaker 00

  Yeah, exactly. So, I mean, Lee Sodol's victory in game four was literally a testament to human ingenuity. Like, move seventy-eight was unexpected and called AlphaGo off-guard. Initially, AlphaGo, like based on its evaluations, misinterpreted as a mistake and thought that it was actually like winning. So that's why it didn't respond appropriately. And, you know, this kind of highlighted a blind spot in the system. So the game showed that like while systems like AlphaGo are extremely powerful, at the same time, they still have vulnerabilities and there were like still areas where we could further improve it.

• 17:45Speaker 03

  But how do you go about improving something like that? Do you need to show a lot more data of, you know, kind of that type of human ingenuity move? Or how do you go about fixing and patching those?

• 17:56Speaker 00

  So, yeah, I mean, it's actually interesting that by the end of the games with like Lee Sodol, we just like put together a benchmark. We're just kind of like trying to quantify and just have a way of measuring the mistakes that like AlphaGo makes. And, you know, these kind of blind spots, let's say. And then it just tried another approaches to just like improve the algorithm so that we can, you know, solve these issues.

• 18:23Speaker 00

  And what happened is that actually, the most effective way of getting rid of them was just like do what we were doing, just like at a higher scale and better. So just like change the architecture of the model, we just like switch to a deep-res net with two output heads. And we also like, we just had a bigger network, trained on more data. Then just like move to AlphaZero and spread the algorithms and that kind of like made it so that we didn't have any hallucinations anymore. So in a way, we just like scaled data, you know, things that are always kind of the well-known recipe in the field of AI. It's exactly what solves it in our case.

• 19:07Speaker 02

  With scale and data, how much did higher quality data, or maybe specifically data from great professional players, the best professional players, make a meaningful difference? Or was it just any data?

• 19:18Speaker 00

  No, for us, what mattered was that we kind of like solved it using self-play.

• 19:24Speaker 02

  Yeah.

• 19:24Speaker 00

  So we actually had access to the most competent Go player in the world. And we just like used it to generate the best gold games. And then it just trained on these games. So I guess like, you know, we didn't need to have like human experts because you had like an experts in-house. It wasn't human.

• 19:45Speaker 02

  Huh, interesting. Amazing. Well, I'd love to move on to the progression from AlphaGo to AlphaZero. And you talked a little bit about this notion of self-play just now. AlphaZero was powerful because it learned how to play the game from scratch, entirely from self-play, without any human intervention. Can you share more about how that worked and why that was important?

• 20:08Speaker 00

  So AlphaZero was a game changer because it learned entirely from scratch through self-play without any human data. And this was like a major leap from AlphaGo. Because like AlphaGo, as I said, relied heavily on human expert games. So two things happened. First of all, AlphaZero managed to simplify the training process and also like showed that AI will literally just like get from zero to superhuman performance just purely by playing against itself. And that allowed it to just be applicable to a whole range of like new domains that were out of reach because like there wasn't enough like human data for it. But I think the more important thing is that we just saw that AlphaZero also solved all the issues that like AlphaGo had

• 20:51Speaker 00

  in terms of hallucinations, in terms of, you know, blind spots and robustness. So like AlphaZero was like a better method, just, you know, full style.

• 21:04Speaker 03

  And you you explained kind of how AlphaGo worked to a fifth grader. What would you tell the fifth grader would be the key difference technically that you that you've implemented with AlphaZero?

• 21:13Speaker 00

  So AlphaZero, just like AlphaGo, uses a policy network and a value network along with multiple characters. So in that respect, it's exactly the same as AlphaGo. So the key difference is in training. AlphaZero starts with random weights and learns by playing games against itself. And by playing games against itself, it iteratively improves its performance. But the main idea behind AlphaZero is that whenever you take a set of weights, a set of policy and value networks, and then you just combine them with search, then you just like end up with a better player. You just like increase your performance. You just like become a stronger player.

• 21:53Speaker 00

  So what that meant is that we can actually use this mechanism to improve the model policy, the role policy. So this is what we call in reinforced learning, a policy improvement operator. Whenever you can just like take an existing policy and then do something, some magic, and then just like come up with like a better policy. And then you can just like take this policy and distill it back to the initial policy and then just repeat this process. Then you have like a reinforced learning algorithm. And I think like, you know, this is exactly what people are trying to do today with like, you know, two star or like, you know, synthetic data. This is exactly the idea of like, how can I take a policy, do something with it, planning, search,

• 22:35Speaker 00

  compute, whatever it is, and derive a better policy, which I can then imitate and just like kind of distill back to the original policy. So this is exactly what AlphaZero is doing. It uses MCT search to produce a better policy. Then it takes these trajectories, it trains this policy and value network on the new better trajectories, and it repeats this process until it converges to the, you know, to an expert level Go player.

• 23:02Speaker 03

  That's fascinating and counterintuitive that kind of like starting without the the weights that you would have from from, you know, professional level players is actually a better starting place.

• 23:13Speaker 02

  The epitome of AI agents in games was achieved, I think, via MuZero, which is the progression even from AlphaZero itself. And it's also where you became one of the co-leads or one of the leads of the game. AlphaZero was obviously impressive because of self-play, but it also needed to be told the environment's dynamics or the rules of the game. And MuZero takes this to the next level without needing to be told the rules of the game, and it mastered quite a few different games: Go, chess, and many others. Can you share a little bit about how MuZero worked? And why was this particularly meaningful?

• 23:54Speaker 00

  Absolutely. So AlphaZero, as you said, was a massive success in games like chess, Go, Shogi. So in games where we actually had access to the game rules, where we actually had access to a perfect simulator of the world. But this reliance on the perfect simulator made it challenging to apply it to real-world problems. And real-world problems are often messy, and they lack rules, and truly hard to write a perfect simulator of them. So that's exactly what MuZero tried to solve. So MuZero masters the games, of course, like Go, chess, and Shogi, but it also masters more visually challenging games, or games like a hot goat like Atari. And it does that without having access to the simulator. It just learns how to build an internal simulator of the world,

• 24:38Speaker 00

  and then just use this internal simulator in a way similar to what AlphaZero was doing. So it does that by using model-based reinforcement learning, where what that means is that you can just take a number of trajectories generated by an agent, and then try and learn a model, learn a prediction model of how the world works. So this is actually quite similar to what methods like Sora are trying to do now, where they just take YouTube videos and they try to learn a world model, by just trying to predict based on, starting from one frame, what's going to happen in the future frames. So MuZero tries to do exactly that, but it does it in a way different from generative models, in the sense that it tries to only model things that matter

• 25:23Speaker 00

  for solving the reinforcement learning problem. So it tries to predict what the reward's going to be in the future, what's the value of future states, what's the value of future... Policy for like future states, so only things that you need within your MCTS. But you know, the fundamental is kind of like remain the same. So how do you just like learn a model based on trajectories? And then once you have these models, you can just combine the search and you know, get superhuman performance. So of course, like you can always decouple the two problems and have like the model being trained separately from, you know, data out in the wild and then just like combine that with new zero.

• 26:02Speaker 00

  And we just found that back then, given the limitations of like our models and the smaller sizes, kind of like made more sense to just like keep those two together and only have the model predict things that matter for our planning. Instead of just like try to model everything because you're kind of hitting the limits of what the capacity of the model could take.

• 26:25Speaker 03

  So interesting. Is it right to assume then that not only Sora, you know, takes the same approach, but maybe other world models or other robotics foundation models?

• 26:35Speaker 00

  Yeah, so anything that tries to just like build a model of how the world works and then just like use that for planning, it's within, you know, new zero like methods. So yeah, you can just like train it on YouTube videos, you can train it on like the inputs coming from like robots. You can train it on, you know, any, any environment. You can even think of like large language models as a form of models of like text. So like the, the model text. But the, the thing about text is that like the model is a bit trivial, like you, you don't need to just, there aren't many artifacts happening when you're trying to predict what's the next world is going to be.

• 27:12Speaker 03

  Right.

• 27:12Speaker 00

  So.

• 27:13Speaker 03

  Have you seen the ideas behind new zero kind of be used outside gameplay or in messy real world environments?

• 27:23Speaker 00

  So yeah, I mean, so as I've said, AlphaZero and new zero are quite general methods. And there were like, there's a number of scientific communities in chemistry. So there's AlphaChem in quantum computing, some people try to use AlphaZero in optimization. Where they just like adopted AlphaZero because it was really powerful in really doing planning. And it's like solving this optimization problems. At the same time, U zero was incorporated in a version of like Tesla's self-driving system. It was kind of reported in their AI day. And it was also used, and I think it's currently being used within YouTube as a custom compression algorithm.

• 28:05Speaker 00

  But I think, you know, it's it's early days and takes time for like this new technologies to be fully adopted from by the industry.

• 28:15Speaker 02

  We'd love to talk a little bit more about reinforcement learning and agents. And you alluded earlier to the fact that reinforcement learning and deep learning back in 2015 were new nascent ideas. They really, you know, grew in popularity 2017, 2018, 2019 onwards. And then they were overshadowed by L L M's, largely because of the G P T and everything else that came out. But now reinforcement learning is back. Why do you think that is the case?

• 28:45Speaker 00

  Yeah, I mean, first of all, L L M's and multimodal models have indeed brought incredible progress to AI. So these models are exceptionally powerful and can perform some truly impressive tasks. But they have like some fundamental limitations, and one of them is the availability of like human data. People just keep talking about the data wall and what happens once you run out of like high quality data. And this is exactly where reinforcement learning science. So reinforcement learning excels because it doesn't rely solely on pre-existing human data. Instead, reinforcement learning uses experience generated by the agent itself to improve its performance. So this self-generated experience allows reinforcement learning to learn and adapt. And to even adapt to scenarios where human data is scarce or like non-existent.

• 29:30Speaker 00

  So if you define the reinforcement learning problem in the right setting, in the in the right way, you can literally effectively exchange computer intelligence. You can just like get to a point similar to where we were with AlphaZero, where we just like the the moment we threw more computer at it, like we made the network speaker, which is like, you know, used more games. We just literally got a better player and was deterministic. You always get a better player. So I guess this is exactly where we want to be with like this synthetic data pipelines. Currently, we have that with, you know, the scaling clause in L L M's that if you have like more data and bigger models, then you get like, you know, you can predict that there's going to be an improvement in performance.

• 30:13Speaker 00

  But, you know, once you run out of like human data, how do you just keep going? And synthetic data is like the answer to that. And the the only way that, you know, you can actually get high quality reinforcement learning, high quality data to just like improve your model is like via some form of reinforcement learning. And just like leaving, I'm just like keeping reinforcement learning as a really kind of blanket term here, where I just like define it as anything that lends through trial and error.

• 30:44Speaker 03

  How do you think reinforcement learning is is being brought into the kind of like L M world? And you mentioned Q star earlier. Like, I I guess in a in a closed form game, you have like a pretty clearly defined policy and value function. How does that work in like a messy kind of real world environment or the L L M world?

• 31:06Speaker 00

  So I mean, I guess like there are two different types of like messy real world, right? Like there is the if you try to just like build a controller or something, that's a really messy environment. And then if you if you operate in the digital space. So my personal, I believe that digital A G I, which has happened much earlier than, you know, robotics A G I. And the reason for that's exactly that you have control over the environment. And the environment is like computers, like the digital world. So even though it's like messy and it's noisy, it's still contained. It's not like the the real kind of like world in that sense. So now in terms of how do you bring like reinforcement learning?

• 31:47Speaker 00

  So reinforcement learning is, we used to say in DeepMind that you have like the the problem and you have the solution. And the problem setting of reinforcement learning is how do I take a model? How do I take a policy and generate synthetic data? Or like I I I learn, I find a way to. Improve this policy via interacting with the environment via trial and error. And this is like the reinforced learning problem setting, right? And then there's like the solution space where you have value functions and have like reinforced learning methods. So I think that there's a lot of inspiration to draw from like classical reinforced learning methods that were developed in the past decade.

• 32:27Speaker 00

  But have just adopt, you have to adjust them to the, to the new world of L L Ms. So methods like Q start try to do that by just taking the idea that if I have a policy and then I do planning, I consider possible future scenarios and then I have a way to evaluate which one is better. Then I can just like take the best ones and then ask the model to imitate these better ones. And this is like a way of improving the policy. So in the classic R L framework, you do that by using a policy and a value network. In the new world, you'll just do that by asking your, by having a reward model or asking your, your L L M

• 33:08Speaker 00

  just like gives you feedback on an output it gave you.

• 33:13Speaker 02

  So interesting. You also talked a little bit about synthetic data earlier. I think some folks are very bullish on synthetic data and some folks more skeptical. I also believe that synthetic data is more useful in some domains where outcomes and success is perhaps more deterministic. Can you share a little bit about your perspective on the role of synthetic data and how bullish you are on it?

• 33:33Speaker 00

  Yeah, I mean, I think like synthetic data is something that's like, we have to solve one way or another. So it's not about like whether, you know, you're bullish or not, it's kind of, it's an obstacle, but you have to just find a way around it. Like we will run out of data, like, you know, we, there is so much data that like humans can produce. And also like it's important that these systems start taking actions, they start learning from their own mistakes. So we need to just find a way to make like synthetic data work. Now, what people have done is that they've tried like the most, I guess, like naive approach, where you just like take the models, they produce something and you try to just like train on that.

• 34:16Speaker 00

  And of course, like, you know, they've seen that there's more collapsing and it's just like doesn't work out of the box. But, you know, new methods never work out of the box. You just like need to invest in it and just like take your time and, you know, really kind of think of what's the best way of of doing it. So I'm really optimistic that we'll just definitely find ways to improve these models. And I think like actually there, there is a number of of methods out there, like the two star and equivalents that just, you know, in the new world where people don't really share their research breakthroughs, the way they used to is probably hidden behind like some company trade secrets.

• 35:00Speaker 03

  I'm gonna ask about reasoning and, you know, novel scientific discoveries. Do you think that that can kind of naturally come out of just scaling L L M's if you have enough data? Or do you think that kind of like the ability to reason and, you know, come up with net new ideas requires kind of doing reinforcement learning and, you know, deeper compute and inference time?

• 35:21Speaker 00

  So I think like you need reinforcement to get better reasoning, because the distribution of like, it's, it's, it's also about the distribution of data, right? Like you, you have like, you have a lot of data out in the wild in the internet. But at the same time, you don't always have like the right type of data. So you don't have the data or like someone reasons and they just like explain the reasoning in detail. You have some of it, you have like, and it's incredible that like the method, the, the, the models have actually mounts to, to pick it up and just imitate it. But if you want to just like improve on that capability, then you need to do that through reinforcement learning. You need to just like show the model how this kind of emerging capability can further

• 36:05Speaker 00

  be improved by just like have it generating the data, interact with the environment, you know, just tell it when it's doing something right and when it starts doing something right. So yeah, I think like reinforcement learning is definitely part of the answer for that.

• 36:18Speaker 03

  AlphaGo, AlphaZero and MuZero are the most powerful agents we've ever built. Can you share a little bit about how some of the lessons and learnings unlocked from that are relevant to how we're pursuing building AI agents today?

• 36:31Speaker 00

  Yeah, so I feel like AlphaGo and MuZero, you know, you have They've actually fundamentally transformed their approach to AI agents. Because they highlight the the importance of planning and scale, in my opinion. That if you actually look at the charts of like different models and how they scale, you can see that like AlphaGo and AlphaZero were like kind of really ahead of their time. Like they were kind of outliers. You had like this this curves of like how compute scaled. And then you have like AlphaZero or like somewhere standing on its own. So it showed that like if you can scale and you can really push on that, then you can get like incredible, incredible results. At the same time, you know, it also showed that you don't have just only train.

• 37:14Speaker 00

  You can also like, you know, have better performance during inference, during test, during evaluation, but just like using planning. And I think that this is something that we will start seeing more and more in the in the near future. Or like this method will just like start thinking more, like planning more, before they're just making any decisions. So I'd say that like this is more of the heritage of AlphaGo and AlphaZero and E0. It's the the the the basic principles. And the basic principles are of that scale matters, planning matters. These methods can really solve problems that we thought that are insanely complex or like, you know, beyond what we can solve on our own.

• 37:58Speaker 00

  Similar problems with the ones that we actually observed today with these last language models are things that we saw back then. Like back in 2016, we actually saw that these models can hallucinate or that like at the same time, they're also creative, that they will just come up with solutions that we hadn't thought of. But they can also like have blind spots or like hallucinate or be susceptible to kind of like adversarial attacks, which I guess like everyone knows now that these neural networks suffer from. So I think that like these are the the the main kind of

• 38:32Speaker 00

  lessons drawn from this line of work.

• 38:37Speaker 02

  What do you think are the biggest open questions from this line of work for the field dancer going forward?

• 38:42Speaker 00

  So the main question is, we had like AlphaGo and UZI, and we just like managed to have like this insanely robust and reliable systems that will just always play Go and at the, you know, the highest possible kind of level, and they'll just like achieve consistently, they will just like be top of the leaderboard, would just like never lose a game. So AlphaGo Master actually like played against 60 people in online matches and just like literally won in every single one of them. So there was like no, there was like, this battle was for like incredibly robust and reliable. And I think like this is exactly what we're missing now with these LLM-based agents. Sometimes they get it, sometimes they don't.

• 39:25Speaker 00

  You cannot trust them. They will just like, you know, you have like some amazing demos, but like, you know, they happen once every two times even, or like once every 10 times, you have like something amazing. And the remaining nine, they just lost their way and just didn't do anything. So I think like what we need to do is just find a way to just make these LLM-based agents equally robust to the ones that we had with AlphaGo and UZI and AlphaZero. This is like the new open question of like, how do we actually do that?

• 39:54Speaker 02

  We'd love to move into some of your thoughts on the broader ecosystem today. You've touched on a few really core problems that people are working on right now. One, the data wall problem that will hit eventually, perhaps by 2028 or so, as some folks predict. Another being the idea of planning as an area that AI agents need to get better at. And then, you know, a third idea that you just described was around robustness and reliability. Can you share a little bit about, and maybe some of these areas that you think the whole field needs to solve that you are most excited about to help us unlock this vision of really getting to the AI agents that we want?

• 40:40Speaker 00

  Yeah, I mean, I'll just like also add another one to the list. So I feel like another major uh uh another major um challenge is like how to improve in context learning capabilities of these models. So like, you know, how do they, how do we make sure that like these systems can learn uh on the fly? And how they can adapt to new contexts like quickly? So this is like another thing that I think is gonna be really important. It's gonna happen the next uh few years, couple of years, actually. So.

• 41:10Speaker 03

  Yanis, what's the term that you used for that?

• 41:12Speaker 00

  In context learning?

• 41:13Speaker 03

  Oh, in context learning.

• 41:14Speaker 00

  In context learning, yeah. So it's the idea that a system can actually learn how to do a new task with like few shot prompting. Uh like it kind of like sees a few examples and on the fly uh it kind of like learns how to adapt to the new environment. It learns how to use the the the new tools that were provided to it. Or like uh it's kind of like lens. It's it's not just all the knowledge it has stored in its weights, but like it can also like acquire new load knowledge by just like interacting with the real world, interacting with the environment. So I think that this is like another um place where uh there is a lot of work happening at the moment and uh

• 41:57Speaker 00

  gonna have like amazing progress there in the next couple of years. And I'm really excited about that. So, yeah, I mean, uh to recap, I think like planning is important. Um, you know, in context, learning is important and, you know, reliability. So the best way to achieve reliability is just like ensure that this model somehow know how to retain from their mistakes. So if they just like made a mistake somewhere, they can just like see that. And they're like, okay, you know, I made a mistake. I'll just like correct for it, the way that uh humans, you know, make mistakes all the time, but like we, you know, you can correct for them. So these are like the three areas which are I'm really, I'm really excited to see progress on.

• 42:41Speaker 03

  Now that you've kind of embarked on your own entrepreneurial journey, uh how do you think about the areas where startups can compete against the the big research labs and like um how how do you kind of motivate yourself for that for that journey?

• 42:55Speaker 00

  Yeah, I mean, uh, it's a completely like, uh, it's a new world for me, but at the same time, it's not that new. Because when I joined DeepMind, it was literally a startup, so. And I was like literally in the first jump place. So I actually like saw that firsthand. But, you know, one of the benefits of like working for a startup is that, uh, you know, the agility and the focus. So everyone really cares, uh, everyone just moves really fast. And there's like a clear focus on what we want to build. So the building is like what's the most important kind of motivation for people, like just like building. And I think that like this is one of the big advantages that like startups have over more established businesses.

• 43:36Speaker 00

  At the same time, you know, it's easier to just like pivot to adapt to new findings and technologies. You're not kind of like tied to, you know, some pre-existing solutions or like, uh, some products that, you know, you don't want to deprecate because like, you know, they bring a lot of revenue to you. While if you're a startup, you know, you have like no such chains, you could just like move fast and, you know, be innovative and, uh, just, you know, break conventions. And at the same time, just like allows you to leverage like open source resources, things that are out of touch for like the big labs. And, uh, yeah, and you don't have like the red tape that like big places tend to have.

• 44:18Speaker 02

  I love the term that you use sometimes, Giannis, main quest versus side quest.

• 44:21Speaker 00

  Yeah, it's the idea of like having a main focus, like, you know, in big places, in big labs, they have like many different, uh, projects that like people are working on. And, uh, it usually happens that they have like the main quest, the main, you know, thing that like everyone is working on, and there's like many multiple like smaller side quests that, uh, the idea is to just like feed into the, the, the bigger quest, but like usually they don't get as much, uh, uh, they don't get like as many resources or like as many, as much focus from like the leadership. So, yeah, they, they tend to, yeah, derotrophy.

• 44:58Speaker 02

  And in the broader field, what are some of the most defining projects that you admire the most? And maybe who are some of the most influential researchers that you admire the most?

• 45:08Speaker 00

  Yeah, absolutely. So, so I actually like started my AI research journey back in 2012. And I've actually like seen some milestones. So it's like I give a list of like what I think are like the main milestones like in AI in the past like 12 years that I've been around. So the first one I'll say is like AlexNet. This is the the first paper that kind of like showed that deep learning is the is the answer. I mean, back then it didn't feel like it. It just like felt like, you know, a kind of curiosity. But like now, I think that most people are convinced that like deep learning is part of the answer. Then it was TQN. I had the pleasure to actually work on TQN and just like see it firsthand, how it started.

• 45:52Speaker 00

  It was actually developed by a friend of mine, Vlad Nye. And it was like the first system that showed that you can actually combine deep learning with reinforced learning to to to achieve super to achieve human performance or like superhuman performance in really complex environments. Then this was AlphaGo. Again, I was like really lucky to just like work on that. And it showed that, you know, scale and planning are really important ingredients. And if you just like do that right, then you get huge success in a incredibly complex environment. AlphaFold, another one. This is again by DeepMind. It showed that like these methods are not just like things that you can use to solve games,

• 46:37Speaker 00

  but they have they they actually will make this world a better place. They will just like ensure that healthcare is improved, that scientific discoveries are being realized, that we'll just like make sure this world is a better place by using AI.

• 46:55Speaker 00

  Then, uh, ChatGPT, uh, it kind of like brought AI to, uh, everyone. Just like made it accessible, uh, to the broad audience. Like everyone knows what AI is now. It's, uh, it has made my life of explaining my job much easier. So, um, and finally, GPT-4. And I think that, uh, yeah, probably GPT-4 is like the latest kind of big advancement, uh, in AI. Because it kind of like showed that, you know, artificial general intelligence is a matter of years. It's, uh, within reach. Um, yeah, we are, we are getting there. Like, I think that, you know, many, most people now believe that we're like

• 47:41Speaker 00

  a few years away from like AGI. And, you know, that's, uh, that's because of like the incredible breakthrough that, uh, GPT-4 was. Uh, now in terms of like some people I really admire, uh, before I forget. So I'd say first, like, uh, David Silver, he's, uh, he was my PhD supervisor. He was my mentor at DeepMind. Uh, he's an incredible AI researcher. Uh, he worked, he led AlphaGo and AlphaZero. And, you know, he is, you know, he, he has an algealing dedication to the field of, uh, reinforcement learning. And he's, you know, probably the, one of the smartest people, or maybe the smartest person I know. And amazing guy, um, in, amazing reinforcement learning engineer. And the second one I'd say is, uh, Ilya Sutskever.

• 48:26Speaker 00

  And, you know, he was a co-founder of OpenAI. I had the opportunity to work with him just a little bit in the really early days of AlphaGo. But, um, I feel like his commitment to scaling AI methods and pushing the boundaries of what these systems can achieve is remarkable. And, you know, he gave nature that like GPT-3 and GPT-4 happen. So, um, yeah, immense respect towards him.

• 48:52Speaker 03

  Thank you for sharing that. Let's close out with some rapid fire questions. And maybe first, what do you think will be the next big milestones in AI? Let's say in the next one, five and 10 years.

• 49:04Speaker 00

  So I feel like the next five to 10 years, the world will be a different place. I actually really believe that. I think that in the next few years, we'll see models becoming powerful and reliable agents that can actually independently execute tasks. And I think that AI agents will be massively adopted across industries, especially in science and healthcare. So in that sense, I'm really excited on what's coming in AI. And you know, what I'm most excited about is AI agents. Systems can actually like do tasks for you. And you know, this is exactly what we're building at perfection.

• 49:42Speaker 03

  In what year do you think we'll pass the 50% threshold on Sweetbench?

• 49:47Speaker 00

  So I think we are one to three years away from the 5% threshold for three agents and three to five years from achieving 90%. So the reason is, while progress is amazing, I think it's like we still need reliable agents to hit these milestones. And it's really, when it comes to research, it's like hard to make precise predictions.

• 50:07Speaker 03

  When do you think we'll hit the data wall for scaling LLMs? And do you think all the research in RL is mature enough to to keep up our slope of progress? Or do you think there will be a bit of a lull as we try to figure out what happens when we hit the wall?

• 50:21Speaker 00

  So I feel like the wall, you know, based on like what I've read, I think like we have at least one more year for text, just like before we hit the wall. And then we have like this extra modalities, which might actually buy us maybe a year extra. And I think we are in a really good place to just like start using synthetic data. So in the next two years, we'll just like figure out the synthetic data problem. So I think that we won't really hit the wall. It's just like we hit the wall, but like no one realizes it because we have like new methods in place.

• 50:55Speaker 03

  Do you think LLMs will have their alpha go moment? And if so, when?

• 51:01Speaker 00

  I think like LLMs had their alpha go moment with the initial release of GPT, where they showcased the power and the progress made over the past decade. I feel like what they haven't had yet is their alpha zero moment. And that's the moment where more compute directly translates to increased intelligence without human intervention. And I feel like this breakthrough is still on the horizon.

• 51:23Speaker 03

  When do you think that will happen?

• 51:25Speaker 00

  I think this can happen in the next five years.

• 51:28Speaker 02

  Wow. Amazing. Yannis, thank you so much for joining us and taking us through the awesome history of AlphaGo, AlphaZero, MuZero, your own journey through DeepMind, and then many of the core research problems that the whole industry is tackling today around data and building for reliability and robustness and planning and in-context learning. We're really excited for the future that you're helping us build and that you're pushing forward in the field as well. So, thank you so much, Yannis.

• 51:57Speaker 00

  Thank you so much for having me.