Reflection AI’s Misha Laskin on the AlphaGo Moment for LLMs

LLMs are democratizing digital intelligence, but we’re all waiting for AI agents to take this to the next level by planning tasks and executing actions to actually transform the way we work and live our lives. Yet despite incredible hype around AI agents, we’re still far from that “tipping point” with best in class models today. As one measure: coding agents are now scoring in the high-teens % on the SWE-bench benchmark for resolving GitHub issues, which far exceeds the previous unassisted baseline of 2% and the assisted baseline of 5%, but we’ve still got a long way to go. Why is that? What do we need to truly unlock agentic capability for LLMs? What can we learn from researchers who have built both the most powerful agents in the world, like AlphaGo, and the most powerful LLMs in the world? To find out, we’re talking to Misha Laskin, former research scientist at DeepMind. Misha is embarking on his vision to build the best agent models by bringing the search capabilities of RL together with LLMs at his new company, Reflection AI. He and his cofounder Ioannis Antonoglou, co-creator of AlphaGo and AlphaZero and RLHF lead for Gemini, are leveraging their unique insights to train the most reliable models for developers building agentic workflows. Hosted by: Stephanie Zhan and Sonya Huang, Sequoia Capital 00:00 Introduction 01:11 Leaving Russia, discovering science 10:01 Getting into AI with Ioannis Antonoglou 15:54 Reflection AI and agents 25:41 The current state of Ai agents 29:17 AlphaGo, AlphaZero and Gemini 32:58 LLMs don’t have a ground truth reward 37:53 The importance of post-training 44:12 Task categories for agents 45:54 Attracting talent 50:52 How far away are capable agents? 56:01 Lightning round Mentioned: The Feynman Lectures on Physics: The classic text that got Misha interested in science. Mastering the game of Go with deep neural networks and tree search: The original 2016 AlphaGo paper. Mastering the game of Go without human knowledge: 2017 AlphaGo Zero paper Scaling Laws for Reward Model Overoptimization: OpenAI paper on how reward models can be gamed at all scales for all algorithms. Mapping the Mind of a Large Language Model: Article about Anthropic mechanistic interpretability paper that identifies how millions of concepts are represented inside Claude Sonnet Pieter Abeel: Berkeley professor and founder of Covariant who Misha studied with A2C and A3C: Advantage Actor Critic and Asynchronous Advantage Actor Critic, the two algorithms developed by Misha’s manager at DeepMind, Volodymyr Mnih, that defined reinforcement learning and deep reinforcement learning

• 0:00Speaker 00

  I think someone needs to solve the depth problem. The field as a whole, I think, or large labs have been really working on the breadth. That's amazing, and there's a big market for that and a lot of very useful things that get unlocked, but someone needs to solve the depth problem, too.

• 0:33Speaker 01

  Hi, everyone. Welcome to Training Data. Today, we're hosting Misha Laskin, CEO and co-founder of Reflection AI. Misha is a former research scientist at DeepMind, and his co-founder Yanis was the creator of AlphaGo and RLHF lead for Gemini. Together, they are building universal, superhuman agents. We'll chat about why we're still far from the promise of AI agents, even with best-in-class models today, what we need to truly unlock agentic capabilities for LLMs, and what we can learn from those who have built both the most powerful agents in the world, like AlphaGo and AlphaZero, and the most powerful LLMs in the world, like Gemini.

• 1:12Speaker 01

  So, Misha, to kick things off, we'd love to learn a little bit more about your personal background. You were born in Russia, moved to Israel when you were one, then the United States, in Washington State, when you were nine. Your parents were pushing forward the field of technology and research in chemistry, and I think that inspired a lot of your love for pushing forward the frontier of technology and getting into the world of AI today as well. Can you share a little bit more about what inspired you to get into this field and what has inspired you throughout your childhood and adulthood so far?

• 1:45Speaker 00

  Yeah, definitely. You know, when my parents, we emigrated from Russia to Israel, it was when the Soviet Union collapsed, and they came to Israel, basically, I mean, with nothing. They had, I think, $300 in their pocket, which was then stolen from them as soon as they landed, because they put down a deposit for an apartment, and, well, that just disappeared. I don't even know if there was an apartment. So, and they didn't speak Hebrew.

• 2:21Speaker 00

  So they decided to kind of pursue a PhD in chemistry at the Hebrew University of Jerusalem. But that's not because, it's not because there was kind of some kind of internal passion for kind of academia at that time. It was more that Israel was giving stipends to Russian immigrants to get further educated. And it was interesting asking my parents about this because they kind of grew to love their craft as they got excellent at it. So I think from them, that might be the, the thing that kind of I took away most. It's not that, it's not that they are particularly like impassioned about chemistry to start. But as they kind of learned about it, got curious about it and really went deeply into it.

• 3:05Speaker 00

  I think they became kind of masters of their craft. And that's something that I found really important myself.

• 3:14Speaker 00

  Moving from there to the States, well, my parents promised that we're moving to this kind of beautiful state with all these mountains, this Washington state. And I remember like we're taking the plane flight, I mean, I bragged to all my friends in Israel and that, you know, I'm up there.

• 3:30Speaker 00

  I, I was really excited. So yeah, we're flying, I do see mountains in the distance, but then the plane does a sort of U-turn. And if, maybe you don't know this about Washington state, but it's a kind of half desert and half, you know, mountainous and forests. And the plane turned to the desert. And so I see it kind of landing in this kind of middle of nowhere. And I asked my parents, like, where are the mountains? Like, well, you saw it from the plane. The reason I'm saying this is because I basically moved to a very boring place.

• 4:06Speaker 02

  Which city?

• 4:07Speaker 00

  There's, there's this area in Washington state called the Tri-Cities. And it has some pretty interesting history. It, the reason it exists is because it was one of the sites for Manhattan Project. So this is where the plutonium was enriched at this place called the Hanford Site, which was a sister site to Los Alamos. So it's a town that was basically built for that in the 1940s. And is in the middle of nowhere, kind of like Los Alamos is, and there's not much to do. I remember seeing my first tumbleweeds. You literally saw tumbleweeds kind of rolling across the highway. I found myself in a place where I didn't really speak the language that well, English.

• 4:48Speaker 00

  I was in this kind of very rural place that was different from where I grew up, didn't have many friends, and had a lot of time on my hands. And the way I got interested in science since the time it was physics is after, you know, after getting sort of video games out of my system, I got bored again. And I found these Feynman lectures that my parents had on the Feynman lectures in physics. And they were so interesting because Feynman had this way of explaining incredibly complex things in a way that basically a person who's, I mean, not that educated mathematically at the time, can really understand something fundamental about how the world works.

• 5:28Speaker 00

  And that is probably the thing that inspired me most. I just got really interested in this idea of understanding how things, how things work at this sort of root level and working on problems that are sort of root node problems. And that, I mean, there are all these examples I was reading about, like the invention of the transistor, which was invented by Joe Bardini, a theoretical physicist, or how GPS works. It turns out you need to understand, you need to make relativistic calculations, which is coming from Einstein's theory of special relativity. And I wanted to work on things like that, so that's why I got into physics.

• 6:10Speaker 00

  I pursued it for a while, got educated in it, got a PhD in it. And I think maybe the critical bit of information that I had not, did not have in my context then was that you don't just want to work on root node problems, you want to work on root node problems of your time.

• 6:27Speaker 01

  Yeah.

• 6:28Speaker 00

  You want to work on the things that are

• 6:29Speaker 01

  You really like

• 6:30Speaker 00

  can be unlocked now. And it's no surprise, you know, when you're, when you're being trained as a physicist, you're doing like these really interesting problems and learning very interesting things about how people thought about these, you know, about physics basically 100 years ago. 100 years ago, physics was the root node problem of our time. And

• 6:50Speaker 00

  I decide, that's, that's why I decided to not pursue professionally. I kind of did a 180 and wanted to do something very practical. So I started up a startup. But as I was working on that, I started noticing deep learning as a field taking off. And in particular, AlphaGo, like when AlphaGo came out, there was something, something just felt very profound about it. That how, how do they get this system that is like a computer to not just perform at a higher level than a human, but do so creatively? In AlphaGo, there was this famous move called Move 37, where the, well, where the neural network made this kind of move that looked like, it just looked like a bad move. Lee Sedol was really perplexed by it.

• 7:35Speaker 00

  Everyone was perplexed by it. It just looked like a mistake. And it turned out that 10 moves later, this was actually the optimal move to kind of put AlphaGo in a winning position for the game. And so you could tell that this is not just, this is not just a brute force thing. This is not a thing that is just, it's obviously, I mean, the system does a lot of search, but it was able to find creative solutions that people hadn't thought of before. And so that kind of made me feel pretty viscerally that solving agency, this was the, this was the first real large scale, superhuman agent. Yeah, that, that seemed profound. That's why I got into AI and, and got into AI to build agents from the first day.

• 8:19Speaker 00

  So there was a kind of nonlinear path where, you know, I was an outsider. I wasn't really, I mean, it was competitive back then too. And OpenAI released these requests for research at the time. This was maybe 2018 or 19. Their requests for research were just things that they wanted other people to work on. I think by the time that I was looking at this list, it was actually already stale. So I don't think they really cared about those problems, but it gave me something concrete to work on. And I started making progress against one of these problems. I felt like I was making progress. I don't know how much progress I actually made. I was kind of peppering a few research scientists from OpenAI with questions and kind of, I mean,

• 8:59Speaker 00

  effectively cold emailing them until maybe I got, you know, too annoying and they started, you know, well, I guess they, they responded rather, I'd say graciously. And I built some, some relationships there.

• 9:14Speaker 00

  And one of them introduced me to Peter Ebel, who is a P I. at Berkeley, one of the greatest, I think, researchers of our time in the field of reinforcement learning and robotics. But his lab kind of does everything. They have some of the most impactful generative model research as well. The one of the key diffusion paper, diffusion model papers came out of there. And honestly, got lucky. He took a chance on me and kind of brought me into his group. He really, he had no reason to. There, after I was on the other side and looking at applicants coming into the group, there was really no reason for him to take someone who was not vetted. So he kind of took a chance. And that, I think, was my kind of foot into the field.

• 10:01Speaker 02

  You and your co-founder, Yannist, have worked on, I think, some of the most incredible projects out of DeepMind and Google. Maybe, can you give the folks here a taste of some of the projects that you both worked on, like Gemini and AlphaGo? Maybe what were the key learnings from each and how they propelled your thinking forward to the present day?

• 10:22Speaker 00

  Yeah, so Yannist was basically the reason I got into A I. He was one of the key engineers on AlphaGo. He was there in Seoul when they played against Lee Sedol. And he's actually, you know, before AlphaGo, he worked on this paper called Deep Q, Deep Q Network, is D Q Ns. And this was, this was actually the first successful agent of the deep learning era. This was an agent that was able to play Atari video games. And it kind of catalyzed this whole field of deep reinforcement learning back then,

• 11:04Speaker 00

  which was A I. systems that were autonomously learning to act in, I mean, mostly video game and robotics environments. But it was the first agent, it was kind of a proof point that you can learn. You know, to act in an environment in a reliable way from just raw sensory inputs coming in. This was, I think, a big unlock that was completely unclear at the time, the same way that neural networks working on ImageNet was an unlock in 2012. And then, yeah, Yanns worked on AlphaGo and the kind of subsequent line of work. There was AlphaGo, there was AlphaZero, there's a paper called MuZero. And I think that really showed how far you can take this idea, like it really scales. Relative to the models we have today, large language models, the AlphaGo model is actually

• 11:49Speaker 00

  really small, and it was so smart at this one thing. I think the key lessons, at least for me, from AlphaGo were kind of encapsulated in this famous essay that Rich Sutton, this reinforcement learning researcher, or I guess kind of a sort of father of a lot of reinforcement learning research, put forth, which is this idea of the bitter lesson. In that essay, he basically says that you want to, if you're building systems that are based on kind of your internal heuristics, those things will likely get washed away with systems that kind of just learn on their own.

• 12:33Speaker 00

  And the, or rather, systems that leverage compute in a scalable way. And he argued that there are two ways to leverage compute. One is by learning, so it's training. That's, when we think of language models today, they're leveraging compute mostly through learning, by training them on the internet. The other way is search, which is leveraging compute to unroll a bunch of plans and then pick the best one. And AlphaGo is actually both ideas in one. I still think it's the most profound idea in AI, that combining learning and search together is the optimal way to leverage compute in a scalable sense.

• 13:18Speaker 00

  And that, and those things together are the things that produce a superhuman agent at Go. The issue with AlphaGo was that it was only good at one thing. And I remember being in the field then, and it did feel kind of stuck, the field of deep reinforcement learning, because

• 13:36Speaker 00

  The goal everyone set out for themselves is to build general agents, superhuman general agents. Yeah. And where the field landed was superhuman, very narrow agents. And there was no clear path to how do we make them general? Because they were so data inefficient that if it takes six billion steps to train on one task, then where are you going to get the data to train on all these others? And that was the big unlock of the language model era.

• 14:08Speaker 00

  Like, one way to think about the internet is, or all the data on the internet, is a collection of many tasks. Like you have, like Wikipedia is a task of kind of describing some historical events. Stack Overflow is a task of Q and A on coding. You can think of the internet as like a massive multitask dataset.

• 14:27Speaker 00

  And that- Interesting. Yeah, and that the reason we get generality from language models is because it's basically a system that's trained on tons of tasks. Those tasks aren't particularly, let's say, directed or, you know, there's no notion of reliability or agency on the internet. So it's no surprise that the language models that come out of that aren't particularly good agents. They're, I mean, they're obviously incredible and they do incredible things. But one of these fundamental problems in agency is that

• 14:57Speaker 00

  you need to think over many steps and you have some error rate over each step. And that error accumulates, that's called error accumulation. And so it means if you have, you know, some percent chance that you're wrong the first step, that'll compound very quickly over a few steps to the point where it's basically impossible for you to be reliable on a task that is meaningful.

• 15:23Speaker 00

  The key thing that I think is missing is that- We have, language models are systems that leverage learning. They're not yet systems that leverage search or planning in a, in a scalable way. And so that, I think, is kind of the missing piece. Okay, now we have general agents, but they're, or we have general, yeah, agents that are not very competent. And so you kind of want to move up the competence. And the only existence proof for that has been AlphaGo, and it's been done through search.

• 15:52Speaker 01

  I really love that, and that encapsulation of how you just shared that. I think that sets the stage wonderfully for reflection. Can you share a little bit more about the original inspiration, this problem space that you're going after, and your long-term vision for reflection?

• 16:07Speaker 00

  I mean, the original inspiration came very much from working, Yannis and I collaborated very closely on Gemini. Yannis led the RL-HREF effort, and I led a reward model training, which was kind of a key part of RL-HREF. What we were working on, and what everyone is working on, these language models is you, in post-training, you align them for chat. So you align them to be good, interactive experiences with, for some end user. So that was, you know, through products like ChatGPT or Bard, which has now been named Gemini. You can, these language models, like the pre-trained ones, are very adaptive.

• 16:46Speaker 00

  And so you can, if, with the right data mix, you can adapt them to be like highly interactive chatbots. And I think our key kind of insight from working on that was that there's nothing specific that was being done for chat. It's just, you know, you're just collecting data for chat. But if you collected data for anything, for another capability, you'd be able to unlock that as well. Of course, it's not so simple. There are a lot of things change in the sense, I mean, one key thing is that chat is subjective. So the algorithms that you train are different.

• 17:25Speaker 00

  And the algorithms that you train for something that has kind of an objective, like this task was done or not. There are also sorts of issues, but. But the main thing was that we think the architectures and models work. A lot of the things that I thought were bottlenecks have been washed away with compute and scale. Like, long context lengths is something I thought would be, something that would be, you need a research breakthrough. And now, all the players are releasing models with extremely long context lengths relative to what we thought was possible even a year or two ago.

• 18:05Speaker 00

  The methods for training these things and aligning them and post-training them are pretty stable. And it's really, yeah, it's a data problem. And it's a data problem and a problem of how do you enable planning and search on top of these objects? And we thought we'd move faster against this problem if we did it on our own. I think we just wanted to move very quickly against it.

• 18:34Speaker 02

  So you've described agents as the dream, both for you and Yanis as researchers, but also for reflection. Can we pause on the word agents for a little bit? Because now it's become, now that it's become the term of 2024, everybody is calling themselves an agent, and the word is starting to lose its meaning a little bit. And I imagine that you have probably a more pure definition of what an agent is. Maybe, could you just explain it? How do you think about what an agent is? And when we look at some of the agents that everyone's gotten really excited about recently, they're still, it seems like they're still very early in terms of being reliable enough to be colleague-level true agents. So where do you think we are on that curve? What is an agent? And how do we get to the promised land?

• 19:19Speaker 00

  Yeah, it's an interesting question, since I,

• 19:25Speaker 00

  the term agents has been floating around within the research community for a while. I mean, I think, well, since the start of AI. But I primarily have been thinking more agents in the context of the deep learning era, so starting with DQNs. And the definition is pretty simple, in that it's

• 19:53Speaker 00

  It's an AI system that is able to reason on its own and take however many steps it needs to, to accomplish some goal that's been specified to it. That's kind of it. And now, the way that goal is specified has changed over time. In the deep reinforcement learning era, the goal was usually specified through a reward function. So for AlphaGo, it's whether you won the game of Go or not. There's no, no one wrote like, "Go, win the game of Go" like via text. So that's how people usually thought about agents. They thought of agents as things that are optimizing a reward function. But there's a whole area of research, even then before language models, on goal-conditioned agents.

• 20:37Speaker 00

  So these would be either in robotics or video games, or you set a goal for the robot, which could be, you give it an image of, you know, an apple having been moved somewhere, and you ask it to kind of reproduce that image. And it has to act on the world and pick up an apple and move it somewhere in order to accomplish the goal. So, as a short definition, it's AI systems that have to act in an environment to accomplish some goal. That's an agent.

• 21:02Speaker 02

  And then, I guess as a follow-up, if you take, for example, coding agents as one potential domain of agents where there's been a lot of activity recently, you could say the goal is, you know, create a calculator app for me.

• 21:18Speaker 00

  Yeah.

• 21:19Speaker 02

  And the agent has to accomplish the task. When I look at what, you know, Sui Agent and what Devin have done, is that, in your mind, is that agentic reasoning? And does kind of scaling that up get us to the promised land? Or do you think there are kind of different approaches that you need, more on the RL side, or more on whatever other techniques you might need to use to get us to the promised land? Because I think those agents are still in the 13%, 14% kind of task completion rate range, and I'm curious how we get them to 99%.

• 21:51Speaker 00

  They're definitely, by the definition of agents, these are agents. They're just on a spectrum of capability, maybe not at a high level of reliability yet. I think the way most people think about agents today in context of language models is prompted agents. So- You take a model, you prompt it, or you set up some flow of several prompts to get it to accomplish a task. That allows anyone to kind of take a language model and take it from zero to something that's kind of working somewhere. So I think that's quite interesting. I think it can only go so far. So this, this I think is actually like a very, I mean, this is a kind of example of what I think the bitter lesson would apply to.

• 22:35Speaker 00

  Because prompting things and kind of really directing them to go in like these specific ways. That's exactly the kinds of heuristics that we have that we're baking into these models to kind of try to achieve higher intelligence. I mean, every major advance in agency since the deep learning era

• 22:54Speaker 00

  has been kind of showing that with learning and search, a lot of that gets washed away. I think that the purpose of a prompt is to specify the goal. So you'll always need a prompt. You always need to tell an agent what to do. But if, once you start deviating from that and the purpose of the prompt is actually to put the agent on rails and, and you know, where you're kind of doing the thinking for it, right? You're kind of telling it, okay, now just go here and do this thing.

• 23:26Speaker 00

  That I think is going to disappear. I think that's a local thing that's happening today. Future systems, I don't think will have that.

• 23:35Speaker 02

  So the key is that the thinking and the planning needs to happen in the AI system, not in the, the prompts layer in order to kind of not hit a wall.

• 23:42Speaker 00

  I think you want to offload as much as possible to, yeah, to the AI system itself. Again, these, these language models have not been, they're, they're never trained for agency. They're trained for chat interaction and predicting things on the internet. So it's almost a miracle that you can prompt your way to getting something that kind of works. But what's interesting is that, once you're able to prompt your way to something that kind of works, that's actually the best place to start for a reinforcement learning algorithm. A reinforcement learning algorithm, all it does is it reinforces good behavior and it kind of downweighs bad behavior. If you have an agent that is doing zero, that is just doing nothing, then there's no good behavior to upweigh. And so the algorithm doesn't work.

• 24:25Speaker 00

  This is known as a sparse reward problem. If you're not hitting your reward, like if you're not accomplishing your task than ever, then there's nothing to learn from. But if you've prompted your way to an agent that is kind of working.

• 24:39Speaker 00

  Like, you know, Sui Agent or something like this, it's getting 13% or something like this. Then you have something that is like minimally capable, where you can reinforce really good behavior. The challenge becomes a data challenge of where do you, where do you get the set of prompts to train on? Where do you get the environment to run these things through? Boom, I guess Sui Agent does come with an environment, but for many problems, like you need to think about that. And then perhaps the biggest challenge is how do you verify that a thing has been done correctly or not in a scalable way? And if you can solve, you see where, where the tasks come from, which usually that's through products that's solvable.

• 25:24Speaker 00

  What environment you run them through, what algorithm you use, but it's really kind of what environment you run them through. And then critically, how do you verify if a thing has been done correctly or not in a scalable way? I think that's a recipe for agency.

• 25:39Speaker 01

  I think that gets to the crux of the problem space in AI agents today.

• 25:45Speaker 01

  Just to set the stage a little bit for the problem that reflection is going after. What do you think is the current state of the market broadly in AI agents? I think many assume that we are capable of more than we actually are with the models that exist today. So what do you think the problem is? And what do you think is, why do you think the current attempts around AI agents are failing us today?

• 26:09Speaker 00

  One way to categorize or classify what it means to be a general agent. And maybe I'll use the term universal agent, since I'll use the term generality to apply to breadth. So a universal agent needs to be a broad, a very general agent that can do many things, can handle many inputs. But it also needs to have depth in the kind of task complexity it can achieve. And so examples are AlphaGo is probably the deepest agent that we, that has ever been built. It can do one task, so not, not that useful.

• 26:46Speaker 00

  It can play Go, but not tic-tac-toe.

• 26:48Speaker 01

  Yeah.

• 26:51Speaker 00

  The current systems, language model systems like Gemini, Claude, ChatGPT, the GPT series of models, lean the other way. They're very broad. They're not very capable in a depth-wise sense. They're extremely impressive and capable broadly. And I think that's one of the things that's been honestly miraculous. Like, as I said, the field felt like we did not have an answer to generality. And then these objects came along.

• 27:23Speaker 00

  But now we're in the opposite end of the spectrum where we have, I think, more or less de-risked as a field, progress towards breadth. That's especially evident with the latest generations of models like GPT-4O and the latest family of Gemini models that are multimodal in the sense that they just understand other modalities at the same base layer that they understand language. You don't need to translate one modality into language. So that's, I would call that breadth.

• 27:56Speaker 00

  But nowhere along this process were things trained for depth. There's no, the internet doesn't have real data around how to kind of think sequentially.

• 28:08Speaker 00

  The way people try to solve this problem is like work on data sets that might have this structure and hope it generalizes. So math data sets, coding data sets, kind of what people refer to reasoning, which usually is reasoning along the lines of can you solve a mathematical problem?

• 28:27Speaker 00

  But that's still not really addressing the problem head on. I think we need methods that you can take recipes, let's say, that are general in that you can take any task category, have a bunch of prompts for it in terms, for your training data, and make a language model kind of iteratively capable, more capable on those things.

• 28:54Speaker 00

  I think someone needs to solve the kind of depth problem. And the field as a whole, I think, is, or large labs have been, have been really working on the breadth that's seen, well.

• 29:08Speaker 00

  That's amazing, and there's a big market for that and a lot of very useful things that get unlocked, but someone needs to solve the depth problem, too.

• 29:16Speaker 01

  I think that takes us really nicely into the unique insight that you and Yannis have from working on AlphaGo, AlphaZero, and on Gemini, and the importance of post-training and data. Can you share a little bit more about how those experiences have shaped the unique perspective that you have that gets us to the unlock with the Genti capabilities?

• 29:38Speaker 00

  One of the things I found very surprising about language models is how close they are to, you know, how oftentimes, even if they're not working on something that you want them to, they're actually quite close. They feel like a nudge away. They feel like they need to be grounded in the thing a bit better. And that's, I think that was the insight that led them to be good in chat. Like, you could play with them and they're, yeah, a bit unreliable and they kind of go off the rails sometimes, but they're almost good chat companions.

• 30:09Speaker 01

  Yeah.

• 30:10Speaker 00

  And so then there's a recipe for how do you take a pre-trained language model and make it a reliable chatbot. So by reliability there, it's just a, the way you measure that is with human preferences. Do people interacting with this chatbot prefer it over other chatbots or other versions of it, the previous versions of itself? So if the current version is much more preferred than the, you know, last few iterations ago, then you know you've made progress. And that progress is made by collecting data for it. So it's collecting data for the kind of, you know, queries that users input into a chat box, the outputs that the

• 30:52Speaker 00

  models provide, and an effective ranking between those outputs so that you push the model to, to, you know, index over on the kind of more preferred outputs. So when we say ranking, where does that ranking come from? Well, it comes from humans. So there are either human labelers, or it's something that's embedded into the product. You sometimes might see thumbs up or thumbs down in ChatGPT. It's harvesting your thumbs to kind of know what your, what your preference is. And that data is used to kind of align the model with the user preferences. That's a very general algorithm. That's a reinforcement learning algorithm. And that's why it's called reinforcement learning from human feedback or RLHF.

• 31:33Speaker 00

  You're just up weighing the things that human feedback is expressing preferences for. There's no reason why the same approach would not be possible for

• 31:46Speaker 00

  for enabling more reliable agency. There's a whole sequence of other problems you need to solve. I think that's what made, the reason this is so hard is because as soon as you go into kind of agent territory, you have more than just the language outputs. You have the tools that they interact with. And, you know, the tools being suppose you wanted to send an email or like work in an IDE or like anything that an agent does, it does in an environment and that requires tools. That requires the environment. And everyone who's deploying agents is deploying agents in different environments. And so there's a challenge of how do you integrate with environments? And how do you onboard agency onto them?

• 32:27Speaker 00

  So I think that's why it's a bit of a schlep if you get into this kind of line of work. And you have to be careful about the environments and kind of, and yeah, the way you structure it. Cause you, you don't want to overfit to some, you know, some particular like environment. But conceptually, it looks very similar to aligning a model for chat. There are just some more integration challenges that need to be solved along the way.

• 32:56Speaker 02

  Since you view AlphaGo as kind of like the pinnacle of building an agent that was, you know, truly capable. I imagine you're trying to usher in an AlphaGo moment with LLMs. What do you think are the differences? Like to me, you know, with gameplay, you have a very clear reward function. You have the ability to self-play. Like is doing kind of the reinforcement learning from human feedback? Do you think that's enough to kind of get us to an AlphaGo moment in LLMs? Or like, I guess, how should I think about the differences here?

• 33:27Speaker 00

  I think what you said around not having a ground truth reward is a key and maybe the key thing. What we learned from the previous era of reinforcement learning research is that if you have a ground truth reward, you're Kind of guaranteed success, like that's kind of um. There have been so many very impressive um projects that showed this at really unprecedented scale. I mean, you think aside from AlphaGo, there was OpenAI's Dota 5 or AlphaStar. And let's say AlphaStar and Dota 5 are a bit more niche in the sense that you kind of have to play those games to understand. But as a former StarCraft player, I was, I still am completely blown away by AlphaStar.

• 34:12Speaker 00

  Like, the strategies that the AI discovered were, it just looked like a very smart, like a smarter-than-us alien game. Like, upon the earth, decided to play this game and completely outcompeted us.

• 34:27Speaker 00

  So that's due to the existence of a number of things, but a ground-truth reward is really, like, is extremely important for tightening that behavior.

• 34:38Speaker 00

  Now, both with human preferences and for agency, we don't have, we, these are very general objects, and we don't have ground-truth rewards for whether something is accomplished or not. Like, even for a coding task, what's the ground-truth of whether this was done the right way? Like, it could pass some unit tests, but it can still be wrong. So it's a really hard problem. And I think it's the, it's, I think it's the fundamental problem for agency. There are others as well, but this is kind of the, the big one.

• 35:10Speaker 00

  The way you get around this problem for chat, this again through RLHF, well, you train reward models. Reward models are, it's a language model that predicts whether something was done or not correctly. The challenge with that, so first, it works well. The challenge with that is that when you don't, in the absence of a ground-truth, when you have this kind of noisy thing, that it can be wrong,

• 35:38Speaker 00

  your policy, or kind of basically the agent,

• 35:44Speaker 00

  Quickly get smart enough where it finds holes in the reward model and exploits them. To give a concrete example in chat, suppose you notice that your chatbot was outputting, you know, was outputting some, let's say, harmful content, or like there are some topics that you don't want it to talk about because they might be sensitive. And so you put in some data into your data mix saying, like where it's examples of the chatbot kind of ignoring, or not ignoring, but saying, I'm sorry, as a language model, I cannot answer this. What can happen is that, okay, you now train a reward model against this. And suppose in your data mix, you really, you only put in like data points that, you

• 36:24Speaker 00

  know, showed, showed instances of like this kind of, this happening, but not instances of a chatbot taking something like, kind of sensitive and actually like, you know, answering it. What that means is that the reward model could think that it's actually like a good thing when you just don't answer the user's query ever, because I've only seen positive use cases of that. And when you train against that, the policy with language model will, at some point, get smart enough and discover that this reward model gives me high reward whenever I just don't answer, whenever I punt the question. And it can collapse into a language model that just never answers your questions.

• 37:06Speaker 00

  And this, this is why it's, it's, it's very finicky and it's very difficult for this, for this reason. I'm sure a lot of users who have interacted with chat GPT or Gemini or these kinds of models, like probably through, through interacting with them, found sometimes that they kind of degrade and they all of a sudden like, don't answer questions, you know, as often as they used to, get slightly worse at something, or, you know, are politically biased in some way. And I think a lot of that is, well, it's artifacts of the data, but the, but the artifacts in the data get amplified by, by bad reward functions. So.

• 37:50Speaker 00

  That is the hardest problem, I think.

• 37:51Speaker 02

  If I view the rough, kind of, large model training pipeline, or large AI system training pipeline as pre-training and post-training,

• 38:02Speaker 02

  I kind of think, like, pre-training seems largely like a solved, like, we're in the, you know, the techniques are solved and we're just kind of in the, the race to scale moment on the pre-training side. Post-training still feels a little bit like in the, kind of, research phase of market where people are still figuring out what techniques will work in a general way. I'm curious if you agree with that. And in an ideal state, like, what is pre-training responsible for doing? How should we, as laymen, think about it? And, and what is post-training responsible for accomplishing? And how should we think about that from the perspective of a five-year-old?

• 38:37Speaker 00

  Yeah.

• 38:41Speaker 00

  I would, I would generally agree with that statement, that pre-training has become... There are a lot of details that need to get right, and it's, and it's by no means easy. So it's, it's a very hard endeavor. But it's a better-understood endeavor at this point.

• 38:57Speaker 00

  And one way that I think about pre-training is, I actually think thinking about it through the lens of something like AlphaGo

• 39:09Speaker 00

  is, is quite simple and clear because it kind of, you know, rather than thinking about this massive internet thing, you just think about a very clear setting, which is a clean setting, which is this game.

• 39:22Speaker 00

  You can think about the pre... AlphaGo has two phases. It has an imitation learning phase, where a neural network imitates a bunch of expert, amateur, like, expert Go players. And then it has this reinforcement learning phase. And you can think about pre-training as the imitation learning phase of AlphaGo. You're just kind of acquiring the basic skill of learning to play the game. You're not, maybe, your, your neural network then is not the best in the world, but it's pretty good. It goes from zero to pretty good. And pre-training for a language model is getting, going from zero to pretty good on everything, which is why it's so powerful. Post-training is...

• 40:04Speaker 00

  I think about it as hardening good behavior. What that means is with AlphaGo, you start off with, you did imitation learning. You start off at a place where you have a neural network that can do something. It can, I mean, it can play a game pretty well. Then you apply this other recipe to it, which is reinforcement learning, which is then the network starts generating, you know, its own plans and kind of acting through the game, getting feedback, and that could, and basically good actions get reinforced. That is, I would say that that's post-training, and you can think about, from a chat perspective, you're hardening the model, like, the good behavior along the chat axis.

• 40:46Speaker 00

  So it's actually quite interesting that the high-level recipe for training AlphaGo and for training Gemini is actually the same. You have this imitation learning phase, and then you have a reinforcement learning phase. The reinforcement learning phase in AlphaGo is just much more sophisticated than what we have now, and the reason comes back to reward models. If you have a reward model that is, you know, that is fairly noisy and exploitable, then there's only so much work, there's only so much you can do before the policy gets smart and finds a way to trick it. And so even if you threw the fanciest, like, RL algorithm at it, like, Monte Carlo tree

• 41:29Speaker 00

  search with AlphaGo, it may not be that effective because it kind of, it collapses into this kind of degenerate state where the policy's hacked the reward model before it could even do any interesting search. Like, suppose you're, like, thinking about, like, if you're playing chess and you're thinking about what to do multiple moves ahead, but your kind of judgment is really bad at every move, then, then there's no, like, there's no point of planning 10 moves ahead. And I think that's where we are with RLHF today. There's this wonderful paper that I think is Very over or underrated. Called Scaling Laws for Reward Model Over-Optimization.

• 42:12Speaker 00

  This is a paper from OpenAI studying this phenomenon. And what's interesting about, I mean, a number of things, but it showed that this phenomenon happens at all scales. And I mean, it tried a couple of different RL, HF algorithms, and it happens for, at all scales for all algorithms that were tried in that paper. And I think it's such an interesting paper because it's the, it's the kind of fundamental problem of post-training. That's, it's that paper.

• 42:40Speaker 02

  Yeah. Just to pull on the thread a little bit, if you follow the results from AlphaZero, though, then we may not need pre-training at all? Is that, is that a fair conclusion of what to make of this?

• 42:53Speaker 00

  I think that the, at least my mental model is that the AlphaGo part, the imitation learning, is necessary. More from a practicality standpoint. When, when we went from, or DeepMind went from AlphaGo to AlphaStar, there was no AlphaZero of AlphaStar. There's no AlphaStarZero that was, you know, released after that or anything like this. And AlphaStar had, like, a big part of it was imitation learning across a lot of games. I think with AlphaGo, it was like this kind of special place where you have, you don't

• 43:35Speaker 00

  only have a zero-sum game, but you can get to the end of that game like fairly quickly. And so it's again, like, you can get that feedback about whether what you did

• 43:44Speaker 02

  Yeah.

• 43:45Speaker 00

  was right or not.

• 43:46Speaker 02

  Got it. Okay, so it's just way too unconstrained of a problem to throw it out generally.

• 43:50Speaker 00

  Yeah, I think in practice, like, yeah, AlphaZero would work generally for everything if we had ground-truth reward functions for everything. But because we don't, you need to, you need to do the imitation learning piece as almost, this is just like a practical. Like, we need to get into the game somehow.

• 44:10Speaker 01

  You described earlier the importance, from a technical perspective, of having an agent in its environment. Also, from a product distribution and getting the product in users' hands perspectives, it's important to think about what the right task categories are for users to first interact with the most powerful agents. What are some of the task categories that are on your mind? And what do you imagine are some of the possibilities that users could use these in their daily workflow?

• 44:37Speaker 00

  If you want to make progress along the depth axis, you could go for like, alpha go first, which is like a really hard thing. Or you could kind of expand concentrically in the sort of complexity that of the tasks you're able to handle. And we are focused on kind of enabling depth, but in this sort of concentric way. And we care a lot about having a general recipe that is not, that does not kind of, you know, inherit heuristics that are special to some tasks. So, from a research perspective, we're building general recipes for this. Now, you have to ground those recipes in something to show progress.

• 45:21Speaker 00

  And at least for us, it's important to show diversity of environments. And so, we're thinking about a number of different types of agents. Web agents, coding agents, kind of OS computer agents. The important thing for us is to just show that you can have a general recipe for enabling agency.

• 45:51Speaker 01

  Switching gears a little bit, you've attracted a seller team already. Who else are you looking to recruit on your team?

• 45:59Speaker 00

  Yeah, we've been fortunate to be able to draw some talent from the top AI labs in the industry. And...

• 46:13Speaker 00

  I think a lot of that has to do with,

• 46:17Speaker 00

  well, with both of the work that Yanis and I did. But definitely, you know, I think a lot of credit goes to Yanis and his reputation. You know that there is this, I was watching the Michael Jordan documentary. And Michael Jordan was, well, one of the reasons he was so effective is because he was such an incredible kind of individual, like basically contributor to the game, maybe the best,

• 46:44Speaker 00

  that he really inspired people on his team to get to his level, even if they couldn't get there.

• 46:53Speaker 00

  And Yanis has this effect on people, like I've worked very closely with him on Gemini, and he had that effect on me.

• 47:00Speaker 00

  I don't know if I ever got to Yanis' level, but I aspired to, and I definitely became a much better engineer and researcher through the process, and I think that's

• 47:13Speaker 00

  a lot of the draw is that, you know, you get to learn a lot from him. We're primarily continuing to look for, so we're not hiring out quickly. We're hiring out, I think, more methodically.

• 47:32Speaker 00

  We're looking for, yeah, other, definitely interested in other researchers and engineers joining, joining us on this mission.

• 47:43Speaker 00

  I'd say a commonality between everyone who's joined is we're all, we're all very hungry. Maybe that's how I'd put it, you know, we could have, Yanis and I could have stayed and tried pushing agents, you know, at DeepMind. And as I said, I think, you know, the reason we decided to do it in our own way is because we, we think we can move quickly, much faster against this goal. And some of this urgency is driven by a, a real belief that we are, we are three or so years away from something that

• 48:20Speaker 00

  resembles a digital AGI. And by that, that's a, that's what I've been referring to as universal agent. Something has both this kind of breadth and depth of knowledge. And that means we're actually on a very accelerated timeline. Yeah. You're, you know, a few months in, you're kind of 5% away from, from hitting that timeline. And maybe Solis is also driven by how quickly AlphaGo went from

• 48:53Speaker 00

  experts in the field doubting this is possible. Yeah. Things kind of decades. Human level or like expert human level Go play was decades away. And how effectively they were able to solve that problem within months. I think we're seeing a similar kind of acceleration happening with language models. There's, I think, one viewpoint you can have is that we've saturated a lot of what we can. We're on the sort of, at the tail end of an S curve. And we don't, we don't view it that way. We think we're still, we're still on an exponential. Part of the reason is that these things are so bulky and slow to train

• 49:33Speaker 00

  that there's no way collectively as a, you know, as a field of researchers and engineers that we've optimized it yet. Like, if it takes a few months to run, and like a few months and a few billion dollars to run the biggest model, then how many experiments can you really run? So, yeah, we, we kind of see things going at an accelerated pace.

• 50:03Speaker 00

  And we think solving the depth and reliability problem is something that is not getting.

• 50:10Speaker 00

  The kind of attention it needs. Like, there are groups that are following this as, I would call it, more like side quests within these big companies. But I think you need a player that is focused entirely on it to solve this problem.

• 50:25Speaker 01

  Yeah. I love the framing of main quest versus side quest. And I love the hunger and the zero complacency and impatience in a healthy way that you and the rest of the team have. And the other thing I'd highlight is the revered reputation that you described for Yannis with inspiring and motivating other people, I think is true for you and Yannis from everyone we've known, we know, in deep mind.

• 50:51Speaker 02

  So, three years until I have an agent that will write my memos for me?

• 50:54Speaker 01

  Hopefully, yes.

• 50:55Speaker 02

  I have three years.

• 50:56Speaker 00

  Yeah. I think the memos might be coming, you know, sooner.

• 51:00Speaker 02

  Because that was one of my burning questions. Is this like decades away? Is this months away? It sounds like you're closer to the, you know, months or small number of years away.

• 51:10Speaker 00

  I think small number of years.

• 51:11Speaker 02

  Wow.

• 51:12Speaker 00

  Yeah.

• 51:15Speaker 00

  Yeah, it's honestly kind of alarming, I think, the speed at which the field is moving.

• 51:21Speaker 00

  And part of, yeah, part of depth and reliability, it's also like, it is, I mean, reliability is safety. So, you want to, you want these systems to be safe. I think that there's a lot of very interesting research in terms of, there's a recent paper from Anthropic on kind of mechanistic interpretability, and that whole line of work is really interesting, and I think starting to kind of get to the point where there's utility in it as well in terms of finding like neurons in the model that are like, you know, lying neurons or, you know, that you can kind of suppress. But...

• 52:02Speaker 00

  To me, safety is reliability. If the thing is kind of running around your computer, breaking all sorts of things, that's an unsafe system. Maybe it's like a utilitarian safety, like you just want these things to work and do what you intended them, but what you asked them to do.

• 52:19Speaker 02

  So I have a few years to find another hobby, other than my memo writing, then.

• 52:23Speaker 00

  Yeah, well, or maybe you'll just have an army of AI interns that, you know, will do all the research work for you.

• 52:32Speaker 02

  Can't wait.

• 52:35Speaker 01

  Wrapping up our topic around reflection, if everything goes right, what is your dream for reflection?

• 52:41Speaker 00

  I think the, there are two angles of this question. One is, we're, we're working on this because this is the kind of scientific, root note problem of our time. We're scientists, that's, that's why we're so kind of interested and committed in it. And it's really, there's a world where you get to be part of one of the most exciting journeys in science ever made. And, and you've, you know, accomplished your goal of building a universal agent. You have highly safe, reliable digital agents running around on your computer, basically doing, doing things that, tedious work that you don't necessarily want to do.

• 53:26Speaker 00

  You kind of, I think, rather than people kind of going and, you know, spending less time working, I, I, I don't think the human need, or like, the human need to be productive and to contribute is going to change. I just think the capacity of each human's ability to produce and, you know, impact the world is, is going to dramatically increase. As, you know, in, in my line of work there, as a researcher, there are so many things that I spend time on that a smarter AI could help me out with to make faster progress towards our, towards our own goal. I mean, this is kind of a circular, but if we had something close to a digital AGI,

• 54:08Speaker 00

  we'd get much faster to solving the problem of digital AGI. That's one angle. I think the, the other angle is from, I guess, I guess we've kind of moved on to the other angle, it's from the user perspective of, a lot of the things that we do on a, on a computer are

• 54:29Speaker 00

  You know, are, maybe you can think about like computers like the first digital tool that we've been introduced to as a, you know, as people, in the same way that there were, you know, hammers and chisels and sickles that people used. And I think we're moving towards the kind of layer beyond that, where instead of you having to kind of learn how to use all these tools with great precision and spend all your time on this, which actually is kind of time taken away from achieving, you know, whatever personal goals people

• 55:05Speaker 00

  have, that you kind of have these incredibly helpful AI agents that can help you bring kind of any goal that you have to fruition. And I think it's very exciting because I think the kind of ambition of our individual goals is going, is, it's already increasing, you know, in this local sense, like a software engineer can get a lot more done today with these tools, but this is just the beginning. And I think we'll, yeah, we'll be able to really set dramatically more ambitious goals for ourselves and for kind of these sorts

• 55:47Speaker 00

  of things we want to achieve, simply because we can offload a lot of the work that's needed to get there to, through these systems. So these are some things I'm really excited about.

• 56:00Speaker 02

  Well, close it out with a few questions that we like to ask everybody about the state of AI. First question, what are you most excited about in AI in your field or more broadly in the next one year, five years, and 10 years?

• 56:14Speaker 00

  I think there are a number of things that, the one, the local one that comes to mind is because the paper is fresh, is this kind of work on mechanistic interpretability in that, I mean, these models are largely black boxes and

• 56:36Speaker 00

  It's unclear. It's really unclear how to study them as like, what's the neuroscience of language models, like if you think about them as brains?

• 56:43Speaker 01

  Yeah.

• 56:44Speaker 00

  And this seems like a really interesting line of work that is now starting to see kind of signs of, well, signs of it working beyond toy settings.

• 56:57Speaker 01

  Yeah.

• 56:57Speaker 00

  So maybe like the sort of neuroscience of language models is, I think, kind of a really interesting.

• 57:02Speaker 01

  Awesome.

• 57:03Speaker 00

  Field in AI to get into. And more generally, you know, if I was in academia, I'd probably be looking a lot at the science of AI. So this is, the neuroscience of AI is one thing, but there are all sorts of, all sorts of things one can investigate in terms of, well, what really determines the scaling laws that these models have, both from a theoretical perspective and from an empirical perspective, how you change data mixes. Maybe kind of taking a step back, we're, we're basically in the like equivalent of what the late 1800s looked like for physics.

• 57:48Speaker 00

  Electricity was being discovered. No one knew why it worked or how. There's a lot of like, there are a lot of empirical results, but there was no kind of theory behind it, which just meant that they were not very well understood. And then this like very rich set of theoretical models were developed that were very simple to understand these phenomena. And that gave rise to actually basically the next wave of empirical breakthroughs. And so I think the science of AI is kind of in that state right now, and I'm very excited to see where that goes.

• 58:30Speaker 01

  So interesting. Who do you admire most in the world of AI?

• 58:36Speaker 00

  I think most people when getting this question, or some people might kind of

• 58:44Speaker 00

  put someone, yeah, I mean, maybe I'll take that back and say like, I want to emphasize people that I admire

• 58:56Speaker 00

  based on having worked with them and kind of see how they operate because through my last kind of, you know, a number of years in AI,

• 59:05Speaker 00

  I think, yeah, there are a handful of people like this who've inspired me. And one of them is,

• 59:15Speaker 00

  so Peter Abiel is certainly one of them.

• 59:20Speaker 00

  He is,

• 59:22Speaker 00

  I think, I've never seen anyone, I think, almost operate as efficiently as Peter. That was something that, and to date, like since meeting him and to date, like, I think there's sort of, you think a lot about research as a creative pursuit oftentimes. And I think what I learned from Peter is just sort of operational competence and efficiency around this. He's very creative as well, and like his lab does a lot of creative work, but I think it's brought, like, these things are very hard and they need to be pushed hard and with with great focus. And he,

• 1:00:05Speaker 00

  he ran his lab, like it's the tightest trip that I've ever been on. And really helped focus like

• 1:00:14Speaker 00

  all the projects. And so, yeah, I think I look up to him a lot, both in terms of his,

• 1:00:21Speaker 00

  the work that he's done, obviously, it's remarkably cross-field.

• 1:00:27Speaker 02

  Yeah.

• 1:00:27Speaker 00

  Like he's both done, well, incredible breakthrough work in reinforcement learning and unsupervised learning, generative modeling. And and a lot of it has been, I think, from kind of recognizing and enabling like talent.

• 1:00:41Speaker 02

  Yeah.

• 1:00:41Speaker 00

  So there is this very much, it was like, the group was, it was a bunch of independent thinkers, students, PhDs, and people are kind of

• 1:00:54Speaker 00

  Pursuing like what was interesting to them, but I the way I saw like Peter is a sort of like great amplifier.

• 1:00:59Speaker 01

  Yeah.

• 1:01:00Speaker 00

  Like he kind of helped people amplify and focus on the thing that really mattered within within their pursuit. I think a few other people that come to mind. So my my manager at DeepMind, his name is Vlad Mnie. He's, yeah, I think also like a very incredible, very creative scientist. He was first author of the DeepQN paper. And then there's actually there are actually two papers at the time. There's like this A2C, A3C papers. These are, these are basically the two algorithms that defined reinforcement learning. And he have a deep reinforcement learning and he kind of really pioneered both.

• 1:01:43Speaker 00

  Yeah, I think his strength is like he was

• 1:01:49Speaker 00

  extremely kind and people oriented, very humble despite his accomplishments.

• 1:01:55Speaker 00

  Yannis as well, I mean, definitely, I mean, Yannis has the the Michael Jordan effect. I think he really, yeah, he just, he just wanted to be the best you could be when you work with him. And there are our L T F team was like quite small and people pushed really hard in order to like largely I think inspired by by him. Yeah, these are some people I I really look up to.

• 1:02:20Speaker 01

  Thank you for sharing that.

• 1:02:23Speaker 01

  It's so interesting to hear you say that about everyone and one comment on Peter Beal.

• 1:02:29Speaker 01

  I tell him all the time that he's also just creating a mafia of founders in the last couple of years, and it's probably because he's taught them how to do many things. And there's a self selection that's naturally happening, the creative thinkers and the independent thinkers who come into his lab. But he's also taught them a lot about how to run a tight ship and how to focus incredibly well. So I'm sure that doesn't come without intention on his part.

• 1:02:53Speaker 02

  Maybe last question, any advice you have for founders building in AI? You are, you're just starting your journey right now, and I'm sure you've asked others for a lot of advice. What advice would you pass on to the next generation?

• 1:03:06Speaker 00

  I think, one, I think I'll be in a better position to answer that question in a few years in a way that is much more meaningful. But I'll actually provide a piece of advice that I lived through through my previous startup.

• 1:03:21Speaker 00

  Which has nothing to do with AI, to to just work on things that are like internally like really matter to you. In a way that is almost independent from what's happening around you. Like, in a way that when things go bad, like it's still interesting to you. Like, it's kind of, there's just some fundamental drive around this problem. That independent of everything else that's happening is just really interesting to you.

• 1:03:47Speaker 00

  Maybe I say that about AIs because there's this.

• 1:03:52Speaker 00

  It's such an interesting, highly capable, cool technology. And so there's this sort of, I think, appeal of taking it and just kind of like, oh, let's just see what we can do.

• 1:04:05Speaker 00

  I think you inevitably find yourself in a hard place without having a very strong internal compass of independently of AI. Like what it is that's important to you and what you want to do. And so having been in that position previously. That's kind of what I would have done differently and what I would advise people to do.

• 1:04:26Speaker 01

  I really love that. The line that I like to think about is play in your own stadium. And don't get distracted by the glitz and glamour of someone else's stadium. You need that internal drive and grit and obsession with the problem to get you through all the tough times.

• 1:04:41Speaker 00

  Yeah, and I think there are things that come with it. Like, if you really care about some problem, like you will care about the customers who you're solving the problem for. Like, like having customers that you don't care about is like a terrible place to be. So I think, yeah, it has to come. And it's not like, I think it's kind of actually hard to control like who you care and who you don't care about. That's like a personal thing. So you can't, it's actually really hard to force yourself to like care about something out of necessity. If it's kind of not aligned with something inside you already.

• 1:05:16Speaker 01

  So no more shopping and retailers for Misha.

• 1:05:19Speaker 00

  Yeah, so you know, I was building software for kind of inventory prediction for retailers. And for some people.

• 1:05:27Speaker 00

  You know, they'd really care about that problem. Like, there's a reason they would. They've kind of seen that problem. They've, you know, maybe if you're a merchant at like, at, you know, one of these retail companies, you've really felt it viscerally.

• 1:05:41Speaker 00

  And in my case, you know, I hadn't. It was like, we were trying to make a, you know, just trying to build, build a revenue generating business, almost kind of independent of like, an internal compass.

• 1:05:55Speaker 01

  Misha, thank you so much for joining us today. You are working on the most ambitious problem of our time. I love your framing of the root node problem of our time. And I think that is today, agents. And it's very clear that both your and Yanis' experiences make you the very best team at what you do. Yanis, obviously, from a RLHF perspective, and yours, from a reward model training perspective, and the insights and experiences that you've both had, working on AlphaGo, AlphaZero, and Gemini, we're so excited for the future of Reflection.

• 1:06:32Speaker 00

  Yeah, thank you for having me.