Ep 87: Gemini Co-Lead on World Models, RL's Next Domains & Continual Learning

Oriol Vinyals, VP of Research at Google DeepMind and co-lead of the Gemini program, joins Jacob the day after Google I/O to unpack the research underpinning Google's latest announcements and where frontier AI is heading. The conversation moves from world models (why Google has uniquely bet on them as a path to AGI, what the "GPT moment" for video and images would look like, and how they connect to robotics and simulation) to agents (the Spark release, why the system and model need to be optimized jointly, and why scaffolding will eventually be written by models themselves). Oriol gets into the mechanics of memory in models, drawing on his cognitive neuroscience background to argue that file-system-style non-parametric memory is more practical than baking memory into weights at serving scale. He shares his views on the limits of RL today (LLMs are data-limited in a way that game-playing RL never was), why training on narrow domains like math and code generalizes surprisingly well, and what a true "Move 37" moment for science or ML research would look like. Throughout, he reflects on the unique advantages of being inside Google (TPU co-design, end-to-end revenue stability, the merger of Brain and DeepMind), the trade-offs between focus and exploration in research orgs, and why he believes AGI in some meaningful sense may already be here, even if the goalposts keep moving.

• 0:00Speaker 00

  Oriol Vinales is the co-lead of Gemini alongside Noam Shazer and Jeff Dean. He's had an incredible career in AI, pioneering many of the breakthroughs in deep learning in the last decade. And it was a ton of fun to get to sit down with him after Google I/O. If you've been following Google I/O, they basically shipped a bunch of products across a ton of interesting surface areas throughout AI. And so Oriol and I hit all of them. We talked about what's required for further advances in multimodal models and what's going to make these world models actually usable. We talked about the increase in memory and the importance of memory and how the advances there will look like reasoning these next few years, as well as what Oriol thinks the path forward is. And we hit on the state of scaffolding today, what folks are building and what Oriol thinks persists.

• 0:44Speaker 00

  It's a ton of fun to get to basically take all the top questions that founders, investors are thinking through and just pose them to Oriol. So I think folks will really enjoy this conversation. Without further ado, here he is.

• 0:57Speaker 00

  Oriol, thanks so much for coming on the podcast.

• 0:59Speaker 01

  Yeah, it's great to be here. Thanks, Jacob.

• 1:01Speaker 00

  Yeah, very exciting to have you a day after I/O. I know things have been, have been busy, but I've been really excited for this because you're one of the people kind of most directly shaping the frontier of, of models today in your work at Google. And you obviously, in the releases that happened yesterday, I/O, they hit on like pretty much all the themes that people are thinking about in the, in the space, where these products and models are going. And so I feel like there's just, our goal today is to, to talk through kind of the research behind those announcements. You know, where, where this is all headed, you know, the kind of future path of RL and post-training and, you know, get your read on the, on the space as a whole. I figured where I'd start was with world models because I think that was just a really impressive part of, you know, of yesterday.

• 1:43Speaker 00

  And also I think a pretty, you know, where Google's pretty distinct from a lot of the rest of the field. So you obviously shipped this incredibly impressive world model in Omni yesterday. And, you know, I think Demis has talked a lot about, you know, seeing world models as a path to AGI. And it's interesting, right? It seems like other labs maybe are more focused on code and, you know, getting to recursive self-improvement. And so I'm wondering if it's a fair characterization and, you know, why you think, you know, you and the team in Google have been somewhat uniquely focused on this world model space.

• 2:11Speaker 01

  First of all, I guess the coding or self-improvement angle is it at a bit of a different layer? Right? So you can certainly bet and believe that, you know, these models can reprogram and improve themselves. And it's something I've been actually quite actively working on at the moment. But then the object that they improve, the model, whether it's multimodal and closer or a word model, as we call it. And that even how to define that is a bit abstract. Since day one and way before actually Gemini program started, we were working on not just language, but you know, understanding the visual world and kind of jointly modeling words in the context of vision, blank video, etc.

• 2:56Speaker 01

  So I think that part, you know, it's been at the core of Gemini and before our research. And I think maybe one way to characterize it is, you know, language, clearly there's a lot of information collectively that we wrote about the world. And so that's clearly paid off big time. And we've kind of distilled in a way all the knowledge written and that is being written at the moment into these weights.

• 3:22Speaker 00

  It was definitely convenient that we put it all on the internet, too.

• 3:24Speaker 01

  Yes, exactly. Right? So and also like this now with users, right? There's obviously like also a flywheel effect. But at the same time, there are, there is lots of knowledge in videos and images. And what I would say it kind of has happened, but softly, I think there probably might be a big moment is how would you extract all the knowledge that you would acquire if you were to look at all the videos and images, which we certainly use right in our, in our training mixtures. But could that knowledge somehow add value and efficiency to the language component? And I think we've seen constructive sort of.

• 4:07Speaker 01

  Let's say, transfer learning, right? From one to the other, we see that and we see generalization. But probably what I would characterize as the GPT moment of video and images. I'm not sure we quite have seen that.

• 4:21Speaker 00

  Do you have any like thoughts on like what that GPT moment might be for for for video and images? As you as you kind of have this intuitive feeling that it hasn't yet been reached?

• 4:30Speaker 01

  Yeah, so at the moment, we train all the modalities, we mix them and we keep enhancing the recipe. So Omni is a good way to see that progress in which we not only input videos and images. We've seen amazing capabilities with long context, understanding, et cetera. But we also now are able to output, you know, video, but also interact with it in a very natural way through language. Editing it, combining, you know, the modalities in a way that feels almost almost magical. Right? So that progress is absolutely there. But maybe one of the, you know, deep learning dreams, and it might be an original kind of dream from way before

• 5:10Speaker 01

  large language models would be, hey, can I train on all the, you know, image data without text, perhaps, as a hard challenge? And still somehow extract all the all the meaning and nuance from from that modality or set of modalities and vast amounts of data, right? So could we train on all the videos ever produced and images? And get to the same level of understanding that clearly the language models using language get to, although probably slightly superficially and some missing links with cause effect and so on, that, for instance, Demi talks about often, right? So that is the moment that have I seen that? Probably not, and most likely we have the most advanced

• 5:53Speaker 01

  or one of the most advanced, you know, multimodal recipe that mixes everything. But that pure transfer is, I think, one of the core quests of machine learning for the last decade plus.

• 6:05Speaker 00

  I mean, to the extent you you can talk about it, I'm curious. Could you give our listeners some context on like, what are still the key problems that need to be solved around this? Or as you think about like the the kind of, you know, the the types of problems that you're you're trying to, you know, work on to further advance this?

• 6:18Speaker 01

  It's hard to describe like the the the solution space, but the idea of, um, you know, you could imagine observing or learning from all the video data and then somehow deriving, you know, the rules of gravity is one that is used often, right? Like, how could you precisely describe how the world works based only on images, right? And so the the issue there is linking language or these concepts, as we sometimes call them, to what you see in the image without the explicit language linkage is fairly tricky, right? So so what you end up doing is trying to explicitly create data sets where there's some sort of correlation or connection between the

• 7:03Speaker 01

  images and video and some language, like maybe it's labels or descriptions and so on. But of course, the amount of data now at your disposal is much less because we haven't clearly described and transcribed every single piece of media out there. So I think that's kind of extracting that those concepts in the purest form, not in just some language that we associate to the words and what we see would be very, very powerful. And there's lots of early research on discrete representations, representation learning. And I mean, that's one of the things that probably I would say is in fairly research stage. So it's not something we can possibly scale up. But I think that's one of the possibly, I'm not sure it's needed. I mean, whether we agree with that

• 7:46Speaker 01

  or not, it's another question. But it's if it was to be unlocked, it would be massive.

• 7:51Speaker 00

  You mentioned kind of this the term world model and how it's thrown around a bunch. And, you know, obviously, you you kind of Omni was was positioned as a world model. And I'm curious, you know, how you thought about that categorization versus, you know, you obviously had really good video models for a while, right? What makes Omni like a world model? And, you know, how is it different from kind of the the generation of video models that you guys have been working on?

• 8:10Speaker 01

  I guess a pure aspect of world model would be representation learning, right? So so you could imagine we take these modalities like the the videos, which are like a sequence of sequences of images or even just images, and then compressing that into sort of a set of concepts and what that those, you know, the movements, the objects, etc., are within those, that's kind of called representation learning, and it models the world in a very compact way that compresses away what's probably not relevant, right? So

• 8:45Speaker 01

  Probably that one is the more classical, but also probably not exactly what we mean or we see or we feel when we interact with Omni. Right? What you see there is a bit more about

• 8:58Speaker 01

  you being able to really change how the video behaves or the kinds of videos you're getting out of an initial maybe image that you ask to animate, you explicitly ask all the movements or even like actions that would be like move forward and you can see that being kind of precisely simulated. And so that is more of like the world model itself is acting as a renderer of the world that you can really just change by a language. And then having that now object, besides being a cool product to play with, of course, like we love to generate, you know, all sorts of different, you know, movements or situations and so on very richly.

• 9:41Speaker 01

  It could also meaningfully add maybe a dimension of simulation that could make us, you know, use, for example, things like prediction before acting in the world. And of course, obvious applications for for these kind of 3D or video world models would be clearly, you know, self-driving cars or robotics.

• 10:04Speaker 00

  It seems so relevant to robotics and it feels like, you know, everyone's kind of still trying to figure out the right data mix of simulation data, you know, versus versus forms of teleop data and egocentric video data. But it seems like as these simulations continue to get better, you know, it's more and more of a compelling thing to put in into the data mix. And I'm curious, like, you know, does this work then directly intersect with, you know, the broader robotics work you all are doing? And how do you think about what's actually required to, you know, append robotic actions onto, you know, these types of models?

• 10:33Speaker 01

  There is a bit of also. Beautiful connection, because of course if we acquire, even if it's obviously a bit more expensive or time-consuming, but if we get more data that is captured from robots that we certainly are investing in, you know, that data could make it into the model, enhancing the world model capabilities themselves. And then the other direction, which is kind of what you're asking about perhaps is, OK, now we can simulate and we could create lots of different scenarios in which these robots or, you know, whatever, 1D, 3D groups, et cetera, could be training on without the cost and the time latency of the physical world, right?

• 11:18Speaker 01

  So for the latter to work better, I mean, it's still a very open problem. There's also all sorts of issues with transfer. But the more powerful these models get, clearly there's kind of an inflection point where things start to be worth doing and we might see an acceleration in robotics indeed. You know, definitely we're seeing in the hardware space lots of investments. So things are accelerating and picking up there. But for the world models to be useful, at least from my limited knowledge, but of course I've, you know, I've been able to interact with these systems and see them, the precision of even grasping a model which we get for granted as humans, the visuals, the exact, you know, how would this feel to your hand,

• 12:03Speaker 01

  which is a modality we currently obviously don't even have data for. And then the exact forces, how things would move, it needs to be very, very accurate, right? So that's where there is a gap and perhaps then some creativity and research is still required and lots of investment in robotics over the years. But it's promising and at some level, maybe not at the precise. Motor control, but at the kind of planning and gross level, we are gonna start seeing how these models accelerate our progress into the quest of robotics.

• 12:37Speaker 00

  A huge part of these models is kind of like, you know, learning, implicitly learning physics through, you know, consuming lots of video data. And so I think you mentioned gravity is like the canonical example of what people look for. Do you have any kind of gut sense being so close to these models of like, when you think that will just be a solved problem within, within world models?

• 12:53Speaker 01

  Yeah, it's a good question. Actually, you're kind of, you made me think about evaluation, right? Like, how would you evaluate if you train a very good, you know, video?

• 13:01Speaker 00

  Yeah, how do you evaluate physics in a model?

• 13:03Speaker 01

  Yeah, it is a good question, right? You could imagine, the problem is as soon as you add language, all of a sudden, that knowledge is there in the weight. So if you ask basic questions about gravity, of course, you would answer them by just having read, you know, explanations of them online and so on. So you would need to somehow connect the, the concept of gravity, which could be present or not in a world model, to then decoding that into an explanation that would satisfy, you know, maybe initially would be some basic explanation later on, could even derive like the, the equations and so on. That's how can you, you could build an EVA.

• 13:43Speaker 01

  I don't think, to my knowledge, we, we've been thinking about this from this point of view. There's definitely lots of early work on unsupervised machine translation, where you, you would try to translate to a language that you would never see during training, and you, you could align the representation. So there's probably some ideas on, you get a language model that can speak, or you can decode from, you get these world models that would create this kind of conceptual, conceptual level understanding and aligning both. You know, there's some papers, I mean, these are like old papers, the one I remember from, I think it was Stefan Gauz and et al. It was from 2014.

• 14:24Speaker 01

  But then you could try to start decoding that and converting that to an EVA seems then, you know, a trivial step. But again, these EVA's need to then be meaningful from like an application point of view. So, so ultimately, you could also say, I mean, we have a world model. Can we decode or I don't know like induce movement in a complex system from its representation, for example, right? That would be another indirect eval. So many ideas, but yeah, EBS are so important always.

• 14:51Speaker 00

  You know, shifting gears to some of the other stuff that you you all shipped yesterday. You know, I definitely want to talk agents. You shipped some really interesting consumer agents in Spark, right? As part of as part of I O. And, you know, I think it's so interesting because it seems like from the outside, at least, like a really improved version of some of maybe the stuff you guys had explored in Project Mariner in 2024 and some of this other like computer use work. And so it does feel like there's been a real step change in what the capabilities are. And so I'd love to hear you just riff on, you know, the research breakthroughs that enabled that. And then kind of how people should think about what these agents can and can't do today.

• 15:26Speaker 01

  We knew that was going to be a very important modality actions, right? Acting and changing a state of, let's say, a digital computer. And then I think the as you evolve and make the model better, you start realizing sort of out, you know, you get the model really good. And then you focus on the system, building a system around the model, then optimizing the system and the model jointly as much as you can and so on and so forth. So in terms of what creates the delta or the increase in capability, it's mostly focused on about it's about sequencing sort of releases. And also, in some sense, the model capability needs to reach certain,

• 16:09Speaker 01

  you know, level for you to then be able to dream about what's the next stage of capability, what the model might do next.

• 16:15Speaker 00

  Yeah, I guess, you know, one thing that's so interesting about the consumer footprint is like there's just such a broad array of things people want to do with it. And so, you know, I wonder like to date and how you see this evolving over time, that work of model plus system, how like bespoke is it to subcategories of the problems people want to do versus, you know, incredibly general and like, hey, you're just optimizing a system and model combination that works across pretty much everything you might want to do in Spark.

• 16:39Speaker 01

  There's always a sequencing to specializing to something that feels controllable and that, you know, very useful already. Where, you know, if you look at Spark, I mean, it has access to, you know, information that it would be needed for it to be able to assist you in sort of scheduling and organizing your day and even thinking about how how you should, you know, tackle different problems because it has this very rich context.

• 17:07Speaker 01

  So it's useful to build kind of the system slightly more narrowly around something you care deeply about. But if you if you look at the history of machine learning and deep learning, we always go from, you know, the components we're building are general. So so and and there's a big hypothesis, which goes a bit to the world model point, actually, that training on everything jointly must be better than just focusing narrowly on just one domain. So even from the modeling perspective, that is very clear. But even from the system's perspective, a system that is fairly generic and then based on how you instruct it or you interact with it, you can then, of course, put it in the

• 17:50Speaker 01

  space of like, hey, I mean, this user wants to do this, but I have all these capabilities. Let me just figure out which ones to use kind of at train time, not necessarily building it for that, but building something generic. And then the specialization happens through a layer of intelligence, right, of the model and the generality of the system. I think that's fairly clearly, you know, already here. And then maybe sometimes in practice, you know, limiting or making it maybe more efficient still makes sense to specialize. But the the special leads to to general, we've seen it just keeps happening from even architectures, right? I mean, the transformer was a machine translation, you know, that, right? And now it does everything from Omni to, you know, controlling your computer.

• 18:35Speaker 01

  So, yeah, I think that's that's a step that I expect.

• 18:39Speaker 00

  You've been vocal about the bitter lesson over the years. And I'm curious, like, as you look out at the field, are there places where you think, like, it's not currently being followed or, you know, basically places where you look out and you see kind of structure or clever scaffolding that you think scale is just kind of eventually going to wash out?

• 18:53Speaker 01

  Yeah, I think I think so. I mean, one one area that I find exciting, there's some research on this already kind of published is that. You know, in the limit, the system that we built now sort of by coding, sometimes a complex scaffold around the model, you know, multi-agent, sub-agents, delegation, very long running. That system itself is a piece of code that eventually the model itself could write on the fly. So you could imagine not having just a system that is very general, but actually maybe no system and just the model being able to write those depending on what is it's being asked to do.

• 19:38Speaker 00

  Like almost the most token efficient, like highest quality output set of sub-agents and whatever it is around around a set of problems.

• 19:46Speaker 01

  Yeah, exactly. I mean, we've seen this also in the kind of one of the paradigm shifts we've seen in the last one year and a half or so is, of course, the reasoning models that, you know, can can reason for a long time in token space. But of course, eventually what becomes more important is should you reason for how long should you reason? And adding that level of intelligence based on the complexity of what a user might be asking will make it more efficient. So I think what you do this, what you do around these systems, there is going to be a level of not sure exactly if it's going to be right to hit from scratch or some automation that will make it smart to create the right scaffold for the right task.

• 20:28Speaker 00

  On the agent side, you know, I think there's a lot of, and everyone's messing, you know, and experimenting with building these kind of long-running agents. And I think, you know, obviously they run into all sorts of issues trying to get them stable across hundreds of steps. How do you think about what's required to get to like further agentic reliability?

• 20:44Speaker 01

  Yeah, I mean, I think that the answer to these questions in the most obvious way is is kind of improving both the scaffold around the model. If you think of how you train a neural network, it trains on some distribution of tasks or modalities or, you know, how to connect different words to video or whatnot, right? All these are all about how you train or pre-train or post-train these weights. So if you think, well, there's a new type of work or modality that requires these very long-running, you know, systems that need to somehow.

• 21:23Speaker 01

  Learn from this very long context, which we have also always kind of innovated and pushed in 1.5 was kind of our long context breakthrough. Then then it becomes obvious that the model also will catch up, right, to to to meeting the users and the futuristic use cases. And that's a bit of the researcher challenge, right? Predicting what can be possible and then focusing not only on building a system that is robust to that, but also how how would the weights get less unhappy or happy about when you push all the contexts and all these crazy things that you you do and not just hoping on generalization from the prompt that induces that behavior, so to speak.

• 22:06Speaker 00

  A pattern everyone's trying to figure out is like memory, right? And how to kind of like solve this across across these agents. You have any any like gut instincts on on where you think that ultimately gets solved?

• 22:16Speaker 01

  Yeah, I mean, memory is fascinating. I think since very early days, right? You you you can sort of think of, I think, I think initially we characterize this, and this is probably through biases from actually Demis having had a PhD in memory systems in the brain, right? But, you know, there's, there's a few ways to think about memory. But I mean, the simpler one that I like is, you know, just working memory, right? Things that are very present because of what we're doing or we're talking about. And then what's called episodic memory, that is kind of kind of a retrieval system that you can access. And it's probably less precise, but of course,

• 22:59Speaker 01

  longer context or potentially has all the context that, you know, you or I care to remember, right? Holistically with all our experience that is accumulated. Now, there's not only two levels of memory, but it's useful to think about these kind of levels of memory. Computers have the same with cache L1, L2, and so on. So so when it comes to models, I think working memory. Because of transformers and so on, we have a very powerful mechanism to to kind of use that memory, you know, have hundreds, thousands, millions of tokens at our disposal to modify that memory and then do amazing things with it, proving complex theorems, you know,

• 23:44Speaker 01

  gold medal level maths and so on. And I think where I'm seeing a lot of momentum is through then, how can we consolidate then things that happen either previously in different interactions or throughout an interaction that might be longer than you could possibly remember in this working memory? How do we store that knowledge? And, you know, through different experiments, I think other, you know, like I think now that the standard name is called like what we call skills, but but more general than that, we do have access because it's an agent to a memory system, which is the computer itself. So you can start thinking about writing, you know, your thoughts into files,

• 24:28Speaker 01

  structure it into directories or folders, and doing that as you either interact with the same user multiple episodes or a very, very long episode. So the mechanism that is fairly good at the moment, but again, I don't think the weights of the model have caught up to this, is this adding this kind of knowledge base into a file system or, you know, any format that is storage that you can modify and read from with some basic retrieval mechanism. So that's very powerful already, yet I think there's still a lot to be untapped there. I think many, many of us call this kind of a form of continual learning. But I think the mechanism I wanted to work kind

• 25:12Speaker 01

  of, or I mean, it's going to clearly work better and better, is this kind of file system style, like nonparametric. It's a bit more convenient than integrating those back into the weights,

• 25:25Speaker 01

  Even from a practical point of view, we try to serve one model at scale. So it would be really painful to have to serve one model with different memories to users. So even from a practical point of view, I think we'll see better evaluations and ways in which these models accumulate this knowledge as they interact. And I think that's probably paradigm shifting as well in a way, similar to how we saw reasoning a year and a half or so ago.

• 25:54Speaker 00

  Does that look like everyone having models that then have their own, the file systems themselves being distinct, or do you think over time people have models whose weights look different based on what they've done?

• 26:03Speaker 01

  Well, as I said, the weights different would be a challenge.

• 26:07Speaker 00

  Yeah, hard to serve.

• 26:08Speaker 01

  Yeah, it would be, I mean, if it's the best way, then we'll find a way, right, to have hardware that, of course, we have lots of investment as well on hardware design that would allow you to have more personal weights, so to speak. But at the very least, of course, you will have your own knowledge base that is maybe personal to you. You're seeing already like many examples of this realized over the last, maybe even years in LLM space. And then perhaps there's another layer of knowledge which is more common to all the users for a given model that you could imagine having access to and enriching or enhancing the model capabilities without touching the weights.

• 26:49Speaker 01

  So that's very interesting and getting to that would be awesome.

• 26:54Speaker 00

  I feel like continual learning has been the topic de jour and everyone's talking about it. And you've seen kind of a few interesting examples now and high profile examples of folks spinning out of OpenAI or other places and saying, hey, sure, you can keep scaling what we're doing now. And I think that no one's denying that those scaling laws are there, but they're saying it feels like you need kind of almost a new research bet to achieve like real continual learning and maybe it makes sense to pursue that outside of the kind of path of continually improving these core LLMs. I'm curious what you make of that whole dynamic and yeah, maybe your reflections on that.

• 27:30Speaker 01

  I was in Google Brain very early days and then moved to DeepMind in 2016. And at the moment, I think the

• 27:39Speaker 01

  You know, there is a challenge and an opportunity on to, you want to obviously have, investigate some research questions that might not be ripe for, hey, in the next three months, this makes it into the next training run. But at the same time, this cannot be very disconnected from the head, right, where the LLMs are moving. I mean, we're improving Gemini. I mean, it's fascinating to see Flash outperforming Pro of only a few months ago, and that keeps happening. So keeping kind of at the head of capability, which might enable or disable certain research, whilst having the protection for research. And of course, that's not multi-year anymore. Things are moving fast.

• 28:23Speaker 01

  But but kind of combining these two is kind of the magic of building these organizations. And all of us, of course, have different angles and have, you know, can kind of figure out how to bridge this and identify the opportunity. That's a bit of what it takes, right, to, I mean, not have full visibility. This is too large of an organization. But have some intuitions and then be able to pull in these ideas eagerly sometimes, right, because it feels like the right thing to do. So that's really what defines actually organizations at that level, right, from a research perspective. So I can see from investment in robotics to, of course, the peak of the LLMs, to research that either has made it or will make it through, right?

• 29:08Speaker 01

  So but it's challenging. It's, Yeah. It's resources are constrained, so it is an interesting trade-off. And not one always you get right, but I mean, I think it's, it's a fascinating kind of different angle of research. Not just what is the idea that will make it to the next paper or now into the model, but actually how to even organize this whole. Yeah, whole organization. It's fascinating.

• 29:32Speaker 00

  And this feels like one of the most interesting questions for someone in a role like yours, where it's hard not to feel excited about the like so many things that you can advance with these models today. And there's obviously so much going on. And I feel like, you know, even take an organization like OpenAI, they've kind of oscillated between like, hey, we should go, there's just so many low hanging fruits and things to go do on the AI side to, you know, now this kind of more focusing moment where it's like, God, we've just got to really nail code and catch up to Claude code. I'm wondering how you think about the trade offs of like, you know, focusing on the one thing and then, you know, having the all rowing toward that versus maybe a broader surface area, all of which are super interesting.

• 30:08Speaker 01

  You know, Google is in the unique place for a couple of reasons. First, we indeed have a lot of surface area in Gemini at the moment, right? This literally powering everything. But we have the advantage that it's already like people, like the other parts of the organization are completely bought into the LLM era. So in a way, they take the model and then they might do something. But if you feel like that's not the next way to advance frontier capabilities, then, you know, you can just rely that there's a very good group that will take the model to where it needs to go, right? At the same time, we

• 30:47Speaker 01

  have stability from hardware procurement and obviously like also investment of capital given like it's we're very end to end in terms of, you know, revenue streams and so on. So you can probably push a little further the risk taking for certain research areas, which which needs to be done with taste as well. So you have kind of this, it's not focused, but it's scalable because of how Google is organized. And then you can still invest in innovation, which is at the very core of what we've always done, right? Like, I mean, if I look at Brain and DeepMind, the two organizations, I've been part on Of like now called Google DeepMind, which is, which I appreciate given I've obviously been in both

• 31:32Speaker 01

  over different periods of time. Then I think, you know, like there is in our DNA to keep innovating. But at the same time, I think what Gemini created is focused and unifying force, which was fascinating to do. It was very helpful that me and Jeff had known each other for many years and had gone on trips together like just for fun, right? So I think that that time though was very special and I think that's the center being the Gemini kind of core modeling effort. Being very focused on frontier capability and then having kind of these inputs and outputs is a fairly, you know, reasonable way to, to, um, to go about, um, being focused, but also being able to leverage, right?

• 32:15Speaker 01

  A bit of exploration, uh, which might still be needed or not, right? I mean, I, I think, do we need world models? I mean, if we make it work, definitely we'll need it. If we don't, maybe it's okay, you know? But it's good to have the bets as well placed rightfully.

• 32:30Speaker 00

  On the model side, maybe switching gears to, to kind of just Gemini models and, and, you know, the, the path forward. You know, I think you called post-training before still kind of a total greenfield. And I feel like what we've seen, you know, clearly, uh, there's incredible progress on post-training in RL in, you know, coding and math. I think there was just a new math problem solved hours before we, we came on this podcast. What everyone's trying to, you know, figure out, and I'm curious for your intuitions is the characteristics of like the next set of domains, where we'll see, uh, RL really take off. It feels like we're on this crazy exponential path on the, on the coding math side and, um, curious for your intuitions on, uh, on, on what makes other domains, you know, good fits.

• 33:07Speaker 01

  Yeah, I mean, it's a good question. I, I, I mean, one must be quite humble in terms of the models are really good at many things. So, so it's very hard to say.

• 33:18Speaker 00

  Insanely good.

• 33:18Speaker 01

  Oh yeah, you know, like, yeah, this, this doesn't work at all, right? Like, I mean, almost via prompting and a bit of like, you know, smart prompting, maybe building the right system.

• 33:30Speaker 01

  Lots of amazing things, at least on the digital world, as I call it, like digital AGI, if you will, are very impressive. Right, so I think there's, when I said that post-training is greenfield, I think that's less about like a capability that I feel is kind of very far from being, you know, at the level that is acceptable from a, hey, like this is fairly intelligent and fairly advanced, and more about just mechanistically looking at how some other efforts that have leveraged imitation learning or pre-training plus post-training, right? Like, and how much investment there has been compute-wise in post-training versus the,

• 34:12Speaker 01

  relatively speaking, smaller amount that today's models use currently. And I mean, the reason there is kind of clear, and I'm not sure it's easy to fix, but the fact that even if you take a very narrow domain like Go, as you play the game of Go in reinforcement learning, right, you have now a system that can play it, it places a few moves, and a few moves into the game, that scenario, that game is now unique. I mean, you've never had seen that particular configuration. So the environment's complexity, as you play, makes kind of training data infinite for free, right? Like, I mean, you play a few moves, now you're in a new situation, and so you can learn from it.

• 34:57Speaker 01

  And the more you play, the more hours you put into your RL algorithm, the more knowledge you get, right? So that is what we've seen kind of in the game reinforcement learning era. And in LLMs, we are data limited, and it's what is the source of infinite complexity is not so clear. I mean, there are some ideas, but I think cracking that recipe could be big, at least in terms of the beauty of the algorithm. It would be like, it would be much more satisfying knowing how this has worked in the past to see it work now in LLMs. Now, is it needed? Are the capabilities not there? That would be hard to say.

• 35:41Speaker 01

  But since you ask about which capabilities, I think the capabilities in terms of what the models do that are most fascinating to me is like, I call these meta capabilities. They're not math or coding. They're like, kind of, what are the traits or attributes of intelligence? And can these models do it? Right? So actually, the ability to continually learn or learn from experience very efficiently, that would be one, you know, in context learning, we used to call them meta learning, whatever, right? This is a capability that I can sort of measure or feel, and probably it's not super good yet, right?

• 36:20Speaker 01

  For example, of course, instruction following is a capability that you could argue is the ultimate capability, because if I ask a model, BAGI, it either follows that instruction or doesn't, right? So, but I mean, trying to look at these capabilities that are less about one particular domain or, you know, vertical and more like, okay, that is intelligent behavior. And so the ability to learn and adapt rather than the ability to, you know, be a professional player or IMO gold medalist or whatnot, is what I think fascinates me the most when I look at new releases and models that we are getting our hands on to every time we train a new model, etc.

• 37:04Speaker 00

  Do you have a go-to way to like test that?

• 37:06Speaker 01

  I like games, so I usually my define a kind of a new game in context, right? Or, you know, this, this is a fairly classic way to do it. Of course, you need to be careful, because if the game is in the weights.

• 37:20Speaker 00

  If anyone else has put that game on the internet, you're in trouble.

• 37:23Speaker 01

  Yes, but I remember, I think there was an EVA. That's not exactly how I do it.

• 37:27Speaker 00

  Yeah, actually, I realize I'm being rude by asking you to talk about it, because then this podcast will be out there, and then the next models will know how to do it, no problem.

• 37:34Speaker 01

  Yeah, maybe, yeah. Hopefully, we, unless, we need to crack world models, right? Unless it's, you know, fully transcribed, which I'm sure it will be, so maybe we don't even need that. But I really like an EVA. I think this EVA is actually very old, and, I mean, way older than like LLMs. It must be like, let's say, 2015 minus, like probably before 2015. And the EVA was simple. You give the instruction manual for, I think it was Civilization, the game, and then... You know, you're meant to be able to play, right? So I like that style of EVA. Like, you know, you can kind of create this differently. But that's one test that I like to test the models.

• 38:17Speaker 01

  And they're not that good, especially as the game has become either something I just invented and whatnot. And the ability there is dual, right? Like, you could imagine first, can you understand the instructions? And from there, follow the instructions to play the game. But there's another aspect, which is as you play the game, you learn to play it better. So can you, do you see that happening in practice? And it's impressive. But again, if you go very out of distribution of a game that could be real, but still not in the training.

• 38:52Speaker 01

  This one in particular is not an easy test for the models to pass. For example, right? There's many others, but this one I really like. And it brings games in a way that is useful. Yet you will not train on the game at all. It's not about go where you only train on go. It's like the opposite. But I like this kind of thinking for a capabilities point of view.

• 39:11Speaker 00

  I mean, obviously, it feels like there's been a lot of effort. There's, you know, games were kind of the canonical first example of like a verifiable domain. And, you know, you've had this with coding now and math. And I wonder, I feel like the big kind of outstanding question in the field is the extent to which we'll see like generalization across RL, right? It feels like sometimes these models will climb incredibly well on the domain that we're arling on. And, you know, you'd have a better insight into me into whether you see that then, you know, flow through to other aspects of the model. But like, you know, it feels like in some ways it's almost an interesting, you know, as we talked about, you know, the most general bitter lesson type moments. This is a moment of like, you know, find data in a particular domain, RL against that data and like improve the model on that one thing. I'm curious, does that feel like a fair characterization of what's happening today?

• 39:52Speaker 00

  Or, and, you know, yeah, have you seen kind of signs of that generalization?

• 39:57Speaker 01

  Yeah, you look hard for sources of hard problems that will induce indeed either deep reasoning that we see generalization from actually. So reasoning models. Reason mostly through, you know, let's say coding and math, but then you see how how they reason about, you know, a question about whatever like, you know, I just recently moved back to the US. So I asked a lot of questions about moving and like taxes and whatnot. And you can see the reasoning is is pretty good. And that's a very hard hard to believe that there's it's been, you know, trained on on that kind of question. So we're seeing definitely generalization and you're you're creatively trying to get,

• 40:41Speaker 01

  you know, more data that induces, you know, deep reasoning and also deep indeed agentic behavior. Right. That's part of like the, you know, the recent the recent improvements that we're seeing is just finding those those sources. But being limited to just verifiabilities is is definitely unsatisfying because most of the times for the things I want the model to do, I would not even be able to write a verifier if I had all the time in the world. Right. So so I think, but but it feels like that there is a bit of an asymmetry between creating the solution and evaluating the solution. And even evaluating the solution is indeed simpler than creating the solution,

• 41:25Speaker 01

  which arguably, if you think of some arguments on, for example, you know, NP hard problems, which are very hard to create solutions for, but trivial to verify. It gives me hope that the models themselves will be able to judge, even if there's no, you know, fully verifiable way to judge, you know, whether a piece of code creates a beautiful game or an engaging game, you know, all these kinds of things. And I think that's a very interesting research and also in practice, seeing lots of impact there already from these kinds of ideas. So the more we do that, the more we can train on more domains. The question is, do you even need that? Or is just focusing on certain math and coding problems enough

• 42:10Speaker 01

  to induce this meta capability of being intelligent at problem solving? Right. I don't know. I mean, I think it could go either way. Do you have a gut instinct one way or another? I I want to believe like you you need to train on a broad distribution and that should help the model. But It is very strong how much you get generalization, possibly through pre-training. So maybe it depends on the level of ambition of superhuman or what's the upper bound that these models can achieve. But ultimately, I feel like training kind of as much in distribution as possible seems desirable in machine learning. So that's one of the quests for researchers to crack in the next few months and years.

• 42:55Speaker 00

  One thing a lot of our listeners are thinking through that are founders or building companies is figuring out the extent to which they should be doing work at the model layer versus purely building the application on top. And so I'm wondering, there's obviously been a trend of some companies are doing their own RL on top of models and saying, hey, there's this specific class of problem we can go solve. Or even, obviously, maybe most notably, Cursor is kind of in the coding space, been like, we need to go train our own base model. I'm curious your intuition on like, when, if that does make sense or when that might make sense and when it doesn't.

• 43:29Speaker 01

  What I would tell folks is the value, and we discussed this a little bit, the value of evaluations and as a sequence of data, basically. Like, those two are very tied to each other. There is a huge amount of value there. So no matter, even if you don't build your own model because maybe it's very early stage or you just can't get access to the talent, resources, all the things, thinking very carefully about how to evaluate progress on whatever the thing you try to do will be actually very valuable, right? And something that might even become a standard eval that folks like ourselves might even adopt or monitor.

• 44:14Speaker 01

  And of course, the value of data is immense, given what we were discussing about post-training in particular and the scarcity of enough data to be able to run these kind of months of go-training that we happily did like a few years ago. So I would say that's where the opportunity is. And I know there's a lot of energy in the space as well in terms of people that are building. At the same time, I think building on top of a model, even though the model capabilities will keep moving. And again, not being obviously an investor, a professional investor or product person actually, but even just focusing on something that you truly believe in

• 44:59Speaker 01

  might create kind of some opportunity for you to have that space at your disposal, understand it, you know, get the users, get critical mass. And if it's something that others are not focused on, big players, let's say, I feel like there's a lot of value to be created by specializing even the product, even if you don't do any of the other things.

• 45:22Speaker 00

  No, it seems almost like certainly in the early days, you specialize the product, you know, build on top of the models, get to some level of scale, learn the evals. And I think a lot of these companies are starting to try and figure out like, do we, you know, do we then use that to, you know, to post-train a model or to, you know, to do something. And obviously, the trade-off of that is as these models generalize and the capabilities improve, they're never going to, you know, be training across the broad swath of things that like the largest labs do. And so you're probably in a treadmill of every two, three months. Even if you get slightly ahead of state of the art, you probably have to keep constantly redoing it.

• 45:56Speaker 01

  Yeah, perhaps the angle here to, and again, another topic we discussed, right? As these models are more capable to continually learn or use a knowledge base that is possibly very complex, then, you know, building that knowledge base for a certain application can also be, is not like training weights, it's a bit more efficient. But there might be a lot of kind of uniqueness that you can add to it that maybe just make protect you, right? From let's say, someone who hasn't spent a lot of time to think carefully about how that interacts with current models. And that capability will only get better. So perhaps that angle is a bit more scalable as well for kind of a bit of early players in the game.

• 46:40Speaker 00

  I mean, I guess obviously it seems that there's such a compelling path forward on so many of the research directions we've talked about. What's the capability you're like leash sure how to get to from here? I guess, you know, where you maybe don't yet see the research path, but what you think is pretty important.

• 46:53Speaker 01

  I think I see the research path for quite a few capabilities. I mean, the one that's fascinated me the most over the years, especially when I joined DeepMind in 2016, meta-learning or the ability of the models to learn. That is, I mean, that's such a beautiful capability since you work on machine learning. So that one is one that I feel like there's a path and there's some base like now and it will keep improving. But perhaps another one that I feel it might be a bit more, I mean, there might be a path, but I'm not sure how practical it is at the moment is, I think people mention, hey, like, can these models truly innovate?

• 47:38Speaker 01

  And I think that part is important because, for instance, when you work on, hey, can you come up with new ideas in machine learning? And then we implement them, coding is excellent, deploy them and so on, right? We're experimenting with this. You know, many folks are quite a bit. Truly taking all the knowledge we have now and innovating with taste is something that is hard to come by, even for humans is fairly special. And to be honest, sometimes random. It's not like, oh, this person is so smart. Look, I mean, you just 10,000 people are trying and you obviously pick the one that was right and then glorify it, right?

• 48:18Speaker 01

  So I think that ability is probably quite important for certain things like self-improvement. And yet, I mean, it's obviously difficult to try to even evaluate. And when something is hard to evaluate, it probably means it's hard to also heal climb on. So the ability to innovate in any aspects, but specifically on science, for example, is a good one that I think more progress is required.

• 48:48Speaker 00

  I mean, obviously, I feel like Move 37 was a canonical example of this in the previous world. Like, is there anything you've seen recently that feels closest to this? I mean, I think even before we started recording, I guess, you know, OpenAI talked about this kind of combinatorial geometry problem they just solved.

• 49:02Speaker 01

  If I look inward to machine learning, there is That's kind of the point. I don't think I've seen truly outstanding ideas that a model has generated yet, but I am sure I will very soon because there are some insights and ways in which the models understand how, let's say, a model is being trained that feels superhuman because mechanistically these models have access to a bandwidth of information we don't. So maybe that part has been impressive, but I would like to see it at the ideal level as well, that level of impressiveness. And machine learning is the obvious thing I can

• 49:43Speaker 00

  Yeah, of course.

• 49:43Speaker 01

  more accurately evaluate, right? So yeah, more to do.

• 49:48Speaker 00

  Yeah. How do you reason about like, you know, when we get to this level of genuine insights into into machine learning research and kind of this like world of workers' self-improvement? I'm curious how you reason about like what that even, you know, or how you even think about what that looks like over time. And, you know, even just like basic questions like, does the bitter lesson still hold? Or like, you know, what happens when like we get into that world? I'd love to hear you just riff on that.

• 50:12Speaker 01

  There's certain efficiency level that probably will be enhanced. So I mean, there's a level in which you as the researcher or engineer use these tools to enhance your own productivity. We've seen that a lot.

• 50:27Speaker 00

  No, it's always impressive to talk to someone at the cutting edge of their field. And, you know, they're always like, you know, yeah, the numbers always vary, but some pretty large percent improvement in productivity across the board.

• 50:37Speaker 01

  Yeah. So I think that one is already happening and obviously very powerful, but there's going to be certain, you know, almost physical limitations to how much this process can keep going, right? Because, I mean, models need to be trained, there's energy, hardware limitations. So.

• 50:57Speaker 01

  I definitely, I'm very keen to see what kinds of problems that are to be kind of more, let's say, automated and enhanced can be done more autonomously. But at the same time, there's probably going to be a natural limit to the speed at which things can happen and also a natural upper bound, right? Certain things, I mean, that was already more than a year ago. Someone reflected something on me, which I now feel very much, which is, I mean, at the point a model writes English better than you, I mean, that's maybe too good and it shouldn't be that good, right? And I'm like, okay, that's an interesting realization that even if you could improve

• 51:36Speaker 01

  that capability and maybe there's no ceiling or the ceiling is, you know, still far away, it might not even be that we need to see that ceiling. So there's the performance of the whole system overall, which is very impressive already and there might be upper bounds, obvious, obvious upper bounds in some cases. But yeah, I think that the physical limits on the models and how you train them, even if you think we knew exactly the recipe, we could iterate very quickly and train the next generation models, there is some acceleration, but there's some upper bounds and rate limits that are still fairly fundamental.

• 52:14Speaker 00

  Well, I always like to end my interviews with a, with a quickfire round where I basically just stuff in all the broad questions that I haven't had time to fit in elsewhere. And so maybe to kick things off, I'm curious, what's one thing you've changed your mind on in AI in the last year?

• 52:28Speaker 01

  What I have changed my mind, I think the fact that even though I want to believe that training on a broad distribution is, is probably going to enhance the model, training on narrow kind of points of great difficulty, like, you know, maths or coding, creates this generalization. I think that is not something. I quite predicted to work as well as it did.

• 52:57Speaker 00

  I think Demis said at I O that we're at the foothills of the singularity and AGI could come in the next few years. Do you feel similarly?

• 53:06Speaker 01

  I feel similarly, and I'll say more, right? Like, if you, I mean, even with someone in the field, like, close to these models, and neural nets in general, if seven years ago, I mean, I'm using a time that is clearly like pre, you know, all that happened in L L M's, if seven years ago I had to experiment with a model that we have currently, would I have declared this is AGI, right? Like, I mean, and I would say probably yes. I mean, it's

• 53:32Speaker 00

  An ever-moving definition.

• 53:33Speaker 01

  It's very impressive, the progress. Yeah. So I think just because now we're seeing it closer, and it is a good thing to be more ambitious about like what it is that we're building. But again, based on different definitions, or perhaps even the expectations we might have had about what AGI meant even only a few years ago, I would say in some way AGI is here, right? So, I mean, all I'm saying is like, I don't think it is here in the way I want to see it, but it is fairly close. And maybe this ability for the models to truly learn from experience is what is missing in my mind. But everyone will have

• 54:13Speaker 01

  their own kind of test or bias, I guess, onto what the models still feel like capability gaps exist.

• 54:22Speaker 00

  And we'll get there, and then we'll move the goalposts again, and have some other reason. I think one huge advantage that you all have is, you know, certainly incredibly bullish on the models you're building. You have, you know, your own hardware. And I think a question a bunch of, I know my listeners will have in the back of their head, so I'll ask it, is, I think one thing that you've done that a bunch of people were curious to better understand was, you know, taking some of the compute you have and selling it to Anthropic, right? And I think there's been this narrative on Twitter of like, well, if you were so bullish on models and the research, like, why not just keep all the compute yourself? And so I'm sure our listeners would just love to hear your perspective on that.

• 54:55Speaker 01

  Yeah, how to invest kind of compute even within, you know, like ourselves. I like the compute is used to serve. We train small models, even smaller models, then trying to train frontier models. I think this is all like a, you know, a fine equation to balance. And I think just in general, like one way to think about Alphabet is there are things that create revenue and economical impact that then you can't reinvest. So it's not just being greedy about, hey, what should we do now and take all these things together? And that's it, right?

• 55:33Speaker 01

  I think the the strategy, which again, like I, I kind of think often about is just multi-pronged. And I think the timelines, although we are bullish, of course, on the technology advancing, you just think of like, you know, revenue streams and so on. And I think hardware is a very important asset. And I think, you know, there's probably a trade off in which you don't use it all, but use use it strategically to create, you know, to reinvest it basically, right? And I think that's what the current, what makes, what seems to make sense. And I mean, the, the, the, obviously, calculations behind these are complex, right?

• 56:16Speaker 01

  So I'm not going to enter into exactly the rationale, but I think in general, it's just a strategic choice to have different levels of investment and timelines in mind.

• 56:25Speaker 00

  What's so interesting about your position is you are like the, like the only frontier model provider with your, your own like cutting edge, you know, or state of the art chip. What is that like collaboration actually look like? Because it's such a unique motion, right? I mean, obviously, like Nvidia works closely with other labs, but like they're not sitting under the same company. And so what does that look like when it works really well?

• 56:44Speaker 01

  As I was explaining before, I reflected on several moments, right? And this was early days. I mean, even, even deep learning internally at Google had to be still proven out. But I remember it must have been 2013, maybe 2014, where a bunch of us, I think it was me, Jeff Hinton, Jeff Dean. And Ilya that were in a room trying to decide, hey, what should the servers have? I mean, how many at the time we had obviously some CPUs, some GPUs. And you're trying to make a guess based on what you know about the research, where the models are going. And you can literally have that impact. But of course, there's delayed rewards because this is just an investment and only, you

• 57:29Speaker 01

  know, a few months, if not years, this will materialize in data centers. So I've been sort of, and I thought that was amazing, right? Like, I mean, obviously, hard to answer the question. I think we tried, I mean, you try to predict what's going to happen in research. And I mean, in the early days, that was even harder. But I think it's a very privileged position to be able to really influence. And we certainly do that. And especially with Jeff, who's obviously been in the, thinking about infrastructure quite a bit for many, you know, for basically the existence of Google. It's very interesting to then think about, hey, like these models are going this way. And then these investments, because they have certain latency, you know, being under

• 58:12Speaker 01

  the same roof and seeing what we see just really, really helps. And again, I've seen it in the very scrappy early days, and it keeps happening and getting better. And of course, uncertainty in some way it reduces, which makes the job easier, but still a fascinating choice that has, you know, deep consequences on then the, you know, the faith of the company, et cetera.

• 58:33Speaker 00

  Well, this has been a fascinating conversation. I feel like I could talk to you for a long time, but I'd be delaying our path toward AGI. And so maybe I just want to make sure to leave the last word to you. Anything you'd like to share with our listeners or research you'd like to point them to? Anything in I/O? The floor is yours.

• 58:49Speaker 01

  I think it's a fascinating time as in anything in AI. So if you're a user, use the models. If you're a builder, use the models to build. Anything you do, even if you think there's no remote connections to AI. So please, you know, play with these models. They're amazing and they will only get better.

• 59:07Speaker 00

  Awesome. Well, thank you so much. It's been an awesome conversation. I'm Jacob Effron, and this has been Unsupervised Learning, a podcast where I get to talk to the smartest people in AI. And ask them tons of questions about what's happening with models and what it means for businesses in the world. As I hope is clear, I have a ton of fun doing this. It's a nights and weekends project in addition to my day job as an investor at Redpoint. But our ability to get these incredible guests on really comes from folks like you subscribing to the podcast, sharing it with friends. It's really what ultimately makes this whole thing work. And so please consider doing that and thank you so much for your support and listening. We'll see you next episode.