DeepMind Genie 3 [World Exclusive] (Jack Parker Holder, Shlomi Fruchter)

This episode features Shlomi Fuchter and Jack Parker Holder from Google DeepMind, who are unveiling a new AI called Genie 3. The host, Tim Scarfe, describes it as the most mind-blowing technology he has ever seen. We were invited to their offices to conduct the interview (not sponsored). Imagine you could create a video game world just by describing it. That's what Genie 3 does. It's an AI "world model" that learns how the real world works by watching massive amounts of video. Unlike a normal video game engine (like Unreal or the one for Doom) that needs to be programmed manually, Genie generates a realistic, interactive, 3D world from a simple text prompt.**SPONSOR MESSAGES***Prolific: Quality data. From real people. For faster breakthroughs.https://prolific.com/mlst?utm_campaign=98404559-MLST&utm_source=youtube&utm_medium=podcast&utm_content=script-gen***Here’s a breakdown of what makes it so revolutionary:From Text to a Virtual World: You can type "a drone flying by a beautiful lake" or "a ski slope," and Genie 3 creates that world for you in about three seconds. You can then navigate and interact with it in real-time. It's Consistent: The worlds it creates have a reliable memory. If you look away from an object and then look back, it will still be there, just as it was. The guests explain that this consistency isn't explicitly programmed in; it's a surprising, "emergent" capability of the powerful AI model. A Huge Leap Forward: The previous version, Genie 2, was a major step, but it wasn't fast enough for real-time interaction and was much lower resolution. Genie 3 is 720p, interactive, and photorealistic, running smoothly for several minutes at a time. The Killer App - Training Robots: Beyond entertainment, the team sees Genie 3 as a game-changer for training AI. Instead of training a self-driving car or a robot in the real world (which is slow and dangerous), you can create infinite simulations. You can even prompt rare events to happen, like a deer running across the road, to teach an AI how to handle unexpected situations safely. The Future of Entertainment: this could lead to a "YouTube version 2" or a new form of VR, where users can create and explore endless, interconnected worlds together, like the experience machine from philosophy. While the technology is still a research prototype and not yet available to the public, it represents a monumental step towards creating true artificial worlds from the ground up. Jack Parker Holder [Research Scientist at Google DeepMind in the Open-Endedness Team]https://jparkerholder.github.io/Shlomi Fruchter [Research Director, Google DeepMind]https://shlomifruchter.github.io/TOC:[00:00:00] - Introduction: "The Most Mind-Blowing Technology I've Ever Seen"[00:02:30] - The Evolution from Genie 1 to Genie 2[00:04:30] - Enter Genie 3: Photorealistic, Interactive Worlds from Text[00:07:00] - Promptable World Events & Training Self-Driving Cars[00:14:21] - Guest Introductions: Shlomi Fuchter & Jack Parker Holder[00:15:08] - Core Concepts: What is a "World Model"?[00:19:30] - The Challenge of Consistency in a Generated World[00:21:15] - Context: The Neural Network Doom Simulation[00:25:25] - How Do You Measure the Quality of a World Model?[00:28:09] - The Vision: Using Genie to Train Advanced Robots[00:32:21] - Open-Endedness: Human Skill and Prompting Creativity[00:38:15] - The Future: Is This the Next YouTube or VR?[00:42:18] - The Next Step: Multi-Agent Simulations[00:52:51] - Limitations: Thinking, Computation, and the Sim-to-Real Gap[00:58:07] - Conclusion & The Future of Game EnginesREFS:World Models [David Ha, Jürgen Schmidhuber]https://arxiv.org/abs/1803.10122POEThttps://arxiv.org/abs/1901.01753[Akarsh Kumar, Jeff Clune, Joel Lehman, Kenneth O. Stanley]The Fractured Entangled Representation Hypothesishttps://arxiv.org/pdf/2505.11581TRANSCRIPT:https://app.rescript.info/public/share/Zk5tZXk6mb06yYOFh6nSja7Lg6_qZkgkuXQ-kl5AJqM

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:01Speaker 00

  Life is all about making decisions. Sometimes you make questionable ones. Like buying a new TV that's too big for the wall because the electronics store employee was a little too good at their job. Then, sometimes you make great ones. Like protecting your home and auto with Progressive. You get customized coverage options so you can be sure you have the right coverage. Get your quote today at progressive.com. Progressive Casualty Insurance Company affiliates and other insurers, not available in all states.

• 0:30Speaker 01

  The best part of waking up? A full cup of Folgers coffee and music on full blast.

• 0:56Speaker 01

  Shop Folgers instant coffee and more at your nearest retailer.

• 1:00Tim Scarfe?

  By the way, look at this dog. This is amazing. This is insane.

• 1:07Tim Scarfe?

  What was the prompt to create that? Today is a world exclusive of what is, in my opinion, the most mind-blowing technology I've ever seen and the most poggers I've ever been. You're not going to believe what Google DeepMind showed me in an exclusive demo in London last week. This technology might be the next trillion-dollar business and might be the killer use case for virtual reality. Google DeepMind has been slaying so hard recently that even Gemini DeepThink can't count the number of wins in the context window. Let me explain. Today, we're going to talk about a new class of AI models which are called generative interactive environments. They're not quite like traditional game engines or simulators or even generative video models like Vio,

• 1:52Tim Scarfe?

  but they do have characteristics of all three. They're basically a world model and video generator which is interactive. You can hook up a game controller or any kind of controller for that matter. DeepMind say that a world model is a system that can simulate the dynamics of an environment.

• 2:08Speaker 03

  The consistency is amazing. emergent, there is nothing explicit. The more the model doesn't create any explicit 3D representation.

• 2:17Tim Scarfe?

  How do you square the circle between like a stochastic neural network? And yet it has consistency, right? So I look over here, I look back, I look there again. The thing is back, like, isn't it a bit weird that a sub symbolic stochastic model can give us apparently consistent, like, solid maps of the world? Do you remember the Quake engine? In 1996, it required explicit programming of the physics and rules and interactions. But this new generation of AI systems learned real world dynamics directly from video data. You can control an agent in the world in real time. The move towards generative world models was born from the limitations of hand-coded simulators.

• 3:01Tim Scarfe?

  Even their most advanced platform, Xland, which was designed for general agent training. It was the frontier for embodied agent training with curriculum learning. But it felt far from the real world. It was almost cartoon-like. It could model 25 billion tasks, but it was still handcrafted. It was constrained to the rules of that particular domain, and it was janky. Imagine if you could just generate any interactive world you wanted to train your agents on with a simple prompt. Now, cast your minds back to last year when I interviewed Ashley Edwards at ICML. This was the first version of Genie, which was trained on 30,000 hours of 2D platformer game recordings.

• 3:42Speaker 04

  When we're generating next frames, the objects that are further away are moving more slowly than objects that are closer. And this is a sort of effect that you would often see in games such that you can kind of simulate depth. It's something that we also haven't, you know, like when we observe things moving, we see things moving slowly when they're further away. So, yeah, the model learns that.

• 4:02Speaker 05

  Just being able to be that good at understanding the physical world was not something we were expecting it to be that good at that quickly.

• 4:10Tim Scarfe?

  The core innovation of Genie 1 was a spatial temporal video tokenizer that converts raw footage into processable tokens and a latent action model that discovered

• 4:20Tim Scarfe?

  Meaningful controls without labeled data and an autoregressive dynamics model which predicted future states. The latent action model, a form of unsupervised action learning, was the core innovation. Genie discovered eight discrete actions which remained consistent across different environments purely by analyzing frame-to-frame changes in game recordings. This means it knew what jump meant or what move left meant without being explicitly trained on those actions. This was an OMG moment for me. I mean, how was that even possible from training on offline game episodes? Even more surprising was how it seemed to have emergent capabilities like 2.5D parallax.

• 5:04Tim Scarfe?

  Just 10 months later, Genie 2 arrived with 3D capabilities and near-real-time performance. The visual fidelity was much higher. Now it can simulate realistic lighting like the Unreal Engine. You know, things like smoke, fire, water, gravity, pretty much anything you might see in a real game. It even had a reliable memory. You know, you could look away from something and then bring it back into view and it would remember the thing. This is GigaChad Jack Parker Holder. He's a research scientist at Google DeepMind in the Open-Endedness team, talking about Genie 2 with Demis, no less.

• 5:39Speaker 06

  This is a photograph taken by someone in our team somewhere in California. And what we then do is ask Genie to convert this into an interactive world. So we prompt the model with this image and Genie converts it into a game-like world that you can then interact in. Every further pixel is generated by a generative AI model.

• 5:57Speaker 07

  So the AI is making up this scene as it goes along?

• 6:02Speaker 06

  Exactly, yes. Someone from our team is actually playing this. They're pressing the W key to move forwards. And then from that point onwards, every subsequent frame is generated by the AI.

• 6:12Tim Scarfe?

  Around the same time last year, you'll probably remember this by the way, DeepMind's Israel team led by Shlomi Fruchter showed diffusion models simulating the Doom engine. The system was called GameEngine. It's almost a meme at this point how Doom runs on, you know, calculators and toasters. But here is a neural network confabulating a Doom game frame by frame in real time. Like, look at how it just knows what the health is. You can shoot characters, you can open doors, navigate around maps. You know, occasionally it was slightly glitchy, but this is just unreal. You know, you could just simulate Doom at 25 frames a second on a single TPU. The only limitation, of course, was that it could only do Doom and nothing else.

• 6:57Tim Scarfe?

  So, last week, we waltzed our way into London and Jack and Shlomi gave us a demo of Genie 3. Honestly, I couldn't believe what I was seeing. The resolution is now 720p, which is firmly in the good enough territory to suspend disbelief. It's real time, it can simulate real-world photorealistic experiences, which can continue for several minutes before running out of context. Shlomi had his hands all over VO3, by the way, and they seem to have combined elements of the Genie architecture with VO, producing something I can only describe as VO on steroids. Unlike Genie 1 and 2, the input is now a text prompt, not an image, which they argued is a good thing from a flexibility perspective, but it does mean that you can no longer

• 7:40Tim Scarfe?

  take a photo of a real place and generate from there. One of the main features of Genie 3 is that it has a diversity of environments, a long horizon, and promptable world events. Now, on the world events, let's take this ski slope example. We might type in "another skier appears wearing a Genie 3 t-shirt", or "a deer runs down the slope", and there you are, things just happen in the world. They say that this might be very helpful for modelling things like self-driving cars, where you can simulate rare events. But I was left thinking that this is just turtles all the way down. How can we write a process to prompt the potentially infinite number of rare things which could happen in a scene?

• 8:23Tim Scarfe?

  There was an example they showed of flying around a lake, and it was amazing, but I was like thinking, "Where are the birds, mate? Like, can you type the birds into the prompt?" The team believes that we haven't yet had the move thirty seven moment for embodied agents, you know, where an agent discovers a novel real world strategy. They see Genie three as the key to enabling that. But the real world constantly surprises us because the real world is creative. Creativity simply means that the tree of things which can happen keeps growing. New branches and leaves just keep appearing. Perhaps in the future, we might have an outer loop, which makes the system more open ended.

• 9:04Tim Scarfe?

  But right now, in my opinion, Genie three, like all AI, gives you exactly what you ask for in the prompts and isn't creative on its own. Currently, the system only supports a single agent experience. But imagine how cool it would be if you could extend that to a multi-agent system. Apparently, they are working on that. I mean, personally, I'm most excited about a new modality of interactive entertainment. You know, just imagine YouTube version two. DeepMind sees the main use case of being able to train robotic simulations as being the real game changer. This seems plausible to me. I mean, like, the miracle of human cognition or in brains is that we have evolved to simulate the world without direct physical experience, which is expensive. This is basically the same idea, right?

• 9:49Tim Scarfe?

  Why train in the real world if we can just simulate any possible scenario in a computer? Just like that Black Mirror episode. Here's a couple of examples they gave of using simulated environments to train an agent to do some specific language tasks. Now with Genie two, they said they were happy if it was consistent, even for twenty seconds. But now, when you notice something inaccurate, it's very surprising. The key thing is that it now extends beyond the prediction horizon of the average human, and the glitches are getting harder and harder to spot. They said that Genie two wasn't actually real time. You had to wait a few seconds between taking different actions. You know, it was low resolution, had limited memory.

• 10:30Tim Scarfe?

  You know, I mean, it was superficially really good, but it didn't look particularly photorealistic. Genie three changes all of that. So Genie supported around ten seconds of generation, Genie two around. Around twenty seconds. Genie three is able to simulate interactive environments for multiple minutes. This time around, they were a little bit more tight lipped around the architecture. They wanted to focus on capabilities in the interview. And that's fair enough. I mean, it's understandable, given that this is potentially a trillion dollar business. And Zuck will be sniffing around like a truffle hound. My my biggest concern with this is that as soon as Zuck gets wind of this. He is going to be getting out his checkbook. He's going to go straight to Jack and Shlomi. And he's going to be like, come on, boys.

• 11:15Tim Scarfe?

  Hundred million dollars. Come and work for me. Um, Zuck, mate, seriously, no, don't do it. These guys, they they're doing God's work over here. You need to just let them let them do what they're doing. You can make it yourself if you want, Zuck. Leave them alone. I should say, I did joke at the end of the interview that if you are learning Unreal Engine right now, you might want to pivot to a different career. But the Google guys were quite grounded. They argued that this is a different type of technology. There are pros and cons, you know, which is fair. I should stress that as amazing as this technology is, it's still a neural network and it still has many important limitations. Certainly, though, just imagine how easily you could generate an interactive motion graphics with this technology. You know, that's something that Unreal Engine is leaning leaning hard towards in version five point six.

• 12:00Tim Scarfe?

  So do I need to fire my motion graphics designers, Victoria?

• 12:05Tim Scarfe?

  Will users be able to use this? Not anytime soon. This is still a research prototype and given the obvious safety concerns. They're going to open this up progressively through their testing program. One question did come up in the press conference yesterday, though, like. Could it generate an ancient battle? And Shlomi said that it's not trained on that kind of data, wouldn't be able to do that yet. So, I mean, certainly not a specific historical battle anyway. So it does sound like there are still some limitations. How can a system like this ever be fully reliable? Well, they did say that with better models, the trend is that they get more and more accurate, the glitches become fewer. And they expect to see further improvements. You know, there's this annoying phrase like this is the worst the model will ever be.

• 12:46Tim Scarfe?

  But even as I said, they can generate some edge cases using a whole bunch of prompt augmentations, but. It might just be turtles all the way down. And how do you come up with all of the rare black swan events that might happen? So what data was it trained on? They were quite cagey about this as well. It's probably safe to assume that it's been trained on all of YouTube and lots more besides that.

• 13:09Tim Scarfe?

  How much compute does this thing need? Well, I asked them that and they were a little bit vague about it. They said that it ran on their TPU network. So I'm inferring from that it needs a crap ton of compute. However, I can say that it was demoed in front of me. It was very responsive. You put a prompt in, it thinks for about three seconds and then you're just in and it just works. They also mentioned some cool stuff about how, you know, like Genie can be used to train agents, as we said, but the agents themselves could be used to better train Genie 3, creating this virtuous cycle of iterative improvement.

• 13:41Speaker 06

  If you're in a world walking around and say you go to cross the street, you sort of check the queues of the drivers, for example, maybe there's not a crosswalk and you need to know when to stop. You can see that they're slowing down, so that's when you would go. And the other agents should be simulated in that fashion.

• 13:56Tim Scarfe?

  Genie 3 and other similar models would be impossible without at least some human feedback in the training loop or the data curation or the evaluation. Prolific is a human data platform and they are sponsoring this video today.

• 14:10Speaker 07

  My name is Enzo. I work at Prolific. I'm the VP of Data and AI. I support everything from AI, data, research and the likes. For those unfamiliar, Prolific is a human data platform for working with everything from academic researchers, but also small and large players in the AI industry.

• 14:31Tim Scarfe?

  Visit prolific.com. Yes, this is the demo where they've got Genie 3 memory test on a blackboard. You see there's like an apple in a cup and then you kind of like go out, you look out the window, you see there's like a few cars. And the purpose of this test is to kind of say they've got such a long context window, you know, similar concept to a large language model, that it still remembers all of the things that it generated, you know, even if it's minutes ago. We've got the the blackboard over here, we look up and there it is, it remembered it, Genie 3 memory test. I've also noticed that this model is even better than VO3 at things like text. It's because you would think that they would dumb down the model to make it interactive

• 15:13Tim Scarfe?

  and to make it this sophisticated, but even as a video generator model, it seems almost better than VO3 for doing a whole bunch of stuff.

• 15:21Speaker 03

  Alright, so I'm Shlomi Fuchter. I'm a research director at Google DeepMind. I'm the VEO co-lead. And I'm basically working at Google for about 11 years. Recently on diffusion models and the various modalities, image, video. And we'll tell you more about what we're working on right now.

• 15:38Speaker 06

  Hey, I'm Jack Parker-Holden. I'm a research scientist at Google DeepMind in the Open Endless team. Originally working on open-ended learning and open-endedness, and more recently working on world models.

• 15:48Tim Scarfe?

  We are here at Google DeepMind in London. And you guys have just demoed to me something which I think, I'm more impressed with this, I think, than anything I've seen probably ever before. I think it's a paradigm-changing moment. Shlomi, can you tell us a little bit about this new version of Genie?

• 16:08Speaker 03

  Sure. So Genie 3 is our most capable world model. And by a world model, what we mean is basically a model that is able to predict how an environment would evolve and also how different actions of an agent would affect this environment. So with Genie 3, we're able to basically push the capabilities of a world model to a new frontier. That means higher resolution, much longer horizon, and better consistency, and all that in real time. Basically allowing, whether it's an agent or a person that interacts with the system, to walk around it,

• 16:52Speaker 03

  navigate it, affect it, while the generation happens in real time.

• 16:57Tim Scarfe?

  Genie 3 is just ridiculous, right? It's just on a completely different level. But maybe we should just contextualize that around Genie 2. So what was Genie 2?

• 17:06Speaker 06

  That's a great question. So Genie 2 was sort of the culmination of two years of research in what was quite a new area, which is foundation world models. We called it at the time. So essentially, in the past, world models had modeled a single environment. So the canonical world model's paper in 2018, from David Haarn and Jürgen Schmidhuber, modeled the car racing environment, which was a Majoko environment. And it could just model that one environment. It could predict the next states, given any actions in that one world. We've seen with the Dreamer series, also from Google DeepMind, Daniel J. Hafner, with Atari games and other kind of environments as well. But no one had ever done something that could create new worlds. So with Genie 1, the real novelty there was that we had a model that, for the first time, could be prompted to create completely new worlds that didn't previously exist.

• 17:51Speaker 06

  But that being said, they were fairly rudimentary, like they were low resolution. You could only play with it for a couple of seconds. And so agents couldn't really learn long horizon behaviors that wanted to. And the diversity was still fairly constrained and it required some form of image prompting. With Genie two, we really pushed that to the next level, right? So we trained it on a much larger distribution of three D environments. And we moved to three sixty P from, I think, ninety P before. So it was more like close to what we see now, but it was still sort of scratching the surface because we didn't really know that this approach could scale the way we've seen other methods have. So we wanted to really like test this from a research standpoint. But then I think for this year, we wanted to really take that to the to the next level. And that's what we think we've done.

• 18:36Tim Scarfe?

  Yes, and it's now seven twenty P. It's interactive, so Genie two wasn't wasn't interactive. It wasn't fast enough. And you know, Steve Jobs said, there's something magic about the touchscreen, right? There's something magic about it. And of course, the magic happens when it's interactive. And some of the demos that you showed me were just just insane, right? So photo realistic, I mean, it's kind of like a fusion of V O, I suppose, that you can now understand the real world and you can build essentially a foundation model for the real world, which is interactive. That's mind blowing. And just tell me about some of the examples you showed.

• 19:12Speaker 03

  Yeah, so I think what you said about V O or more generally about video models is right. Like we, there is a way we can think about them as somewhat a world model, but it's not really, it doesn't allow us to actually navigate or interact with it completely interactively. And I think that's that's one of the limitations of of video models that with Genie free, we're trying to address. And basically, in the examples that you've seen, we're able to, because Genie freeze generates the experience and what's the what we see frame by frame, it lets the user or the agent that is using it basically control where it wants to go in every, like in a very low latency.

• 19:56Speaker 03

  That allows basically exploring the environment and.

• 20:00Speaker 03

  And creating new trajectories that are not predefined like video models. So in the examples that you've seen, for example, you can see the character or the agent in this video moving around, maybe going back to the place they're already been in before, and everything remains consistent. And I think that's a very remarkable property or capability of the model, the ability to preserve and the consistency of the environment along very long trajectories.

• 20:30Tim Scarfe?

  Yes, and even Genie two had some kind of object permanence inconsistency, but nowhere near as much as we have now, but we'll come back to that in a second. We can't say too much about the architecture for Genie three, but in Genie two, there was a an ST transformer, so a special temporal transformer, which was conceptually quite similar to like a V I T. And there was a latent action model, which means even from like, you know, non-interactive data, you could infer some low cardinality action space. And then those went into a dynamics model.

• 20:59Speaker 03

  I I think we can, what we can say about the architecture that, you know, might be interesting is that definitely because of the interactive nature of the problem or the setup, then the model is ultra-aggressive. So what it means that, it means that the model generates a frame by frame and has to refer back to everything that happened before, right? So if, for example, we're walking around some auditorium or what some some other environment, basically, if we revert, if we revisit a place that we've already been to, the model has to look back and and understand that this information has to be consistent with what's happening

• 21:38Speaker 03

  in in the next frame. So I think the interesting style, the interesting point here is that everything here, like the consistency is emergent, there is nothing explicit. The more the model doesn't create any explicit 3D representation. And it's unlike, unlike, you know, methods, other methods like nerves and Gaussian splatting. So I think that's that emergence, a kind of like a capability is very interesting and surprising for us.

• 22:03Tim Scarfe?

  Yes, and and even Genie two had emerging capabilities like parallax and, you know, it could model certain forms of lighting and. And so on. But this just blows my mind. Yeah, Shalom, you're you're involved in that Doom simulation last year. And even that just blows my mind, right? So we all played Doom in nineteen ninety three. It was one of John Carmack's finest. And now you're saying that, I mean, certainly the work that that you folks did last year. You've got a neural network model which is sub symbolic. So there's no explicit model of the world. You don't know where the doors are. You don't know where the lakes are, where the maps are and so on. You just kind of take a, you know, a sample, a traversal through this space. And and it just produces the game in pixel space. I mean, that's, yeah.

• 22:48Speaker 03

  Yeah, you know, this is really, you know, I've been playing games, obviously, including, you know, Doom and others. And I also worked some on on game engine development at some point, very early in my teens. And I I think what what I really like about this project is that we now are now able to run models that actually generate consistent three D environments, as in, you know, game engine and the Doom simulation. And they run on GPUs or TPUs, while in the past we were running, you know, this like game engines on the same hardware. So I think there is, it's really something very interesting and it kind of closed this circle for me.

• 23:25Speaker 03

  And in particular, in the case of game engine, we tried to push on the real time, interactive kind of like aspect. So we basically said, OK, would a diffusion model be able to simulate a game environment end to end with nothing explicit, no code, nothing except for actually generating the pixels and getting the inputs from the user? And, you know, we weren't sure if it's going to work. So I think there is like with this kind of research, we try and it doesn't work. And then it all of a sudden, something happens and it's we we we see that it does work and that's a very rewarding moment. And I think in this case, once people saw, I think this was really even the the the reception of that was a bit surprising.

• 24:10Speaker 03

  Because there is something about the real time, interactive capability that really sparks the imagination of, oh, I can actually walk into this environment, maybe generated environment and actually experience it, right? So I think this was a moment that I'm later, when I think about it, like we were. Kind of like excited about the real time nature of the simulation. And we really wanted to bring it to higher quality, more general purpose simulations.

• 24:41Speaker 00

  Progressive's Accident Response can contact emergency services if you're in a major accident. The only thing worse than someone you love being in an accident is not being able to help them. So Progressive is here to help when you can't. So get peace of mind knowing your loved ones are protected with Progressive's Accident Response. Learn more at progressive.com. Progressive Casualty Insurance Company and Affiliates. Limited availability in California. Driver is responsible for emergency and towing charges not covered by the policy. Coverage subject to policy terms and conditions.

• 25:11Speaker 01

  The best part of waking up? A full cup of Folgers coffee and music on full blast.

• 25:38Speaker 01

  Shop Folgers K-Cup Pods and more at your nearest retailer.

• 25:42Tim Scarfe?

  So Jack, I mean, one of the million dollar questions is, you know, like even with the language model, it's stochastically sampled, you know, with this temperature parameter. Same thing here, I mean, with Genie 2, the dynamics model is using this masked GIT. And it was, it was run iteratively. And how do you square the circle between like a stochastic neural network? And yet it has consistency, right? So I look over here, I look back, I look there again. The thing is back. Like, isn't it a bit weird that a sub symbolic stochastic model can give us apparently consistent, like solid maps of the world?

• 26:19Speaker 06

  That's a really good question. I think probably similar to language models that there are some fundamental things about the world that you want to remain consistent. So with a language model, I think even though, as you said, they can be sarcastic models, if there are things that are stated as facts in their context, they will still probably recall them correctly, whereas new things are where they maybe have more degrees of freedom to change things like that. So I'd imagine in a world like a Genie three-generated world, if you were to move around, then maybe new things would have some degree of sarcasticity to them. But then, once they've been seen once, then they should be consistent from that point forward, because the model knows when to use this sarcasticity. And this is

• 27:02Speaker 06

  kind of an emergent property from the scale that we train at.

• 27:06Tim Scarfe?

  Yes, and we'll save the emergent discussion. I was just telling the guys about my conversation with David Krakauer the other day, but maybe you won't go there. But the other really interesting thing is, so you said David Harg, 2018 with Shmid Huber, the world models thing. And Shlomi, in the presentation, you defined a world model as essentially being able to simulate the dynamics of something, right? If a world model simulates the dynamics of a system, how could you, for example, how could you measure that?

• 27:36Speaker 03

  So I think it's very hard to exactly measure the quality of world models in general. And I think when it comes, especially to visual generation, if it's image models and generative models in general, very difficult to measure their quality, because some of it is very subjective, right? So I think for LLMs, actually, we're in a better place, because we can measure their performance. First, of course, there is perplexity, just the next token prediction problem. But later on, we actually care about how they operate for the task that we care about, right? So we measure, for example, a downstream performance on various tasks.

• 28:17Speaker 03

  But when it comes to world models, and today, we focus mostly on the visual aspect, right? So it's important to highlight that the world is more than just visuals, right? Um. But but again, for Genie Free, we're focusing more on that. Um. Because a lot is captured in the visual in in the interaction of the world. Um. So measuring how well a model is is doing really depends on the context and and also on how we want to use it later. I think that's something we have to keep in mind when we evaluate things um and models. So we have in mind one particular application that we think is really key,

• 28:56Speaker 03

  and that's to be able to actually uh train and and let A A I agents interact with simulation environments. I think that's something that, um, you know, I'm coming more from this kind of like a kind of maybe uh simulation a background, but not so much from the um maybe training agents in simulation environments. That's not so much wasn't wasn't my my original background. But, you know, I think through the interaction with, you know, other people in DeepMind that are kind of like exploring that for a long time. I was really, um, you know, over the last few years, I kind of came more and more to realize how much potential there is in in that. Because if we really think about it, AI would be limited by the ability to perform physical experience

• 29:40Speaker 03

  experiments, right? Because imagine that you want to develop a new drug or a new um kind of treatment. Um. You cannot really do it in uh in the real world if it takes months for every step in the way, right? And the same, we can think about, you know, if we want to learn how to assemble something, then again, if I have to train the robot and um in the real world, it might take very long. So that's why the simulation of the real world is really key. And that's what we hope we kind of like push a bit further with Genie Free.

• 30:10Tim Scarfe?

  Yes, very exciting. I I spoke to a startup recently and and they they sketched out this future where we'll have um essentially um a a model platform where people doing robotics can download the policies, you know, so I'm I'm in a factory and I need a policy for doing this particular thing. But of course, you know, they imagined that it's so scarce, it's so difficult to get real world data that you know, there would be a marketplace and everyone would train their own policies and they would like sell it to other people in the market. This is a slightly different vision. You're saying that now we have a a world foundation model and essentially I could say, well, in this situation, I need to have a robot policy for doing this particular thing, so I can just spin off a job. I can create the policy and and away we go. So is is that roughly correct?

• 30:49Speaker 06

  I think that is kind of the vision that we have. So I think in robotics in particular, there's a lot of focus on deploying robots in somewhat constrained settings, right? So it might be, for example, in someone's apartment that's very staged, right? Almost a stage as a podcast recording, you know, you've got like all these support staff watching around this robot achieve one goal, right? And from a control perspective, it might be very impressive, but in terms of the stochasticity of the world that it's in, it's very limited, right? And if we look at simulation environments, they might accurately model physics, but they definitely don't model things like weather or other agents or animals, all these kinds of things, right? Whereas a model like Genie 3, because it has world knowledge, that world knowledge extends beyond physics, actually, to also the behavior of other agents.

• 31:34Speaker 06

  And as we showed you in that example at the beginning, with the world events that we can also inject, right? You can actually prompt to have, you know, another agent crosses in front of you or like, you know, we had a herd of deer run down the ski slope or something like that. And I think these are the kind of things that for robots to be deployed at large scale in the real world, the real world is fundamentally populated by people and other agents. And this is something that we can gain from training on this general purpose world model, that we just have no other approach, I think, to scalably get this data in a safe way as well, right? Because the safety is a critical element of this, that we can simulate things in a realistic way without having to actually deploy agents in the real world.

• 32:16Tim Scarfe?

  Yes, and that was a very important detail. So you can put a prompt event in, and you gave me an example of there's a skier going down a slope and then here's a guy with a Gemini t-shirt. And I guess what I'm thinking about here is if we did train these robot policies, we would need to do probably some kind of curriculum learning and some kind of diversity. So, you know, we would start off with a simple environment and then we'd add the guy with the Gemini t-shirt and then there'd be a car coming along. And maybe in reality, there would be some kind of meta-process, you know, creating some gradient of complexity and, you know, diversifying environments. I love that Ken Stanley paper, you know, the poet paper doing something like that. But is that like a fairly? Reasonable intuition.

• 32:53Speaker 03

  So I think it's still early to say exactly how word models like Genie free will be actually used for AI research. I think we can only kind of directionally say, we, in general, I think we still, we see it in also in other generative models, that there are some capabilities that we actually discover, right? And we don't necessarily know that they're there. And then through the interaction development, we're actually seeing them emerge. For example, you know, with VEO, we just recently, like a few days ago, we then, we kind of shared that you can write like some text on a, on a photo and provide it to VEO and it just

• 33:32Speaker 03

  like, it reads the, the text and it follows also the spatial instructions, right? And I think that's, that's for example, something that we didn't necessarily explicitly train the model to do, but it's capable of doing. And I think here as well, the capabilities of Genie free that we're exploring are still, we still discover new things. And I think that's something that we hope that by, first by having more like, you know, testers and external testers, that we already shared some, some, like, we basically previewed the model to, and give us feedback. So we hope that through this kind of engagement with the community, and we can better see how those models will be useful. And that's

• 34:16Speaker 03

  something that I expect to take some time, as we basically try and understand the best application.

• 34:21Tim Scarfe?

  You know, I'm a huge fan of open-endedness, for example. And certainly at the moment, when we prompt models, if we're quite generic, you know, in, in, in what we put in the prompt, then we tend to get quite simplistic answers. So, you know, a lot of people doing computer graphics, when they prompt image models, they, they have so much specificity and they deliberately take it, you know, on, onto the tail of the distribution. So they get something that's novel and interesting and so on. And, and the real world just always produces a sequence of artifacts which are novel and interesting, you know, like you get random NPCs walk onto the, onto the screen and cars go in and so on. And is my intuition correct that, that at the moment, as good as it is with, with, with Genie three, you, you tend to get quite a specific scene and you don't have like random kind of planes

• 35:06Tim Scarfe?

  flying over and sort of, you know, just random things happening?

• 35:10Speaker 06

  Yeah, so that's a really good intuition, right? So it definitely is the case that the model is very, it's very aligned with the text prompts that it's given. So therefore, there is a lot of emphasis placed on the quality of the text prompt to describe the scene. But I actually wouldn't see that as a limitation. I would see it as a strength, right? So firstly, it means that there's actually a lot of human skill still involved to create really cool worlds, right? And you see some of the examples we showed you. We have some very talented people that can do amazing things with these models, right? And there is actually a lot of value add there to do that, right? So it actually is a tool that can really amplify already creative humans in new ways. And I'm definitely not the best at doing this, right? And I can tell you that it does, it is really impressive when someone is able to do that.

• 35:54Speaker 06

  But on the, on the flip side, from the, the agent perspective as well, right? So when we're talking about designing environments for agents, and you referenced Poet, which was, for me, like Poet and World models were the two papers that I just thought were eventually on a collision course, right? And that's where I started my research career.

• 36:13Speaker 06

  And I think Poet was fundamentally limited because of the, the environment encoding being an eight dimensional vector, but also the fact that there was not really any notion of interestingness as well, right? And in your recent interview with, with Jeff, right? He's obviously talked about how this problem is largely now solved with foundation models, right? So these foundation models can not only define what's interesting based on standing on the shoulders of, of human knowledge, right? But they can also steer the generation of worlds in things like Omni Epic to do this. And that's, in that case, it's done through code. But here we have text as a substrate as well. So in theory, this, these kind of open and algorithms that use language could actually be quite strong places to have these kind of like notions of interestingness and agents

• 36:57Speaker 06

  steer tasks through that space as well.

• 36:59Tim Scarfe?

  Yeah, I think, I think this is the fundamental thing, because certainly with creative models at the moment. Like weirdly, counterintuitively, you need more skill to make it do something interesting than you did before. Like the average creative process now for someone designing a thumbnail on YouTube is they they mix together. You know, they might use the contact model, they might use an upscaler, they might then like, you know, use another image generator. Well, you get this huge kind of compositional tree of operations that happen. And it's very, very highly skilled because a lot of the kind of um structure for constraining the generation of these models still comes from our own abstract understanding of the world. And this is kind of what Kenneth Stanley was saying in a sense. He was saying that we have this understanding of the world, which is constrained by things like symmetry and, you know, various different rules.

• 37:43Tim Scarfe?

  And we then kind of, we we hint to the model, we we constrain the model in the prompt using those things. Would the models ever be able to do that without the humans needing to prompt them?

• 37:54Speaker 03

  So I I think what's interesting is that eventually what we find eh like what humans find interesting and worth um, you know, maybe watching or investigating or researching, it's it's eventually being defined by people. And and I think in the case, for example, if it's video, eh if it's it's eh video generation, for example, then we see that people go and and find ways that maybe we weren't like, they use the tool that we put in front of them to generate new things. So for example, we have people try to cut, like we we have the ASMR videos of people cutting, you know eh eh fruits made of glass, right? Which is not something you can do in the real world. And the novelty comes from from the prompt, basically. Um, I think that's what you're alluding to.

• 38:38Speaker 03

  Um, so I think in in the case of, we're still in a similar place, I would say, when we think about the world models, because you have to provide the description of the world that you want to to maybe walk in into an experience. But some elements are not like would kind of emerge from and will be inferred from the prompt that you provide, right? So you can maybe write a very short prompt, but still the world will have much more richness. So I think there's a question of where does this richness richness coming from? And I think we're at different maybe levels of of the the ability of models um to to bring this rich eh richness into your experience. But I think it's kind of like over time, we see that it becomes more and more.

• 39:22Speaker 03

  Higher and higher and little information provided by users can actually generate very rich videos or experiences. So I would say it's not like it's a bit of an evolving answer, I would say, like over time, I expect that more inputs to the model, or you can think about it like the person is providing a seed and from that seed, we can maybe generate more like more elaborate descriptions and finally an experience. So I don't think about it like a one step process, but more of like a series of creative steps. Each one of them can be, can happen by, can be done by a person or by an AI model and together they generate maybe something new.

• 40:06Tim Scarfe?

  Yeah, and that's what we're seeing play out on Twitter, that, you know, because the creative process is like, you know, generate, discriminate, generate, discriminate. And we memetically share all of the prompts that work. And that's why we've just created this beautiful phylogeny of creative artifacts that are exploring the, you know, the space of these models, which is beautiful. And I'm thinking about the future, I mean, I know you probably can't speculate about this, but this could be the next YouTube, it could be a new form of virtual reality. You know, in philosophy, there's this thing called the experience machine, where you plug yourself into this better than life matrix simulation and no one wants to leave the experience machine because it's better than real life. But we could co-create something like that, right? We could, we could have, it could be on a, on a phone or a virtual headset and we could

• 40:50Tim Scarfe?

  create these worlds and portals between the worlds and it would just be a never ending simulation.

• 40:56Speaker 06

  Yeah, so it's a, it's a great question. So, I mean, going back a few steps, I think another really inspiring sort of thought experiment in this space before the generative models really became capable was something like pickbreeder, right?

• 41:09Tim Scarfe?

  Yeah.

• 41:10Speaker 06

  And so in that case, it was a very simple idea, right? It was just, you know, evolving some, some image, images, basically. And some quite surprisingly creative things emerged from that experiment that I don't think many people would have expected, right? So you had these beautiful, beautiful images basically emerging, use the word emerging again, emerging from just evolving, evolving user preferences.

• 41:32Speaker 06

  Over time, right? And we definitely see modern analogies of this, like you described with, you know, social media platforms, sharing prompts and people generating ideas. And then it emerges in different ways or goes in different ways, like, like the VO ones with people generating stand up, for example, and then suddenly there's tons of exciting content in that space. And I think that it's definitely fair to say that what we've done with Genie three is create another form, another platform or type of model where this kind of creativity could happen and it could also lead to some unexpected, exciting things. But I don't think we can speculate too much at this point exactly what those will be, other than say that it should be interesting and humans will likely to call things with it.

• 42:13Tim Scarfe?

  Yes, I was discussing with Kenneth the other day whether, because he's a big fan of, you know, neuroevolution. And I think he's leaning towards the, the evil, you know, like creating an algorithm that represents evolution in of itself as being the way to, you know, explore interesting phylogenies. And for me, pick breeder was like a kind of supervised human imitation learning. So it was almost like a reflection of the constraints and the cognition that we have. And I lean externalist a little bit. So I think that a lot of semantics is about this embodied physical interaction with the world and that, you know, just via osmosis perhaps gets represented in our brains. But do you have a position on that? You know, do you think that just, just pure neural networks simulating the world could, could understand the world in the same way?

• 42:57Speaker 03

  So maybe first to like about kind of like the immersion or the, the maybe potentially using, you know, this kind of, kind of models for actually, you know, being immersed in it. Like, I think this is where it's still very far. Like, I think it's really, I said before that I think the visual aspects are pretty much primary, right? We're generating pixels and, you know, we've free, we also added audio, but our embodied existence is so much more than that. And I think sometimes we, you know, it's, it's getting lost, right? Like, because eventually we, like as people, we feel a lot, we walk around, we have other senses, we, we have this sense of like where I am right now. And, and, and I think that's, and of course, the physical interaction, which is also applicable to robots.

• 43:42Speaker 03

  Right. So there's still a large gap between where we are right now and where and building, you know, a real full simulation of the world that can actually provide all of the information to an embodied agent. So I think there is definitely a gap there that is interesting, but it does show that we're still very far, you know, in those in that regard. And but I think, as Jack said, basically, building those kind of like experiences, we do, we do see people try to come to build experiences together and kind of explore worlds together. And I think that's a very interesting direction for us. Yes.

• 44:19Tim Scarfe?

  Yes, yes. Yeah, so many things to talk about that. I mean, I suppose one, one important step is this multi-agent simulation thing, right? So quite a few people have spoken about this. Certainly David Krakauer, he said that a lot of, you know, emergent intelligence is about coarse graining when you have these systems that can, you know, through a variety of tricks accumulate information over time. So, you know, eventually, like, we developed a nervous system and culture and language, and that allowed us to communicate, you know, to accumulate information, sort of like, you know, transgressing the, the hardware, the DNA evolution speed. So it's evolution at light speed. And Max Bennett spoke about that in his Brief History of Intelligence, how, you know,

• 45:02Tim Scarfe?

  like, a lot of the evolution of the brain and culture was about the propagation of information without needing to have direct physical experience. So we can implicitly share simulations with each other. So, you know, when we start to build these multi-agent simulations, do you think that similar things might emerge where, you know, almost irrespective of the lifespan of an individual agent, that the system could accumulate information and develop forms of agency and dynamics that, like, simple systems couldn't?

• 45:28Speaker 06

  That's a really question. So I think the way I would see it from the standpoint of Genie three, where it is right now, is that it's sort of, it's a multi-agent world, but there's only controllable enough in a, in a single agent setting, right? So a lot of the multi-agentness about the world is kind of baked into the, the simulation around you. They're almost like additional characters in the world, rather than being like controllable agents. You can control them if you wanted to with the world events, right? So you could actually control what the other agents are doing. But otherwise, it's always kind of implicit in the weights. And what you see is that there is some sort of like natural behaviour of them. So if you walk through a crowd, people will move out the way.

• 46:08Speaker 06

  For instance, if you're, if you, if you create a driving world, then when you drive around, the other cars move in a sensible fashion. And to go back to your actual question, I think you're saying almost the system can almost like bootstrap from itself and learn to, to learn, to learn sort of across the different agents in the system. I think the way I would see it right now is more that the, the model's sort of knowledge of, of human behaviours can distill into the, the ecocentric agent. And that's actually something quite powerful that we haven't really got with any other simulation tool, right? Because if the other agents are moving around sort of in a way that we do, then I think it might even be a way of, of our embodied agents learning things like theory of mind. Because they know, for instance, that if you're in a, if you're in a, if you're in

• 46:53Speaker 06

  a world walking around and say you go to cross the street, you sort of check the queues of the, of the drivers, for example, maybe there's not a crosswalk and you need to know when to stop. You can see that they're slowing down, so that's when you would go. And the other agents should be simulated in that fashion. So actually you can learn these kind of cues that you can't really learn any other way other than being deployed in the real world. And that obviously has safety, safety risks and probably wouldn't be an advisable thing to do with an agent that's learning from its own experience. So what we think with this kind of model is that agents can really learn these sort of social cues, things like theory of minds, how to operate within human-like other agents. But it's not the case that the model itself is then learning back from the agent that's collecting experience.

• 47:36Speaker 06

  That might be a future step, but not something we've really considered in this work yet.

• 47:40Tim Scarfe?

  Yes, fascinating. I mean, Shlomi, what do you think about that? I mean, certainly we, we use tools, you know, we have sextants and GPSs and computers and calculators and all these different things. And I mean, do you, do you think about the, the locus of intelligence being in our brains? Or do you think if we built rich multi-agent systems, or maybe even if we look at humans and L L Ms now, where do you think of the locus of intelligence in that system being?

• 48:02Speaker 03

  So I think there is a different types of intelligence eventually. And you know, as we make progress towards understanding intelligence and building intelligence, we end up building like initially separate models that can can like accomplish different tasks along different dimensions of intelligence. So as I said before, I think there is like, in a way, if you really think about it, like generating and simulating a world is not necessarily something that a person can do, right? We're not actually, that's something that some people say, okay, we have a word model, but definitely we don't have the same word model or inability of like Veo or Genie Free, right? We cannot really simulate, like if you tell me a sequence of events, I want output pixels, right?

• 48:45Speaker 03

  I can maybe imagine at the lower level of detail, what would happen if, for example, you would get up or if something happens in the environment, right? And I can plan accordingly. So I think there is like, it's not completely parallel. Like, we cannot just say that, okay, those models are exactly how we operate. But I think what we do see is that some capabilities that we wouldn't, you know, a few years ago, if you would come to me and say, you know, we'll be able to generate videos from text, I would say, okay, I, you know, it doesn't make sense to me. Like, I don't think that's gonna happen in few years. But it did happen. And other things that people thought's gonna happen way, way before, like maybe self-driving cars, now we are much better progress towards that. But they didn't happen as fast as people thought.

• 49:29Speaker 03

  So I think in this case, different types of intelligence made progress in different ways. And what I'm really interested in is seeing how those types of intelligence can work together. For example, if we have models that can simulate the world in a just in a different level than was possible before, and we have other models, for example, Gemini, that is able to maybe reason about the world in a different, maybe less visual way. When we bring them together, for example, what would happen? Would be, we'll be able to, and so like the examples that we've demonstrated of the SEMA agent that is interacting with Genie Free, right? Those are two separate models, trained completely separately. But then when they are put together, they can accomplish maybe a new thing. So I'm really excited about that.

• 50:12Speaker 06

  Yeah.

• 50:12Tim Scarfe?

  Yeah, that's amazing. There's also this notion that, so it's seven twenty P, you know, Genie three was just creating these these immersive, and I use the word immersive intentionally because as a video editor, I know that it's all about, it's a bit of an illusion. So you are trying to create a creative artifact that is just beyond the predictive horizon of the consumer and then they suspend disbelief. Right. And so in a sense, like we are cognitively bounded as observers. We see the world macroscopically. We see chairs. We don't see particles, you know, and so the world can have descriptions at different levels. And when when you interact with Genie three, do you see it kind of like traversing the levels?

• 50:56Tim Scarfe?

  Like if you zoom in on something, does it have a different description or is it limited in some way? How do you think about that?

• 51:03Speaker 06

  So in some of the examples we showed, there's the one where you're controlling this drone in the like by a lake and there's some trees and it's like very beautiful scenery. And you do notice in that one that when you focus your view in different areas, it definitely hones in on detail more. And so I think the model kind of learns that sometimes you don't need all this detail, right? And actually it should focus its efforts sort of with the focus of the agent. And I think this comes a bit from, I mean, our emphasis on this with this model is to have an agent centric, sort of egocentric often. They also can do third person, but a model that really feels like it's it's your view of the world, right?

• 51:42Speaker 06

  Rather than as a contrast to V O videos, right, which are more a much more sort of like like cinematic in quality, right? The whole video is very high quality, whereas Genie three, often it does feel much like a your own personal view in the world, which I think is is quite a different experience, right? And it does have this different levels of detail to it, right?

• 52:04Speaker 06

  In terms of things like more abstract representations, I think we're still kind of exploring it, to be honest. But it definitely has a slightly different feel to it, because especially if it's the first-person view that you get often when you're experiencing it.

• 52:17Tim Scarfe?

  How do you think about that? I mean, it's so difficult for us to know how these inscrutable models work, but do you intuit that it's simulating the world at multiple levels of resolution?

• 52:29Speaker 03

  So, yeah, it's a really interesting question and way of thinking about it, because when I first saw video models and in the simulation of, for example, fluid dynamics and other aspects of reality, I was like, how is it even possible to do that in so little time or compute, right, comparing to actually running the entire simulation? So I think that's first a surprising aspect of these models, but it does come with some limitations. I think what we basically see is that the models somehow find ways to simulate, as you said, in a way that looks good, looks reasonably realistic.

• 53:09Speaker 03

  I think we see it with video models, but as they get better, those kind of approximations become even better. And maybe that's a good opportunity to think about the difference in when we simulate the environment in an interactive way, it becomes much harder, because if, for example, you want to just spill water, like a video of someone spilling water on some surface, right? So if the model is a video model, it can just think about, like, try and generate the entire video end-to-end, past and future can be modified at the same time, and eventually you get some video that maybe looks real. But with Genie free, we have, because it's an interactive model, then the user or the

• 53:52Speaker 03

  agent that controls it can decide to intervene, they can maybe look from a different angle, and we have to create the entire simulation frame by frame in a causal way, and that makes the problem much harder for the model. Basically, it cannot change the past, right? Once the past happens, you cannot change it, like in the real world, right? And I think that's where we hope to see better physical simulation, but it also makes it much more challenging. So your question about different levels, for example, of reality, would it work? If I just zoom in and look at the molecules, right? And we're like, I think this is, this highlights the amount of computation that actually happens

• 54:36Speaker 03

  in the real world, that if we actually had to simulate it completely, that's probably be impossible. But some of our models find ways to, to approximate it to a certain degree that looks reasonable to, to the observer, which is us basically, yeah.

• 54:51Tim Scarfe?

  So another interesting thing is, um, we, we do this thing called thinking. And, and we know that neural networks, they, they are sort of roughly computationally limited, so that they, they can be trained to do a certain amount of computation in a certain amount of time. And, and that means that we can, you know, we can do lots of things, but there might be certain types of things, like for example, if you simulated someone solving a Rubik's Cube, you might find that for whatever reason, it just doesn't have enough computation to do that thing. So would there be an opportunity to create a variable computation version where for doing certain types of things, it could think more about it?

• 55:27Speaker 06

  That's a really interesting question. I think some folks in the team were also talking about this. So for example, um, if in the, in the future, if you wanted to be able to even write code inside the model, um, at that certain point, um, maybe that requires some different approaches, right? And I guess then we already do have models that can write very good code, I think, um, quite widely available now. Um, we also have models that can, you know, win a gold medal, the IMO, for example, right? And, um, maybe eventually you want to be able to do this inside the simulation, um, because that might be the next level, is to be able to really develop embodied agents that can blend these two different, like, tasks of physical tasks and thinking-based tasks. Um, and so, yeah, at a certain point, I think we will probably need to kind of cross that gap.

• 56:12Speaker 06

  Um, but for now, I think we're probably focused much more on the, the visual, uh, quality and more like physical simulation, rather than, um, this sort of like, um, more math and code type problems, which typically have more thinking style models. Um, but I definitely think it is. It's an interesting question. I think it's also something where the model definitely has this like physical knowledge in it, but I don't know if the model itself could describe it. It probably just has it implicitly in the weights. So another agent could actually learn probably about about the physical world from the model, but the model doesn't necessarily know it and can't tell you it, but it just sort of implicitly has that in the weights somewhere. And there's this kind of interesting duality in a sense, which goes back to the

• 56:56Speaker 06

  agent environment thing, which I think is much more of our belief is that right now this is kind of nice setup to have models that can focus on different strengths and simulating the future versus thinking and understanding the present.

• 57:09Tim Scarfe?

  Yes. What's your philosophy on that, on that, Shlomi? Because in a way, you've built something which is even higher resolution than a language model. So in principle, all the things a language model could do, as you were just saying, Jack, could kind of emerge from a model like this. So is your philosophy to kind of build a massive model that does everything?

• 57:29Speaker 03

  I'm typically thinking about this more from a practical point of view. I think there is definitely this kind of like a puristic approach, or we should have just one model to do everything. But I think when, when, you know, a lot of the challenges with modern machine learning comes from actually building, there is a lot of engineering and, you know, software and hardware design that actually, you know, to build those things, right, train them and running friends. And I think when we actually, we try to actually design those systems, there are a lot of constraints. And those constraints basically, basically kind of like impose on us some, some ways in which we can, we have to prioritize what we want

• 58:13Speaker 03

  the model to do. I think especially for Genie free, when we're bringing the real time capability, right? Real time is basically means that we have to generate frames very fast, right? Multiple times per second for the person or agent that interacts with it to feel like this is actually, you know, I can, they can move around and feel the responsiveness of the, of the model. So that sets some constraints on what the model, how much capacity we actually have. So I think when it comes to the point of basically to your question, can we have one model to encompass all of the aspects of intelligence we discussed before?

• 58:57Speaker 03

  I think it boils down to what are the set of requirements that we have. If we don't care about real time interaction, maybe we can do that. If we don't care about things like how expensive it is to run. But ultimately, we're trying to build models that are not just some, you know, they don't end up just being as a theoretical kind of exercise. We hope to actually bring them like other models for people to use and for to advance actual applications. And I think that's where we kind of have to make those decisions. And ultimately, we pick the type of capabilities we want to emphasize.

• 59:32Tim Scarfe?

  Very cool. And a 20 second answer, Jack, is there a sim to real gap?

• 59:38Speaker 06

  Well, it depends how you define it. I think that there's currently, sim to real is actually a bit of a conflated term. It's more sim to lab what people currently do. I think sim to really real can only really be achieved with a photorealistic world simulation tool like Genie 3.

• 59:53Tim Scarfe?

  Yeah. So you think this is actually a big step in the direction of?

• 59:56Speaker 06

  I think it's the only way to solve it, to actually get in the real world where there's people and other agents in general moving around, rather than just a very constrained lab-like situation, which has real world physics, but nothing else that's real.

• 1:00:08Tim Scarfe?

  Amazing. Guys, this has been an absolute honor. Thank you so much for coming on. And for folks at home, if you're developing on Unreal Engine, it might be time to, you know. Yeah. Anyway, cheers.

• 1:00:23Speaker 00

  Progressive's Accident Response can contact emergency services if you're in a major accident. The only thing worse than someone you love being in an accident is not being able to help them. So Progressive is here to help when you can't. So get peace of mind knowing your loved ones are protected with Progressive's Accident Response. Learn more at progressive.com. Progressive Casualty Insurance Company and Affiliates. Limited availability in California. Driver is responsible for emergency and towing charges not covered by the policy. Coverage subject to policy terms and conditions.

• 1:00:54Speaker 00

  When everyone's finally together, what makes mornings even brighter? Keeping things deliciously simple. Whether you're pouring up a glass of morning refreshment. Or topping off festive drinks for a family brunch, Simply Orange brings that 100% fresh squeezed taste to every moment. Deliciously bright and delightfully easy, Simply Orange is made for all your seasonal gatherings. Simply Orange, say yes to simple.