Fei Fei Li: The Race to Build World Models For AI

World Labs co-founders Fei-Fei Li, Justin Johnson, and Ben Mildenhall join a16z General Partner Martin Casado to discuss Atlas, their latest world model, and what it reveals about the pursuit of spatial intelligence. At the center of Atlas is what the team calls “new view prediction”: given images or views of a scene, the model predicts what that environment should look like from a different position in space and time. This brings generation and 3D reconstruction into the same model, and raises a broader question about whether predicting views could become a useful primitive for understanding the physical world. They discuss the technical bets behind the model, what it can and can’t yet capture, and the importance of dynamics, editability, and simulation as world models develop. The conversation also explores applications in creative work, architecture, and robotics, where Fei-Fei argues that one of today’s biggest constraints is access to real-world training data.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Fei-Fei Li?

  On the path to spatial intelligence, generating pixels that are truly spatially contextualized and grounded, that is the very hard step that Atlas has taken.

• 0:10Speaker 01

  We know LLMs are built on next token prediction. We've seen video models as being built on next frame prediction. Atlas is really new view prediction.

• 0:16Speaker 02

  This is the real place where AI can actually unlock a ton of value for people and their process. We're saying like 50, 100x reduction.

• 0:23Speaker 01

  There was a famous shot in the first Matrix movie where Neo is falling down. Exactly. They had hundreds of cameras viewing that angle on a green screen. On Atlas, we can do this with just three cameras. No studio capture, no green screen, no expensive calibration.

• 0:36Fei-Fei Li?

  No one has ever seen this result.

• 0:39Martin Casado?

  When you set out to do this, did you know it was gonna work?

• 0:41Speaker 01

  I was pretty sure. Each time we made the model bigger and each time we trained it for longer, it got significantly better.

• 0:46Martin Casado?

  Does that mean we're gonna get 40 video? Like go walk around?

• 0:50Erik Torenberg?

  Language models are built around predicting the next token. What happens when a model instead learns to predict the next view of the world? In this episode, Martin Casado sits down with World Labs co-founders Fei-Fei Li, Justin Johnson, and Ben Mildenhall to discuss Atlas and the broader challenge of building AI that can reason about physical space. They explain how Atlas combines generation and 3D reconstruction, why the team chose new view prediction as its underlying primitive, and what they learned trying to scale an approach that hadn't been tested before. They also get into what's still missing, including richer dynamics and interaction, and how world models could eventually connect simulation with robotics and planning.

• 1:32Erik Torenberg?

  Underlying it all is a bigger hypothesis. Could new view prediction play a similar role for spatial intelligence that next token prediction has played for language?

• 1:44Martin Casado?

  It's a big day yesterday. You launched a new frontier model, which has got amazing reception, which is still coming in. I think maybe a good way to structure this conversation, let's just talk about exactly what that was and then we'll go back to history and work our way back up. So maybe Justin, do you wanna talk about what was launched yesterday? Why it's significant?

• 1:58Speaker 01

  Yeah, so Atlas is our new next generation world model. It has three basic things. It can generate, reconstruct, and simulate the world. So within that, there's a couple of different major capabilities. It has really good camera condition generation. So you could input an image together with a camera trajectory and steer the model and have it generate video frames from any perspective you want. It's really good at sparse 3D reconstruction. You can input one or multiple, up to 100, frames that are views of the real world and use those to reconstruct the real world. And that reconstruction can take the case either of novel of video flying through the space or an explicit 3D reconstruction of the space. Then finally, it can be used for simulation. And for this, we show off these awesome bullet time videos, which got a lot of attention online. And then also robotic simulation.

• 2:37Martin Casado?

  What's a bullet time video?

• 2:39Speaker 01

  A bullet time video. This comes from The Matrix. There was a famous shot in the first Matrix movie where Neo is like falling down.

• 2:45Martin Casado?

  Oh, yeah.

• 2:46Speaker 01

  So and then remember in that famous shot, he's falling down, it's in slow motion, and the camera flies all the way around. The way that they did that shot is they had a ring of hundreds of cameras. So then he fell over in the studio. They had hundreds of cameras viewing that angle on a green screen. And then they used those hundreds and hundreds of cameras to make that famous shot in Matrix. But now with Atlas, we can do this with as few as three cameras. So no studio capture, no green screen, no expensive calibration. We can literally stick three iPhones on tripods, use these to take sort of a video of something happening, like someone shooting a basket, someone dropping a strawberry into a bowl of milk. And then from those three iPhone videos, we can then reframe the shot and imagine like freeze time, have the camera fly in as the milk is splashing up and get these amazing frozen time views. And we can do this with just a couple of cameras.

• 3:30Martin Casado?

  What is the simplest description of what Atlas does? What goes in and what comes out?

• 3:34Speaker 01

  Yeah, so one of the really core principles of Atlas, the most fundamental thing, is it does new view prediction. And this is a really fundamental primitive that we think is super exciting, a super new primitive for base models that no one's ever done before. Right, so we know LLMs are built on next token prediction. We've seen video models as being built on next frame prediction. Atlas is really new view prediction. Right, that given some number of views of a scene or a description of a scene, those go into what we call a spatial context that describes implicitly what is the world that we wanna talk about. Then you can point a virtual camera at an arbitrary point in space and time, and Atlas will understand what that world is supposed to look like from that position in space and time.

• 4:12Martin Casado?

  Ben, the video models out there all claiming to be world models and all claiming to have novel views. And can you maybe tease apart kind of more concretely how this is different from the myriad models that have come before?

• 4:24Speaker 02

  Yeah, I think what Justin was saying about the spatial context aspect is super important here. So there's many video models. A lot of video models actually got their claim to fame from their single image input or their start to last frame interpolation. Now we're starting to see models that can do this kind of omni-reference thing with 20, 30, 50 images. But what's key with Atlas is that it actually has a kind of like spatially grounded meaning to every frame you put into it. So it's not just an image that the model's gonna interpret whatever way it wants or you can kind of try to argue with it in the text prompting and get it to do something specific. With Atlas, every image actually has an associated three-dimensional camera pose. And that means that you can perform this task of reconstruction with an extremely high degree of precision, right? So if we had four views of this room, one in each corner, you can put those into the model

• 5:08Speaker 02

  and then get an exact replication of everything you see in this room. And it's not going to guess what's in the other corner, like a relationship between things. It's just gonna reproduce exactly what you give it. And you can also do that in a kind of creative or imaginative sense, too. If you take two photos from different AI generations or real world locations, you can actually position and stage those to build these kind of intentionally directed fly-throughs that are really governed by exactly the precise place that you put the content you want and where the camera's gonna look and travel, which is very different, I think, than the kind of like more slot machine effect you get of having to retry generations over and over with just that kind of higher level of text control you get with video models.

• 5:47Martin Casado?

  Is this just kind of an obvious, scaled up version of a traditional video model? Or is it a new architecture?

• 5:54Speaker 01

  I think it's a pretty new thing for a couple of different reasons. One that we talk about is it does both generation and reconstruction jointly in the same model. Like Ben was saying, this thing can take a couple of views of this room and then reconstruct everything in this room exactly as you see it. And historically, reconstruction has been its own subfield in computer vision with its own specialized task, its own specialized models. And generation is what all the text to video models are really good at, like with all the big diffusion models we've seen in the last couple of years. And those are great for creative applications. I wanna imagine something that's never been there before. But now with Atlas, for the first time, we're putting these two different parts of visual intelligence together in one model. So it can do both 3D reconstruction and generation together in one architecture. So to do that, we had to make a couple of changes.

• 6:37Speaker 01

  One is we had to make it multimodal from the start. So this thing natively works on text. It works on images, it works on videos. It also works on camera poses as a native input to the model, which I don't think anyone's ever done at the pre-training phase before. And it uses 3D as a native modality that it works on. So this thing from the beginning was designed to be natively multimodal in a way that no one else.

• 6:56Martin Casado?

  Sorry, I just, I don't know this space super well. By 3D, is this like depth or models? Like, what does that mean?

• 7:04Speaker 01

  Yeah, so the formulation we used so far is depth maps. Right, so right now, when you have a frame that has a virtual camera telling its position in 3D space, that camera position and camera parameters are a native input to the model. And then what attach to that camera position, you can have both like RGB telling you what does that position in space look like? And you can have a depth map that tells you what is the spatial structure of that position in 3D space. So then text, image, video, 3D cameras are these modalities that this thing all does jointly in a multimodal way.

• 7:32Fei-Fei Li?

  I wanna add something because I think what Justin just said is actually so important and also what Ben said, that it's underappreciated. It's the first time we have a unification of pixel generation and pixel reconstruction. In the world of computer vision, this field has been around for more than half a century. Sitting here, having been in this field for decades, I cannot tell you how many PhD theses have been written on the problem of reconstruction or novel view synthesis. And also, our field traditionally have multiple tracks. You go to a computer vision conference, you have the pixel generation track, you have some recognition track,

• 8:16Fei-Fei Li?

  and you have 3D reconstruction track. This is an elegant model that combines or unifies the problem of reconstruction and generation by anchoring on viewpoints, the viewpoint estimation. And that's just incredibly powerful.

• 8:33Martin Casado?

  Can we take a step back and then maybe you just fill something out? So when you started the company, I remember you saying you wanna tackle spatial intelligence, right? And now we have this new model. And so as a layperson, it feels very gentle to me. You've got next token prediction and this is next?

• 8:47Speaker 01

  New view prediction.

• 8:48Martin Casado?

  New view prediction, right? And so like you can get one viewer set of views and you get a new view. Can maybe you pencil out like how this is a significant step to this general problem of spatial intelligence? And maybe by starting to describe what spatial intelligence is?

• 9:01Fei-Fei Li?

  Well, spatial intelligence eventually must enable us to both generate what the space is, reason within it, and being able to edit and interact within it. Now, we can argue is it 3D or 4D? Ultimately, it's 4D with the time dimension. But even just 3D, these are the fundamental tasks that one has to do or spatial intelligence has to enable. And then we talk about with that, you can render, you can simulate, and you can plan actions. But to do that, a fundamental problem to solve is to understand the geometry and structure and the physics of the space. And I do believe Atlas is a significant step forward

• 9:45Fei-Fei Li?

  because now with every single frame, you can generate and estimate an important piece of information, which is the viewpoint, the camera pose. And that is the most critical information one needs about the geometry of the space. And that can lead to all the emergent behaviors we see in the downstream of the model, which we showed in the blog. So on the path to spatial intelligence, generating pixels is definitely a early step, which we have seen with what you call it, gazillions of models. But generating pixels that are truly spatially contextualized and grounded is absolutely another major step. And that is the very hard step that Atlas has taken.

• 10:30Fei-Fei Li?

  We definitely have, you know, we can just keep going here, right? Like there is the fourth dimension of time, which will bring in dynamics and there is more higher fidelity simulation and delineation of the space. So this is part of the roadmap of spatial intelligence.

• 10:50Martin Casado?

  Great, yeah. I mean, I definitely wanna like dig into like where this is going. But first, maybe let's talk about getting here. How long has World Labs been in existence?

• 11:01Fei-Fei Li?

  Two and a half. Two and a half, yeah.

• 11:03Martin Casado?

  And so you've actually released models before. So why didn't you just jump right to Atlas?

• 11:10Fei-Fei Li?

  Great question.

• 11:11Martin Casado?

  It's so magic, right?

• 11:12Fei-Fei Li?

  Justin's team needs a lot of chips.

• 11:15Speaker 01

  Yeah, we need a lot of GPUs to actually scale this thing up. So what one of the last year, we released our marble world model. And that was the first kind of big, major world model that we put out. That powers our current marble product. And marble and marble is really cool. Marble can take images, it can take videos, it can take text prompts and use these to generate 3D worlds. But one of the biggest differences between marble and Atlas is exactly what is that output modality? So marble was really focused on Gaussian splats as an output representation. So whatever you're inputting, it's going to output a 3D world represented as a Gaussian splat. And Gaussian splats are really useful, right? They're really nice, they're easy to render, they are, they can, they can render efficiently on mobile devices, on VR devices. They can interoperate with other, with game engines, with simulation engines. So there's a lot of nice things about Gaussian splats.

• 11:57Speaker 01

  But, you know, I think that was kind of a bottleneck in the previous marble model. So what we did with Atlas is redesign the thing a little bit. And we realized that we need to bifurcate these modalities earlier. And actually have these things, all these modalities working in a more unified way in the model. So now with Atlas, the fundamental primitive is not like a good generate a Gaussian splat world. The fundamental primitive is, as we said, new view prediction. And that can generate RGB frames, that can generate 3D. And we can use those to generate a beautiful Gaussian splat worlds when you need them. But we don't need to bottleneck our outputs through the Gaussian splats when we don't need to. And that was, that actually took a lot of, you know, blood, sweat and tears to understand like, what are all the pros and cons of these different representations? But that's one part of it.

• 12:41Speaker 01

  The other part is you gotta like climb the scaling ladder, right? You gotta like work your way up and like do smaller experiments, do smaller models, like to build your conviction on what's going to work and what's going to scale. And there's, there's, you know, if you could instantly know the right thing that's going to scale, you know, you should just do that. But when we started the company, the world was a very different place.

• 13:00Fei-Fei Li?

  There is no scaling law of spatial intelligence.

• 13:03Speaker 01

  Right. So like when we started the company, like the world was in a very different place. The tech was in a very different place. We had a lot of ambitions, but where we wanted it to go. But it took a couple, it took a couple iterations for us to hit upon this formulation that we thought is actually, is actually like, this is the one, this is the one that can scale out.

• 13:19Martin Casado?

  You know, Ben, you know, being the creator of Nerf and doing a lot of 3D and reconstruction. So it's not so obvious to me that like, if you have multiple views, that you actually end up with a 3D thing. But like, you've kind of like made a career of ending up with a 3D thing. So maybe talk a little bit about like kind of that step.

• 13:35Speaker 02

  Yeah, yeah. Yeah, I mean, as you said, I've spent many, many years of my career, the vast majority of my career, actually working on producing 3D things from images. And this is actually something we talked a lot early on in the company, even of like, is this going to be the approach that produces 3D, right? Are we going to synthesize multiple views and then build 3D out of that? Are we going to try to go direct to 3D? Like, there's been a lot of uncertainty in the field around like, which of those approaches kind of will win out? Or will kind of like reap the best advantages earlier on? But I did have a lot of conviction just from seeing the kind of power of what I would almost call the brute force scaling. Scaling in a very, very, very small baby scale, not like a real model scaling, but the scaling of dense reconstruction that we had seen happening over the past three years before.

• 14:20Speaker 02

  So basically we put on-

• 14:21Martin Casado?

  Why is dense reconstruction dense?

• 14:24Speaker 02

  Yeah.

• 14:25Martin Casado?

  Because I know we're going to talk about sparse and I want to make sure that people understand what is dense and what is sparse.

• 14:30Speaker 02

  Yeah. So I think this is actually, even on the kind of like business and commercial side, I think been one of the challenges of productizing 3D reconstruction technology, like at a fundamental level, right? People kind of don't, in a, in a casual sense, like you think I took three photos of this object or I took six photos of this room. Like, I, I look at the photos, I can understand in my mind, like how those piece together, I can kind of fill in the gaps and get it. But there's just never been really any kind of reconciliation between those like really data-driven priors. And then the kind of brute force dense reconstruction, which it actually is much more akin to almost like scientific or medical imaging, what we did in dense reconstruction, right?

• 15:11Speaker 02

  You basically have to say, every single thing I want to appear in this reconstruction, I need at least three or four views of it. And if you think about that, like even just in this room, right? There's like, under the microphone, under the table, between every different crack and crevice and the plant leaves, right? To actually truly get a picture that covers every one of those spots. It's this like very tedious and exhaustive effort to walk around the room. I think you've all seen me running around various places like capturing them. It takes, you know, for, for someone who's well-trained per se, like it can take minutes. But if you hand a casual consumer or even some like kind of professional trying to do this for the first time, an average like cell phone camera or capture device, like it's going to take them probably an hour.

• 15:51Speaker 02

  I've seen someone for their first time trying to scan a multi-room environment, spend like two hours walking through it and get enough coverage. And that's just, this like very, very exhaustive and tedious loop. And so, yeah, when we say dense, we really mean dense. It's like this room, I want like a hundred photos, so many photos, right? I want like 100, 200, 300 photos of this room to capture it. And what we're trying to do is bring that down to like three, right? Wow. We're saying like 50, 100x reduction. And then that's at that scale where it just completely like flips that calculus on its head of like, what type of captures you reconstruct? You can go back to existing imagery you have, you can go to stuff you find on the internet and even build scenes out of that. You can go to casual videos and like kind of unearth a lot of footage in the past we would never have treated as reconstructable. And go back and like bring it to life as 3D potentially.

• 16:37Speaker 02

  This is something we've been playing around with a lot with Atlas, right? Like taking old clips, like I've taken a bunch of my own old captures that never worked before. And then put them through the system and then kind of seen a reconstruction of the first time. Or taking my old captures and throwing away 95% of the photos I took. And, you know, imagine angles that I never would have gotten from a traditional kind of like Nerf or splat type reconstruction.

• 16:58Fei-Fei Li?

  One thing that's underappreciated on the website of the demos is the Stanford demo where Ben showed anywhere between three to 25 images, you can reconstruct that entire Stanford quad. But the thing is, we had to show it from an aerial view. But every single input image is Ben standing on the ground, taking a picture from the ground. So everything you see are generated, but according to the laws of reconstruction. And this is really magical.

• 17:33Speaker 01

  And this is where like generation and reconstruction need to interplay in a really fundamental way to solve this problem. Because under the classic kind of reconstruction stuff that Ben was talking about, like the reason you need so many views is because I need like multiple images. And I need to triangulate this point in 3D space and see it from multiple viewpoints. So that means like, that's required in the traditional version. And on the flip side, anything that wasn't captured in these views, like any pixel that was not visible in one of the input views will be a hole in a 3D reconstruction. Because fundamentally, like if a thing wasn't visible in the input views, you know, you need to imagine it to fill in the gaps. And that's fundamentally a generative process. So even in this room, even if we set Ben loose with a DSLR and like let him like capture like hundreds of views of this room.

• 18:16Speaker 01

  Even the world expert on doing these dense captures is still going to miss some spots. Like he's not going to get like underneath all of the microphones or underneath all the tables or like in between all the chair legs. You're always going to miss something, no matter how many views you get. So that, that's where you need generation as another mechanism in the model. Because you're never going to get everything. So you need to have some generative capacity for the model to imagine, oh, based on what I'm seeing, then like first triangulate what I, what I can see. But then fill in the gaps of the stuff that inevitably, inevitably was not captured.

• 18:47Speaker 02

  Yeah. And there's something like super cool about this, that LLMs have really understood this for, for a long time, right? There was kind of almost these like context wars, like the first couple of years, I was like, oh, we got to 128 to 26 to like 512, we got a million, right? And everyone kind of understands now at a pretty tangible level the value of, you know, you crank your context length to high when you're using your coding model. It's a hard problem. Like everyone has a feel for that. But like, no one has pushed that at all on the image and video model side in the same kind of like principled way. Like, no one's out there trying to like put a, like an hour long video through and do a needle in a haystack retrieval of like a frame at the 37 minute mark. Whereas with reconstruction and generation, you actually have the same exact thing of like reconstruction is just like generation with a really long context. And you put a lot of stuff in it, right? Like, that's the way to actually build this continuum where you kind of bridge between those two things.

• 19:32Speaker 02

  And like Atlas, like being able to, like, this is something we can never do with marble. Marble had this kind of fundamental blocker of like, you couldn't really jam more than honestly, like a couple of images in. But Atlas, I can go and I can actually take like a 64 image capture and do like a fly through of an entire house. And everything is grounded by being, you know, seen or like almost seen or like slightly extrapolated from what's not there. But you're just getting these, you know, I'm taking captures I did with 2000 images of a multi-room house and taking it down to like 30, 40 inputs. And the fly through looks like basically the same. And this is just like totally inconceivable before. And it's all enabled by building this gracefully scaling kind of context window that you can dump stuff into.

• 20:09Martin Casado?

  And so the way to think about it is like the, the sparseness of the pictures that you physically took. And then Atlas as a model creates the rest of the views. And then you use classic reconstruction techniques. Is that roughly the way to think about it or?

• 20:23Speaker 02

  In some sense, yeah, yeah. I mean, that's a video of Atlas, it's like, you can take however many inputs you use, you have down to like a single view. And then you can almost use Atlas as this, this rendering engine to produce anything else you want, right? You can, you can navigate it.

• 20:35Martin Casado?

  Like the virtual camera.

• 20:36Speaker 02

  Yeah, exactly. You can just, you can say like, okay, I have a picture here, I have a picture there, there, there. You can make a couple of those, then you can stage a dense fly through. You can do this in sequence because it's an autoregressive model. It's up to you, right? To kind of pick and choose what you add interactively into the context as you generate.

• 20:52Martin Casado?

  I mean, the thing that I just blows my mind is, listen, I just have a very simple mental model. I, I, I have four pictures and then I've got to like have the model extrapolate between them. And then it has to fit when you're trying, like it's got to be 3D. Cause like, and I always think of these diffusion models as like being visually great, but not accurate. And so like, and I don't even know if there's a question here, but like, how is, like the room fits? Like, how is it that it's 3D, 3D consistent? Is it just lots of data or?

• 21:25Speaker 01

  Yeah. I mean, it's a, partially it's a belief in the scaling hypothesis, right? Like, you know.

• 21:30Fei-Fei Li?

  It's faith.

• 21:30Martin Casado?

  Did you, by the way, but I have to ask, when you set out to do this, did you know it was going to work?

• 21:35Speaker 01

  I was pretty sure.

• 21:36Fei-Fei Li?

  That's, so, so, okay.

• 21:40Martin Casado?

  Were you sure?

• 21:42Fei-Fei Li?

  I think three of us have total conviction about the scaling law. That, that I think we do. I do think the exact architecture choices and data mixtures is where the, the devils are in the details. I, you know, have watched Justin and his team going from, we really don't know how long this is gonna take, to, oh, maybe sign of life, to, wow, this is gonna work. So it, it, no one, what, no one has done it. But I think the hypothesis, two hypotheses, one is scaling law hypothesis, another one is uh next viewpoint prediction. We had a conviction of these two things from early on.

• 22:23Speaker 01

  So I think I was very convicted that it was going to work. I was not sure it was gonna work this well at this fast, right? Like, I thought there's a chance that we do this, maybe, like, it's not clear that, like, the first cycle of pre-training a new model with a new architecture and a new paradigm. Like, the first cycle of that working is insane. So I thought there was a chance in which we had to, we might have had to do a couple more turns of that, of that pre-training cycle before we got to the level of quality we were, we, we wanted.

• 22:47Martin Casado?

  Is there, is it, uh, are we kind of like at the end of like the scaling for this architectural approach? We need another breakthrough, or is there?

• 22:54Fei-Fei Li?

  No, no, we're at the beginning.

• 22:55Martin Casado?

  Really?

• 22:56Fei-Fei Li?

  Yeah.

• 22:56Martin Casado?

  Without changing, without changing the architecture?

• 22:57Speaker 01

  Yeah, I think we're basically at the beginning. I think we're basically at the beginning and are basically limited by compute at this point. Right? Like, data is very important, as Fei Fei likes to point out, but like, everything has a bottleneck. And I think the main bottleneck on continuing to scale this thing is actually training compute. Right? Like, during development, we trained a sequence of models. We read about this in the blog post a little bit, but we trained a couple of models, the like, the first couple of rungs of the scaling ladder. Um, and each time we made the model bigger, and each time we trained it for longer, each time we put it on more chips, like, it got significantly better. And the model size that we, like, the model that we showed in the blog post is obviously the biggest and best one that we trained. But the thing that was limiting it was not the scale or the data or anything like that. It was literally like, we had a deadline of when we wanted to release this thing, and therefore we backed up what we could afford to train in that deadline.

• 23:41Fei-Fei Li?

  But here's a little bit of an insider story, right? Like, Justin and team are, are training the, from the smaller and slightly bigger, you know, are train, having these roadmaps. And then there was one day in summer, early summer, that it's not even this, the current Atlas model size, it's a smaller model. And then Ben, Justin, Ben feed it into, you know, the viewpoint generation. And remember that famous table, the garden table for NERF paper and many papers? That overnight I got a slack, I mean, we all saw the slack from Ben that our camera flew through under the table.

• 24:22Martin Casado?

  With the soccer ball?

• 24:23Fei-Fei Li?

  Uh, yes, with the soccer ball.

• 24:24Martin Casado?

  Is the soccer ball emergent, or is that in the?

• 24:27Fei-Fei Li?

  It's, it was real, right?

• 24:29Martin Casado?

  Okay.

• 24:29Fei-Fei Li?

  That morning, the three of us looked at each other in the eyes and say, that's it, this is, we're gonna build this. Like, we, we made a decision within five seconds, how, this is just a, no one has ever seen this result.

• 24:44Martin Casado?

  Ben, um, can you talk through maybe more specifically the use cases? So, so, uh, World Labs has historically had a lot of users that were creatives. And they use it for like consistency and, you know, like whatever, 2D images and for movies and for 3D and for games, et cetera. And so maybe can you talk about how this extends use cases or cater to the existing ones? And then we'll, I would like to talk about robotics, actually.

• 25:07Speaker 02

  Yeah, sure. Um, yeah, I mean, it, it's kind of funny, actually, one of the kind of main ways we even saw people using Marvel plays exactly into this new view prediction case. Like, a lot of our.

• 25:17Martin Casado?

  Marvel being the previous, sorry, yeah, Marvel, our previous product.

• 25:20Speaker 02

  Like, people would take that product, put an image in, get a full 3D scene as a Gaussian splat, take a couple screenshots of it from different points of view at least, right? And we're like, we can just make those images that that's a different data role, right? So, so I think like, and you know, there's a lot of degradation there, they're like, oh, this flat could look better. And it's like, okay, what if we just generatively modeled those viewpoints with that exact modality of control? So I think like, even that core capability of just like view synthesis, um, it's sort of been this academic problem for a long time. But in the sense of, oh, you're gonna do this really dense capture, like, like generative view synthesis is relatively quite a new problem. And we just see so many people who, uh, in, in this creative pipeline, right? People have a multi-stage workflow, right?

• 26:01Speaker 02

  I don't think there's a single person out there using one model, ethic model, not even C dance or whatever, for, for their entire, uh, task. Uh, people will have this like, you know, kind of a bunch of storyboards and mood boards of images they pull out from like their favorite collection of image models. And then they'll go to different video tools and like, build those together as keyframes, then they'll go and like clip and edit those later, right? So we were seeing this like sort of, you know, niche, but very specific use case for Marvel as just providing that like sanity that you can ground your generations in some kind of 3D consistent world, right? People, you know, I don't, I, I've thought with, with various image models to ask them to like, give me different viewpoints of a room. And every time you can just look and see, oh, things kind of moved around, like, it's not stable. And like, even that one seed of a use case, I think, kind of signals that there's this value.

• 26:44Speaker 02

  And there's hiding under the surface there, like, there's just decades of people being used to persistent 3D state, like virtually modeling what they would be doing in the real world and having, you know, a stage and props and like elements there, whether it is for a, a movie or a show or a marketing shot or like building out game environments. Like, this, this statefulness and persistence is so key in how people think about spatial reasoning and like developing an environment over time. Like, people don't think in this ephemeral, like, generate a thing, generate a thing, like, just throw it away, keep my text prompts. Like, people want to build this like collection of assets and like model a world in that way. So we're, we're trying to provide like, again, with, with this spatial context mechanism and other things, like, we're trying to provide that level of control and precision,

• 27:29Speaker 02

  uh, and the ability to ingest different modalities of input, starting with the posed images. But, you know, we want to give people more control over the elements of the things in the scenes they're looking at and editing and interaction and all that as we go forward. Um, and I think that that, it unlocks like further use cases in those areas we're already seeing. But, um, also expanding out into kind of any place people want to create a virtual replication or, or like, you know, a pre-imagination of a real world space they need to build, right? For architecture and construction, like, I talked to a guy at some point building booths for conferences, right? There's just so many things in the world you don't think about and need to be fabricated. And every single one of those basically goes through this like pretty painstaking virtual design phase.

• 28:08Speaker 02

  Uh, and of that process, like, the part where you go into 3D software is kind of one of the most like arduous and like labor-intensive parts right now. Like, like taking feedback on a 3D design from kind of like verbal commentary or sketches or really, really quick, um, stuff you got from like a creative director or like a design director or an architect or whatever, like, mapping that back into the 3D representation is like 95% of the work, right? You can have a meeting, get feedback, and then you go back and do a week of revisions. And that's just because like, our software is kind of decades old at this point. And it's, it just never became as intuitive as, you know, playing with Legos or like pottery or doing this stuff with your hands or sketching with a pencil.

• 28:48Speaker 02

  Um, and this is the real place where AI can actually unlock a ton of value for people in their process, whether it's a creative application or something more industrial or design or, or whatever. Uh, and that like really motivates me to kind of build different flavors of, of our model to cater to those kind of people.

• 29:05Martin Casado?

  Yeah, I, I, I can understand how it helps with the creatives, because like Marvel did that, and also how that extends to things like design or architecture. Uh, Fei Fei, you acquired a robotics company, it's just like.

• 29:16Fei-Fei Li?

  We just talked about it too.

• 29:18Martin Casado?

  I know, I know. But it's less obvious to me, especially in the context of Atlas, like, how that maps to robotics. So if you wouldn't mind just penciling that out.

• 29:25Fei-Fei Li?

  Yeah, actually, Atlas is a, a, a key part of the puzzle. So, um, we acquired this company that was formerly known as Scenics. And what is their key technology? Right now, their key technology is a system that goes from real to sim and then, and then sim to real. And what does that mean in robotics situation? You want to train a robotic arm to, you know, uh, figure out how to, um, do cabling, let's say, in a, in a industrial setting. Well, you need a whole bunch of data to first train a robotic policy to do these cable, cables, cabling activity. And then you want to evaluate if the robotic policy is doing a good job.

• 30:09Fei-Fei Li?

  And then you deploy the robot into the cabling environment.

• 30:13Martin Casado?

  Yeah.

• 30:13Fei-Fei Li?

  In order to train, what you, what this company, uh, uh, Scenics and now our robotics team, uh, used to be doing is do exactly what Ben was saying, dense reconstruction. You take, you know, pictures of a situation and then, and try to reconstruct that environment. It's excruciatingly painful, takes a long time, laborious, and it really blocks the velocity of robotics simulation, real to sim, right? So Atlas really is the next generation technology for, for that. And this is not just for robotics cabling or anything.

• 30:52Fei-Fei Li?

  We should zoom out and recognize the biggest problem right now in robotics is actually data. One day it'll be chips, but for now it's data. Because it's so hard to collect real world data where robots, uh, you know, are operating. Um, and, uh, and, uh, in order to, uh, not only you need to, uh, collect the data of, let's say, the cabling situation or dishwashing situation or whatever. There is also a very important step called randomization, is that you have to take the same environment and then randomize the conditions. So the cable doesn't literally only, you know, uh, bend this way, it can bend a different way.

• 31:35Fei-Fei Li?

  Or the box can have different sizes, colors, different lids and all that, and, and or in different parts of the scene. So you have to go through a real to sim situation in order to get enough of that data in addition to other data you can get from internet. So this real to sim, uh, step will be, um, you know, really helped by, by Atlas. That's just the first part of this, is meeting the, the, the robotics needs in the current technology. Cause we don't yet have a, a frontier foundation model that's robust enough for, for, uh, robotics. But Atlas is a, uh, only model, is a multi, uh, um, multimodal model.

• 32:19Fei-Fei Li?

  It takes up different kinds of input and generates different kind of output. You can totally imagine the next step is Atlas, um, taking in da, uh, data that's in the, uh, in, that's dynamical. And that can really start to bridge the gap between, you know, action planning and, uh, and, um.

• 32:44Fei-Fei Li?

  And robotics and the Atlas output. So that's on that roadmap. What were you gonna say?

• 32:50Speaker 01

  Yeah, I was gonna say there's something fundamentally different about training a robotics policy compared to really any other application in AI we've seen before. And that's like, if you're generating a piece of code, like you're generating an image, you're generating a video, the model is fundamentally creating this artifact. And that artifact, like, there's a lot of examples of artifacts that you can go out on the web or somewhere and collect, right? You want to generate images, there's a lot of images out there. You want to generate videos, there's a lot of videos out there. You want to generate a code base, there's a lot of code bases out there you could learn from. A robotics policy is something fundamentally different. It's not producing a static thing. It's instead a policy that's gonna go out into the world, make actions, and like, try to achieve a goal. And the world's not always gonna respond the way you expect, right? Unexpected stuff is gonna happen.

• 33:31Speaker 01

  So a robotics policy is like fundamentally an agent that is out in the real world, interacting with the real world, and stuff happens. So you need, like, a critical part of that is those policies during training need to be exposed to every possible thing that could go wrong during their deployment. And that's where simulation is really key for robotics, right? So there's the, and then there's two angles on that. Like, one is the kind of classical simulation. You can go out, you can go and like, go to your favorite physics engine and like, try to imagine creatively as a human designer, what are all the scenarios that might happen in this, in this, in this, when achieving this task? And then try to write explicit code that models them all. That's one angle, and that's an interesting angle with coding agents, like, that actually gets supercharged too.

• 34:15Speaker 01

  But there's another angle, which is try to, more data-driven simulation, right? Like, maybe we can, can we have a learned model that can understand how the environment, how the world is gonna respond to actions? And maybe it might respond in unexpected ways sometimes. Then could we build these neural simulators that are trained on as much data as we can, then use these neural simulators, these learned neural simulators, you know, as a simulation bed to train robotic policies? And that's, that's a really interesting future direction, Valis. But, but then, it doesn't stop there, right?

• 34:47Speaker 01

  So, but once you have, you know, this learned simulator, like, this learned simulator kind of already has in its like, mental brain, like, it understands the world, it understands how the world is gonna respond to actions, and why doesn't the simulator itself become the planner? Right? Like, the same, and that's kind of the core thesis that we've had around world models and their generality, that there's some core stuff that a model should understand around generating worlds, simulating them, understanding how they appear in different search situations, and, you know, understanding how the world's gonna respond to an action is highly related to imagining what kind of action I need to take to make the world respond in a particular way.

• 35:25Martin Casado?

  What, what, um, one piece of feedback that I got, I think, by the way, congrats on the launch, it was overwhelmingly positive, I think it was probably the most significant model launch this year. And, you know, every single glowing thing, but one person, who's an expert in the space, who I texted was like, "What'd you think?" And it's a great person, "It's fantastic, it's amazing, but there needs to be more dynamics."

• 35:46Martin Casado?

  And so it seemed to be, at least in the robotics case, but generally, it was kind of ideal to actually have a world that moves. And so, maybe talk a little bit about that, and then any other future directions that, A, you're comfortable sharing, but you think are worth talking through?

• 35:59Speaker 01

  Yeah, I mean, like, dynamics is clearly gonna happen, like, actually-

• 36:02Fei-Fei Li?

  We have baby dynamics.

• 36:03Speaker 01

  We actually do have baby dynamics already, and this is something I think people didn't quite appreciate, we didn't really highlight in the blog post, but like, the previous Marvel world model, it was like fundamentally static. Like, the, the model just like could not handle any dynamics at all. And that was just like baked into the model architecture, baked into the training, like, the whole thing was fundamentally static. We already knew that that was a big problem post Marvel, and we already fixed it in Atlas, right? Like, the Atlas architecture is already fundamentally supports dynamics. The Atlas training data fundamentally has dynamics. And if you look carefully in some of the videos that we even posted-

• 36:34Martin Casado?

  I saw, I saw, you see it from the little bit.

• 36:37Fei-Fei Li?

  Yeah, the waves, the water waves.

• 36:38Speaker 01

  Yeah, so like, some of the examples, there's like waves in the water, like, in some of the like air-generated aerial views, there's like little cars moving around. So like, dynamics is actually already in this model.

• 36:48Martin Casado?

  But I said, by the way, dynamic seems very problematic to me if you're trying to reconstruct 3D from multiple views, right? And so like, are these things like at odds, or?

• 36:56Speaker 01

  No, so it's actually, one of our theses here is that like, you know, if you're gonna do fundamental 3D reconstruction, you actually want to have no dynamics. Like, you want to be able to model like, exact views of the scene with exact frozen time. But then, like, this is actually kind of a problem with our previous Marvel approach, right? Like, there, like, you can try to find data that's fully static, but that's really hard to scale and really hard to get more of. And the thing we realized is that, even in the case where I want static output in the end, the best way to get it is actually expose the model to dynamics, right? Like, expose the model to as much dynamic stuff as you got, as much static stuff as you got, and let the model figure out how to factor out the dynamic stuff. So, especially in like the-

• 37:33Martin Casado?

  Oh, interesting.

• 37:34Speaker 01

  This is like, and this is actually, so in, in the Atlas pre-training already, like, it, it saw a ton of dynamics in the pre-training already. Then the post-training that we did specific to this checkpoint in this release, was focused a lot more on, on static stuff, focused a lot more on spatial movement and less on temporal. But like, we already had, we already, like, I'm pretty sure this, the, the pre-training checkpoint already has a lot of latent dynamics in it. And this is something we're gonna improve quite a lot going forward.

• 37:58Martin Casado?

  So Ben, does that mean we're gonna get 4D video? Like, go walk around?

• 38:03Fei-Fei Li?

  You can see the smile on their face.

• 38:06Martin Casado?

  So I, I actually can see if, like, you just stopped now and you only did kind of bigger, you know, faster, better, you could build almost an entire industry. Like, it feels like a very horizontal primitive, and then, if you did nothing else. But are there other things that are not just kind of bigger, faster, that you're excited about for the applications you're focused on, which tend to be kind of more on the kind of content created 3D side?

• 38:27Speaker 02

  Yeah, I'm really excited about pushing that kind of multimodal aspect. I think different modes of control is so critical here. I think like, it's super underappreciated, especially in the academic community, how critical it is to add control conditioning to these models to kind of get out what's inside. I mean, honestly, this dynamic versus static-

• 38:42Martin Casado?

  I'm not sure I even understand what those words mean.

• 38:44Fei-Fei Li?

  To translate in layman's language is to start with editability. I think editability is, is the key here.

• 38:51Speaker 02

  Yeah, so I mean, this is something we've seen in like, sort of, single image models, and starting this year in video models is starting to be unlocked in terms of, oh, like, getting that flavor of like multi-turn, or really like intuitively interpreting, like, I want like this person and this object and this thing to happen, and kind of combining those all together in like one pastiche without having to do a lot of like manual work with the system, like, it just interprets it, like, kind of frontier image models are kind of there, right, for, in terms of editing. But we haven't seen that propagate out as strongly into video yet, and then into world models, right? We've seen some really kind of toy examples of, oh, I can like, put in a sentence and like, you know, a dinosaur appears or something with these like sort of real-time models. But I want to like turn that up to really industrial strength and make that, because

• 39:34Speaker 02

  like, the trick here is, you gotta add control but not compromise the quality of the model, or it just becomes a party trick, basically. Like, it's like, no one is going to seriously think about swapping their like, cutting-edge frontier video model usage for your model if you give them extra knobs, but the quality degrades. So I think it's really that game of like, how can we maintain like the high bar we've set, was the outputs we're able to get in the current model? And then add all kinds of interesting stuff that people will ask us for in terms of like, I want to interact with the scene, or control the layout, or control like, the identity of the objects and the things that we're seeing within there, or control time, right? And I think that's like an access where it opens up like a ton of really interesting product and interface work, the more complexity you add there and richness in terms of, kind

• 40:17Speaker 02

  of like enabling you to really think about like redesigning almost from scratch the way people interact with sort of like stateful, you know, 3D worlds in the computer. Like, that's, that's really the end goal here, is getting like all the capabilities you need to build that kind of system.

• 40:31Martin Casado?

  Awesome. Anything you'd add to that as far as new functionality that you'd be excited about that's not just bigger, better?

• 40:37Fei-Fei Li?

  I think for me, let's go back to the first principle of intelligence. Intelligence is not sitting there stacked and just seeing something or interpreting something when it comes to space and physical space, right? It's really this, uh, closing the loop between seeing and experiencing and interaction. So just thinking about going up that ladder is exactly what Ben said.

• 41:00Speaker 01

  I think one interesting notion there is this notion of AI completeness. Have you read this before?

• 41:03Martin Casado?

  Yeah, yeah, I have, yeah.

• 41:04Speaker 01

  So like, everyone-

• 41:05Martin Casado?

  Yeah, I, I hear about AI complete, by the way, in terms of LLMs, which is like, you have to be basically, you know, like the smartest LM to answer the question, what the smartest LLM will need to answer, or you have to solve general intelligence.

• 41:16Speaker 01

  No, no, it's obviously, it's a, it's a connection to Turing completeness, right? Like, the idea of being that like, a, a, a task is Turing complete, like in classical complexity theory, if like, I can take any class and any problem in this category, reduce to that one problem, right? Like, three sat, the fast example, right? So you can take any NP-hard problem and reduce it to three sat, therefore you can use three sat to solve any, any problem.

• 41:36Martin Casado?

  Yes, it's a, yeah, yeah.

• 41:38Speaker 01

  So then like, the, the kind of like, the soft definition of AI completeness is like, there's this fundamental primitive, that's, that's an AI task, but if I could solve this AI task in its full broadest generality, it would solve any intelligence model. And like, the classic example in LLMs is like, next token prediction is AI complete, because I could like, you know, there's the classic example, I think from Ilya, where like, I, there's a mystery novel, and like, the thing has to read the whole mystery novel, and the final sentence of the mystery novel is like, "And the killer was..."

• 42:05Speaker 01

  So like, you could basically like frame any kind of intelligence task in terms of that. So clearly, next token prediction is something that people believe is AI complete. But I think there's something we're kind of realizing, and Ben was talking about this earlier today, is like, new view prediction, this primitive that we have in Atlas, especially generative new, new view, new view prediction, this is also AI complete, right? And because I could take something like-

• 42:25Martin Casado?

  You can, you can have the, the movie, and you do all of the frames of the movie, and then like, the killer walks out, and then you predict exactly who walks out.

• 42:33Speaker 01

  Exactly. Not just that, like, I want, I want to have a world where like, Martine is like writing a proof of the Rema hypothesis, I don't know. And then the camera pans over to the next whiteboard.

• 42:43Fei-Fei Li?

  And he solves the, so, okay, so to take an evolutionary view, right? That new viewpoint prediction is exactly evolution had to solve by making animals move.

• 42:57Fei-Fei Li?

  You, you, nature give animals eyes, but nature didn't give trees eye, eyes. Why? Because when you move, you see a new viewpoint. And that is the, the, whether you call it AI complete or intelligence complete. So, so we do believe very strongly that next viewpoint prediction is, is the equivalent of next token prediction.

• 43:22Martin Casado?

  Amazing. Well, with that, congratulations to all of you on a phenomenal model launch. We're very excited for future model launches, and thanks for coming.

• 43:30Fei-Fei Li?

  Thank you.

• 43:33Erik Torenberg?

  Thanks for listening to this episode of the A16Z Podcast. If you liked this episode, be sure to like, comment, subscribe, leave us a rating or review, and share it with your friends and family. For more episodes, go to YouTube, Apple Podcasts, and Spotify. Follow us on X at A16Z, and subscribe to our Substack at A16Z.substack.com. Thanks again for listening, and I'll see you in the next episode.

• 43:59Erik Torenberg?

  As a reminder, the content here is for informational purposes only, should not be taken as legal, business, tax, or investment advice, or be used to evaluate any investment or security, and is not directed at any investors or potential investors in any A16Z fund. Please note that A16Z and its affiliates may also maintain investments in the companies discussed in this podcast. For more details, including a link to our investments, please see A16Z.com/disclosures.