#303 Fei-Fei Li: Spatial Intelligence, World Models & the Future of AI

This episode is sponsored by AGNTCY. Unlock agents at scale with an open Internet of Agents. Visit https://agntcy.org/ and add your support. How will AI evolve once it can understand and reason about the 3D world, not just text on a screen? In this episode of Eye on AI, host Craig Smith speaks with Fei Fei Li about the rise of spatial intelligence and the world models that could transform how machines perceive, imagine, and interact with reality. We explore how spatial intelligence goes beyond language to connect perception, action, and reasoning in physical environments. You will hear how models like Marble build consistent and persistent 3D spaces, why multimodal inputs matter, and what it takes to create digital worlds that are useful for robotics, simulation, design, and creative workflows. Fei Fei also explains the challenges of long term memory, continuous learning, and the search for training objectives that mirror the role next token prediction plays in language models. Learn how spatial reasoning unlocks new possibilities in robotics and telepresence, why classical physics engines still matter, and how future AI systems may merge perception, planning, and imagination. You will also hear Fei Fei's perspective on the limits of current architectures, why true understanding is different from human understanding, and how world models could shape the next generation of intelligent systems. Stay Updated: Craig Smith on X: https://x.com/craigss Eye on A. I. on X: https://x.com/EyeOn_AI

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  The spatial intelligence work I've been thinking about in the past few years is truly a continuation of my entire career's focus in computer vision and visual intelligence. Why did I emphasize on spatial is because we've come to a point in our technology that the level of sophistication and profound capabilities of this technology is no longer at the level of staring at an image or even simple understanding simple videos. It is deeply, deeply perceptual, spatial, and also connects to robotics, connects to embodied AI as well as ambient AI.

• 0:40Craig S. Smith?

  Welcome to the podcast. In this episode, I have the honor of talking again to Fei-Fei Li, a pioneer in artificial intelligence and computer vision. I had Fei-Fei on the podcast a few years ago, and I invite you all to go listen to that episode. I'll put a link at the end of this video. We'll explore her insights on world models and the importance of spatial intelligence, crucial elements for creating AI that truly understands and interacts with the world around us. While large language models are amazing, much if not most of human knowledge is not captured in text.

• 1:22Craig S. Smith?

  And to reach a more general artificial intelligence, models need to experience the world firsthand, or at least through video. We talk about her startup World Labs and their first product, Marble. It generates incredible, complex 3D spaces from the model's internal representations of the world. Plus, she tells us her guilty pleasure, watching her favorite TV show on airplanes. And you probably won't be surprised what that show is. Stay tuned for an enlightening conversation.

• 1:58Speaker 00

  I'm Fei-Fei Li. I'll be joining the podcast of IONAI and discussing spatial intelligence and world models. Come and join us.

• 2:08Craig S. Smith?

  Build the future of multi-agent software with AGNTCY.

• 2:15Craig S. Smith?

  Now an open source Linux Foundation project, AGENCY is building the Internet of Agents, a collaborative layer where AI agents can discover, connect, and work across any framework. All the pieces engineers need to deploy multi-agent systems now belong to everyone who builds on AGENCY, including robust identity and access management that ensures every agent is authenticated and trusted before interacting. AGENCY also provides open standardized tools for agent discovery, seamless protocols

• 2:58Craig S. Smith?

  for agent-to-agent communication, and modular components for scalable workflows. Collaborate with developers from Cisco, Dell Technologies, Google Cloud, Oracle, Red Hat, and more than 75 other supporting companies to build next-generation AI infrastructure together. AGENCY is dropping code, specs and services, no strings attached. Visit agntcy.org to contribute. That's A-G-N-T-C-Y dot O-R-G.

• 3:39Craig S. Smith?

  I wanted to talk not so much about Marble, your new model, which is amazing, that generates a sort of consistent and persistent 3D worlds that the viewer can move through. I want to talk more about why you're focusing on world models, on spatial intelligence, why that is necessary to go beyond learning in language.

• 4:12Craig S. Smith?

  And how your approach differs from the Coons. So can you talk first of all about, did the world model work emerge from your work on ambient intelligence? Or has this been a parallel track?

• 4:31Speaker 00

  The spatial intelligence work I've been thinking about in the past few years is truly a continuation of my entire career's focus in computer vision and visual intelligence. Right? And why did I emphasize on spatial is because we've come to a point in our technology that the the level of sophistication and profound capabilities of this technology is no longer at the level of staring at an image or or or even simple understanding simple videos. It is deeply, deeply perceptual, spatial and also connects to robotics, connects to embodied AI as well as ambient AI.

• 5:16Speaker 00

  So so from this point of view, it's a continuation really of just my entire career in computer vision and AI.

• 5:27Craig S. Smith?

  And the importance of spatial intelligence. And I've talked about this, I've had this podcast for a while.

• 5:37Craig S. Smith?

  Human knowledge, language models learned from human knowledge encoded in text. But that's a very finite subset of human knowledge. And as you've pointed out, many people have pointed out, humans learn a lot from interacting in the world without language. So it's important, we're going to move beyond the current L L Ms, as amazing as they are, that we develop models that have more direct experience of the world, learn more directly from the world. Your approach has been, well, certainly with model with marble.

• 6:21Craig S. Smith?

  To take the internal representations of the world that the model learns and create an external visual reality with those. Le Cun's approach is to create internal representations from direct experience or video input that allows a model to learn physical laws of motion and things like that. Is there a parallel between that? Are the two approaches complementary or are they overlapping?

• 7:01Speaker 00

  So yeah, so first of all, I actually would not pit myself against Yang or vice versa because I think we are, we're on this, you know, on this continuum of intellectual

• 7:16Speaker 00

  just approaches to spatial intelligence and world modeling. So if you, you probably read my recent long essay that I called Manifesto of Spatial Intelligence. And there I was very clear. I actually think both implicit representation and eventually some level of explicit representation, especially in the output layer, are both probably needed if we are considering a universal omnipotent world model eventually. And they both play roles. For example, our current world model in World Labs, Marble, does explicitly output 3D representation.

• 8:00Speaker 00

  But within the model, there is, there is actually also implicit representation, you know, along with the explicit output. So it is, it is.

• 8:17Speaker 00

  Eventually, I think we need both, to be honest. And also, in terms of the input modality, yes, learning from videos is very important. You know, um, the, the whole world is, is a continuous input of lots of frames. But the whole world, too intelligent, or just to animals, is not just watching passively. It's also, you know, uh, an embodied experience of movements, of interaction, of, um, of tactile experience, of sound, of, uh, of, um, you know, uh, smell and, and, and, and also just, just forces, uh, you know,

• 8:59Speaker 00

  physical forces of, uh, and temperature and all this, right? So I think it's deeply multimodal. And if you even look at, of course, marble as a model is just the first step. But in our tech, tech piece that we released, uh, just this few days ago, we were pretty clear that we believe in multimodality. Um, as a, as a learning paradigm, as well as, uh, as an input paradigm. So, uh, yeah, so I, I think, you know, there's a lot of intellectual discussions on this. It, it also shows the excitement in the early days of, of this, uh, model.

• 9:41Speaker 00

  So I would not say that, um, we have finished exploring, um, exactly the model architecture, the representation and, and all that.

• 9:54Craig S. Smith?

  In your, uh, world model, the inputs are primarily videos. Um, and then the, the model, uh, builds an internal representation of the world.

• 10:06Speaker 00

  Not quite. In, if you experience mar- marble, our world model, our inputs are generally pretty multimodal. You can do a pure text, you can do one to multiple images, you can do videos. Uh, you can also input coarse 3D layouts, you know, like boxes or voxels. So it's, uh, it's multimodal and I, I think we're gonna deepen that as we, we go.

• 10:33Craig S. Smith?

  And is the ambition beyond, you know, it's a fantastic product with many applications. Is the ambition to, to build a, a system?

• 10:45Craig S. Smith?

  Uh. And when I said the the input is video, but to build a system that can learn uh from direct experience, whether it's through video or or or some other modality, but uh it's it's not learning through a secondary uh uh medium like text.

• 11:06Speaker 00

  Yes, I think world model is about learning about the world and the world is very multimodal. And whether it's machines or animals, we are multi-sensory as well. So the learning is through sensing. And and the sensing has different modality. So text is one form. Um. Again, this is where we depart from, uh, you know, uh, animals, because uh most animals don't learn through sophisticated language, but but but humans do. But um, you know, our today's AI's world model will learn from a lot of language input as well as other modality, but it's not squeezed through only language.

• 11:51Craig S. Smith?

  Yeah, and uh uh one of the the limitations of L L Ms uh is that the uh the models are the model parameters are fixed after training. So the the models don't learn continuously. There's a certain amount of learning at uh test time inference. I uh is that a uh something that that you're tackling with world models? Uh, because presumably a world model is learning as it encounters new environments.

• 12:24Speaker 00

  Yeah, the the the continuous learning paradigm is definitely a very, very important one, especially for uh for um, you know, living beings. That's how we uh how we do it. And even there, there is you know, online learning versus offline learning. So um in the current form of our um world model, we are still more in the batch or offline uh learning mode, but I I I'm we're definitely open to, you know, continuous, especially uh eventually online learning uh modality.

• 13:02Craig S. Smith?

  Yeah, and and how would that would it? I mean, it wouldn't be a a very different architecture. It would.

• 13:11Craig S. Smith?

  It would simply be a matter of engineering, is that right?

• 13:14Speaker 00

  Well, I would be open minded, you know, yeah, yes, I think it's gonna be a mixture of both. You know, obviously, good engineering, good, you know, fine tuning, online, you know, uh uh learning can already happen, but maybe there will be new architecture.

• 13:32Craig S. Smith?

  Yeah, yeah, uh, can you talk about the real time frame model which which underline, underlies um uh marble and and your work with world models?

• 13:46Speaker 00

  Yeah, so you're referring to a tech blog that we also put out a couple of weeks ago that specifically double clicks on our um real time frame model. Um, World Lab is a um a a a very research heavy org- organization. We also care about product, but uh so much of our uh work right now in this stage is model first, so we're definitely looking at how we can uh uh we can push forward spatial intelligence and this particular line of work, which definitely uh connects to marble is really focusing on how we can um achieve frame based generation with as much

• 14:31Speaker 00

  pick uh uh geometric consistency and permanency as we can, because uh some of the early work in frame based generation, you lose that um permanency as you you you move forward, but in this particular case, we try to really balance and also uh do it in a compute efficient way during inference, which in this case we achieve through uh a single H one hundred in in uh inference time. We don't really know. Some other uh frame based models, they don't tell us how many uh chips they use during their inference time um. So we don't, we we have a hypothesis, it's it's it's a uh quite a number,

• 15:16Speaker 00

  but we don't know that information to compare ourselves.

• 15:20Craig S. Smith?

  Yeah, and and in your your um manifesto, I think you called it, uh you talk about the need for a universal task function of

• 15:31Speaker 00

  Universal what function?

• 15:33Craig S. Smith?

  Universal task function.

• 15:36Speaker 00

  Oh, yes, yeah.

• 15:38Craig S. Smith?

  Analogous to the next prediction, next token prediction in language models. Does RTFM does have a prediction element to it? What are you talking about when you say universal task function? Beyond what the predictive element of RTFM?

• 16:04Speaker 00

  Well, so one of the biggest breakthrough of Gen AI is really this

• 16:13Speaker 00

  discovery of the objective function of next token prediction, right? Because it really is such a beautiful formulation because language is in this kind of sequential, you can tokenize language in this kind of sequential representation. And your learning function of next token prediction is precisely what you need during inference, which is as you generate language, whether humans generate or computers generate, it really is putting token after token forward. So having an objective function that is 100% aligned

• 16:56Speaker 00

  with the actual eventual task that is supposed to achieve or carry out is just, is just great because it makes the optimization

• 17:10Speaker 00

  just on target. In computer vision or in world modeling, it is not as, it's not as simple. Because if you look at our relationship with language, it really is just to speak it or to generate it. There's no language in nature that you're staring at it. I mean, eventually you learn to read, but that's because it's already generated. So, but there is, you know, your relationship with the language is deeply generative. It literally is a generated thing by humans. But the relationship with the world is so much more multimodal, I would say, right?

• 17:55Speaker 00

  There is a world out there for you to observe. For you to interpret, for you to reason with it, for you to eventually interact with it. But there is also a mind's eye that can formulate different versions of reality as well as imagination. And also allow you to also generate the story, the the the the the the ima- imagine world or or. So it's it's much more complex. So what is the task that defines or the objective function that defines a universal function

• 18:40Speaker 00

  that we can use that is as powerful as next token prediction? The it's actually a really profound question because is it, let's say, is it 3D reconstruction? You know, just uh some people actually would argue that the universal task for world modeling could be that it's just be able to 3D reconstruct the world because if if that is the objective function and if we achieve that, a lot of things will fall through uh uh it just fall naturally out of it. But that is, you could also argue, I don't think so, because most

• 19:22Speaker 00

  animal brain don't necessarily do precise 3D reconstruction. You know, like, it's not clear a tiger or a person actually can reconstruct the world, yet we're such powerful visually intelligent, spatially intelligent uh beings. So maybe that's not necessarily the right task. Then is it um is it the next fr- uh frame prediction? As a a way to, as if it's uh you know, the next token prediction. Well, there's some power to that, right? Like, first of all, there's a lot of training data for for this.

• 20:00Speaker 00

  And second is that it uh in order to predict the next frame, you you you you have to learn the structure of the world because Worlds are not white noise, so there's a lot of structure that connects one frame to to the other. And if you if you do it well, maybe that is um maybe that is the right universal task or objective function. But maybe it is also unsatisfying because you treat the world as two D. And and and the world is not two D. It's so so you do you really force the representation, collapse it in a very unsatisfying way?

• 20:44Speaker 00

  And also, even if you manage to, you can say if you manage to do it perfectly, three D is implicit. That's true, but it's also very wasteful because uh with the three D structure, there's actually a lot more information that you don't have to lose in a way that frame-based uh prediction might. So so there is a lot of uh, there's still a lot of exploration on that.

• 21:13Craig S. Smith?

  RTFM, and I have to ask you, was that a joke uh naming it RTFM?

• 21:20Speaker 00

  It was a it was a brilliant play with the, I did not even, this one of our researchers is really brilliant in naming, right? Like, as you know, you know, I don't know how much we're allowed to say those words.

• 21:35Craig S. Smith?

  I'll say it, yeah, read the fucking manual and see the acronym, yeah.

• 21:41Speaker 00

  Every computer scientist knows that, so we we find that really fun to to just play with that name.

• 21:49Craig S. Smith?

  But but RTFM is predicting uh the next frame, right? That's that's uh and.

• 21:56Speaker 00

  With three D consistency.

• 21:58Craig S. Smith?

  Yes, and that's what's interesting about the internal representation that uh that the model learns. I mean, sitting. Here I'm looking at my computer screen. I know what the other side of the computer screen looks like, even though I can't see it. And I can, there's an internal representation in my mind of that. And, and your model does that. I mean, that's why you can move around objects in the, in, even though it's a, it's a 3D representation on a 2D screen, but you can move around, see the other side of

• 22:42Craig S. Smith?

  things. So the model is, has an internal representation of the 3D object, even if its current view is, you know, can't see the backside of something. And that really interests me. That's essentially what you're talking about when you say spatial intelligence, understanding that, that 3D world.

• 23:08Craig S. Smith?

  Is, is that, does that include, does that learning include things like the physi- physical laws of nature? I mean, does it understand that,

• 23:23Craig S. Smith?

  that you can't walk through a solid object, or that, I think on one of the podcasts I saw you on, somebody was talking about using this for people with, with fear of heights. So you could look over the edge of something. But if you create a, a physical representation, I mean, an explicit representation of a cliff, let's say, and, and you move the

• 23:59Craig S. Smith?

  Uh, the point of view of of the agent or the viewer over that cliff. Will it know that it's no longer standing on solid ground or will it float in uh in the space?

• 24:13Speaker 00

  Yeah, so what you are describing is both physical and semantic, to be honest, like, you know, of course, falling off a cliff is very much depend on the law of gravity and all that, but um, and but the the fact that you the the, you know, going through a wall, it's very much material based and and semantics based, right? Solid object versus non-solid object. So our TFM as a current model right now is not focusing on the physics yet. Most of the physics, to be honest, coming out of the the the the gen AI models are statistics.

• 24:56Speaker 00

  If you look at, for example, you know, um, a lot of these gen video models, where you do see water running and trees moving. That is not based on a Newtonian law of forces and uh uh masses. It's really based on, I've seen plenty of movements of water and leaves in this particular way, and I'm just gonna follow that statistical pattern. So we have to be a little careful. Uh, right now, World Labs is still focusing on on generating and exploring and uh static uh worlds.

• 25:35Speaker 00

  We are going to explore dynamic and there, uh, a lot of that will be, you know, statistical learning. Um, I don't think any of today's AI, language AI or pixel AI has the capability to abstract at a different level and deduce physics, like at the level of a Newtonian law out of out of it. Uh, everything we've seen is statistics based, statistical uh based uh physics and dynamic, physical and dynamical learning. We could, on the other hand, put these worlds.

• 26:20Speaker 00

  Into physics space, physics engines and these.

• 26:23Craig S. Smith?

  Yeah, like unreal.

• 26:25Speaker 00

  Right, have the, the, the laws of physics and they, they eventually these uh physics engines, game engines and, and also world generation, eventually they're gonna come by into neural engines. I don't even know. Maybe we should call them neuro spatial engines or, or something like that. I think that we are moving towards that direction, but it's still early days, right?

• 26:50Craig S. Smith?

  Yeah, yeah. And I didn't mean to pit you against Yan. Uh what, what I was driving at is it, it seems like you're focused on explicit representations coming out of inter, of, of an abstract internal representation. Yan is just focused on, on the internal representation and, and learning uh and, and that's where the learning takes place. Uh and it, it, it just seemed to me that they would marry beautifully.

• 27:23Speaker 00

  That's a possibility. Like I said, like you already repeated, we are, you know, we are exploring both. The explicit output is actually a very deliberate approach because uh we want to be useful for people. We wanna be useful for people who are creating, who are simulating, who are designing. And if you look at today's industry, whether you're creating uh VFX effects or you're creating games, or you're designing interiors, or you're simulation, uh simulating for robots or autonomous vehicles, or, or industry, digital twins, or whatever you're doing.

• 28:04Speaker 00

  It's very three D, right? So it's the, the, there's entire industry after industry that's very much three D uh in the, in the workflow of three D and we wanna be absolutely useful for, for, for people and uh businesses to, to use these models.

• 28:23Craig S. Smith?

  Uh and the reason I brought up continuous learning uh.

• 28:29Craig S. Smith?

  Right now, uh, the, the model understands, uh, you know, depth of field and, and other sort of spatial properties. Yeah. Um, what is the, uh, the learning that, that is implicit in the model? Or that is, is, uh, that the, the model has that allows it to, uh,

• 28:57Craig S. Smith?

  to generate that explicit representation? I mean, does it? Be- because the ultimate goal is to build a model that presumably can learn, uh, over time. And then I'm sure everybody, you and everybody else. I mean, I imagine it having a model that maybe it's on, uh, you know, robot, or maybe it's, uh, in, maybe it's attached to a, it's getting data from a video camera that moves around in the world, but that, that eventually learns not only the, of the scene that it's, it's seeing,

• 29:38Craig S. Smith?

  but understands the, uh, the physicality of the space. And, and eventually then you marry that with language and you've got a really powerful intelligence. Uh, is, is that something that, that you think about? Uh, and, and that would require continual learning? That, that it's, it's not a finite data set? That you're feeding the model, that the model is continuously learning from its interaction in the world?

• 30:09Speaker 00

  Yeah, definitely, especially as, as one comes close to a use case, especially if the use case requires continuous learning. You know, there are many ways of continuous learning, right? Like in language model, we see that, um, taking into context itself is continuous learning. Right? Like having the context as memory, uh, so it helps, uh, inference. And then of course, there is other methods of online learning, of fine tuning. So continuous learning is a Is a term that can encompass multiple ways of of doing it. And I think in spatial intelligence, especially like you said, some of these

• 30:50Speaker 00

  some of these use cases, whether it's robots in personal or in customized situations or or you know, artists with particular styles, creators with particular styles. These will all eventually um push the technology to be able to be more responsive in whichever time horizon that the use case requires it to be. And some are real time, some are are maybe just, you know, more, you know, segmented, you know, in in the time horizon point of view. So it depends.

• 31:27Craig S. Smith?

  Build the future of multi-agent software with agency. That's A G N T C Y. Now an open source Linux Foundation project. Agency is building the Internet of Agents. A collaborative layer where A I agents can discover, connect, and work across any framework. All the pieces engineers need to deploy multi-agent systems now belong to everyone who builds on agency, including robust identity and access management that ensures every agent is authenticated and trusted before interacting.

• 32:11Craig S. Smith?

  Agency also provides open standardized tools for agent discovery, seamless protocols for agent to agent communication, and modular components for scalable workflows. Collaborate with developers from Cisco, Dell Technologies, Google Cloud, Oracle, Red Hat, and more than seventy five other supporting companies to build next generation A I infrastructure together. Agency is dropping code, specs and services, no strings attached. Visit agntcy.org to contribute.

• 32:53Craig S. Smith?

  That's A G N T C Y dot O R G. Yeah, and you know, you've come a tremendous way of since you're running a dry cleaning shop, as I recall, in New Jersey. And in a very short period of time.

• 33:15Craig S. Smith?

  And you're moving very quickly now. Do you do you have any sense of where you'll be in five years with this technology? Will you

• 33:28Craig S. Smith?

  have you know, some physics engine built into the model or will you have you know, the ability to learn in longer timeframes and build up richer internal representations that the model can can begin to understand the physical world?

• 33:53Speaker 00

  Yes, I I actually think, you know, it's not as a scientist, it's very hard for me to give a prediction in terms of precise time because some part of the technology has moved much faster than I thought, some are much slower than I thought. But I think that is very much a very good goal. And also, it's five, actually, five years is kind of a fair guesstimate. I don't know if we'll be faster, but it's a it's a more, it's a better guesstimate than fifty years in my mind, or five or five months.

• 34:32Craig S. Smith?

  Can you talk a little bit about why you think spatial intelligence is the next frontier?

• 34:42Craig S. Smith?

  I mean, we we've already said that human knowledge is contained in text is only a subset of all human knowledge. And while it's very rich,

• 34:55Craig S. Smith?

  you you can't expect

• 34:59Craig S. Smith?

  an AI model to understand the world simply through text.

• 35:04Craig S. Smith?

  Can you talk about why that's important and and how marble and world labs relates to that larger goal?

• 35:13Speaker 00

  Yeah, so I think fundamentally technology should help people. In the meantime, understanding the science of intelligence itself is

• 35:27Speaker 00

  The most fascinating, audacious, ambitious scientific quest I can think of myself, and it's a quest of twenty first century. Both of these, whether you're fascinated by the curiosity of the science or you're motivated by using technology to help people, it points to one fact is that so much of our intelligence and so much of our intelligence at work goes beyond language. Right, it's, I, I kind of lightheartedly say that you cannot use language to put down a fire. And in my manifesto, I, I described examples,

• 36:08Speaker 00

  whether it's the spatial reasoning deduction of DNA double helix structure or a first responder putting down fire in a rapidly evolving situation with a team of coworkers.

• 36:27Speaker 00

  A lot of this goes beyond language. So it is obvious, both in terms of use case point of view as well as in terms of a scientific quest, that we should try our best to unlock how to develop spatial intelligence technology that, you know, takes us to the different, to the next level. So that's really the, the, the, the thirty thousand feet view that, that describes how I'm motivated by these dual purpose of scientific discovery as well as making useful tools for, for people. And then we can dive,

• 37:12Speaker 00

  and then we could dive deeper into, for example, being useful, like we alluded to a little bit earlier, whether you're talking about creativity or simulation or design or, or immersive experiences or education or healthcare or, you know, manufacturing.

• 37:31Speaker 00

  There's just so much that you could do with spatial intelligence. I actually, I'm very excited that so many people who are thinking about education and immersive learning and immersive experiences are telling me that marble are. First release of the model is inspiring them to think how to use it for for immersive experiences in, you know, to, to, to make learning more interactive and fun. And, and that's just so natural, cause you know, pre-verbal children learn entirely through immersive experience. And, and even today, as, as, as grownups

• 38:16Speaker 00

  our life, right? Like so much of our life is immersed in this world and, and involves, of course, it involves speaking and writing and reading, but it involves doing and interacting and, and, and enjoying and, and, and all that. So it's, it's so natural.

• 38:37Craig S. Smith?

  Yeah, I mean, one of the things that, that has struck everybody uh, is that this marble, I'm not sure it's marble or, or the itself, or the, or the RTFM, the, the, the generation of uh, next frame in, in a moving, moving through a space uh, is that it's running on one H one hundred GPU. Uh, it's, and, and I've heard you in other talks refer to uh, experiencing the multiverse,

• 39:17Craig S. Smith?

  uh, which, you know, everyone was very excited about until they realized how much computation it was gonna take, uh, and how expensive it is. Uh, it is, is, do you really think that uh, this is a step toward creating worlds? For education, for example, and because you've been able to reduce the computational load?

• 39:42Speaker 00

  Um, not just that. First of all, I really believe at the inference front, we will be speeding up, we will be more efficient and we'll be also better, bigger, higher quality, um, longer, you know, um, experiences, all this, I, that is the trend of technology. I, I really do, things are moving fast. I also do believe in multiverse, a multiverse experience. You know, humanity, um,

• 40:14Speaker 00

  As far as we know, the entire history of humanity, our world experience is in one world. And it's literally physically this earth, right? Okay, a few, a handful of human beings have gone on to the moon, but that's about it.

• 40:33Speaker 00

  And that is the one shared uh three D space. And we build civilization, we live our lives, we, we, we, we do everything in it. But with the digital revolution, digital explo- explosion, we are moving part of our lives in digital world. And there is also a lot of crossover. This is not, I don't wanna paint a dystopian picture of we have abandoned our physical world. Uh nor am I gonna paint a utopian, total, hyperbolic, utopian world of everybody wear a headset and never even enjoy and look at our beautiful real world. And that's, that's the fullest of life.

• 41:18Speaker 00

  I, I, I would reject both notions. But both pragmatically speaking, as well as just projecting to the exciting future. The digital world is boundless, it's unlimited. And it gives us so much more dimension and experiences that our physical world will not allow us. For example, we already talked about learning. Just, I would love to have learned chemistry in a much more interactive and, and immersive way. Because I remember my college chemistry class has so much to do with arranging molecules,

• 41:57Speaker 00

  understanding the parities and the asymmetry of the molec- uh molecular structure. Man, I wish I could just experience that in a, in a, in a experi- uh, you know, in an immersive way. I would love for creators to um.

• 42:16Speaker 00

  You know, creators, I meet these creators, I I realize in their minds, I they simultaneously, every given moment, they have so many ways to tell their story. There's so much in their head, but they are, they're rate limited by tools. You know, if you use Unreal Engine, it's gonna take weeks, hours after hours to express one world in your head. And and and whether you're making a fantastical musical production or you are, you know, designing the the bedroom for your newborn child. So many of these moments, if we allow people to use the

• 43:02Speaker 00

  digital universe as much as the physical world to experiment, to to iterate, to communicate, to to create, it's just so much more fun. And also, we also are, digital age is also helping us to break the physical boundaries of of labor, right? Like, I can tell, I mean, we're looking at um um robots that can be teleoperated. I can also totally imagine um creators collaborate across the globe through embodied, uh, you know, um robotic arms or whatever form factor

• 43:47Speaker 00

  and through the digital space, uh, so that they're both, they can both work in the physical world as well as work in the in the digital world. And movie, right? Movies will completely change. Right now, it's passive experience as as much as it's beautiful, but we're gonna change the way we get entertained. So all this required multiverse.

• 44:11Craig S. Smith?

  And and in in teleportation or to, you know, the teleoperated robots, um, you could, you know, there's a lot of talk of mining rare earths on asteroids and stuff. Um, you know, you don't need to physically be there if you if you can, uh.

• 44:31Craig S. Smith?

  Operate a robot remotely, that's in those spaces. What in what you're talking about is again creating

• 44:42Craig S. Smith?

  explicit representations of of 3D space that people can can experience. How much in your models does the model itself understand

• 44:59Craig S. Smith?

  the spaces that that it's internalizing before it explicitly

• 45:08Craig S. Smith?

  projects them? I mean, and that's again, one thing that that I think about more than the the productization or practical uses of this is working toward an AI that truly understands the world, doesn't

• 45:31Craig S. Smith?

  just have a representation of 3D space that really understands not only the laws of physics, but but what it's seeing and and maybe the values or the the usefulness of what it sees and ways that it could manipulate the physical world. And how much of that understanding do you think is there? And what needs to happen for the models to really understand the world?

• 46:02Speaker 00

  Yeah, okay, great question. So this word understanding is a very profound word. You know, when AI understands something, it just fundamentally is different from how humans understand it. If partially because we're very different beings, right? Humans has a level of consciousness and self-awareness in the embodied

• 46:30Speaker 00

  in the embodied. Body, for example, when we understand something, when we understand my friend is really happy. It's not just a abstract understanding, my friend is happy. If there you, you actually have chemical reactions in your body that releases happy hormones or or whatever chemicals that your heartbeat might increase, your, your, your, your mood might. And so, so that level of understanding is very different from an abstract AI agent that has the capability of correctly assign meaning and and link uh meaning to each other.

• 47:13Speaker 00

  For example, in marble, our product, uh, our model product, you can actually go, go to a advanced mode of world generation. And in, in this advanced mode, it asks you, it allows you to edit. You can take a preview of the world and say, I don't like this couch being pink, change it to blue. And it, it changes it to blue.

• 47:39Speaker 00

  Does it understand at the level of blue couch and the world change? It does, because without that understanding, it wouldn't change the pink, the couch to the blue couch. Does it understand it that, that the same way you and I understand it? Including everything about this couch, couch, including every useful or not even useful information of the couch? Does it have a memory of the couch? Does it, you know, the, take the concept of couch to afford us to many other things? No, it doesn't.

• 48:20Speaker 00

  It, as a model, it's limited to allowing you to, you know, do whatever it's necessary the model needs to do, which is to create a space that has the couch that that looks blue. So I think.

• 48:34Speaker 00

  So the answer to your question is, I do think AI understands, but let's not mistake that understanding from a anthropomorphic human level understanding.

• 48:49Craig S. Smith?

  Yeah, and in fact, the whatever understanding there is, when you tell it to swap a red couch for a blue couch, that's semantic, that's semantic understanding, it's not understanding at the level of, you know, light and hitting the retina and knowing and having a concept of blueness as opposed to redness. You know, I saw your talk with Peter Diamandis and Eric Schmidt in Saudi Arabia.

• 49:31Craig S. Smith?

  And I think you were wonderful in that, much more grounded than some of the questions, or even Eric's views.

• 49:44Craig S. Smith?

  But one of the things that struck me is there was a brief discussion about the potential for AI to be creative or to help in scientific research and the analogy that is given, you know, if there had been AI and, you know, before some, like Einstein's relativity theory, or I can't remember the other one that was referenced, could AI reason to that discovery?

• 50:22Craig S. Smith?

  What's missing for AI to be creative? I'm not talking about creative in the arts, I'm talking about in scientific reasoning. It seems to me that that should be within reach. I mean, I'm, again, with no timeframe, but that seems just intuitively something that should be possible.

• 50:54Speaker 00

  Yeah, that's a great question.

• 50:59Speaker 00

  You know, I would think we're closer to AI ded- ded- deducing double helix structure than AI formulating special relativity. Um, partially because, I mean, we already have seen a lot of great work in protein folding. Um, partly because deducing double helix structure is still, the representation is, is more grounded in, in, in, in space and, and geometry. Whereas

• 51:40Speaker 00

  the formulation of special relativity is on the abstract layer that is not just expressed in

• 51:54Speaker 00

  unlimited amount of words. That, that's just not how it works, right? It, everything we see in physics, from Newtonian law to, to quantum, we abstract it to a causal level. That

• 52:14Speaker 00

  the, the, the relationship of the world, the concept, whether it's mass or it's force or

• 52:26Speaker 00

  is abstracted at a level that is just no longer

• 52:31Speaker 00

  pure statistical pattern generation. Like, language can be very statistical. Um, 3D world or, or 2D world, doesn't matter, can be very statistical. Dynamics can be very statistical, but the causal abstraction of forces and masses and

• 52:55Speaker 00

  Magnetism and all this are not purely statistical. Um, it, it, it, it is very profoundly causal and abstract. So I, I'm more in the pontification mode than answering your question mode. I, you know, I think Eric and I on stage were saying, you know, we've got plenty of celestial body data, movement data in the world now. Just aggregate all the satellite data and all that. Give it to today's AI, can it come up with Newton, Newtonian law of motion, motion?

• 53:36Craig S. Smith?

  See, yeah, and actually I found that really, that was the other example I was trying to think of. Relativity, yeah, there, there's, there's no physical system that you can observe, at least in, in day to day life that would, would lead to that deduction or inference, but, but with the,

• 54:02Craig S. Smith?

  with the data of, of the movement of, of celestial objects. I, I, it, I'm, I'm not sure, just again intuitively, I'm a journalist. Uh, it's, it seems that maybe not today's AI systems, but that an artificial intelligence uh, could deduce the laws of motion. If, if given the data and, uh, and given the, the, the time to think, uh, why do you think it would not be able to deduce those, those laws?

• 54:43Speaker 00

  When we say those laws are deduced, Newton had to deduce, has to abstract concepts like forces and mass and acceleration and, and, and fundamental constants. Those are at an abstract level that I have not yet to seen today's AI can take that whatever vast amount of data and abstract

• 55:15Speaker 00

  representation or variables or, or relationships at that level. It's a

• 55:22Speaker 00

  There isn't much evidence yet. I'm, you know, I don't know everything that's going on in AI. So I'm happy to be proven wrong. I just haven't heard any work that has done that level of abstraction. And in the architecture of a transformer model,

• 55:44Speaker 00

  I don't see where that abstraction can come from yet. So, so that's why I'm questioning that.

• 55:54Craig S. Smith?

  Yeah, yeah, it just from my, my sense, you know, you're building these internal abstract internal representations. And as that knowledge accrues,

• 56:09Craig S. Smith?

  I, I just, I don't see why

• 56:13Craig S. Smith?

  an AI model wouldn't be able to apply the, you know, rules of logic, knowledge.

• 56:21Speaker 00

  I'm not saying AI shouldn't or shouldn't try, but it probably takes more progress in our fundamental architecture of our algorithms.

• 56:33Craig S. Smith?

  Yeah, yeah. And that was something I wanted to ask. I mean, you use transformers in, in your, in your models. And I've been talking to people about post transformer architectures. You know, we're certainly not at a breakthrough, but you know, you, we started talking about this, this predictive function,

• 57:02Craig S. Smith?

  a universal task function, as you called it, is, is, do you have a an expectation that there will be that kind of a breakthrough? A new architecture that will unlock some of these capabilities?

• 57:20Speaker 00

  I do, I, I, I do think we will have architectural breakthroughs. I, I do not think transformer is the last invention of AI. And

• 57:34Speaker 00

  You know, in the grand scheme of things, humanity hasn't been around for that long compared to the entire history of the universe we know of. But in this short history of thousands of years, we have never stopped innovating. So I do not think Transformer is the last algorithm, architecture of AI.

• 57:58Craig S. Smith?

  In one of the conversations or something you wrote, I saw you, you, you were talking about your, the history of your research and that at one time you felt if you could get an AI system to label or caption images, you would have, that would have been the pinnacle of your career. And of course, you've blown past that. What, what is, what do you imagine as the, as the crowning achievement of your future career today?

• 58:35Speaker 00

  I do think unlocking spatial intelligence, creating a model that really connects perception to reasoning, spatial reasoning seen to doing, including planning and imagine, imagining to creation would be incredible, a, a, a model that can do all three of that.

• 59:02Craig S. Smith?

  I have a couple of silly questions I'm going to ask you. What's your favorite food or one of your favorite foods? I'm just curious. I mean, you, you were born and spent some of your early years in China. Is it Chinese food? Is it, you know?

• 59:20Speaker 00

  So I'm married to an Italian and he's a phenomenal cook in addition of being a phenomenal AI scientist. So I, I love Italian and in our family, when we are being nonpartisan, we love Japanese food.

• 59:40Craig S. Smith?

  But not, but these days, yeah, that's, yeah. And then what's your guilty pleasure? Do you read romance novels on the plane or, or binge watch?

• 59:53Craig S. Smith?

  Uh, you you know.

• 59:55Speaker 00

  Yeah, that's a great question. First of all, I don't have too much leisure time, because what I do is so much fun, but I do have a guilty pleasure. I don't think I've ever told anybody, um.

• 1:00:11Speaker 00

  If I'm really tired in a plane. I watch Big Ben Theory. I love that show. And I graduated from Caltech. I was a physics major. Everything of that show, I just identify so much with with the with the people there. It's it's my I don't binge watch because it depends on the duration of the flight, but that's my totally if I'm totally exhausted, I watch that.

• 1:00:42Craig S. Smith?

  Yeah, yeah, and you get the jokes, right?

• 1:00:45Speaker 00

  Oh, God, I I can tell you, I love every single jokes and every single character, the nerds in the show.