Luma's Dream Machine and Reasoning in Video Models

a16z General Partner Anjney Midha speaks with Luma Chief Scientist Jiaming Song about the company's recently released Dream Machine model. Among the amazing features of Dream Machine are how it learned to accurate generate 3D structures and elements from 2D videos, and how it is able to reason about things such as shot consistency and fictional worlds without explicit priors.

• 0:13Speaker 2

  Hey Jia

• 0:14Speaker 1

  Ming, thanks for joining us. I've been very excited for this conversation for a while. So what is Dream Machine?

• 0:19Speaker 0

  So Dream Machine is a foundational video generator model. At the release, we were having two like features that are critical to it, which is text to video, where you can type in the text prompt to generate a video, or uh image to video. And you can have an image plus text to generate uh a video.

• 0:42Speaker 1

  Why don't we start with the story of 3D capture, right? Because that's really the first research and product milestone that led to the development ultimately of Dream Machine.

• 0:51Speaker 0

  Luma was mostly doing things like 3D reconstruction, 3D capturing. So in this video, we are showing you can take a drone and you fly the drone around some landmark to capture a video, and then the Luma application will transform that video into a 3D scene that you can interact with. And of course, you can also do that on your cell phones to capture objects that are closer to you. But it essentially making things representable in 3D and being able to interact with them was the initial kind of a product that we have. Then we were thinking about how we can not only do reconstruction but also do uh generation. So we then released this app called Genie, which allows you to generate 3D assets from text prompts.

• 1:41Speaker 0

  And it was definitely the first one of the product in kind that we got there. And I think we started releasing the preview at around November, early November last year. But then as we moved along into like working more and more on 3D with a lot of the best people in 3D in our company, we realized more and more that 3D data has this scalability issue. And you if you compare that with images, everyone can you know use their cell phone to take a uh photo or you know take a video. Uh whereas if you try to do the same with 3D, it's very difficult. Um you either have to you know have professional knowledge about how to use these tools, or you have to you know use uh you know multi-view capture techniques that are not still not as nearly accessible to you know everyday uh users.

• 2:37Speaker 0

  Right. So the there is you know a vast gap between um you know image video data and 3D data. How did you solve that problem? Yes. One thing that we realized is that in the last year, there has been many works that use it that tries to solve 3D from a quite a different angle. Instead of trying to use only 3D data, they try to use 3D as like an application of a 2D foundation model, such as uh image model. So in this case, you first train uh diffusion foundation model and you leverage that vast amounts of data, and then you try to fine-tune this model on multi-view images. So just like the problem where we mentioned earlier about you know the model not being able to reason about different size of an object.

• 3:27Speaker 0

  It actually turns out that being able to do fine-tuning on top of it leverages benefits from the both worlds. So it not only gets this semantic knowledge about what an object in the world should look like, it also leverages the 3D data to know about like how these objects look in like multiple size. So uh a very common solution nowadays is to use like multi-view like 2D generative models that are fine-tuned from the flat non-multi-view versions of that, and then try to convert these like representations into a 3D object. And here I'm showing two kind of representative works on this direction.

• 4:10Speaker 1

  Okay. So once once um it became clear that using 3D reasoning about 3D as a fine-tune of a 2D foundation model was a feasible path. What came next?

• 4:22Speaker 0

  So for us, it's uh quite natural. So uh you know like the limitations of images was that it wasn't able to reason about how you know the camera works in the world, because it only has like a relatively independent shots of different objects. The next thing that is quite natural for us is to see if we can learn and uh and capture 3D knowledge from like by learning from videos. Because videos naturally have a lot of these init observations. For example, the camera has there's uh camera movements, there's object movements, and we are hoping that the having a video model can leverage these things and try to learn something 3D about the world.

• 5:03Speaker 2

  Yeah.

• 5:04Speaker 0

  And it turns out from these uh type of videos, we are able to show that a video model is able to reason about 3D quite well, uh, which in some sense is unexpected before in in the community.

• 5:16Speaker 1

  What was

• 5:16Speaker 0

  most surprising about this? People will talk about different things about what surprises them. But I think what's really surprising to me is how uh large scale of compute is mostly all you need to capture a lot of the like intricate effects that people spend years to develop in like graphics and you know physics simulation community, and I will show some like more examples in details in just a second.

• 5:43Speaker 1

  Great, yeah. Uh so just to just so I'm playing this back correctly, you're saying when what what we're looking at here is a few samples of Luma dream machine generations. Yes, right. That were able to reconstruct a 3D scene.

• 6:00Speaker 0

  Yes. For example, in this case, what we do is uh we literally took one of the videos from the last site and we put this video into our 3D reconstruction pipeline, and it turns out that it is able to reconstruct uh 3D scene at this direction quite reasonably well. So the the pipeline was quite simple. We had take some image and then we convert that image to a video, and then we put the image video into 3D reconstruction. And uh it turns out that it was pretty 3D consistent.

• 6:34Speaker 1

  Yeah, so this is pretty mind-blowing. Yeah. I think we should take a second to just talk about this. Yes. You just said your the pipeline started with an arbitrary image. Yes. That image then became the prompt for the Dream Machine video model. Yes. Which then outputted a video that you were able to pass through into a reconstruction pipeline and get a structurally consistent 3D scene outside the other end. Yes. What does that tell you about what Dream Machine is doing?

• 7:01Speaker 0

  So I think that tells me that Dream Machine is definitely able to reason about 3D better than any of the models that we've you know worked with before. And uh this is very much unlike like even the models that you try to obtain by fine-tuning on the you know, like raw, like you know, fine-tuning on 3D uh like data because there is just a lot more videos and it was able to capture things in like much greater detail than the resolution that was possible for multi-view um you know image models. Right. And the also the simplicity about this this pipeline is also um very attracting. And uh that like 3D reconstruction without having to take very complex captures has always been a thing that we are trying to solve. Like we we've uh considered multiple approaches to solve this problem because we acknowledge the fact that despite how Nerf and Gaussian Plus were super um popular in the research literature, uh, when you try to develop the deploy these techniques in in the wild, there are many issues that comes with this imperfect capture like capturing system that comes along.

• 8:08Speaker 0

  Like people when they're trying to capture an object will oftentimes not capture the object fully 360 degrees. There might be motion blur associated with the camera. The object might be moving, so it it would it's better to have them like perfectly still. So all of these are limitations that we were trying to solve like before we even started trying to develop the Dream Machine.

• 8:31Speaker 1

  I

• 8:32Speaker 0

  see. So this is a very exciting to us as well because it also shows that there is a very interesting alternative path towards solving like 3D reconstruction.

• 8:43Speaker 1

  So here this was the first, I guess, attribute or first clue that you had that video models might be doing more than just generating interesting frames. Yes. That there might be an inherent understanding of the world. Yes. Right. And 3D, I guess 3D consistency was the first clue. What were some other clues?

• 9:02Speaker 0

  These are some videos on that we tried on some like NERF data sets. And actually, because a lot of us uh work on Nerf like Gaussian splatting and reconstruction. So the context here is that we put the first frame of an image into Dream Machine, and then Dream Machine will give us the output as a video. So as you can see here, like the 3D consistency of the generated video looks uh quite amazing. And here are some like more examples, all from very popular uh NERF datasets. And if you're working in the domain, you will definitely know what these data sets come from.

• 9:36Speaker 1

  Okay. What is a nerf and what is a Gaussian splendid?

• 9:39Speaker 0

  I see, sure. So a NERF uh is more like um, you know, new it's a full name called the neural radiance field. And the idea is to um assume that the world is simulated by volume rendering techniques. And we uh based on this assumption, which is obviously Obviously imperfect, we try to solve in this 3D field, how much what's the color and what's the opacity of this point. And then the rendering happens by integrating along a ray of line, and then uh oftentimes things are like just non-opaque at some certain like location, and that's how we get the get the rendering out. Of course, having one view is not sufficient enough to solve this problem. So in the case of Nerf, there is oftentimes like a very uh detailed multi-view um, you know, like input, for example, images or video such that these problems can be solved.

• 10:34Speaker 0

  Uh one uh quite interesting breakthrough at that point was how realistic these reconstruction can be compared to the older like photogrammetry-based methods. And uh, because these methods can reflect the lighting changes and uh you know complex physical effects are much better than the existing photogrammetry techniques. But this is also from the 2020. So a lot of more a lot more uh techniques have been improved uh since then. Uh, one example being uh Gaussian spadding. So you can think of it as another type of you know representation based on uh like volume rendering. Um that is you know much efficient, much more efficient than the existing like nerf-based techniques in terms of how like rendering works.

• 11:22Speaker 0

  Right. So it made uh you know kind of interaction with these objects possible because you can render things at much higher FPS than before. Gotcha. So what are we looking at here? So here we are looking at uh video generated by Dream Machine, where the input is uh uh image frame. So this is a very common example that people use in like 3D reconstruction. In the case of Nerf, there will be like many images like that surrounds this object, and the goal is to reconstruct the 3D object and then re-render them like back. It's a very interesting setup because you can see that the table is not fully reflective, but you can see how the lights are like reflecting on the side of the table as you change your like viewing angles.

• 12:10Speaker 0

  And that at that point was quite a revolutionary for Nerf to do because photogrammatic techniques won't be able to kind of reason about this type of physical effects just yet. Um but what's really surprising about Dream Machine is that we almost did nothing with regards to these NERF datasets or have zero 3D priors into how the model works. Right. But the model just by itself learns to kind of uncover these interesting physical um aspects of the world uh just by generating uh image to video. But the second example is from this paper called the zip nerf. And here we are also given first frame of the image and try to generate five seconds of video on top of it. But what's the really interesting thing you can check it is the reflections of the you know background in the TV.

• 13:02Speaker 0

  So as the camera moves, the TV refraction also seems to move uh consistently with what seems to be in the physical world, which might be a window that shines light onto this uh you know TV thing, like there. So that's another kind of an interesting or a few interesting like physical properties that is being like appeared onto this traditionally nerf theme. Again, this is only one image compared to in the traditional nerf sense, you have to take hundreds of images with almost perfect accuracy.

• 13:36Speaker 1

  So I I think it's just worth calling out what's happening here. You're saying a single image was passed through the dream machine as is a prompt. Yes. Dream Machine was then able to reconstruct a scene in video. And that video has consistent 3D light transport.

• 13:54Speaker 0

  I think it is yes. So yes, the it's I'm not an expert on like graphics to fully say how physically accurate it is, but at least I think it looks to me to be quite physically compelling. Right. So maybe if we run this through the perfect simulator, we will see some discrepancies as some now. But still, I I think the results are quite compelling and quite realistic compared to you know any of the models that we see.

• 14:26Speaker 1

  So, you know, one surprising capability that the that dream machine was able to handle was modeling the world in 3D. Yes. What other capabilities were you surprised by when Dream Machine was done training?

• 14:42Speaker 0

  Yes. Because we work on 3D, so naturally there are many properties that are in 3D that we definitely care about. And we are very surprised to see some of these problems being like solved just implicitly in the model. So one example is the depth, which is uh very obvious thing that people care a lot about in like 3D modeling. There's a tons of research on the like topic of depth. Um but we are just gonna show some videos that clearly shows that the model has an understanding about depth. So, for example, in this video, the it's an image to video case, so the model is able to know that the person is in the program that's closer to the camera than the background. A similar case happens in this video as well, where the subject is closer to the camera, and this is reflected when you see the camera starts to turn.

• 15:31Speaker 0

  And uh similarly, in even like more abstract cases, you can see how this notion of depth appear where the center of the like circle turns in the smaller distance than the circles that are like outside it. Similarly, there can be some quite non-trivial camera move changes that are just emerged from this type of reasoning, and not only still things, but also some cases where the subject is slightly moving, for example, this person tilts their camera up on. And there's also cases where the depth is pretty clearly reflected in the case of falling leaves where there are falling leaves that are closer to the camera, and then they're also following these that are further from the camera. Another interesting aspect of this things is also life transport, where this is also a very heavily investigated topic in graphics.

• 16:20Speaker 0

  So, for example, in this case, what we are trying to look at is as the camera turns, the red neon sign of Luma goes from the left to goes from the right to the left of the video. And this is reflected on the person's uh left side as their jacket is slightly reflective of the red color. And you can see more and more of this like red color being seeped in. So this is clearly something that is quite difficult to simulate in traditional graphics cases. Similarly, you can also see that like for example, in this cute animation, you can see that as the bear is wobbling inside the water by itself is moving, but also the reflection of the bear on the water is also moving with the flow. This other video shows how it can simulate how light goes through semi-transparent materials as the camera slowly turns.

• 17:09Speaker 0

  You can see the leaf on the foreground has more light shines through it as the camera turns. So these are also uh all quite tricky. So one we think we want to compare, of course, is how does it work against regular nerve captures? So this is something that nerfs are traditionally much better at than photogrammatry. Uh but unfortunately this case requires hundreds of images to perform well. Uh with Dream Machine, we just took another coffee machine that we have in the office and then just do image to video, and we can actually see a lot of the light reflections on top of the metal surface of the coffee machine being simulated by the model. And similarly, there are also some very non-trivial dynamics such as how water moves, how first move that can be done in the video models.

• 17:56Speaker 0

  And this is also example from like having animals, image to video of animal. It requires definitely a lot of the understanding of how first move and the dynamics of things if we come if we solve this problem from more traditional graphics pipeline. And not only that, we can also simulate like very compelling or interesting like camera motions because of this innate knowledge. Similarly, we can see how as this giraffe moves, the shadows also moves like alongside it, so that the model definitely has a knowledge about how light works like when things are moving. This is another example where we show that it'll like in the exploding room how things are floating together as the camera moves in forward. And this is another case where we show that behind the person there is like interesting like cloth that are moving.

• 18:45Speaker 0

  And if we try to simulate all of these with traditional graphics pipelines, like the cost simulation part will by itself be quite difficult.

• 18:54Speaker 1

  Alright, so fair enough. We've got a video model now that, as you just showed, can handle a ton of capabilities that traditionally would have taken explicit graphics pipelines, right? No game engine required, no volumetric reconstruction required. Yeah. We can handle 3D structure, it can do depth, it can do light transport, it can do dynamics, right? But if I'm being, let's say, ultra um rigorous about whether these video models really understand what's going on in reality. Yes. The holy grail is causality. Am I right? You kind of want to understand and see evidence that Dream Machine, if it's actually doing more than just generating cool frames, it ideally should be able to start predicting what changes in the world must come, must happen.

• 19:44Speaker 1

  Yes when given some input. Yeah. Some indication that it has an understanding of Newtonian physics or or state of the world. All and be able to reason over the entire state space of things that should happen when. And you when you throw a ball or my pen falls off. Do we have any evidence that might indicate that there's something more going on in these models than just pure frame to frame prediction?

• 20:12Speaker 0

  Yes, totally. In the previous types of video models that people have access, typically it's more like an image animator. So the image takes uh part in the majority of the scene, and then uh you know it's mostly like moving without cuts. However, you know, as we know in like movies and such, there's a very common thing, which is you know, things like transition to different cuts of you know, you're they're taking a video of the same subject inside you know different angles. Uh for example, in this case, the task is uh image to video. So we basically give this a close-up shot of this cute, like very little monster, and we just ask it to do image to video. So what's there are two interesting facts about it. The first interesting fact is that it generates a cut, which is uh again not seen in like previous video models.

• 21:04Speaker 0

  The other thing that is also interesting about this is that the background and the subject, which is the monster in the second shot looks basically the same as what is in the first shot. So we can see that this is a blue furry little monster, and then the second shot is that even though it's like taking a shot in the background, the monster is a little blue and flurry. So you get the impression that this is a video about the same monster. And also the environment is also similar where you have this like red ground on underneath and the blue kind of a forest on top of like on top of it, even though we haven't really asked the model to do anything about it. The model is able to reason about the second shot being a cut of the first shot that happens to have the same things in the first shot.

• 21:54Speaker 0

  So this is definitely some kind of a non-trivial cause and effect that happens like even in a not entirely like physically um possible scenario. Another example here I want to show is that again, this is an image to video. So it might look a little bit disturbing in the first frame, but the first frame contains what we want to give to the dream machine. So in this case, we are showing a very unnaturally looking eye looking at a girl that is standing like in front of the eye. In the first shot, that is the input to the dream machine. The model like the image does not contain the front face of the girl. There are two interesting facts about that. First, the dream machine generates the second shot, which is a close-up shot on the girl.

• 22:41Speaker 0

  And secondly, this girl looks very like terrified. Right. And this is possibly caused by this eye being very unnaturally looking. And this is some kind of a cause and effect that is like even harder to reason like strictly in the physics, but more delving into how just human psychology works. So I think the causality is definitely deeper than physics alone, and is going into other aspects that we as humans don't really can explain well about. Another interesting fact about this uh frightened girl uh close-up shot is that if you look at the first shot, uh the girl was wearing a blue dress and in hasing short hair. And the second shot is basically shows the same thing. Uh right. So it's not it's also suggesting that these two shots are basically different views of the same thing.

• 23:30Speaker 0

  Right. So I think this is also another evidence that you know the dream machine is able to reason about things that will happen or like even at like different angles in a matter that is quite non-trivial to reproduce using traditional methods.

• 23:46Speaker 1

  So this is pretty remarkable, right? Basically there's a representation of reality that the Luma Dream Machine model is exposing. That is able to maintain semantic consistency about what should happen in the world. Yes. With zero explicit design from you guys around that. So is causality essentially just an emergent property of choosing the right data?

• 24:11Speaker 0

  I would say yes. Um it seems that all the evidence that we are seeing points toward that.

• 24:19Speaker 1

  Got

• 24:19Speaker 0

  it.

• 24:20Speaker 1

  So this is pretty wild because Dream Machine is able to simulate the causality of reality purely through looking at camera cuts. Yes. Just by observing how camera angles and videos change. Yes. And the consistency of characters across those cuts. What's going on under the hood? What did you have to do to get that causality to show up?

• 24:42Speaker 0

  Again, this is not much. I think the biggest factor are the scale of data and the scale of compute. And I think we also get a lot of questions about whether there are more explicit priors being like baked into this. But um the answer is no. Like there's really not much prior that is um being baked into these models. And uh this follows roughly how um the better lesson tells us.

• 25:09Speaker 1

  Right. Cool. Okay.

• 25:12Speaker 0

  Um any other capabilities? Yes. So I think the other interesting thing that we showed a slightly earlier, but also want to kind of re-emphasize here is how Dream Machine is able to reason about the non-physical world as well. Even in cases where it's in entirely a scene of art, it is able to first of all do camera changes and also introduce like new concepts into the scene by just using the model. So again, if we were trying to reason about the world via the traditional kind of physics techniques, it is very difficult to kind of reason about things inside a totally fictional world. So I think this is something that more happens like inside our dreams rather than happening like inside a physical world, and it is like something that it's very close to what humans imagine.

• 26:04Speaker 0

  And therefore, it's also evidence that shows that maybe trying to be too fundamentalist about physics might not solve these problems. Um this is another example like from the same friend that like we're showing. This is like a very abstract figure of the input image, it's just a line inside the circus. And the dream machine was able to imagine the scene of this is inside a performance, and this concern was just uh vanished under this uh magic trick. So uh again, this is a totally kind of abstract scene that would not make any like physical sense. Um, but uh overall it uh has this artistic effect that is going on.

• 26:46Speaker 1

  Where do we go from here? What comes next and what have you what are your conclusions from seeing all the sort of emergent properties of reality that Dream Machine can simulate? Yes.

• 26:57Speaker 0

  So I think you know obviously one of the next steps to do is to um improve the model on all fronts, including uh resolution, uh, you know, efficiency, uh prompt following, uh being able to control with more precision and so on. And the team is actually you know kind of working on it. Um but I want to take it a step further um about like what's you know what we have in mind. So one of the examples here that we are showing is we mentioned earlier that from you can actually get 3D from 2D. So I don't think it's a much of a stretch to say maybe we can get from like videos to 4D. So that's being able to do world simulators meaning meaning that you might be able to simulate multiple angles at the same time.

• 27:46Speaker 0

  So I think towards that goal, I think it's definitely something which very much be interesting to have. Another really interesting aspect is to see how much uh unlock we can get by having a more intelligent multimodal agent. So you know, in this case, we've only demonstrated uh you know the interaction between uh text and uh video and image modalities, but human uh human perceive the world in you know many different factors. Uh for example, humans also you know hear the world and also you know uh you know interact with the world um you know differently. So uh I I think it will also be very um interesting to see how we can make further unlocks by going into this uh multimodal uh AI route where you know we combine all the signals that we can get uh and try to develop uh AI that you know learns from these uh signals uh together in a joint manner.

• 28:46Speaker 0

  Uh and I think currently you know we are only at a very very uh you know basic stage at you know being able to like generate uh videos from text and images uh at this stage. I would say this is more like a as you said, research preview or version zero of the model. I think the most more intelligent model will be able to kind of handle more complex inputs and more complex requirements that arise uh both from the you know product and requirement standpoint and also from you know what it means to be intelligent. So uh our our goal is to be able to you know create these multimodal uh AI that are uh like that can achieve this you know in a efficient and compelling manner.