How Physical AI Learns Across Language, Video and Action — Ming-Yu Liu

The car making a left turn at the start of this episode was never filmed. Cosmos 3 generated it. Ming-Yu Liu, who leads the Cosmos research at NVIDIA, explains how one model can describe a video, generate one, and produce robot actions. He walks Tim through the architecture. A vision language model reasons one token at a time; its weights then initialise a bidirectional diffusion generator for video, audio and action, and a shared temporal position scheme lines up signals that run at different rates. Ming-Yu treats "world model" as a set of tools, not one definition: forward dynamics, inverse dynamics and policy, trained together under a capacity limit so that each helps the others. He also explains why plentiful first-person human video carries over to robots, which have far less data of their own, and why a Cosmos model post-trained on the DROID dataset is a good starting point for pick-and-place policies. The most practical thread is testing. A neural simulator does not need accurate success rates. It only needs to rank policy A above policy B the way the real world would, so a team can narrow down which checkpoints deserve a real trial. Cosmos Dreams applies that closed-loop idea to driving and robotics, and Ming-Yu argues that humanoids around children and pets make safety matter even more than it does for cars. The conversation ends on the Super, Nano and Edge sizes (Edge targets Jetson Thor, Orin and DGX Spark) and where to find the open weights, code and data. This episode is a paid partnership with NVIDIA. Learn more about Cosmos: https://nvda.ws/4cJoY1S Explore Cosmos Lab: https://research.nvidia.com/labs/cosmos-lab/cosmos3/ --- TIMESTAMPS: 00:00:00 A road that was never filmed 00:02:28 Inside Cosmos 3: reasoning and generator towers 00:05:02 World models: dynamics, policy and one clock 00:08:59 Learning robot skills from human video 00:11:06 Ambiguous tasks and system 2 planning 00:12:53 Neural simulators for policy verification 00:16:41 Cosmos as a starting point for robot policies 00:19:00 Cosmos Dreams and robot safety 00:22:04 Super, Nano and Edge model sizes 00:24:24 Open models, the Cosmos repo and feedback --- REFERENCES: tool: [00:00:13] Cosmos 3 (NVIDIA Cosmos Lab project page) https://research.nvidia.com/labs/cosmos-lab/cosmos3/ [00:18:27] NVIDIA Cosmos GitHub repository https://github.com/NVIDIA/cosmos [00:22:05] Cosmos3-Edge model card https://huggingface.co/nvidia/Cosmos3-Edge [00:22:15] Cosmos3-Super model card https://huggingface.co/nvidia/Cosmos3-Super [00:22:16] Cosmos3-Nano model card https://huggingface.co/nvidia/Cosmos3-Nano [00:22:50] NVIDIA Jetson Thor https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-thor/ [00:22:52] NVIDIA Jetson Orin https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-orin/ [00:22:53] NVIDIA DGX Spark https://www.nvidia.com/en-us/products/workstations/dgx-spark/ [00:24:42] Cosmos 3 collection on Hugging Face https://huggingface.co/collections/nvidia/cosmos3 other: [00:01:07] Cosmos-Dreams closed-loop simulators (NVIDIA SIGGRAPH 2026 blog) https://blogs.nvidia.com/blog/siggraph-news-2026/ paper: [00:08:54] Cosmos 3: Omnimodal World Models for Physical AI https://arxiv.org/abs/2606.02800 [00:17:43] DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset https://arxiv.org/abs/2403.12945 --- RESCRIPT: https://app.rescript.info/share/e2385948cf465f0d6a2c0930150fc3ab

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:01Speaker 00

  This episode is brought to you by Michelob Ultra. Good times mean competitions between you and your friends. Like seeing who can throw some pigskin the farthest, or rush the field the fastest. Winner gets an ultra. Because it's not just a superior-like beer, it's also giving fans superior access to their favorite sporting events. Superior is worth playing for. Visit michelobultra.com/superioraccess to enter. Enjoy responsibly. Copyright 2026 Anheuser-Busch InBev Michelob Ultra Beer, St. Louis, Missouri.

• 0:30Speaker 01

  Flowing ad budget on metrics that look great, till the CFO sees them, that's bull spend. And marketers are calling it out, in. Dashboard, confessions.

• 0:39Speaker 02

  I remember telling my boss, "It'll be good for the brand," when leads were slow. Yeah, it, it wasn't.

• 0:47Speaker 01

  Cut the bull spend. LinkedIn lets you target by company, job title, and more. Advertise on LinkedIn. Spend $250 on your first campaign and get a $250 credit. Go to LinkedIn.com/campaign. Terms and conditions apply.

• 1:00Speaker 03

  This car is about making a left turn. Can you envision the trajectory the car is taking, and the video generated?

• 1:08Tim Scarfe?

  Guess what? That road was never filmed. It was made by Nvidia's Cosmos 3. It reads video, it simulates roads, and it's learning how to handle objects. Mingyu Liu leads the research, and this is a paid partnership with Nvidia. Now, Cosmos, it takes text, video, audio, and actions as input. Let's start with the simplest case.

• 1:37Speaker 03

  You ask the model, what happened in the videos? How many cars in the video? Whether the robot succeed in complete and action in the video? Okay, there are three cars, and the robot succeed in pick up the apple and put in the basket. When you have a video and tested inputs, and generate task, it becomes a vision language model.

• 1:58Tim Scarfe?

  So why don't we flip this thing on its head? The model stops describing the scene, and starts generating it.

• 2:07Speaker 03

  It device Cosmos 3 is a set of skills to build a closed-loop simulator, not just for self-driving. I can employ a hundred driver, drive a hundred cars, go to different intersection, and figure out whether it works better. Or I can use this simulator to verify the accuracy. You are not limited by the size of your fleet. Anytime you need to do a lot of testing, you be of the computer, just launch it.

• 2:40Tim Scarfe?

  Okay, so what if we took the same idea, but on an even harder problem? Like robots, for example?

• 2:46Speaker 03

  You know, navigation, you don't want your physical device to touch any other things. But manipulation task require all kind of interactions. When there's an interaction, there's a cushion. There's a potential deformation, you know, when things are manipulated. So it's more challenging. We will reach a state where we will be able to use neural stimulation to handle this complex manipulation well, right? Once we reach the status, you can test your robot policy with the wall simulator.

• 3:20Tim Scarfe?

  Okay, so one model, Cosmos 3, handling three jobs. Handling objects in robotics is the future. It's very exciting.

• 3:28Speaker 03

  So you can see that you are looking at a very diverse, versatile model architecture.

• 3:34Tim Scarfe?

  And we now have action as a first-class citizen, which means, you know, the model might actually take an action, and then its next observation could be caused by the thing that it did before.

• 3:46Speaker 03

  We start with a language model. We, we either train it from scratch ourselves, or take it in open model, right? Start with the language model. It has a basic understanding of the language structure, you know, understand language. Then, we have a vision encoder that can take a video or a set of images inputs. Then, we connect this vision encoder to the larger language model. Connecting this vision encoder to the language model, we kind of build a vision language model, right? And sometimes you can also start with a pre-trained vision language model. Okay, now, this language model,

• 4:31Speaker 03

  you know, operating in the autoregressive manner. We did a token at the time, right? Okay, it learn to connect, you know, visual output to the language space. It thinks in language, it spend what it see, right? So you have a vision language model. Okay, now, we take this vision language model, you know, the pre-trained weights, and then initialize a generator, right? So, one key thing is that this vision language model is autoregressive. And our generator is actually diffusion, is bidirectional.

• 5:14Speaker 03

  Now, this bidirectional generator tower learn to generate video, action, audio. Okay, in this tower, every token attend to every other token. So when you generate the video or the action, there's a coherence among the chunk it generates. Among each generation, the signal within this chunk is coherent. And it also diverge the token from the vision language model part, the recent tower, which we call it the recent tower. And so it also understand the instruction, you know, what you want to generate, okay? And in that generator training, there are two stage.

• 5:57Speaker 03

  Pre-training, learn the basic. And mid-training, we add action in.

• 6:02Tim Scarfe?

  In your estimation, what is a world model?

• 6:05Speaker 03

  This is a very challenging question, and I think earlier, there's something similar. People tried to give a definition to AGI. I think several years ago, this started. But I think till today, there's still no kind of common agreement what AGI is, right? People have their own definition. And I think a world model gonna be the same, right? So, I think a world model is a collection of useful tools, right? We model something because we are trying to achieve some goal, right? For example, it could be predict the future, or it could be understand why this happened.

• 6:48Speaker 03

  There are three important modeling in robotics. Forward dynamics, you know, given what the starting point and the action taken, what will be the future? Predict what gonna happen. Inverse dynamics, given the visual transition, you know, infer what was the action taken. And policy, what should the robot do to achieve the task?

• 7:15Tim Scarfe?

  We're combining these different modalities, like, you know, audio and action and vision and so on. And they might run on different time scales. How do you make them all run on the same time scale?

• 7:26Speaker 03

  Different signal have different, you know, frequency, right? Even for video, we have come with a different frame rate. High-speed video or low-speed video. And the audio, you know, there's also different hertz, and action also different. So, we have a temporal position embedding scheme, where we pretty much normalize all the signal into the same essence, the same scale. So that's when the token represent different time chunk, or different special time chunk of the signal,

• 8:08Speaker 03

  better know which other token are in the same time instance, at the same time instances. And also know the relative distance between different time instances, yeah. So that is a critical part to make this only model work. You know, handling audio, video, and action all come with different frequencies.

• 8:29Tim Scarfe?

  You were talking earlier about these different modalities. Forward dynamics, inverse dynamics, and policy in one single model. How exactly do they reinforce each other?

• 8:38Speaker 03

  When we train the model, we put them together, right? And we give the information bottleneck. You only have these amounts of capacity to expand all them. These are all trying to connect video, visual observation to action. So in the world action model, right? You have a test instruction, what you want to achieve, and this is why you observe. And you start to generate action and possible outputs, right? And possible output is similar to forward dynamics, right? When you have action in, what could be the output? Inverse dynamics, when you have this visual transition, what could be the action, right?

• 9:22Speaker 03

  Also similar to what happened in the world action model, right? You have the video and action, the correlation between them. Yeah, so I think these three, they are trying to, fundamentally trying to capture the correlation between the observation and action, right? It's just a different slice, different perspective, yeah. And once you put the information bottleneck, you put the capacity constraints, you know, the model learn. And our result in the paper show that there's a synergy. One does help the other.

• 9:59Tim Scarfe?

  And I think it's fair to say there's a bit of an asymmetry in the training data, right? So there's lots of human-generated video data, and then there's far less action data. And, but you get this virtuous transfer between the modalities. But could the model ever know when not to transfer? When transfer could be harmful?

• 10:18Speaker 03

  I think the model doesn't really know. It's, so, as you are talking about, we have a larger amount of egocentric video, right? The first-person view and the two hands trying to complete certain task. And we have really small amounts, actually much smaller amounts of the robot video. You know, robot also have head camera, you know, gripper camera, and, you know, see it's complete certain task, right? So, I think it's more about our shared vocabulary for different embodiments, right? So, you know, human or robot looks like a human, right? And how human hand manipulate one object, the visual pattern, the visual...

• 11:04Speaker 03

  Action correlation, how human manipulates an object. The pattern, you can see it's very similar to how a robot would manipulate objects. Even though the action space doesn't really correlate precisely. Once you correlate one set of action, the pattern is similar. It's easier to generalize to the other. And it also give you, when you incorporate a couple of different embodiment in, it also help you to generalize to unseen embodiment. Because, I mean, if humanoid robot, they're gonna look like human. Maybe some have a bigger hands, some have a longer hands. But it gonna be similar, right?

• 11:48Speaker 03

  So, and gripper can be different, but you know, the function are also similar. The key is really the the pattern, the correlation between the visual observation and action, right? And the translation from one action embodiment, one embodiment to another embodiment, I think is really really easier.

• 12:06Tim Scarfe?

  Apparently, one of the failure modes is when tasks are not well enough specified.

• 12:11Speaker 03

  I think a system two is the key to make ambiguous tasks more concrete for system one to execute.

• 12:21Tim Scarfe?

  Yeah, it's fascinating, because I'm thinking that we want to use this technology in safety critical environments, right? And some tasks are just quite ambiguous. But presumably, we we could add in layers of, of, you know, maybe maybe we don't understand exactly how to specify the thing that we're doing, but we could add in layers of tests and checking, and we could red team it and make it improve over time.

• 12:45Speaker 03

  You give a task, it will be similar to L N agents, right? You give a task, first initial call of the agents or the model was to find a second of task. Need to be complete, right? And it would execute one by one. And after each execution, it will check whether this is complete. And at some point, there may be choices need to be need to be made, right? So, okay, should I do this or the other, right? I think this will require us to think this robotic task more in the system level.

• 13:30Speaker 03

  A layer above the model level. You have some harness to kind of gather the resources together. Either your past memory or the tools that you can use, right? So, and seeing this this gonna be a more more complex way to look at the physical task.

• 13:53Tim Scarfe?

  One thing we we've not spoken about enough is is using Cosmos as as a simulator essentially to train a policy. And I'm just thinking whether there would be some potential problems with that. Because you know, in reinforcement learning, we have the sim to real gap. And could you have a situation where the policy might learn to exploit features inside the simulator?

• 14:13Speaker 03

  Yeah, it it's possible. The way I look at how this neural simulator progress is that I think it will be first, it will be more useful for passive verification in the beginning. Passive verification is a task that's, you know, if you are a model builder, any model wrong, you're gonna have a lot of checkpoints. And you might do a version study, and you're gonna, every, you know, variance, you're gonna have many checkpoints. And how do you know which one is better, right? An ideal case is to deploy each policy in a real robot, in a real environment, and make sure the complete rate, right?

• 14:57Speaker 03

  Success rate, in all the hope happen to self-driving car, right? Every time you update your driving policy, how do you know it works? So you need to have a fleet of driver and go to different challenging scenario and check whether this policy is desirable, right? It's so costly, there's so many checkpoints. So, if we can use a war model as a war simulator, as a replacement of the real world and real driver, right? You have you have the the policy directly interact with the war simulator. And and the war simulator in the end of the rollout, you can measure whether this test is completed or not.

• 15:42Speaker 03

  Then you have a success rate, right? If the success rate of the war simulator, the neural simulator, correlates with the real world, you know, testing, the ranking preserve. You don't need them to be precise. You just need to know if policy A is better than policy B in neural simulator. And most likely, policy A gonna be better than policy B in the real world. You just need this ranking preserve. Then with this, you can quickly narrow down on the number of checkpoint you need to do real developments. It gonna largely improve your development velocity, help you to find a better policy sooner. Okay, and and in this passive verification,

• 16:25Speaker 03

  you don't directly train the model using the visual outputs, the rollouts, right? And the model, I mean, if what happened in large language model gonna happen in war model, we're gonna see rapid improvements of of war model, right? So even before it's mature enough to feed the passing model, the training data to train, war simulator can be used as a passive verification. Yeah, that's what I think. Okay, so now we go back to the question you asked, there's a hacking. Hacking gonna happen in, you know, even you use, you know, deep learning,

• 17:05Speaker 03

  you know, if there's some pattern, some scene that can exploit, right? There're gonna be adversarial samples, right? So I I think a neural simulator, you know, gonna be exploited, and other simulator gonna also be exploited. And and I think as we reach that status, I I think there will be ways to to to regularize the model training, to make sure those can be used. Yeah, so I don't know precisely the approach. I think this is also a very exciting area. And but I I just believe that, yeah, people will figure out way.

• 17:41Tim Scarfe?

  Yeah, and and on on this thing about using Cosmos as as a teacher essentially for for policies, you you guys have got a a set of recipes around this, and and you should you should talk to that. But one interesting thing is the ability to imagine, you know, like the the notional elephant in the room. So something that that is quite unnatural, that would be quite low frequency in the training data. You can imagine those, but you could also have this adaptive system where you could recognize edge cases, and then you could improve the policy over the over over time.

• 18:12Speaker 03

  The Cosmos as the teacher, so in Cosmos, right? So we try to help the ecosystem in three ways. Better data, better environment, and better starting points, right? So we believe that a a customer model, a only model, war model, let's link video action test together to learn the shared representation. It's a great starting point to build a policy model. And not just a theory, we actually post-train a customer model on a on the ZOE dataset and choose still the art, pick and place policy results. And the reason, I mean, our intuition is that if we can predict the pixel dynamic well, and we can learn the correlation between the pixel and action,

• 18:57Speaker 03

  then it gonna help you to build a policy model. When when we kind of keep advancing Cosmo model, when we bring more embodiment in, it it gonna become a better and better starting points. And the amounts of effort required for adapt to a new embodiment will be less.

• 19:19Tim Scarfe?

  Well, maybe you should comment on the recipes, because this will really help folks at home get to get up to speed with it.

• 19:25Speaker 03

  As we build Cosmo as a platform, we also provide post-training recipe in our kind of Cosmo repo. So that's a user, a developer can leave you a recipe to reproduce the results we have, starting with the Cosmo model, right? So, and we do forward to add more recipe there, maybe also skills to help you, you know, provide data, and then agent help you to post-train Cosmo and to towards something you like, yeah? So that's trying to serve the community this way.

• 19:59Tim Scarfe?

  Amazing, and can you tell me about Cosmos streams?

• 20:02Speaker 03

  We define Cosmos streams is a set of skills to build a closed-loop simulator for, not just for self-driving, it's for all sorts of embodiment, right? So, it it's a fall into the conventional, like traditional definition of war model in robotics, right? Action in, you know, future observation out, forward dynamics. You know, you're not limited by the size of your fleet, right? And anytime you need to do a lot of testing, if you have the computer, just launch it. And it gonna help you the development velocity. You know, self-driving, very useful.

• 20:46Speaker 03

  And I think this gonna also be true for robotics. I think a war model now is good enough for navigation task. In robotics, the other challenge is the manipulation task. You know, navigation, you don't want your physical device to touch any other scenes. But manipulation task require all kind of interactions. When when there's an interaction, there's a cushion, right? And there's a potential deformation, you know, when when scenes are manipulated, so it's more challenging. I'm very optimistic with the excitement the whole field has in war model

• 21:25Speaker 03

  and and you know, the continued advance of the the deep learning research and also computers. We will reach a state where we will be able to use neural simulation to handle this complex manipulation well, right? Once we reach the status, you can test your robot policy with the war simulator, right?

• 21:52Speaker 03

  You know, self-driving car require a lot of passive verification today because car is moving around in our human space, right? So we need them to be safe enough, right? So when robot, when humanoid robot reach the status where you're gonna be deploying on many people's houses, safety is even more important. You don't expect to see a car, you know, in your house. But once humanoid is everywhere, you're gonna expect to see humanoid surrounding you and maybe your kid, your pets, right? So you do want them to be safe enough. And how do you know, right? So a developer of humanoid robot,

• 22:35Speaker 03

  how do you make sure your every iteration of your policy, you know, or the whole system is safe enough? You need the policy verification. And you won't have enough space for setting up all kind of kitchens, all kind of different task. And so I think using new simulator is the only way to give you the development velocity. You know, we need this.

• 23:04Tim Scarfe?

  Very cool. And you also have this Cosmos 3 Edge model. So this runs on a single device?

• 23:09Speaker 03

  Yes, Cosmos come with, models come with three different sizes. We have a super, nano, and edge, right? Super is the, you know, the, give you the highest fidelity. It's a frontier model. If you want very high accuracy and you can, you have to come. The resources definitely go with super. Nano is in between, smaller, easier to pull train with all sorts of GPU resources you have. We want to make sure it run really well on edge devices

• 23:49Speaker 03

  like the Jason Thor, Orin, or Digi Spark, right? So the reason we build this one is that we want the world model deeps where robot lives, right? So that you don't need to have a wrong chip to data center when you want to take a certain action. And I think it's important, you know, when you come to real deployments, right? You may not be able to count on the, you know, the network connection good enough to do all the tasks, right? And sometimes it's sensitive, critical, right? You have to complete the task at a time. So we develop edge so that it's small enough, but also powerful enough to run on those edge device.

• 24:34Speaker 03

  And because the model is smaller, you know, take less compute resources, so it can be easily find you. And earlier, we have a recipe find you the Cosmos Edge model in one day to boost its visual understanding capability. So I think this all three version gonna be super important. Yeah, and I also envision that they may need to work together in some form for future robotic development. Very challenging task. You probably want a very powerful model. You know, it's like call out, call for help. And for simpler task, you know, you have to finish right away.

• 25:19Speaker 03

  You know, you want to depends on the model that is on the edge device.

• 25:24Tim Scarfe?

  And for folks at home, where can they get their hands on these models and find out more about it?

• 25:30Speaker 03

  Yeah, so all our models are open and all our code and even some of the training data we created are open. So the models and the data live in Hugging Face and the code live in GitHub repo. Yeah, so we have github.com/media/cosmo as our landing repo. From there, you can see links to the Hugging Face models, the data sets, and also the training code, the post-training script. Yeah, and we also have skills in the repo to help your agent quickly pick up the offering in Cosmos.

• 26:13Speaker 03

  We build Cosmos to help the ecosystem, to help a wide range of physical eye developers. We need your feedback, you know. I have my email box, you know, I'm watching every issues come to this Cosmos repo. And we do want to see your feedback, you know, good or bad, both are very welcome. And we are committed to make it better and better.

• 26:47Tim Scarfe?

  Mingyu, it's been a pleasure and an honor having you on MLST. Thank you so much for joining us today.

• 26:52Speaker 03

  Thank you, Tim.

• 26:57Speaker 05

  Now, more than ever, we need someone on our side. But Rob Whitman? Whitman's votes drove up costs, cut Medicaid, and risked 350,000 Virginians' health care to pay for tax breaks for himself and his donors. He's not on our side. Shannon Taylor will fight corruption, cut taxes, and bring down costs. This November, vote for Shannon Taylor. She's for us. Paid for by HMP, the House Majority PAC dot com. Not authorized by any candidate or candidate's committee.

• 27:27Speaker 06

  Your heart can tell you a lot about your health. Apple Watch Series 12 measures your heart rate every five seconds with the most accurate heart rate sensing in wearable. So your vitals app, now with heart rate variability, can tell you when something is off. And your readiness score can let you know when to rest and when to push. Hear the story in every heartbeat with Apple Watch Series 12. The features described are for wellness purposes only and not for medical use. iPhone 11 or later required. Based on Apple conducted study of heart rate accuracy, August 2026. Visit apple.com slash apple watch series 12.