Physical AI at NVIDIA

In recent years, the main impact of AI has been on the digital world, and most notably software engineering. However, its longer-term effect on the physical world may prove even more profound, reshaping how cars drive themselves, how factories run, and how robots move through homes and warehouses. World models are at the center of the movement to bring AI into the physical space. These models take in observations of the current state of the world and predict what comes next, which lets a robot plan its next move or a car anticipate the road ahead. NVIDIA has been building a family of these models called Cosmos, and sits at the frontier of physical AI and robotics. Amit Goel https://www.linkedin.com/in/amit-goel-759146ahas been atNVIDIA https://www.nvidia.com/ for fifteen years, and he is currently the Head of Robotics Ecosystem and Edge AI Product. In this episode, Amit joins Matt Merrill to discuss what separates physical AI from digital AI, how world models plan and predict, the two-tower architecture that lets a robot reason and act at once, why training on video demanded a new data pipeline, and how any developer can start building in this space. Free Physical AI Courses: https://nvda.ws/4xqCfo9 Cosmos: https://nvda.ws/4zJ4iRj GR00T Teleop + LeRobot: https://huggingface.co/blog/nvidia/nvidia-isaac-teleop-and-gr00t17-in-lerobot Sponsorship inquiries: [email protected] mailto:[email protected] The post Physical AI at NVIDIA https://softwareengineeringdaily.com/podcasts/physical-ai-at-nvidia/ appeared first on Software Engineering Daily https://softwareengineeringdaily.com.

• 0:00Speaker 00

  In recent years, the main impact of AI has been on the digital world, and most notably, software engineering. However, its longer term effect on the physical world may prove even more profound, reshaping how cars drive themselves, how factories run, and how robots move through homes and warehouses. World models are at the center of the movement to bring AI into the physical space. These models take in observations of the current state of the world and predict what comes next, which lets a robot plan its next move or car anticipate the road ahead. Nvidia has been building a family of these models called Cosmos and sits at the frontier of physical AI and robotics. Amit Goel has been at Nvidia for 15 years,

• 0:45Speaker 00

  and he is currently the head of robotics ecosystem and edge AI product. In this episode, Amit joins Matt Merrill to discuss what separates physical AI from digital AI, how world models plan and predict, the two tower architecture that lets a robot reason and act at once, why training on video demanded a new data pipeline, and how any developer can start building in this space. Matt Merrill is a software engineering leader with over 20 years of experience building and scaling software teams across enterprise and product focused organizations. His background is in backend development, cloud architecture, and distributed systems design. He currently architects and delivers software products

• 1:29Speaker 00

  and leads a team of engineers at Deft Agency. You can learn more about his work at code.theothermattm.com.

• 1:48Speaker 01

  Hello, everybody. My name is Matt Merrill, and I am here today with Amit Goel from Nvidia, who is head of robotics and edge AI. And I'm really excited to be here to talk to you about physical AI. So why don't I turn it over to you, Amit, to introduce yourself?

• 2:06Speaker 02

  Hey, Matt. Great to be here. I'm Amit Goel, head up our robotics ecosystem at Nvidia, which means my team gets to work with all the developers who are doing some amazing work with AI and robotics. And I also lead our product team that builds the computers for running the AI on the device on the edge. It's called the Jetson family of products. I've been at Nvidia for 15 years. Very excited to talk about physical AI today.

• 2:35Speaker 01

  All right, so I am super excited to talk about this because I think that a lot of folks listening here are probably, at this point, pretty familiar with the software side of Gen AI and LLMs. And what I kind of want to do today is just start from zero, right? Like, and that's actually, I'm not playing a role here. I purposely didn't dive into this deep so that I could kind of be fresh. So when Nvidia or anybody says physical AI, what do you mean? What makes it different from LLMs and coding agents that most of our listeners are using every day?

• 3:09Speaker 02

  Yeah, the big difference between LLMs or in general, digital AI and physical AI is that you're working with atoms. And what that means is you are dealing with sensors. You're looking at spatial information, 3D information. You're looking at videos. You're understanding dynamics of the world. So physical AI essentially is a model that can take as inputs the information of the world around you and predict what's going to happen in that world. So everybody is probably familiar right now with self-driving cars.

• 3:51Speaker 02

  If you live in San Francisco Bay Area, you'll see the Waymos driving around. So that's probably the closest way to think about physical AI in the real world, right? It's there, you have a machine that is taking the information from the cameras, the lidars, and the actuators in the car, and making a decision on what it should do next based on what the driver wants out of the car, right? So it's taking the data from the observations from the physical world and determining actions that have to be executed in the physical world. And it's been around for a while. The simplest form is probably your Nest thermostat that's controlling the temperature of things in your house based on your activity.

• 4:37Speaker 02

  And even like a small traffic intersection that's observing what's the activity that's happening and controlling that particular traffic light. So physical AI has been around for a while, but it's been very narrowly defined for one particular use case. So now what's happening with the large, really large models and general models is that you can have a single model that can understand all of this. And similar to what happens with LLM, right? You don't have a sentiment analysis model. You don't have a translation model. You don't have a summarization model. You have one big model that understands everything in the textual world, and you can program it to do what you want it to do.

• 5:18Speaker 02

  We are getting to the point where we can build this general physical intelligence models that you can use for a lot of applications in the physical world.

• 5:27Speaker 01

  Got it. Yeah, that was one of my big questions. I never really thought about that, like a self-driving car or even a Nest thermostat is technically physical AI. But when I was prepping for this, seeing some of the stuff that is currently happening is so much more advanced than that. And it makes sense that in the past year, the explosion and what these models can do enables a lot of this. So let's dive in and try to give folks like a mental model of how these things break down. And so one thing that I was reading about prepping for this was a concept of a world model. Can you tell folks about that and like how that fits in here and how it might drive one of these devices?

• 6:05Speaker 02

  Yeah, so the way to think about the world model is in the context of what are the inputs and what are the outputs, right? So for a world model, the inputs are observations of the current state of the world. It could be a sensor coming data from a motor of a car, of a robot. It could be a signal that is audio command that has been given as an input. It could be a textual prompt. And of course, it has to have the cameras and the visual representation of the world that's coming into it. The next question is, what does the world model predict, right? And inherently, the world model, because it's dealing with the physics of the world, is a physical AI model.

• 6:51Speaker 02

  It is multimodal because the real world has multiple modalities, right? We are hearing, we are listening, we are processing information. So it's multimodal, many inputs, and also many outputs. The typical output of a world model is a video, which, given a prompt, given a current state, if the world model understands what happens in the world, it can predict, based on what you told me, this is what's likely to happen. In the case of a car, you can say, I'm going at this particular speed. This is what I see is happening around me. Tell me what is likely going to be the state of the world in the next five seconds. Based on that, I can then control the car to what action it should take.

• 7:35Speaker 02

  Same analogy applies to a robot, right? You asked me to pick this thing and put it over there. This is what I see is my current state of everything around me. This is the object that I see. How should I move? Should I go left? Should I go right? Should I go backwards? Do I need to bend? It can predict what is the next few things it needs to do. So the world models can predict the world around you. It can do a 3D representation of the world. It can also understand, hey, I want to be there. This is my objective. How do I get there? What do you think about, in our head, it's called planning, right? So the world models can predict the plan to go from the current state to the desired state.

• 8:20Speaker 02

  So this is essentially the planning function of the world model. And the other thing is, you can say, hey, this is my current state. I'm planning to take this action. Can you tell me what is going to happen if I do this? And that is super important for robotics, right? Like, if you can say, I'm going to do this and tell me what's going to happen, you can actually decide whether that's the right thing to do or not. So rendering the entire world around you, given an action, predicting what the world is going to look like after you take that action, or given the current action and the desired state, predicting the action I should take to get to that desired state. These are all of the outputs of a world model today.

• 8:59Speaker 01

  That sounds like it could be, especially for something like a self-driving car, an absolutely enormous model or set of models? Yeah, it just seems like a lot, like a video, potentially audio? I don't know, just a lot. I mean, I'm almost having a hard time wrapping my mind around how much stuff is going into that.

• 9:19Speaker 02

  No, you're right. I mean, the amount of data that gets fed into a world model is orders of magnitude more than what you would feed into a text LLM, right? Because just a video of 15 minutes is several gigabytes of data. So of course, the amount of data that gets processed in a world model is a lot. But at the end of the day, I think the size of the model we see varies depending on what you want the model to do. You were asking about physical AI and difference between L L M's. And one of the things is that you have to run some of these models on the device itself, on that particular robot or on that particular car.

• 10:02Speaker 02

  You do not have the luxury to say, it's okay if it takes one minute to get back to with an answer. I'm going to throw all the compute available at it. So you need to think about how you're going to make distilled version of these world models that can run on device. But for something like evaluation, which can happen offline, right? If you're training a model, you collected some amount of data, but you want to train what it is going to do if it was in a particular street of New York. You can just ask the world model to give it a particular image and say, can you create a world around me and I want to drive this thing in New York, right? That part is super heavy, but that part can also run offline. You can throw a lot of compute at it.

• 10:47Speaker 02

  So at Nvidia, we have been working on the Cosmos models and a variety of those exist. And we have many different sizes of the model, starting from things that can run onto the edge, all the way to really, we have nanos and supermodels. They can be used for creating data. They can be used for evaluating things offline. So you're a hundred percent right. These models are processing a lot of data.

• 11:11Speaker 01

  You anticipated my next thought comment, which was just the speed. These things can't wait. It's not like me waiting for Fable to analyze my next presentation or something. Yeah.

• 11:21Speaker 02

  Yeah, you cannot just leave it around and come back and see what happened. You gotta, especially if you're deploying it on the device itself.

• 11:29Speaker 01

  Yeah, that's amazing. So I'd love to walk through, you know, you get these interview questions of, I remember back in the day, it used to be like, what happens when you press enter on Google? Right? So I kind of want to give you this for one of these situations. So we can pick a type of robot or whatever. What happens in a hundred milliseconds between that thing capturing, let's say, an image or some sort of multiple sensor inputs? And then a motor command going out of that thing to perform an action?

• 11:58Speaker 02

  Yeah, so typically, if you think about, let's say, a robot, the moment you are capturing the information from the robot's camera, typically you need more information than just the camera. You need to also understand what is the state of each of its actuators. You need to understand some of the other sensors like IMU, which is an inertial measurement unit that tells you the pose of the different parts of the robot. And once you capture this information, first you need to synchronize all of this. What that means is my camera is observing it this particular time. At the same time, what was the state of my motors? What are the state of the IMUs?

• 12:42Speaker 02

  And you need to fuse them together. The next thing you have to do is, if you're given a task, let's assume we're running a vision, language, action model. You need to tokenize this data, all of this data, the data from the current frame, but also the data from the past frames. Because your next action is dependent on not just what you're currently seeing, but also a little bit of history. So you, you have to take the data, you have to tokenize it, and then you feed it into this model, which is now going to do two things. First thing it has to do is it has to do a planning, which is the task of taking my current observation, my past observations, and my objective function. What should I do, right?

• 13:28Speaker 02

  If I was asked to pick up that particular cup, do I need to move left? Do I need to move right? Do I need to go up or down? How far do I need to reach? So this is the planning task. And today, this is something that is done by like an order regressiveness model that will predict the sequence of tasks that you need to do based on the objective function and current input. Now that's your reasoning and planning brain that kicks in. And the planner tells you, okay, based on the task that's given to you, this is the next step you need to do. Now comes in the second piece of the model, which is generating all the actions. And that part is very complex because depending on the robot type, you could be controlling thirty to forty motors.

• 14:13Speaker 02

  And you need to generate actions of all of that synchronously. You cannot serially generate that, right? You need to control the output. And one thing that's very unique about physical world is that a lot of things are continuous. You have to control the current and you cannot send it for like this much current. It has to be continuously generated. So the kind of models that you need is different than what you need to predict a discrete output. A sequence of text versus the current for fifty motors that you have to send it at every millisecond. So the frequency at which the output must be generated

• 14:55Speaker 02

  is very different for a robotic and a physical AI model. So you have to make these two models work together. A reasoning model that is planning can exist at a low frequency. The inputs of that goes to another model that actually generates the motor control commands. And that one is running really, really fast because it has to control all of that at high frequency. And once you have that output, typically that's happening in the brain. But typically, at the end of that, it goes to a small computer that is attached to the motors, which does your PID control based on the input that it got from the model.

• 15:32Speaker 01

  I'm not familiar with PID. What's up?

• 15:35Speaker 02

  So PID is the closed loop control, that low frequency control loop. That is, you have a target position, you get the current up position and you control the current. So that is running right onto the motors. It does not need to know what's happening on the other thirty motors, right? Because its objective is to control that one motor. And that runs on a very small computer or a microcontroller.

• 15:58Speaker 01

  I'm thinking of this in terms of like a BIOS or something like that, like.

• 16:01Speaker 02

  Yeah, yeah, yeah. It's just a very simple firmware that's running.

• 16:05Speaker 01

  It's just taking commands. I'm listening to this and I just can't help but think like this is the world's most insane state machine ever.

• 16:14Speaker 01

  That's just so much to keep track of.

• 16:17Speaker 02

  Yeah.

• 16:17Speaker 01

  And so you talked about this in a way that almost seems like there's two different models happening. There's a reasoning and a planning, but yet they're all sharing the same context, right?

• 16:31Speaker 02

  That's right.

• 16:31Speaker 01

  How does that work? How does it share that?

• 16:34Speaker 02

  Yeah, so that's the interesting part. There are many ways these models are being architected. And one of the ways that we are looking at this is having a shared latent space where the attention from one model and there's a cross attention between one model and the other model. The embeddings are shared so that you have a really high bandwidth communication between these two brains. If you look at the latest Cosmos, three family of models that Nvidia released, we have these two heads, right? A reasoning head and on top of that is the, we have the reasoning tower and the action tower on the output side. And they talk to each other.

• 17:17Speaker 02

  They have shared context in which they exchange information in the latent space. If you were running them distributed, you may want to take the output of one and decode it and send it to the other one and again encode it. But that slows down the bandwidth, right? As I was telling you, for a robotic system, they need to operate really in a closed loop. So that's why we build Cosmos with that shared context so that both these can coexist and exchange information with very high bandwidth.

• 17:46Speaker 01

  I'm almost picturing this just like, I don't know, I'm trying to think of like a CPU analogy, but it's logical, it's almost like a logical division inside the same hardware. Is that correct or do I have that wrong?

• 17:57Speaker 02

  You're absolutely right. Think about like in a physical space, we have a shared memory, right? Where you have two different processes, you don't want to send the data over because it wastes your bandwidth. The same thing we are doing at the model level now, right? Where with in the latent space, we can share the information and the context between these two models. So they can operate, although they're operating in different frequencies, they can share the same context.

• 18:22Speaker 01

  Got it. That's really interesting. I was going to ask about that two tower design because that was something I came across. It didn't make sense to me, but as you explained that reasoning step, that does make more sense.

• 18:35Speaker 02

  Yeah, and you know, the best, another way to think of it is like is an architect, which is planning everything that you need to build. And then there's an executioner, which is given the plan is doing the execution. They both need access to the same information, but what they do with that information is slightly different.

• 18:55Speaker 01

  This episode of Software Engineering Daily is brought to you by VariaCode. If you're an engineering or IT leader, you know how painful this is. An open rec sits unfilled for three months. Your tech lead is burning out, covering the gaps, and your product roadmap gets pushed back again. VariaCode solves that. For over 10 years, they've been embedding senior Latin American engineers directly into US engineering teams in under 10 days at up to 40% less than local hiring. We're not talking about generic contractor pools. These are pre-vetted, high-quality engineers with real on-call and SLA experience. They work directly in your Slack, Jira, and daily stand-ups in your exact US timezone with zero friction. If you're ready to stop losing sprints to open recs,

• 19:37Speaker 01

  head over to variacode.com/sed right now and get a custom shortlist of senior candidates in 48 hours. That's V A R I A C O D E dot com slash S E D.

• 19:51Speaker 00

  Every visitor on your platform looks the same at first. Can you trust them or are they a threat? Fingerprint lets you answer this question right away with one API call, no model to maintain, and implementation in minutes. Fingerprint recognizes bots, AI agents, VPNs, tampering, and over 100 more unique signals to give you durable risk scoring and threat detection in real time. Even as human and agentic activity blends together, you can identify suspicious traffic faster and build safer and more secure sites and apps. Over 6,000 companies rely on it every day. Join them now at fingerprint.com. Think about your mobile app's source code.

• 20:34Speaker 00

  Once it hits the App Store, it's out in the wild. And without the right protection, decompiling is easy for malicious actors looking to steal your IP or tamper with your software. That's where GuardSquare comes in. GuardSquare provides the highest level of mobile app security for Android and iOS applications and SDKs. Their advanced tools integrate seamlessly into your CICD pipeline. We're talking polymorphic, multi-layered code-hardening techniques and automated runtime application self-protection, paired with mobile application security testing and real-time threat monitoring to deliver the highest level of mobile app security without compromise. Don't leave your hard work exposed.

• 21:17Speaker 00

  Secure your mobile applications today. Go to guardsquare.com to learn more.

• 21:24Speaker 01

  So, one of the things here that's like, I mean, we talk in textual models about using GPUs because they're really good at this task, but I can't help but think like this requires just so much more horsepower, so to speak, than what those need. So like, if you can simplify it, like, what is the difference in the hardware that's powering these things?

• 21:46Speaker 02

  There are a few things that is different. As I mentioned, when you're thinking about the world model, the number of modalities that you're dealing with is quite different. What that does to your architecture is when you're training a model, when you're dealing with the text inputs, the bandwidth that you're processing for data itself is not that high. When you're working with the world models and you're training a world model, you need to retrieve the images. You need to retrieve the actions or the motor control commands. You need to retrieve the text annotation for all of that in parallel and bring it into your model.

• 22:27Speaker 02

  That creates pretty unique challenges from the hardware perspective as to what do you keep in the storage? How do you manage the data? We in fact had to go ahead and build out a completely different stack in order to process this data because GPU resources are super scarce and precious. And a lot of this preprocessing typically was done on a CPU. And so you would have your model waiting for the data to do the training, but it's just not possible to do that with the existing pipelines that were built for text-based model training. So we created something called Curator,

• 23:11Speaker 02

  which was the ability to now take large amounts of videos, process it, store it in a way that now you can keep the training data coming in. So that's on the data side of things. On the hardware side of things, when it comes to like training these models, as you know, with LLMs also, there are multiple phases, right? You have this pre-training, there's mid-training, and then there's post-training. You see a lot of post-training happens when using reinforcement learning. You're using different tools. The same is true for world models. But when you're training a world model, let's say it's a robotics world model. How do you verify it? How do you do the reinforcement learning? For a coding model, reinforcement learning is you go run it,

• 23:56Speaker 02

  compile the code, verify it, works or not. For a word model, its job is to simulate the word. You actually need to go to a physics simulator as a tool, right? And that physics simulator is best run on another kind of GPU, which has the ray tracing cores that can do the sensor simulation. So it becomes a heterogeneous computer architecture. On the one side, you need something that can process all of this data. Then, of course, you have your transformer model that needs to get trained. And then after that, you have to continuously call a simulator where you go and do the reinforcement learning and testing whether if it's working or not. So it becomes a pretty complex architecture.

• 24:39Speaker 02

  And inherently, it has become the heterogeneous computing problem, right? That's one other thing that developers who are used to LLMs get access to a whole bunch of computers, all, you know, latest Blackwell clusters. They can go and just train their models, test the models, evaluate it, serve it all on that cluster. But physically, AI requires actually three computers. You need a computer for training the model. You need a computer for where you do simulation, either to create data or to do reinforcement learning post-training. And then last piece of it, deployment. You cannot deploy it on the same infrastructure that you trained it on.

• 25:21Speaker 02

  You need to deploy it on a small computer that's sitting inside the brain of a robot. So you need to go do some more work on the model, quantization, distillation, compression to go deploy it. So it requires a very different way to think about the compute hardware that you need if you're building a physically high model.

• 25:40Speaker 01

  I'm joking here, but I'm imagining you being at like a cocktail party, talking with a software developer about textual LLMs and going, that's cute.

• 25:48Speaker 02

  Yeah.

• 25:50Speaker 01

  Just kidding, but.

• 25:51Speaker 02

  Yeah, I think Jim Phan, who leads our research team for physically AI and humanoid robotics, he often jokes like, I'm always envious of my friends in the LLM world. Their problem is I don't have more GPUs. And my problem is like, okay, first of all, the data that I got, like half of the sensor data is missing, the synchronization is not there. And then I'd have to, I get the GPUs, I train it, but next I have to find a simulation environment to go test it, deploy it on the real robot, the cables are breaking and, you know, so it's a very different world. Yeah.

• 26:26Speaker 01

  That is something, the simulation piece of it, never even, I mean, yeah, the horsepower even just to do that simulation is mind-boggling.

• 26:33Speaker 02

  Yeah.

• 26:35Speaker 01

  So, switching topics a little bit, like, when these things are deployed, they need to react to somewhat unpredictable things, right? Like, they effectively need to learn. And I read a little bit about imitation learning and watching humans do tasks. And can you say a little bit more about how that happens in, at least in current research and out in the real world and how that operates?

• 26:59Speaker 02

  Yeah, so two things that's being done right now to make these models more generalizable. First is to collect data in diverse environments, right? So this is if you are supposed to assemble, let's pack a bag of shoes, right? Often, the size of the bag, the shoes is going to be different. The size of the box is going to be different. So one way to do this is you have a lot of people collecting data in different environments, in different factories, bringing the data of doing the task in different environments and training the model so that it now has seen enough distribution

• 27:40Speaker 02

  so that the real world falls somewhere within that distribution, right? So that's why when you see imitation learning, that's what is happening. We are collecting the data, increasing the distribution in which it's been collected.

• 27:55Speaker 02

  But as it turns out, however much data you collect in the real world, the real world has an infinite distribution. It's continuous, so you can collect it in ten different lights, but there's going to be something that's always going to be out of distribution. That's where the second part comes in, where you need to think about having reasoning model paired with a model that has seen enough distribution. They go together, right? You cannot just have a reasoning model but not have enough real world data because the reasoning model can tell you this is closest to this particular real world distribution and then you can go act on that, right? So the reasoning model is learning from data in the wild.

• 28:40Speaker 02

  It's using all the text data that's available. It's using the video data that's collected across different domains, right? And learning from that. So combining this pre-training with data in the wild, textual, video data, along with a real data, action data, which is people doing the task, robots doing the task. Combining these two, the world models are now trying to solve this problem of, how do I know enough that I'm physically capable of doing the task? I know how to pick up an object. I know how to slide something. I know how to bend over and perform a particular task.

• 29:22Speaker 02

  And how do I do it in different environment, in a different context, right? So both of these things are being trained together. The first one, how do I do a certain particular task? Is being done combination of either imitation learning and reinforcement learning. And then how do I do this in any environment? This is where the data in the wild is playing a role.

• 29:43Speaker 01

  Okay, so the reasoning models, are these very task specific?

• 29:48Speaker 02

  The reasoning model, this is where the generalization is coming from, right? So you're trying to collect diversity of information. Your robot might be designed to work in a factory, but you're putting in all the data from homes, from outdoors, from indoors. Because that's when you build a reasoning capability that can generalize and you can draw parallels.

• 30:09Speaker 01

  I see, yeah, that makes sense to me that you need to give it tons of different inputs to be able to react to different things. So, okay.

• 30:19Speaker 02

  Yeah, because if you were just looking at factory, you would probably not see a cat or a cow come in. And so, you know, when you're doing the reasoning model, you expose it to all elements.

• 30:30Speaker 01

  This is a little random digression. I was on Reddit the other day and somebody posted, I live near Boston, so it was on the Boston subreddit. Somebody posted a picture of this guy with his wacky looking backpack on. It was like a crazy backpack and people were making jokes like, oh, he's the rocketeer or whatever. And somebody was like, no, I think that they're gathering world data for AI models. I see you nodding your head that that's probably what that guy was doing, so.

• 30:56Speaker 02

  Yeah, one of the things is happening is simulation is important, right? As I said, for for physical AI, because that's where you can safely test your models. That is where you can do reinforcement learning faster than real time speeds. But the challenge with simulation models is the cost of creating that simulated world is quite high. So what you probably likely saw was somebody, you know, carrying a backpack with sub cameras and lidars. And what they are doing is capturing the real world and passing it into a model that can create a 3D representation of that world. Right? What we like to call real to sim, right?

• 31:40Speaker 02

  You bring the, take the real data, run it through a model and creates a simulated version of it. Now, once you have the simulated version of it, you can, you know, drive in that simulated version and test how it's going to have, you know, how your robot or your car is going to behave. You can use that and do augmentations, right? What if this building's colors were different? What if the lighting was different? What if the road was narrower? So you are closer to the actual distribution. And this data is often used for post-training. This is where, you know, I want my car to operate really well in this particular block. Okay, I'm going to collect the data and let the my general model post-train in this world. Now it becomes really good at that environment.

• 32:25Speaker 01

  That's really interesting. That's actually the opposite of what I might have thought, so thank you for clarifying. Cool. One thing that just came to mind here is, you know, like hallucinations and text models or, you know, you just kind of chuckle at them. But hallucinations in a physical space sound a little scary. I would imagine that there's better guardrails for things like that. And I'm curious how, you know, you handle that in the physical AI space.

• 32:51Speaker 02

  So one of the things that is important for physical AI is, as you mentioned, that if you're out of distribution, if you're hallucinating, the impact is pretty significant. So you have to do two things for that. Either you can make your model really conservative and says, okay, this is out of distribution. I see something that I've not seen before. Or, you know, this is not part of my learning curriculum. I'm not going to act on it. The other thing you can do is, well, I'm going to act on it. And in fact, the way I'm going to make sure that I know how to handle, you know, this out of distribution scenarios is

• 33:33Speaker 02

  I'm going to actually give those and simulate in that environment, right? I'm going to evaluate my policies or my models in a lot of different environments, which some of them may not even be real, and then score them on their performance. So one of the ways to make sure that your robot policies are not hallucinating or if they are out of distribution, they're still safe, is to expose them to those adverse environment in simulation and see how they perform. That's one of the ways to kind of make sure your policies and the models that you're training are within the guardrail. The second thing also is, you remember we talked about a planning brain and an action brain, right?

• 34:18Speaker 02

  The planning brain is sort of like a VLM or an LLM where you can do the guardrails into that so that it does not plan something that is not acceptable. If it does not plan, then the action brain cannot, you know, it is operating on the plans that are coming out of it. How do you put the guardrails around your planning? It's very similar to how do you put guardrails around in LLM to, you know, not give you an answer or an information that is not suitable, right? So you can just train things at the planning stage. On the action stage, again, you know, it is mostly learning in domain. You can go to simulation and test it.

• 34:59Speaker 02

  And the third thing that we are also thinking about is when it comes to safety and actions is you have to have some level of redundancy, like some level of external element that is observing and verifying. So because that kind of separates the concerns, right? The person that is acting and the person that is monitoring, if that's, there's another AI that can observe what is happening and then, you know, have a way to control the robot or the car, that's another way to manage safety, right? So I think it's a combination of all of these things that are coming together to make sure that as we are building these models, as they go into the real world,

• 35:43Speaker 02

  we can make sure that, you know, we're not being extremely conservative and not being able to do half of the tasks. We don't want that, right? But we want them to be able to do that safely. So we are bringing this outside in element of safety where you're monitoring, you're looking at it and making a decision. You make sure that your reasoning brain, planning brain has the guardrails and then extensively test in simulation.

• 36:11Speaker 01

  That's one thing I wanted to dig in. When you say, so when you say it evaluates those plans, it makes sense in training that it does that. But I want to clarify, is it also doing this in the real world too? Like if it is unsure about its action, is it actually going to run simulations in real time to evaluate which one may be the best?

• 36:32Speaker 02

  That's the biggest value of the word from word models is, you know, they can do counterfactuals. They can simulate, if you look at the work that we published recently with our self-driving car team called Alpha Mayo, it's a reasoning model. And within that, it can reason, it can predict ten different futures and choose one of them, right? Because it is a word model, it is not limited to only the right thing that it does. It's not this imitation learning on, you know, this is the right way to do it. It can explore ten different directions.

• 37:10Speaker 02

  Ten different outcomes, score them and act on just the safest and the one that gets you the most to your objective function.

• 37:17Speaker 01

  I see.

• 37:17Speaker 02

  So that's the beauty of the world models. It is doing that at evaluation time.

• 37:23Speaker 01

  Wow.

• 37:24Speaker 02

  Yeah. And it's important to have these models open for that reason as well, right? Because what is acceptable in one part of the world versus what's acceptable in another part of the world? Thinking about driving rules, right? What is an acceptable driving behavior in Boston is likely different from the acceptable driving behavior in the Bay Area. And if you go to a different country, of course, it changes. So taking these world models that can predict all of the different counterfactual scenarios and then a post-training them to choose, you know, what is acceptable is important.

• 37:59Speaker 01

  When I heard you talk about the security, I can't help but think like this is security in layers, security in the planning stage, security in the, yeah, in the execution stage evaluating options. So that's

• 38:11Speaker 02

  Yeah.

• 38:11Speaker 01

  really good. And that is not something that, well, I don't know. I was about to say, that's not something you hear too much about in the L L M's, but I, that's probably not true. We're starting to do things like that as well.

• 38:21Speaker 02

  That's right.

• 38:22Speaker 01

  So you mentioned the open models, which is really exciting and also a little bit scary because, you know, one of the things where, you know, my catastrophizing brain goes is like, what happens if this tech gets into the wrong hands, right? Like, and I know, like, you're not a soothsayer. We, you know, we all, at least, you know, our listeners have the best of intentions. But like, is there anything being done in the research community to help get, get ahead of this in terms of ethics, legalities, and just, you know, human safety in general?

• 38:52Speaker 02

  Yeah, I think fundamentally to think about it, if these models are open, right? As long as we have more people who have access to them with the right intention, they will outnumber the people who will have access to this with, with a bad intention. Hopefully that's the world we live in, right? So in that case, I think being open and accessible actually gives us more tools for everybody to build systems that can monitor them. As I said, you know, the thing with safety is you, you have to assume that what if it happens? How am I going to counteract? I think we saw recent, what happened in a couple of weeks ago, you know, with the agent

• 39:35Speaker 02

  getting, and getting attacking the Hugging Face and they were able to use an open model to actually go and triage that and, you know, fix it, right? So I think that's why we believe that keeping, having a pretty strong, open ecosystem is vital to make sure that there are checks and balances. It's vital for people to, you know, create models that are better aligned to their philosophy, right? And it's especially true for like physical world. I was off kidding with somebody that me and my wife can't agree on how we load a dishwasher, right? Like, that is the right way.

• 40:21Speaker 02

  So we need an open model where, you know, I can, she can reinforce, train it to do it the way she likes it, or I can do it. So if it doesn't, you know, somebody decides for us how the dishwasher is going to be loaded, there's no way that's going to work.

• 40:34Speaker 01

  I mean, do you have a dishwasher loading robot? Like, is that, are you doing that at home? Like, because I'm totally jealous. Like, is that a perk of working at Nvidia? No, okay.

• 40:43Speaker 02

  Not yet, but I'm looking forward to it. I think it'll happen soon.

• 40:47Speaker 01

  Give me a prototype.

• 40:48Speaker 02

  Yeah.

• 40:49Speaker 01

  That's really interesting. And tell me if I'm completely off here, but like, I almost think of this in terms of like human morals, right? Like you, the deeper the moral is set in someone, the harder it is to compromise it. And I think what you're saying is that these models are open and they are deeply ingrained in ethical things. It's very hard to unprogram that. Is that kind of what you're saying?

• 41:13Speaker 02

  I think if these models are open, you can introspect them and you can steer them. If they're not, then you can put monitors, you can put all kinds of things, but you cannot, you know, you're limited in what you can do. And then you also introduce a single, single point of failure as well.

• 41:31Speaker 01

  Yeah.

• 41:31Speaker 02

  Right? That's like, so I think diversity and choice and, you know, one model checking the other one is the best way for making sure that you have a better answer to the security problem.

• 41:46Speaker 01

  I can see what you mean by the transparency being huge because, you know, if there's, if there's a bad guy, you don't know what he's thinking, but if you could see what he's thinking, that's a little bit of a different story. It's really interesting. I want to talk a little bit about, so you released Cosmos three edge. You mentioned this Cosmos model. And for somebody who's not familiar with this, right? Like, I don't really know what that, I think you said it's a model, but I don't really know what that means edge. So can you talk a little bit about what that is?

• 42:14Speaker 02

  And I was telling you earlier, right? The, the world models can do many different functions. They can create, predict random worlds in which you can, you know, evaluate your policies. They can take real world data and augment it to change the environments. But some of these things are pretty compute intensive, right? But one of the functions of a world model is also, what we were talking about earlier is that they can be the planning brain for your robot because they can predict a lot of different scenarios and pick one that is most appropriate, that gets you to their objective function. That model needs to run on the robot, right?

• 42:58Speaker 02

  That part of the world model. This model is not really generating, you know, pretty videos, 10 second long videos. It doesn't have to do that. This model does not have to, you know, you can still down to certain particular domains that you care about, but it should be able to, you know, do the task of predicting the future and controlling the robot. So we took our Cosmos three model, which comes in different sizes, and we created a four billion parameter version of it. And the main purpose of the Cosmos three edge is that it's a, it's a world model that can take the inputs as text or actions or videos and generate, you know, the next action that the robot needs to do, right?

• 43:43Speaker 02

  And by making it a four billion parameter model, we can now deploy it onto the robot. So it can be used as the policy of the robot itself. And, you know, we, we optimize it. It can run real time on a Jetson Thor, which is, you know, a computer, which is the size of passport that has 2,000 tera ops of compute.

• 44:05Speaker 01

  Did you say the size of a passport?

• 44:07Speaker 02

  Yeah, it's the size of a passport. If you're passport, yeah.

• 44:10Speaker 01

  Wow.

• 44:10Speaker 02

  Yeah. That's what is, you know, inside a lot of robots. If you look at the one X robot, the Boston Dynamics, yeah, humanoid, they are all powered by this small computer, but it has 2,000 tera ops of compute. It can run 10 billion, four billion parameter model on it. So that way is, you know, it can do the reasoning you were asking about counterfactual, it can do all of that on device.

• 44:34Speaker 01

  That is wild. That is really cool. So I can see why that's a big deal. Some mini model that can run on robot.

• 44:42Speaker 02

  On the robot.

• 44:43Speaker 01

  It's so funny because when I think edge, I think of, you know, like, oh, CDN distributed geographically, but when you guys talk about edge, it's like, nah, it's in a factory or it's, yeah.

• 44:52Speaker 02

  Yeah. Yeah. It's battery powered, you know, running on device, plus running all the time.

• 44:58Speaker 01

  Yeah. And the power consumption of those things has to be reasonable too. Not only is it small, but the power consumption. Wow.

• 45:06Speaker 02

  Yes.

• 45:06Speaker 01

  That's crazy. Okay. So I'm a normal developer. I want to start playing around with this next weekend. How do I get my hands on one of these passport sized computers? They're only like 20 bucks, right? I'm just kidding. How accessible is this? Like, I would love to play with something like this. Like, how does a normal dev get into this?

• 45:28Speaker 02

  Yeah, I think it has become much easier in the last, I would say 12 to 18 months. You know, in the past, the hardware was very expensive. It was because it was designed like for robotics, right? And one of the things we started working with, with the team at Hugging Face was to target and enable all the software developers who are used to, you know, writing programs to start programming robots. And to do that, you need to have a robot, right? That you have to have, you're building physical AI. Of course you can do a lot in simulation, right? That's the first step. You have to get started with this in a, you use your existing computer, you download

• 46:08Speaker 02

  a simulator and you train, you download a existing model, connect it together and see how it works, right?

• 46:15Speaker 01

  And this is all just running on your laptop or?

• 46:17Speaker 02

  Yeah, you can, you know, you can run on a decent laptop today, which has a 4090 GPU. Yeah, I can get a DGX spark, run it on that. So you can run the simulator, you can train the models, test it. But the interesting thing that is about physical AI is that experience of, you know, touching a real thing and dealing with the connection and the motors and the, you know, the variations there. So now it's like, you know, you can, for a couple of hundred dollars, you can get a robotic arm that you can train, you can tele-operate it, collect data, format it, publish it to Hugging Face, train a policy, convert it, deploy it onto the small Jetson and let the

• 47:02Speaker 02

  robot do its thing, you know? And all of that, I would say like for you to get started, if you have a good computer, a thousand dollars budget, take a couple of days and you can have your first physical AI model doing things.

• 47:17Speaker 01

  Wow. What are the arms? Is that something Nvidia produces or is that something that, that other like third parties produce?

• 47:25Speaker 02

  It's, it's just third party. There are lots of projects. If you're into 3D printing, you can just order the material, 3D print these things, get it together.

• 47:33Speaker 01

  That's really cool. Okay.

• 47:34Speaker 02

  Yeah. Pretty big open source ecosystem. You know, the, if you look at SO 101 arm, that's the most popular with the makers right now. You can, you can 3D print it, a lot of vendors do the getting of all together and they ship it to you, but you still get the joy of, you know, connecting it and building it all together. Yeah. So that's there. And then, you know, you can start from there and go to a humanoid and get one of these, you know, humanoid is for 15,000, $10,000 these days that you can make it do whatever you want, dance or, you know, or clean your home.

• 48:07Speaker 01

  Wow. When somebody's like looking into the, like those, I don't know what we call them, like classes of product, like what would I Google for or what would I ask Claude for in terms of trying to find that? That's something that to me, if I were to step away, I'd be like, okay, I, I guess I could search like open source robotic arm, but I feel like there's probably something more specific that people could search.

• 48:29Speaker 02

  Yeah. I think if you're interested in using this.

• 48:34Speaker 02

  New age of robotics, which is end to end models, right? That is taking pixels and controls out. You want to look for models that are supported by, that support Nvidia's like crude or physical intelligence or, you know, these popular AI models, right? So those ro- robot hardware that has been tested with these models.

• 49:01Speaker 01

  Okay.

• 49:01Speaker 02

  The way to search for is look for different robot policies, AI models that exist and say, I'm looking for a hardware that works with this particular model, right? You don't want to have to customize the model for a different hardware, right?

• 49:17Speaker 01

  Yeah, that's a bridge too far for somebody trying to get into it.

• 49:20Speaker 02

  Exactly. That's where the, the hugging face checkpoints are really good because they're, they already have checkpoints for this SO one on one arm. If you're looking at action checking transformer model, you are looking for, you know, a group model, you're looking for any of the popular AI models for robotics. You can find a checkpoint for specific hardwares and that's one way to, you know, go find it. Because then you can just cobble it together and have your first, you know, roll out. And then you can say, okay, I want to go customize it, right?

• 49:52Speaker 01

  Yeah. Oh, that sounds fun. Okay.

• 49:54Speaker 02

  Yeah.

• 49:54Speaker 01

  Cool. I hope that inspires some people. All right, let's close it out. I got a couple of open ended what if questions. So what sounds like science fiction right now that you actually think might come true in five years with this?

• 50:06Speaker 02

  If you think about robots today, what comes to your mind?

• 50:11Speaker 01

  Probably that Boston Dynamics thing that can like, you know,

• 50:13Speaker 02

  Yeah.

• 50:14Speaker 01

  jump up stairs and jump up on platforms and things like that. Yeah.

• 50:17Speaker 02

  Yeah. I think for most people, when I think about robots, it's the embodiment that comes to mind, right? As you said, that humanoid or the big arm that they saw in an automotive plant that was carrying the entire car and turning it upside down. But I think in five years, we will see definitely robots that are more around us, more ubiquitous, that are multifunctional and can do many things. Robots today are purpose built. You think about robot, you think about either that humanoid or you think about a Roomba or you think about, you know, that industrial robot. You don't think about a brain that can do 10 things for you. Right?

• 51:02Speaker 02

  You think about the body, not the brain. I think in five years from now, I expect that we will have robotic embodiments that are going to be, you know, capable of doing multiple things. So when you think about robot, you will not think about the body, but you will think about, oh, that thing that can do all of these tasks for me. The embodiment is just, you know, an artifact. So that would change. You would not think about a robot for A, but you would think about a robot that can do a class of things for you. The other thing that's going to change is for the robotics people is the task of programming was very intimidating. It's like you need to be an expert, you need to be either, you know, a technician or PhD

• 51:47Speaker 02

  to be able to program the robots. That is going to look very different, you know, with these foundation models. You would be able to likely show a task, do it one time, do it with the robot two times and teaching the robot with data is, is what is going to be magical. It seems impossible if you tell a robotics person today that, hey, you know, programming a robot is just like, go show it. Hey, this is the task I want you to do. It is a video. Can you go do it? Right? That seems like magic today. But that's going to be how we wouldn't tell robots how to do a task. You're going to either demonstrate it, opti-operated just one time, two times, and the robot will go do it.

• 52:32Speaker 01

  I mean, if we didn't have this inflection point with software based L L Ms in the past year, I probably would not believe you, but seeing what happens in the past year, I, I believe it. Yeah.

• 52:44Speaker 02

  Yeah.

• 52:44Speaker 01

  It's, it's crazy.

• 52:46Speaker 02

  So I think those are the two things that I, you know, I want thing would happen.

• 52:49Speaker 01

  Yeah.

• 52:50Speaker 02

  Multifunction robots that can be just taught by telling them what to do, showing them what to do.

• 52:56Speaker 01

  Right? You make such a good point about, I mean, for the history of robots, it's always been the thing with the arms and the face and it talks. Doesn't have to be like that, right? And so, yeah, I can't wait until I have a disembodied arm robot that can do whatever I show it to do in my house, like load dishwasher. So. So AI is reshaping a lot of software engineering jobs, right? I mean, it's a little scary and also enthralling at the same time. Some people are wondering where their careers are going to go and, you know, maybe you're, you've inspired some people to look into this. So if I'm a senior engineer today, I have no robotics background, but I have a heck of a lot of software background.

• 53:38Speaker 01

  What would you start learning besides the basics that we just talked about playing around? Like what, what's the best way to really get into this?

• 53:46Speaker 02

  The best way, if you want to, you know, get into robotics, it's just starting, I believe with simulation because that is where, you know, it still feels like you're working with software, but you get to experience the physical challenges where you don't think about, just like you get exposed to gravity, you get exposed to friction, you get exposed to all the real world challenges that you will encounter when you get to the hardware. So I think that's the first thing is get out, start working with the simulator, get

• 54:26Speaker 02

  exposed to those terminologies and the, you know, the datas that, that you're not used to when you're working with code and software. And then when you get comfortable with the simulation, then, you know, deploying it on hardware, as I mentioned, that there is a lot of different hardware modalities that are available now where you can deploy the models that you've trained, you can get one of those and start deploying them. You have to get physical when you're working with the physical AI. So don't be intimidated by it, embrace it. And I think the more and more like software developers and hardware developers work together, I think we'll have better systems.

• 55:11Speaker 02

  Today, still I feel that a lot of times the hardware developers who are building the mechanical team is a little bit operating differently, decoupled from the software team. But I think when they start understanding each other's languages, which is what, what would happen if, you know, as AI, software developers get into physical AI, they need to understand, you know, all the mechanical constraints. That would be pretty exciting and I think we'll have much more capable systems.

• 55:42Speaker 01

  Before we talked, I was wondering if you're going to say like, oh, you need to go get a PhD in math or some, I'm joking, but like, I think what, what I'm hearing is like, no, that's starting to change, just like in software, when you wanted to mess around without, with, let's not even talk about L L Ms, but machine learning, it used to be, you gotta know a lot of math, you gotta really get in there with the concepts and now it's becoming a lot more user friendly. Sounds the same thing.

• 56:06Speaker 02

  Yeah, I think the open source ecosystem around robotics is pretty strong. You have everything that you need from tools to assets to models to, it's all available. So you can really go ahead Zen and get, get started.

• 56:20Speaker 01

  All right, well, I mean, this was fantastic. Super, super interesting. I really thank you for being here and, and explaining this to us.

• 56:29Speaker 02

  This is great and I really enjoyed being here and hope you have a good rest of your week.

• 56:35Speaker 01

  You too. Send me that dishwashing robot when you get it.

• 56:38Speaker 02

  Yeah, I will.