Open models and the future of Physical AI with NVIDIA

As AI moves beyond the cloud and into robots, vehicles, and other physical systems, new approaches to models, simulation, and inference are emerging. Daniel and Chris are joined by Ming-Yu Liu, Vice President of Cosmos Lab at NVIDIA, who shares his perspective on open models, world models, and the future of physical AI. They explore why open models matter for innovation, how world models can help AI understand and simulate the physical world, and what it could take to build increasingly capable physical AI systems. Featuring: Ming-Yu Liu – LinkedIn https://www.linkedin.com/in/mingyuliu/Daniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaChris Benson – Website https://chrisbenson.com/, LinkedIn https://www.linkedin.com/in/chrisbenson, Bluesky https://bsky.app/profile/chrisbenson.bsky.social, GitHub https://github.com/chrisbenson, X https://x.com/chrisbensonLinks: NVIDIA Cosmos https://www.nvidia.com/en-us/ai/cosmos/Sponsors: Midwest AI Summit: Join AI practitioners on October 15 in Indianapolis for practical sessions, hands-on discussions, and real-world AI solutions. Use code PracticalAI20 to save 20% on your registration. https://midwestaisummit.com/#ticketsPrediction Guard: A self-hosted AI control plane for running agents in high impact environments. predictionguard.com/practicalai https://predictionguard.com/practicalaiResources and Events: Register for upcoming webinars here https://practicalai.fm/webinars! Prior Webinars from our partner Prediction Guard https://www.youtube.com/playlist?list=PLRpRdFmBfdF6boDgAD3nNaLm8Yx4KN2VtMidwest AI Summit 2026 https://midwestaisummit.com/

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:02Speaker 00

  Welcome to the Practical AI Podcast, where we break down the real-world applications of artificial intelligence and how it's shaping the way we live, work, and create. Our goal is to help make AI technology practical, productive, and accessible to everyone. Whether you're a developer, business leader, or just curious about the tech behind the buzz, you're in the right place. Be sure to connect with us on LinkedIn, X, or Blue Sky to stay up to date with episode drops, behind-the-scenes content, and AI insights. You can learn more at practicalai.fm. Now, onto the show.

• 0:41Daniel Whitenack?

  Welcome to another episode of the Practical AI Podcast. This is Daniel Whitenack. I am CEO at Prediction Guard, and I'm joined, as always, by my co-host, Chris Benson, who is a Principal AI and Autonomy Research Engineer. How are you doing, Chris?

• 0:56Speaker 02

  Hey, I'm doing great today, Daniel. Really excited about today's conversation.

• 1:00Daniel Whitenack?

  Yeah, well, I think in the world that both of us inhabit, we are open to interesting discussions. And I think we'll talk about things open and things related to the world in today's episode, because we have with us Mingyu Liu, who is Vice President of Cosmos Lab at Nvidia. Welcome to the show, Mingyu.

• 1:25Speaker 03

  Hello, sent to have me here. Great to be here. Hi, Daniel and Chris.

• 1:29Daniel Whitenack?

  Yeah, yeah, great to have you. And one of the things I kind of alluded to is maybe something open, which I know is dear to Nvidia's heart, which is open models. And I've seen in the news recently, in what Nvidia has posted, obviously, they are promoting open models as a key piece of what is important in the AI ecosystem. I'm wondering, Mingyu, if you could help us understand, maybe from your perspective, from Nvidia's perspective, why in today's world is the idea of having open models and promoting open models,

• 2:11Daniel Whitenack?

  why is that such a critical thing in the ecosystem?

• 2:15Speaker 03

  Yes, okay. So when I was a student, I learned a lot about computer vision and machine learning. At that time, there are great researchers, students, open source layer code and model. And looking at their code and model actually helped me understand the concept and helped me to innovate, right? And the academy always worked like that and had been years, you know, even before GPT. And GPT also have open source model in the past. And that had been, you know, pushed the field forward, allow innovation. And it's just until recently,

• 3:00Speaker 03

  the model is getting bigger. The amount of resources required to build a model is tremendous amount. And hence, you know, the amount of open model reduced, right? But open model generally is there for the field. And it's been very useful to push the field forward. And so Nvidia, one, this continue to happen. I think the world will be in a better place if we have open model to enable innovation. So people, you know, API are great. They solve problems, but they don't give you the insight. You know, it didn't tell you. It doesn't allow you to tear apart

• 3:44Speaker 03

  and miss your idea insight, you know? And also, the kind of thing you want to do, maybe suit about, you know, you want to have a local compute and you don't have the internet connection. So API may not always be the solution. And that's why, you know, I think the open model is important to enable people. And Nvidia is a celebrated computing company. So we build this computer, great computer, GPU, networking devices. And now we also build CPU SoC, right? We build this computer, and we want to enable the whole ecosystem, right? And we generally believe that,

• 4:26Speaker 03

  you know, to better build these computers, we need to know more about the applications, right? And in the old day, you have this computer, and you have library, and you have software applications, right? And in the modern world, you have computer and you have library still, but you now have models. Model is a new kind of libraries people can use and to build great amazing applications. You know, maybe all kind of agents to help you buy stuff or help you to code all on top of models. And we, Nvidia, like to build models, like to build open models

• 5:09Speaker 03

  on top of Nvidia's computer. There are a couple of reasons why we do that. First, we, through building the model, we better understand what kind of computer we should build, what kind of GPU architecture we should have, so that Nvidia can always be there to help the whole ecosystem. And second, we also want to make sure people are enabled to do amazing applications, right? So when we have open model there, and people can test this out first, you know, and combine them to do application good for their targeted business. And again, some familiarity, and maybe later they can build their own model. Yeah, so there's a lot of reason why, you know, open model is critical,

• 5:53Speaker 03

  both for academy and also for the whole industry.

• 5:55Daniel Whitenack?

  Yeah, you were kind of getting to a follow-up that I wanted to have, which is it's important to have open models, to your point, to promote innovation. But I'm wondering, in what you're seeing at Nvidia, is it increasingly important also on the commercial side? So like for Nvidia's customers, right? What is the implication of having access to open models, maybe on the commercial side? Do you have any thoughts on that? So it's great if, like, I'm in a lab and, you know, I have, you know, five years to work towards, you know, the next innovation, the next types of models. But if I'm in a commercial company,

• 6:40Daniel Whitenack?

  maybe I'm a customer of Nvidia. I run Nvidia hardware. How does the existence of open models help me as, you know, in the commercial world, maybe?

• 6:51Speaker 03

  Innovation, freedom. Innovation, freedom. I think that is maybe the two words to summarize, right? So, you know, you can leverage API to solve the problem. And but with the open model, you can do more than what's available in API. Yeah, so and see it be the field is finding more and more important application of AI, right? You know, from earlier day of chatbots and now the coding agent, and we also see more and more agent that can do things that we didn't think is possible. People are working very hard to try to figure out

• 7:35Speaker 03

  how AI can make our life collectively better, right? And there will be application that was not foreseen by the foundation model of the frontier companies who are building AI, right? And then requires some customization. Open model allow them, give them options, give them a choice to do the innovation they need for their target business. And this way, I think it's good, you know, for the whole society.

• 8:04Speaker 02

  I think, I know I was, you know, kind of following up on your point there. I was very happy recently to see Nvidia kind of step, you know, there was a moment some months back where, you know, open models in the West were starting to struggle a bit, you know, and some certain unnamed organizations were closing and that had been leading the way. And so it was a delight to see Nvidia step in and take the leadership there and follow through with some other, you know, things. The Hugging Face acquisition that's coming in, the work that you're doing. So there's, we're kind of, you know, going through this turning point of, you know,

• 8:48Speaker 02

  toward the next generation of open models and what that looks like, especially with regard to the landscape of physical AI out there, which is my personal passion, as most of our viewers know. And so would love if you could kind of, as we are pivoting into this new generation of open models in a physical AI context, where it's not all API-based, it's not all frontier models at big services and stuff like that. And we're all getting excited about drones in everyday life, robots in everyday life, and all the different capacities

• 9:32Speaker 02

  that that can bring benefits. Could you kind of lay out the landscape? What is this, as you mesh these things together? What does that look like to you now? You know, what are you seeing as the opportunities in the near term? And kind of how are organizations that work with Nvidia starting to position to take on this next generation of physical AI that's upon us?

• 9:55Speaker 03

  Yeah, thanks for the great question. So I think physical AI is a term covers several important verticals. Like automotive is one. Uh, factory automation is another uh and robotics is, you know, a third pillar. There could be more agriculture, construction, many of them. And physical A I are A I deployed in physical device. And physical device uh perturbed the state of the world and complete uh certain tasks. And the result is shown uh as material, as the final uh uh something you can touch, right?

• 10:37Speaker 03

  That is how I I would like to uh, you know, define physical A I. So uh nowadays, uh, you know, we have some uh A I in physical device, right? But they they are preliminary, right? And and I think the most exciting form of physical A I uh will be one of the most exciting forms of physical A I. It's a human knowing robot uh and me envision that human knowing robot will be able to help people accomplish a lot of task. Uh, and now we don't have a human knowing robot surrounding us. But one day we have human knowing robot surrounding us, they will require a computer. They will require a powerful computer to help us process visual input, understand people's instruction and then self-correction uh in completed task.

• 11:22Speaker 03

  And and these are great business opportunity for Nvidia, right? So that's why Nvidia care about physical A I. If we can make physical A I uh come, you know, the the gene of physical A I come true. Uh, there will be allow opportunity for Nvidia. So we are very aligned and so we want to help the field uh moving forward. Yeah, and yeah, go ahead.

• 11:40Daniel Whitenack?

  Yeah, and I was wondering, just tying that back to where we started the conversation with open models. Do open models have certain implications specifically in the physical A I world? Because of the constraints or requirements of the physical A I world?

• 11:56Speaker 03

  Yes, it's a good question. So uh I I think uh, for example, one thing uh you a physical A I model, you know, one popular physical A I model is a parsing model. A parsing model is a model based on the visual observation and the intrusion given by the uh user or some machine. Uh, and you know, that it it generates on action to complete the task, right? So observation intrusion in and action out. And every robot have a different uh kind of sensor setup, right? Some robot have two eyes, some robot have cameras mounted on the gripper. And self-driving car, some car has eleven cameras, some have seven, you know, some have LiDAR sensor.

• 12:41Speaker 03

  So every physical device, every physical, every robot see the world differently, right? And and some of them are unique, right? So it's difficult to have one A I model that can cover all these, right? And when you have open model, uh, the model come with building capability like war understanding or simulation. And from that prior, people can leverage the data they collected from their device and fine tune. You know, maybe this uh initial model only handle one camera. Now you have multi-view, you know, enable this through post-training and open model. And it's easier, it's more achievable with open model, right?

• 13:25Speaker 03

  So I think open model give you the the innovation power. Uh, so you can customize uh open model uh to better tailor for your uh physical use case. And this is important, right? It means uh better leverage the observation, right? And and also give you opportunity to make it more efficient. Uh, eh, yeah, so in the end, uh, uh, the primary uh evolution will be uh intelligence per per watt, right? And and so with this uh open architecture, uh you will be able to do more.

• 14:03Daniel Whitenack?

  It's amazing to consider this topic of physical A I, how A I agents are being embedded in the physical world around us. But it just emphasizes this point that we need to have operational control of the agents that we're deploying. Whether that be in the case of physical A I or in your digital infrastructure within your company, where you're deploying agents as a part of your digital workforce. Operational control is critical so you can ensure that those agents are operating based on the policies that you set. And you can alert when those agents go off-script and actually quarantine agents that are behaving in some destructive manner.

• 14:47Daniel Whitenack?

  That's exactly what we're offering with Prediction Guard, the self-hosted A I control plane. This is a company that I'm leading personally and I'm just so thrilled to see that this operational control of A I agents is at the core of what we're offering. We're seeing companies build and deploy fleets of A I agents without losing control of those agents. And we'd love to show you how we do that. Check us out at predictionguard.com-practicalai to book a time with myself and the team to talk more about how we're enabling this operational control of agents. Again, that's predictionguard.com-practicalai.

• 15:34Speaker 02

  So I'd like to uh, I'd like to take you uh kind of expand on what you were talking about going into the break uh in terms of, you know, you talked about, you know, a couple of things like fine-tuning models and stuff like that, which I, which I think uh maybe to some people, maybe a little bit of a lost art because people have become so A P I dependent on a lot of the closed models because they're used to cloud environments. And as we move into uh, you know, into physical A I and you're taking these open models that give so much potential on what's, what you can do with your own creativity and innovation uh in uh in these physical devices. And you have to add in um, you know, the the sensors, as you talked about LiDAR and other things,

• 16:19Speaker 02

  and and whatever the robot's effectors are, how it's manipulating the real world uh uh or or perturbing the real world, I believe is is how you put it earlier. Um, could you talk, you know, there's inference involved, which now is on a device versus out in the cloud. Um, and there's power, you know, you have to have power to drive these things. So there's all of these considerations that a lot of folks in the A I space haven't had to deal with in the past. Um, and that that that gets something out there that you can touch uh working toward your whatever end you're trying to achieve. Could you talk a bit about how you see that and like what, how how you put that together for?

• 17:02Speaker 02

  So maybe kind of give a a quick primer uh to folks that are doing A I, that are that are watching or listening, but may not have done physical A I with open models and just kind of lay out all the concerns that they have to start thinking about there. I think that would be really useful, um, but you know, just as a, as a, as a layout there.

• 17:21Speaker 03

  Hmm, okay. So basically, it's been uh the difference in physical as compared to the uh the digital A I that people are very familiar with. Okay.

• 17:32Speaker 02

  Yeah.

• 17:32Speaker 03

  Uh, okay. So uh, let's say you have a robot uh and trying to complete certain task, right? And robot need to react very fast, right? So the the processing had to be done real time uh on spot, right? Uh, and also, uh, a lot of task is a batch size one inference problem. So A P I, you can aggregate users, uh, A P I code from, you know, from different users and then process then in a whole, right? So when you do that, um, you know, maybe I can go back a little bit uh uh a bit, right? So uh L M is a very popular architecture, uh, you know, for coding, for chat, right?

• 18:13Speaker 03

  And L M is based on the auto-regressive transformer, so it's a memory bound. And uh to more efficient, to use the G P U power more efficiently, you can aggregate the A P I code from different user to to fill in the compute, right? But the same architecture if you put uh in this uh uh embedded device, right? Uh, you don't have A B I code to aggregate, right? So you are not fully utilize the compute power in in a machine. So for some other architecture might be more efficient, right? So I'm talking about because the application is different, right? Uh, so it it it require different architecture uh and different thinking uh so that you can use your uh power,

• 18:58Speaker 03

  your watt more efficiently. And and also there's a real-time constraint and safety constraint. So uh, so because of the setup difference, uh, the uh it it gonna promote uh different architectures.

• 19:11Daniel Whitenack?

  And maybe that leads naturally into talking about some of the architecture. Well, there there's all sorts of architectures that Nvidia has been working on for for quite a while. Obviously, you're in the Cosmos lab. I I know one of the things that you're working on is quote-unquote world models. Um, you know, uh Cosmos meaning uh uh world as people might uh might um infer. But uh help, help those who maybe, maybe imagine someone who has just used L L M's or maybe that's their only perception of of models.

• 19:52Daniel Whitenack?

  Help connect what a world model is and why it might be necessary in this world of of physical A I.

• 19:59Speaker 03

  Hmm, okay. So uh, language model um is a model uh that's a learn great internal representation uh through processing human knowledge written in the form of language, right? And and it's optimized uh for this uh language representation to generate languages for chat, for code. You know, code is written as a language as well. Uh, and world model is found a different perspective. It is from the perspective of uh, you know, you you have a sensory observation of the world. And and you know, those sensors are are capturing uh different uh uh angles of the world, right?

• 20:43Speaker 03

  And and those sensor, those camera also uh capture the dynamics, how physics evolve. And people believe that uh uh from modeling these signals, you you can in, you know, implicitly encode the physics uh of the world. And with the physics kind of something you model, then you can start to do simulation. And if you uh connect this world representation with language presentation, then you can generate explanation that's help you understand why this thing happened. So it's more like one is coming from a language perspective. The other is more like a uh from the observation of the world of physics and try to represent it.

• 21:26Speaker 03

  So different way of learning the representations and of course they have different use cases.

• 21:31Daniel Whitenack?

  Yeah, and and I guess language.

• 21:35Daniel Whitenack?

  Is a represent or it represents part of the knowledge or how people experience the world, but it's only a small piece of how people experience the world, right? Like I'm, I'm looking at you with my eyes, I'm, I'm hearing things. I, as you mentioned, I see things from different perspective. I see how physics is evolving. Is, is it true? So, so in an example world model, to just make it tangible for people. In an LLM, you would basically put in text and get out text, or a next, at least the next predicted token, right? Or in a image generation model, you'd put in text and then get out, get out an image.

• 22:17Daniel Whitenack?

  In the world model case, what, what is an example sort of input and output? One of those might be, as you mentioned, maybe it's just a representation of the world, and then you build additional things on top of that. But maybe help people get some intuition about what do you put into a world model? And what do you get out when you're actually operating the world model? Or performing inference or performing prediction with the world model?

• 22:44Speaker 03

  Yes, so I want to touch a point before answering this question. So people build a model for a purpose, right? And we did in the world, so we are initially building some model of the world, so we can do a lot of different tasks, right? One is predict how it can happen, and some is to explain what happened, right? Or sometimes you will need to do navigation, so you have a 3D representation of the world. So some people call it world model. To me, world model is a way to solve some tasks, right? And so it's difficult to, to me, difficult to use like the input, output

• 23:26Speaker 03

  to kind of define what is a world model. So I would like to say some sort of modeling of the world is a world model, okay? And from my perspective, because our mission is to build physical, the mission of Cosmos is to build a, you know, physical foundation model to help the whole ecosystem. So we realized that, you know, in the early day, that you need to have world simulation capability. Given the current observation and that there's, you know, the action input, what can happen in the future? For this world simulation capability, the input is the current observation and the action sequence. Sometimes the action can be describing tasks, right?

• 24:11Speaker 03

  And the output is the, the future, that what can happen? Maybe in video, maybe in audio, in the future when you have other modality, you know, other modality, so world simulation. And the other thing is the world understanding, right? So, so you want to understand, you know, why this, you know, you ask a robot to complete a task and it didn't complete, right? You want to understand what happened, right? So there's an obstacle blocked, blocked the pathway of the robot on, so it couldn't, right? So the, when the situation like this happen, you find the sensory input, right? You kind of have a modeling what happened, right?

• 24:56Speaker 03

  But in order to, for people, you know, the receiver and the question asked is a human, right? So you need to connect those world representation to language. So it can generate some description that people understand. Yeah, so, and so this is another side of a world model understanding. So it's more similar to multimodal LLM. So you have the video and you have the description and you have the test as outputs. And in Cosmos, we believe that, you know, one is for world understanding, the other is for world simulation, right? Although they for different purposes, but they model the same world. So, so in Cosmos 3, our latest Cosmos model, we actually fused them together.

• 25:40Speaker 03

  Because we believe that the representation can be shared, because, you know, we live in the same world, right? It's the same stuff, yeah. So, yeah, and for the world understanding, the, the input is video, text, output is text. And for world simulation, the input is, you know, observation, action, output is test, output is video, sorry. And for me, put together, it becomes an Omni model, right? You have different inputs. You know, in Cosmos, we actually support text, action, audio, video as the input. And also supports task, audio, video, action as outputs. So you can find all combination, and I think a developer can choose the best way

• 26:24Speaker 03

  they, they want to build on top of it.

• 26:27Daniel Whitenack?

  That was really good. I'd like to ask a clarifying question, because there's a couple of terms out there that I'd like to, I'd like to get your, kind of your definitions and how they relate a little bit. One is obviously this notion of world model, which you've been diving into. But a number of times, obviously, you've mentioned simulation. And for, for those who, who may not be familiar with this, like how, how do you define or relate the notion of simulation? You know, where you could either have a more of a simulator, like what people may be already familiar with, which also is obviously in gaming and things like that, with the notion of the world model, as you've just explained it? How do those two terms relate to each other?

• 27:11Speaker 03

  Okay, good question. So, so we have, you know, people are spending, you know, centuries of years studying how the world work. And they sometimes put that in equations, right? And we have, you know, simulated environment governed by program built on top of those equation, you know, how, you know, when two things, you know, hit each other, how thing gonna happen? Those are, we consider classical simulators, right? So they are based on physics, the rule, you know, people's interpretation, equations, and then to govern how things.

• 27:55Speaker 03

  The thing I described earlier, people generally call it neural simulation. So it's a data-driven way of doing simulation. Instead of directly put the physics in, you saw a tons of observation, you know, showing different, you know, how two things hit together, they, they kind of separated, or, you know, when you pour water to the ground, how water spread out. It show a lot of different scenes, you know, visual observation to a model. And the model then later can approximate, you know, predict what can happen when they see similar pattern.

• 28:36Speaker 03

  So it's more like a pattern recognition based way of doing the simulation, right? So one is a rule-based, the other is more like a data-driven approximation.

• 28:48Daniel Whitenack?

  And I'm, I'm coming from a perspective that I'm not an expert like you or Chris in this area of, of physical AI and world models. And I'm wondering in my mind, maybe as a non-expert, you're talking about world models, you're talking about simulation, et cetera. If we go all the way back to kind of one of those concrete examples, like the robot or the manufacturing case or the automotive case, where does the world model come into play? As you're creating the models that are actually, you know, driving decision-making or movement or actions in the, in the robot in the physical world? Where does that fit in versus some other things?

• 29:31Daniel Whitenack?

  Like you mentioned, policy, you know, models, et cetera?

• 29:35Speaker 03

  Okay, it's a good question. So in Cosmos, we build this foundation model and try to learn a good world representation, right? And this model can be used in many different ways. From the simulation perspective, okay? Let's say I'm building a self-driving car policy, right? A policy that can navigate, navigate to drive the car from destination A to B, right? So as developing the model, I might have many different checkpoints, right? And how do I know that checkpoint A is better than checkpoint B? In the old day, what people do is that they have a fleet of cars and driver.

• 30:18Speaker 03

  Whenever I have a new policy and we deploy the policy to the car, and that the driver kind of, you know, take the car to the field and see, you know, record how good is the policy model? And you can clearly see this is, you know, not scalable, right? It's limited how many iteration you can do, right? But with a world model, right? Instead of having your policy, you know, deploy the real car, you know, driving the real world, you can have your policy interact with the world model, right? When you still left, what gonna see, still right, what you gonna see, right? And from this simulation, you can verify the accuracy of a checkpoint.

• 31:02Speaker 03

  It can help you quickly narrow down a couple hypotheses you have. And then use, then you have few one that you think is truly valuable, and you do the real test. It help you to get this iteration faster. And now we know in technology, the most important thing is how far you can do iteration, right? Then, then you can better develop better policy for your self-driving car. It can be the same for, for robots, right? And, and this is the world simulation. The representation allow you to do simulation, what it can do for physical AI. The other thing is that, you know, this can be also a great backbone for a policy model, right?

• 31:44Speaker 03

  In a world model, it has world representation, it, it's understand, you know, it's able to understand the instruction given by human, right? And it's, when you model the world, right? The, it generate the piece of space evolution, right? And the piece of space evolution has a strong correlation to the, to the control signal you might need to use to complete certain manipulation tasks. And that's correlation become a great regularization to help you build a better policy model when the data is more limited. So I think world representation is the key. And, and the things physical AI gonna operate in the real world.

• 32:29Speaker 03

  And this world representation encapsulated in this foundation model gonna be useful in many different way. And it's also kind of important, we need to make it open. So that's, you know, people can, can, can test out and share feedback with us. And we can then double down on the capability that's more important to help the physical AI ecosystem.

• 32:53Daniel Whitenack?

  Wow, it's just amazing to hear Ming Yu talk about Nvidia's effort to make sure that there is a community of open source enthusiasts, those that are working on open models, to foster a real community of innovation around AI and agents. And that is so practical. That's what we're about here. Practical AI is that sort of practical value. And we share that with an amazing event that's happening October 15th, just here in a couple of weeks, October 15th, 2026, called the Midwest AI Summit. We're one of the media sponsors of this event, and I'm personally going to be an emcee at the event.

• 33:36Daniel Whitenack?

  And it's just filled with amazing practical value. There's an AI engineering lounge where you can sit one-on-one with AI experts that actually can practically provide amazing feedback on your roadmap, toolset, architecture, et cetera. There's gonna be amazing talks from leaders in the field from all over, talking about why they've made certain decisions in their AI roadmap, how you can deploy agents off of your laptop into cloud environments, so much more, so much practicality. Plus, there's free food and other amazing things. This isn't one to miss, so check it out, Midwestaisummit.com.

• 34:15Daniel Whitenack?

  And you can use the code PRACTICALAI20 to get 20% off registration. Do not miss this. Register today. Use the code PRACTICALAI20 to get 20% off registration for the Midwest AI Summit. Find out more at Midwestaisummit.com and use the code PRACTICALAI20 for 20% off registration.

• 34:42Speaker 02

  So I'd like to, I'd like to kind of maybe expand, might be the right word, the conversation into another dimension. And that's the fact that by contrast, in kind of the cloud digital world, people are often used to dealing with an LLM either by itself, or it's managing a, you know, a collection of agents that are doing tasks and stuff. But that interface is very much, there's a human and there's that LLM, and they are, you know, agreeing on what's gonna happen with, with the AI agents going and doing things. When you move things into the physical realm, you add the dimension of, you may have all

• 35:24Speaker 02

  of those things happening, but you also have many physical devices that are beginning to interact together, and they're interacting with humans in different ways. And so that the notion of what interactions are gets quite complex. And I was wondering if you could kind of walk us through that a bit, and maybe kind of take us into that, that new dimension of, we're not talking about one robot now, but now, or, you know, or one self-driving car, or whatever you wanna address, but maybe a whole bunch of them. And maybe you have a city full of them, and that's what is mostly there, or, and they're in all the domains, everything, you know, from the ground all the way up and all the

• 36:08Speaker 02

  way down, maybe undersea and stuff like that. We may have robots eventually in our bodies, you know, nanobots that are, that are doing. So like, could you kind of lay out what this, what this implies, and what some of the extra considerations are that you have to start thinking about to set this new reality up?

• 36:28Speaker 03

  Good question. So I've been asking myself, what the future gonna look like, right? So I believe, let me take one example, right? So let's say I, I'm a manufacturer. I build some very cool toy called robot dock, okay? And I got an order of, you know, people want a minion robot dock, right? And I might have a digital agent powered by LN, help me understand, look at my inventory, help me understand what the materials I have in different places, right? So there's an agent, LN agent, to kind of do some high-level planning, right?

• 37:11Speaker 03

  And now I know that I need to ship certain material from factory B to factory A, so that I can manufacture the robot dock in factory A. And then that agent send, you know, demand to, to the, maybe the factory floor agent in factory B, and factory B do a hand, easy handshake, right? And that factory four gonna have some robots, some mobile platform, right? And the robot might carry the good to put the mobile platform, and mobile platform put, you know, load in the truck, and the truck go from a self-driving truck, go from factory B to factory A, right? And then we start to do the manufacturing.

• 37:56Speaker 03

  And every station in, you know, in assemble the stuff, they're gonna have visual inspection. They're gonna have a robot arm, right? There'll be some kind of code group, and those gonna be operated by another agent, right? So I see the world gonna be multi-agent, right? And also go back to my previous viewpoints, right? So in the end, the intelligence per dollar, you know, matters, right? And the different tasks, you know, you probably don't need a robot arm to solve, be capable of solving, solve Olympia mass problem, right? You probably want it to be more sensitive in some kind of manipulation task. So I think we are looking at war with the multi-defensive specialized agent, and they're

• 38:40Speaker 03

  gonna work together and to complete certain task. And drone, you know, can be one of them, right? So I might be owning a farm, let's say, produce tea, right? And I can have drone to collect the tea leaf, and then put that in a package, you know? So, so all of them, so I think we like an agent, because, you know, agent is somebody has certain autonomy to behave, to, to, to complete task on behalf of you, right? It requires that agent have some self-curation capability, right? That they probably like. And also, you kind of puts all the problem to agent, and it figures out how you want to solve that.

• 39:25Speaker 03

  And so we're gonna have specialized agent, and there's some agent good at distributing tasks, and some agent good to complete task, and they're gonna talk together, work together to complete the stuff. Yeah, and that's how, I think, that's what I think gonna happen. Yeah. This is my view, by the way.

• 39:42Daniel Whitenack?

  Yeah, I, I love that perspective. It really helps bring a concrete visualization to my mind of what this may look like. And it, but it does, maybe to some people out there, it also triggers some, well, there's obviously challenges, right, to get there. I almost, I think about, about AI agents every day in my day-to-day work, not physical AI, but there's this idea of cascading failure of AI agents, right? Which is this sort of idea, oh, like something small happens over here, but the, all the agents are interacting, and there's this sort of system-wide failure.

• 40:26Daniel Whitenack?

  Maybe, you know, obviously if things are interconnected, things can happen, but from your perspective, maybe it's that, maybe it's something else. What do you see as the challenges that, you know, you mentioned, we're working towards these innovations, right? You are working towards these, Nvidia, the industry. What are some of the main challenges in your mind, from getting kind of from where we're at now to something that looks more like that? What are some of the categories of those challenges in your, in your mind?

• 40:58Speaker 03

  Yeah, so I think there are a ton. And, and the picture I just gave, you know, multiple agent working together to complete the, like, production task, might be decades away, right? So we are not yet there. And verify the correctness is super important, right? So like what you said, so I envision they're gonna be also supervisor agents working in the factory floor, and trying to capture things that not yet been identified, right? And also, you know, building this physical also requires strong, you know, robots, right?

• 41:40Speaker 03

  And I know that the humanoid robots, you know, they evolve very fast, but I think they are not yet kind of ready for major commercial use, right? And also, the, you know, to achieve full automation, we need our robot be smarter, even the robot arm smarter and more, I would say more configurable, so that you can have a new task, and maybe just read the user menu, and the robot arm figure out how to assemble this, right? Those are not yet there. So I think there are a lot of problems need to figure out, and I think it's difficult to, for one company to do it alone. I think we need a whole ecosystem.

• 42:25Speaker 03

  And that's why, you know, I think building this open model is super important, right? Because you enable more people, the tool, to do the exploration, to do the innovation. And I think, you know, to achieve that beautiful future, open model gonna play a critical role.

• 42:44Daniel Whitenack?

  And on that front, I guess, you know, drawing Somewhat closer to an end here. Nvidia is pushing forward this narrative about open models, right? And I'm very personally thankful for that. And you've emphasized that here. How can, maybe there's software engineers, developers, AI practitioners, listening to this podcast? How might they even get started in this, in this area? Find some of the resources, the open resources that Nvidia is putting out there? Any suggestions or tips for people that may want to try to start getting into this, this innovation?

• 43:28Daniel Whitenack?

  You know, in terms of a starting point? And of course, I'll tell our listeners too, we'll include a bunch of show notes as well in, in our episode where you can click and find some of these things. But any thoughts there, Mingyu?

• 43:41Speaker 03

  Yes, so, so maybe I'll also use this opportunity to define, you know, you know, what Nvidia, you know, what open model mean for Nvidia? So, open model is not just have the model, open weight allow you to use. We also provide the training framework so that you can take this open model and your own data and push train to something more tailored for your use case. We even open source data that's recreated, you know, to help you to, to use those data in you, if you want to build your own open model.

• 44:25Speaker 03

  And we also have examples, right? Nvidia is a company that work with all the AI companies. And we also work with all the physical AI companies. And, and people are very willing to share some of their, you know, experience or, you know, some success use case, you know? And we open put them in, in the form of developer blog or, you know, example recipe and the different places so that people can look at those useful examples and then, and to, to start. And nowadays, I think most of people doing research using L N.

• 45:06Speaker 03

  Maybe you can use your favorite agent, no matter it's Gemini, G P T or a cow. As then to look around and find useful materials. And I think in the new world, everybody want to make sure the material is easily accessible to those agent. So they can better convert them to certain form to educate the human developer. Yeah, so, yeah, so, you know, pretty much we put, you know, open models in hacking phase and all the core resources in GitHub. And we have a developer box to talk about how we use this. And in the GitHub, we also have a recipe cookbook to help developer to, to learn from how other

• 45:51Speaker 03

  use, you know, open models. And what I say is true for Cosmo is also true for Nimochuan, for Good and other open models after by Nvidia.

• 46:03Daniel Whitenack?

  That's great. And yeah, from the, you know, just a member of the community, the community, thank you and Nvidia for, for the effort that you're putting into. It's, it's no small effort to maintain all of those things and, and all of those places. So it's much appreciated and this has been a great conversation, Mingyu. Really appreciate you taking time to walk us through all of these topics and of course, the work that you're doing and hope to have you back on the show. Thank you so much for your time.

• 46:33Speaker 03

  Thank you. Thank you. Thank you for having me here and it was a lot of fun.

• 46:42Speaker 00

  All right. That's our show for this week. If you haven't checked out our website, head to practicalai.fm and be sure to connect with us on LinkedIn, X or Blue Sky. You'll see us posting insights related to the latest AI developments and we would love for you to join the conversation. Thanks to our partner Prediction Guard for providing operational support for the show. Check them out at predictionguard.com. Also, thanks to Breakmaster Cylinder for the beats and to you for listening. That's all for now, but you'll hear from us again next week.