Pod Insights Training Data Jim Fan on Nvidia’s Embodied AI Lab and Jensen Huang’s Prediction that All Robots will be Autonomous About this episode AI researcher Jim Fan has had a charmed career. He was OpenAI’s first intern before he did his PhD at Stanford with “godmother of AI,” Fei-Fei Li. He graduated into a research scientist position at Nvidia and now leads its Embodied AI “GEAR” group. The lab’s current work spans foundation models for humanoid robots to agents for virtual worlds.
Jim describes a three-pronged data strategy for robotics, combining internet-scale data, simulation data and real world robot data. He believes that in the next few years it will be possible to create a “foundation agent” that can generalize across skills, embodiments and realities—both physical and virtual. He also supports Jensen Huang’s idea that “Everything that moves will eventually be autonomous.”
Hosted by: Stephanie Zhan and Sonya Huang, Sequoia Capital
Mentioned in this episode:
World of Bits: Early OpenAI project Jim worked on as an intern with Andrej Karpathy. Part of a bigger initiative called Universe
Fei-Fei Li: Jim’s PhD advisor at Stanford who founded the ImageNet project in 2010 that revolutionized the field of visual recognition, led the Stanford Vision Lab and just launched her own AI startup, World Labs
Project GR00T: Nvidia’s “moonshot effort” at a robotic foundation model, premiered at this year’s GTC
Thinking Fast and Slow: Influential book by Daniel Kahneman that popularized some of his teaching from behavioral economics
Jetson Orin chip: The dedicated series of edge computing chips Nvidia is developing to power Project GR00T
Eureka: Project by Jim’s team that trained a five finger robot hand to do pen spinning
MineDojo: A project Jim did when he first got to Nvidia that developed a platform for general purpose agents in the game of Minecraft. Won NeurIPS 2022 Outstanding Paper Award
ADI: artificial dog intelligence
Mamba: Selective State Space Models, an alternative architecture to Transformers that Jim is interested in (original paper here)
00:00 Introduction
01:35 Jim’s journey to embodied intelligence
04:53 The GEAR Group
07:32 Three kinds of data for robotics
10:32 A GPT-3 moment for robotics
16:05 Choosing the humanoid robot form factor
19:37 Specialized generalists
21:59 GR00T gets its own chip
23:35 Eureka and Issac Sim
25:23 Why now for robotics?
28:53 Exploring virtual worlds
36:28 Implications for games
39:13 Is the virtual world in service of the physical world?
42:10 Alternative architectures to Transformers
44:15 Lightning round
Transcript 0:00 Speaker 00 So from the chip level, which is the Jensen Thore family, to the foundation model, Project Groot, and also to the simulation and the utilities that we build along the way, it will become a platform, a computing platform, for humanoid robots and then also for intelligent robots in general. So I want to quote Jensen here. One of my favorite quotes from him is that everything that moves will eventually be autonomous. And I believe in that as well. It's not right now, but let's say 10 years or more from now. If we believe that there will be as many intelligent robots as iPhones, then we'd better start building that today.
0:58 Speaker 01 Hi and welcome to Training Data. We have with us today Jim Phan, Senior Research Scientist at Nvidia. Jim leads Nvidia's Embodied AI Agent Research with a dual mandate spanning robotics in the physical world and gameplay agents in the virtual world. Jim's group is responsible for Project Groot, Nvidia's humanoid robots that you may have seen on stage with Jensen at this year's GTC. We're excited to ask Jim about all things robotics. Why now? Why humanoids? And what's required to unlock a GPT-3 moment for robotics?
1:30 Speaker 01 Welcome to Training Data.
1:32 Speaker 00 Thank you for having me.
1:33 Speaker 01 We're so excited to dig in today and learn about everything you have to share with us around robotics and embodied AI. Before we get there, you have a fascinating personal story. I think you were the first intern at OpenAI. Maybe walk us through some of your personal story and how you got to where you are.
1:50 Speaker 00 Absolutely, I would love to share the stories with the audience. So back in the summer of 2016, some of my friends said there is a new startup in town and you should check it out. And I'm like, huh, I don't have anything else to do because I got accepted to PhD. And that summer I was idle, so I decided to join this startup and that turned out to be OpenAI. And during my time at OpenAI, we were already talking about AGI back in 2016. And back then, my intern mentor was Andrej Karpathy and Ilya Sutskever. And we talk about and we discuss a project together, it's called World of Bits. So the idea is very simple. We want to build an AI agent that can read computer screens,
2:36 Speaker 00 read the pixels from the screens and then control the keyboard and mouse. If you think about it, this interface is as general as it can get, right? Like all the things that we do on computer, like replying to emails or playing games or browsing the web, it can all be done in this interface, mapping pixels to keyboard and mouse control. So that was actually my first kind of attempt at AGI at OpenAI and also my first journey, the first chapter of my journey in AI agents.
3:06 Speaker 02 I remember World of Bits, actually, I didn't know that you were a part of that. That's really interesting.
3:10 Speaker 00 Yeah, yeah, it was a very fun project and was part of a bigger initiative called OpenAI Universe, which was like a bigger platform on like integrating all the applications and games into this framework.
3:24 Speaker 02 What do you think were some of the unlocks then? And then also, what do you think were some of the challenges that you had with agents back then?
3:29 Speaker 00 Yes, so back then, the main method that we used was reinforcement learning. There was no LLM, no transformer back in 2016. And the thing is reinforcement learning, it works on specific tasks, but it doesn't generalize. Like we can't give the agent arbitrary language and instruct it to do things, to do arbitrary things that we can do with a keyboard and mouse. So back then, it kind of worked on the tasks that we designed, but it doesn't really generalize. So. You know, that started like my next chapter, which is I went to Stanford and I started my PhD with Professor Fei-Fei Li. And we started working on computer vision and also embodied AI. And during my time at Stanford, which was from 2016 to 2021,
4:13 Speaker 00 I kind of witnessed the transition of the Stanford Vision Lab led by Fei-Fei from, you know, static computer vision, like recognizing images and videos, to more embodied computer vision, where an agent learns perception and takes actions in an interactive environment. And this environment can be virtual, as in simulation, or it can be the physical world. So that was my PhD, like transitioning to embodied AI. And then after I graduated from PhD, I joined Nvidia and have stayed there ever since. So I carried over my work from my PhD thesis to Nvidia and still work on embodied AI till this day.
4:53 Speaker 01 So you oversee the embodied AI initiative at Nvidia. Maybe say a word on what that means and what you all are hoping to accomplish.
5:00 Speaker 00 Yes, so the team that I am co-leading right now is called GEAR, which is G E A R, and that stands for Journalist Embodied Agent Research. And to summarize what our team works on in three words is that we generate actions. Because we build embodied AI agents and those agents take actions in different worlds. And if the actions are taken in the virtual world, that will be gaming AI and simulation. And if the actions are taken in the physical world, that will be robotics. Actually, earlier this year in March, GTC at Jensen's keynote, he unveiled something called Project Groot, which is Nvidia's moonshot effort at building foundation models for humanoid robotics.
5:45 Speaker 00 And that's basically what the GEAR team is focusing on right now. We want to build the AI brain for humanoid robots and even beyond.
5:53 Speaker 01 What do you think is Nvidia's competitive advantage in building that?
5:56 Speaker 00 Yeah, that's a great question. So, well, one is for sure like compute resources. All of these foundation models require a lot of compute to scale up. And we do believe in scaling law. There were scaling laws for like LOMs. But the scaling law for embodied AI and robotics are yet to be studied. So we're working on that. And the second strength of Nvidia is actually simulation. So Nvidia, before it was an AI company, it was a graphics company. So Nvidia has many years of expertise on building simulation, like physics simulation, and rendering, and also real-time acceleration on GPUs. So we are using simulation heavily in our approach to build robotics.
6:39 Speaker 01 The simulation strategy is super interesting. Why do you think most of the industry is still very focused on real-world data? The opposite strategy.
6:46 Speaker 00 Yeah, I think we need all kinds of data. And simulation and real-world data by themselves are not enough. So at GEAR, we divide this data strategy into roughly three buckets. One is the internet-scale data, like all the texts and videos online. And the second is simulation data, where we use Nvidia simulation tools to generate lots of synthetic data. And the third is the real robot data, where we collect the data by teleoperating the robot and then just collecting and recording those data on the robot platforms. And I believe a successful robotic strategy will involve the effective use of all three kinds of data and mixing them and also delivering a unified solution.
7:31 Speaker 01 Can you say more about, we were talking earlier about, you know, how data is fundamentally the key bottleneck in making a robotics foundation model actually work. Can you say more about your kind of conviction in that idea? And then like, what exactly does it take to make, you know, great data to kind of break through this problem?
7:50 Speaker 00 Yes, so um I think like the three different kinds of data that I just mentioned have different strengths and weaknesses. So for internet data, they're the most diverse. They encode a lot of common sense prior, right? Like, for example, um all the, most of the videos online are human-centered. Uh, because humans, uh, we, we love to take selfies, we love to record each other doing all kinds of activities. And there are also a lot of instructional videos online. So we can use that to kind of learn how humans interact with objects and how objects behave under different uh situations. So that kind of provides a common sense prior for the robot foundation model. Um, but the internet-scale data, they don't come with actions.
8:33 Speaker 00 We cannot download the motor control signals of the robots from the internet. And that goes to the second part of the data strategy, which is using simulation. So in simulation, uh, you can have all the actions and you can also observe the consequences of the actions in that particular environment. Um, and the strength of simulation is that it's basically infinite data. And um, you know, the data scales with uh compute. The more GPUs you put into the simulation pipeline, the more data that you will get. And also, the data is super real-time. So if you collect data only on the real robot, then you are limited by twenty-four hours per day.
9:11 Speaker 00 But uh, in simulation, like the GPU-accelerated uh simulators, we can actually accelerate real-time by ten thousand x. So we can collect the data at much higher throughput, given the same workload time. So that's a strength. But the weakness is that uh for simulation, uh, no matter how good the graphics pipeline is, there will always be uh this simulation-to-reality gap. Like, the physics will be different from the real world, and the, the visuals will still be different. They will not look uh exactly as realistic as the real world. And also, there is a diversity issue. Like, the contents in the simulation uh will not be as diverse as all the scenarios that we encounter in the real world. So these are the, are the weaknesses.
9:55 Speaker 00 And then going to the real robot data. And those data, they don't have the sim to real gap because they're collected on the real robot. But it's much more expensive to collect because you need to hire people to operate the robots. And again, they're limited by the speed of the world of atoms. You only have 24 hours per day and you need humans to to collect those data, which is also very expensive. So we see these three types of data as having complementary strengths. And I think a successful strategy is to combine their strengths and then, you know, to remove their weaknesses.
10:32 Speaker 01 So the cute Groot robots that were on stage with Jensen, that was such a cool moment.
10:40 Speaker 01 If you had to help us dream in one, five, 10 years, like what do you think your group will have accomplished?
10:46 Speaker 00 Yeah, so this is pure speculation. But I hope that we can see a research breakthrough in robot foundation model, maybe in the next two to three years. So that's what we call like a GPT three moment for robotics. And then after that, it's a bit uncertain because to have the robots enter like daily lives of people. There are a lot more things than just the technical side. The robots need to be affordable and mass produced. And we also need like safety for the hardware and also privacy and regulations. And those will take longer for the robots to be able to hit a mass market. So that's a bit harder to predict, but I do hope that the research breakthrough will come in the next two to three years.
11:29 Speaker 01 What do you think will define what a GPT three moment in AI robotics looks like?
11:33 Speaker 00 Yeah, that's a great question. So I would like to think about robotics as consisting of two systems, system one and system two. So that comes from the book Thinking Fast and Slow, where system one means this low level motor control. That's unconscious and fast, like for example, when I'm grasping this cup of water. I don't really think about how I move the fingertip at every millisecond. So that would be system one. And then system two is slow and deliberate. And it's more like reasoning and planning that actually uses the conscious brain power that we have. So I think the G P T three moment will be on the system one side.
12:17 Speaker 00 And my favorite example is the verb open. So just think about the complexity of the word open, right? Like, opening the door is different from opening a window. It's also different from opening a bottle or opening your phone. But for humans, we have no trouble understanding that open means different things when you're interacting. It means different motions when you're interacting with different objects. But so far, we have not seen a robotics model that can generalize on a low-level motor control on these verbs. So I hope to see a model that can understand these verbs in their abstract sense and can generalize to all kinds of scenarios that make sense to humans. And we haven't seen that yet, but I'm hopeful that this moment could come in the next two to three years.
13:01 Speaker 01 What about system two thinking? Like, how do you think we get there? Do you think that some of, you know, the reasoning efforts in the L L M world will be relevant as well in the robotics world?
13:13 Speaker 00 Yeah, absolutely. I think for system two, we have already seen very strong models that can do reasoning and planning and also coding as well. So these are the L L M's and the frontier models that we have already seen these days. But to integrate the system two models with system one is another research challenge in itself. So the question is, for robot foundation model, do we have a single monolithic model? Or do we have some kind of cascaded approach? Where the system two and the system one models are separate and can communicate with each other in some ways? I think that's an open question. And again, they have pros and cons, right? Like, for the first idea, the monolithic model is cleaner. There's just one model, one A P I to maintain.
13:59 Speaker 00 But also, it's a bit harder to kind of control because you have different control frequencies. Like, the system two models will operate on a slower control frequency. Let's say one hertz, like one decision per second. While the system one, like the motor control of me grasping this cup of water, that will likely be a thousand hertz, where I need to make these. Minor like these tiny muscle decisions at a thousand times per second. It's really hard to encode them both in a single model. So maybe a cascaded approach will be better. But again, how do we communicate between system one and two? Do they communicate through text or through some latent variables? It's unclear, and I think it's a very exciting new research direction.
14:41 Speaker 02 Is your instinct that we'll get there in that breakthrough on system one thinking? Like through scale and transformers, like what is it's gonna work? Or is that, you know, cross your fingers and hope and see?
14:53 Speaker 00 I certainly hope that the data strategy I described will kind of get us there. Because I feel that we have not pushed the limit of transformers yet. On the essential level, like transformers take tokens in and outputs tokens. And ultimately, the quality of the tokens determines the quality of the model, the quality of those large transformers. And for robotics, as I as I mentioned, like the data strategy is very complex. We have all the internet data and also we need simulation data and the real robot data. And once we're able to scale up on the data pipeline with all those high quality actions, then we can tokenize them and we can send them to a transformer to compress.
15:36 Speaker 00 So I feel we have not pushed transformer to the limit yet. And once we figure out the data strategy, we may be able to see some emergent property as we scale up the data and scale up the model size. And for that, I'm calling it the scaling law for embodied AI, and it's just getting started.
15:52 Speaker 02 I'm very optimistic that we will get there. I'm curious to hear what are you most excited about personally? When we do get there, what's the industry or application or use case that you're really excited to see this completely transform the world of robotics today?
16:05 Speaker 00 Yes, so there are actually a few reasons that we chose humanoid robots as kind of the main research thesis to tackle. One reason is the world is built around the human embodiment, the human form factor. Right? All our restaurants, factories, hospitals, and all our equipments and tools, they're designed for the human form and also the the human hands. So in principle, a sufficiently good humanoid hardware should be able to support any task that a reasonable human can do in principle. And humanoid hardware is not there yet today. But I feel in the next two to three years, the humanoid hardware ecosystem will mature.
16:46 Speaker 02 Yeah.
16:46 Speaker 00 And we'll have affordable humanoid hardware to work on.
16:49 Speaker 02 Yeah.
16:50 Speaker 00 And then it will be a problem about the AI brain, about how we kind of drive those humanoid hardware. And once we have that, once we're able to have the group foundation model that can take any instruction in language and then perform any task that a reasonable human can do, then we unlock a lot of economic value. Like, we can have robots in our households, helping us with daily chores, like laundry, dishwashing and cooking, or like elderly care. And we also have them in restaurants, in hospitals, in factories, helping with all the tasks that humans do. And I hope that will come in the next decade. But again, as I mentioned in the beginning, this is not just a technical problem,
17:33 Speaker 00 but also there are many things beyond the technology. So I'm looking forward to that.
17:39 Speaker 02 Any other reasons you've chosen to go after humanoid robots specifically?
17:42 Speaker 00 Yeah, so there are also a bit more practical reasons in terms of the training pipeline. So there are lots of data online about humans, right? It's all like human-centered. All the videos are like humans doing daily tasks or having fun. And the humanoid robot form factor is closest to the human form factor, which means that the model that we train using all of those data will be able to have an easier time to transfer to the humanoid form factor rather than the other form factors. So let's say for robot arms, right? Like, how many videos do we see online about robot arms and grippers? Very few.
18:22 Speaker 00 But there are many videos of people using their five-fingered hands, right, to work with objects. So it might be easier to train for humanoid robots. And then, once we have that, we'll be able to specialize them to like the robot arms and more kind of specific robot forms. So that's why we're aiming for the full generality first.
18:45 Speaker 02 I didn't realize. So are you exclusively training on humanoids today versus robot arms and robot dogs as well?
18:51 Speaker 00 Yeah, so for project group.
18:53 Speaker 01 In simulation, yeah.
18:54 Speaker 00 Yes, so for project group, we are aiming more towards humanoid right now. But the pipeline that we're building, including the simulation tools, right, the real robot tools, like those are general purpose enough that we can also adapt to other platforms in the future. So yeah, we're building these tools to be generally applicable.
19:13 Speaker 01 You've used the term general quite a few times now. I think there are some folks, especially from the robotics world, who think that, you know, a general approach won't work and you have to be domain, environment specific. Why have you chosen to go after a generalist approach? And, you know, the Richard Sutton bitter lesson stuff has been a recurring theme on our podcast. I'm curious if you think it holds in robotics as well.
19:37 Speaker 00 Absolutely. So I would like to first talk about the success story in NLP that we have all seen, right? So before the chat GPT and the GPT three, in the world of NLP, there are a lot of kind of different models and pipelines for different applications, like translation and coding and, you know, doing math and doing like creative writing. Like they all use very different models and completely different training pipelines. But then chat GPT came and unified everything into a single model. So before chat GPT, we call those specialists. And then the GPT three and chat GPTs, we call them the journalists.
20:21 Speaker 00 And once we have the journalists, we can prompt them, distill them and fine tune them back to the specialized tasks. And we call those the specialized journalists. And according to the historical trend, it's almost always the case that the specialized journalists are just far stronger than the original specialists. And they're also much easier to maintain because you have a single API that takes text in and then spits text out. So I think we can follow the same success story from the world of NLP. And it will be the same for robotics. So right now in 2024, most of the robotics and applications we have seen are still in the specialist stage.
21:01 Speaker 00 Right? They have specific robot hardware for specific tasks, collecting specific data using specific pipelines. But Project Groot aims to build this general purpose foundation model that works on humanoid first, but later will generalize to all kinds of different robot forms or embodiments. And that will be the journalist moment that we are pursuing. And then, once we have that journalist, we'll be able to prompt it, fine tune it, distill it down to specific robotics tasks. And those are the specialized journalists. But that will only happen after we have the journalist. So it will be easier in the short run to pursue the specialist.
21:43 Speaker 00 It's just easier to show results because you can just focus on a very narrow set of tasks. But we at Nvidia believe that the future belongs to journalists. Even though it will take longer to develop, it will have more difficult research problems to solve. But that's what we're aiming for first.
21:59 Speaker 02 The interesting thing about Nvidia building Groot to me is also what you mentioned earlier, which is that Nvidia owns both the chip and the model itself. What do you think are some of the interesting things that Nvidia could do to optimize Groot on its own chip?
22:15 Speaker 00 Yes, so at the March GTC, Jensen also unveiled the next generation of the edge computing chips. It's called the Jensen Thor chip. And it was actually co-announced with Project Groot. So the idea is we will have kind of the full stack as a unified solution to the customers. So from the chip level, which is the Jensen Thor family, to the foundation model, Project Groot, and also to the simulation and the utilities that we build along the way, it will become a platform, a computing platform for humanoid robots, and then also for intelligent robots in general. So I want to quote Jensen here.
22:56 Speaker 00 One of my favorite quotes from him is that everything that moves will eventually be autonomous. And I believe in that as well. It's not right now, but let's say 10 years or more from now. If we believe that there will be as many intelligent robots as iPhones, then we'd better start building that today.
23:15 Speaker 02 That's awesome.
23:16 Speaker 01 Are there any particular results from your research so far that you want to highlight? Anything that gives you kind of optimism or conviction in the approach that you're taking?
23:25 Speaker 00 Yes, we can talk about some prior works that we have done. So one work that I was really happy about was called Eureka. And for this work, we did a demo where we trained a five-finger robot hand to do pen spinning.
23:42 Speaker 01 Very useful.
23:45 Speaker 00 And it's superhuman with respect to myself because I have given up pen spinning long since childhood. I'm not able to do it.
23:53 Speaker 01 You're going to do a live demo?
23:55 Speaker 00 I will feel miserably at this live demo. So yeah, I'm not able to do this, but the robot hand is able to. And the idea that we used to train this is that we prompt an LOM to write code in the simulator API that Nvidia has built. So it's called the Isaac Sim API. And the LOM outputs the code for reward function. So a reward function is basically a specification of the desirable behavior that we want the robot to do. So the robot will be rewarded if it's on the right track or penalized if it's doing something wrong. So that's a reward function.
24:35 Speaker 00 And typically, the reward function is engineered by a human expert, typically a roboticist, who really knows about the API. It takes a lot of specialized knowledge. And the reward function engineering is by itself a very tedious and manual task. So what Eureka did was we designed this algorithm that uses LOM to automate this reward function design, so that the reward function can instruct the robot to do very complex things like pen spinning. So it is a general-purpose technique that we developed. And we do plan to scale this up to beyond just pen spinning. It should be able to design reward functions for all kinds of tasks, or it can even generate new tasks, right, using the Nvidia Simulation API.
25:20 Speaker 00 So that gives us a lot of kind of space to grow.
25:23 Speaker 01 Why do you think, I mean, I remember five years ago, there were people that were, you know, research labs working on solving Rubik's cubes with the robot hand and things like that. And it felt like robotics kind of went through. Maybe a trough of disillusionment. And in the last year or so, it feels like the space has really heated up again. Do you think there is a why now around robotics this time around? And like, what's different? And we're, you know, we're reading that OpenAI is getting back into robotics. Everybody, everybody is now spinning up their efforts. Like, what do you think is different now?
25:55 Speaker 00 Yeah, I think there are quite a few key factors that are different now. One is on the robot hardware. Actually, since the end of last year, we have seen a surge of new robot hardware in the ecosystem. There are companies like Tesla, working on Optimus, Boston Dynamics, and so on, and a lot of startups as well. So we are seeing better and better hardware. So that's number one. And those hardware is are becoming more and more capable with like, you know, better dexterous hands, better whole body reliability. And the second factor is the pricing. So we also see a significant drop in the, in the price and the cost, the manufacturing cost for the humanoid robots.
26:36 Speaker 00 So back in 2001, NASA had a humanoid developed, and it's called a RoboNot. I remember, if I recall correctly, it cost north of 1.5 million dollars per robot. And then most recently, there are companies that are able to put a price tag of about $30,000 on a full-fledged humanoid. And that's roughly comparable to the price of a car. And also, there's always this trend in manufacturing where a mature product, the price of it will tend towards the price of the raw material cost. And for the humanoids, it typically takes only 4% of the raw material of a car. So it's possible that we can see the cost trending downwards even more.
27:21 Speaker 00 And there could be an exponential decrease in the price in the next couple of years. And that makes, you know, these state-of-the-art hardware more and more affordable. That's the second factor of why I think humanoid is gathering momentum. And the third one is on the foundation model side, right? We are able to see the system two problem, the reasoning, the planning part being addressed very well by the frontier models, like the GPTs and the clouds and the Lamas of the world. And these LOMs, they're able to generalize to new scenarios. They're able to write code. And actually, the Eureka project I just mentioned leverages these coding abilities of the LOMs to help develop new robot solutions.
28:01 Speaker 00 And there are also a surge in multimodal models, improving the computer vision, the perception of it. So I think these successes also encourage us to pursue robot foundation models. Because we think we can ride on the generalizability of these frontier models. And then add like actions on top of them. So we can generate action tokens that will ultimately drive these humanoid robots.
28:25 Speaker 02 I completely agree with all that. I also think so much of what we've been trying to tackle to date in the field has been how to unlock the scale of data that you need to build this model. And all the research advancements that we've made, many of which you've contributed to yourself around Sim2Real and other things. And the tools that Nvidia has built with Isaac Sim and others have really accelerated the field alongside teleoperation and cheaper teleoperation devices and things like that. And so I think it's a really, really exciting time to be building here.
28:51 Speaker 00 Yeah, I agree.
28:53 Speaker 01 I'd love to transition to talking about virtual worlds, if that's okay with you.
28:56 Speaker 00 Yeah, absolutely.
28:58 Speaker 01 So I think you started your research more in the virtual world arena. Maybe say a word on what got you interested in Minecraft versus robotics? Like, is it all kind of related in your world?
29:14 Speaker 01 What got you interested in virtual worlds?
29:16 Speaker 00 Yeah, that's a great question. So for me, my personal mission is to solve embodied AI. And for AI agents embodied in the virtual world, that would be things like gaming and simulation. And that's why I also have a very soft spot for gaming. I also enjoy gaming myself.
29:32 Speaker 01 What did you play?
29:32 Speaker 00 Yeah, so I play Minecraft. At least I try to. I'm not a very good gamer. And that's why I also want my AI to avenge my poor skills.
29:41 Speaker 01 Yeah.
29:42 Speaker 00 So I worked on a few gaming projects before. The first one was called Mind Dojo, where we developed a platform to develop general purpose agents in the game of Minecraft. And for those audience who are not familiar, Minecraft is this 3D voxel world where you can do whatever you want. You can craft like all kinds of recipes, you know, different tools. And you can also go on adventures. It's an open ended game with no particular score to match. And no fixed storylines to follow. So we collected a lot of data from the internet. There are videos of people playing Minecraft. There are also wiki pages that explain every concept and every mechanism in the game.
30:27 Speaker 00 Those are like multimodal documents. And also like forums like Reddit, the Minecraft subreddit, has a lot of people talking about the game in natural language. So we collected these multimodal data sets and we're able to train models to play Minecraft. So that was the first work, Mind Dojo. And later, the second work was called Voyager. So we had the idea of Voyager after G P four came along. Because at that time, it was the best coding model out there. So we thought about, hey, what if we use coding as action? And building on that insight, we're able to develop the Voyager agent where it writes code to interact with the Minecraft world.
31:10 Speaker 00 So we we use an API to first convert the three D Minecraft world into a text representation. And then have the agent write code using the action APIs. But just like human developers, the agent is not always able to write a code correctly on the first try. So we kind of give it a self-reflection loop where it tries out something. And if it runs into an error or if it makes some mistakes in the Minecraft world, it gets the feedback and it can correct its program. And once it's written the correct program, that's what we call a skill. We'll have it saved to a skill library. So that in the future, if the agent faces a similar situation, it doesn't have to go through that trial and error loop again. It can retrieve the skill from the skill library.
31:53 Speaker 00 So you can think of that skill library as a codebase that the L O M interactively authored all by itself, right? There's no human intervention. The whole codebase is developed by Voyager. So that's the second mechanism, the skill library. And the third one is what we call an automated curriculum. So basically, the agent knows what it knows and it knows what it doesn't know. So it's able to propose the next task that's neither too difficult nor too easy for it to solve. And then it's able to just follow that path and discover all kinds of different skills, different tools, and also travel along in a vast world of Minecraft. And because it travels so much,
32:35 Speaker 00 and that's why we call it the Voyager. So yeah, that was kind of our team's, one of our earliest attempts on building AI agents in the embodied world using foundation models.
32:47 Speaker 01 Talk about the curriculum thing more. I think that's really interesting because it feels like it's one of the more unsolved problems in kind of the reasoning and LLM world generally. Like, how do you make these models self-aware so that they know kind of how to take that next step to improve? Maybe say a little bit more about what you built on the curriculum and the reasoning side.
33:05 Speaker 00 Absolutely. I think there's a very interesting emergent property from those frontier models is that they can reflect on their own actions and they kind of know what they know and what they don't know. And they're able to propose tasks accordingly. So for the automatic curriculum in Voyager, we gave the agent a high-level directive, that is, to find as many novel items as possible. And that's just the one kind of sentence of goal that we gave. And we didn't give any instruction on which objects to discover first, which tools to unlock first. We didn't specify. And the agent was able to discover that all by itself using this kind of coding and prompting and skill library.
33:50 Speaker 00 So it's kind of amazing that the whole system just works. I would say it's an emergent property once you have a very strong reasoning engine that can generalize.
33:58 Speaker 01 Why do you think there are so many, so much of the kind of virtual world research has been done in the virtual world? And I'm sure it's not entirely because a lot of deep learning researchers like playing video games, although I'm sure it doesn't hurt either. But I guess what are the connections between solving stuff in the virtual world and in the physical world? And how do the two interplay?
34:19 Speaker 00 Yeah, so as different as gaming and robotics seem to be, I just see a lot of similar principles shared across these two domains. For the embodied agents, they take as input the perception, which can be a video stream along with some sensory input, and then they output actions. And in the case of gaming, it will be like keyboard and mouse actions. And for robotics, it will be low-level motor controls. So ultimately, the API looks like this. And these agents, they need to explore in the world. They have to collect their own data in some ways. So that's what we call reinforcement learning and also self-exploration.
35:00 Speaker 00 And that part, that principle is again shared among the physical agents and the virtual agents. But the difference is robotics is harder, because you also have a simulation-to-reality gap to bridge. Because in simulation, the physics and the rendering will never be perfect. So it's really hard to kind of transfer what you learn in simulation to the real world. And that is by itself an open-ended research problem. So for robotics, it's got the sim-to-real issue, but for gaming, it doesn't. You are training and testing in the same environment. So I would say that will be the difference between them. And last year, I proposed a concept called Foundation Agent, where I believe ultimately we'll have one model
35:45 Speaker 00 that can work on both, you know, virtual agents and also physical agents. So for the Foundation Agent, there are three axes over which it will generalize. Number one is the skills that it can do. Number two is the embodiments or like the body form, the form factor it can control. And number three is the world, the realities it can master. So in the future, I think a single model will be able to do a lot of different skills. On a lot of different robot forms or agent forms, and then generalized across many different worlds, virtual or real. And that's the ultimate vision that the gear team wants to pursue, the Foundation agent.
36:28 Speaker 02 Pulling on the thread of of virtual worlds and gaming in particular, and and what you've unlocked already with some reasoning, some emergent behavior, especially working in an open-ended environment. What are your, what are some of your own personal dreams for what is now possible in the world of games? Where would you like to see AI agents innovate in the world of games today?
36:48 Speaker 00 Yes, so I'm very excited by two aspects. One is intelligent agents inside the games. So the NPCs that we have these days, they have fixed scripts to follow, and they're all manually authored. What if we have NPCs, the non-player characters that are actually alive?
37:08 Speaker 02 Yeah.
37:08 Speaker 00 And you can interact with them, and they can remember what you told them before, and they can also take actions in the gaming world that will change the narrative and change the story for you. So this is something that we haven't seen yet, but I feel there's a huge potential there. So that when everyone play the game, everybody will have a different experience. And even for one person, you play the game twice, you don't have the same story. So each game will have infinite replay value.
37:37 Speaker 02 Yeah.
37:38 Speaker 00 So that's one aspect. And a second aspect is that the game itself can be generated.
37:43 Speaker 02 Yeah.
37:44 Speaker 00 And we already see many different tools kind of doing subsets of this grand vision I just mentioned. Right, like there are text to 3D generating assets. There are also like text to video models. And of course, there are like language agents that can, you know, generate storylines. What if we put all of them together so that the game world is generated on the fly as you're playing and interacting with it?
38:08 Speaker 02 Yeah.
38:09 Speaker 00 That would be just truly amazing and a truly open-ended experience.
38:12 Speaker 02 Super interesting.
38:13 Speaker 01 For the agent vision in particular, do you think you need GPT-4 level capabilities? Or do you think you can get there with Llama-8B, for example, alone?
38:22 Speaker 00 Yeah, I think the agent needs the following capabilities. One is, of course, it needs to hold an interesting conversation. It needs to have a consistent personality, and it needs to have long-term memory, and also take actions in the world. So for these aspects, I think currently the Llama models are pretty good for that, but also not good enough to produce very diverse behaviors and really engaging behaviors. So I do think there's still a gap to reach. And the other thing is about inference cost. So if we want to deploy these agents to the gamers, then either it's very low cost, hosted on a cloud, or it runs locally on a device.
39:06 Speaker 00 Otherwise, it's kind of unscalable in terms of cost. So that's another factor to be optimized.
39:13 Speaker 01 Do you think all this work in the virtual world space, is it in service of, like, you know, you're learning things from it. That way, you can accomplish things in the physical world. Does the virtual world stuff exist in service of the physical world ambitions? Or, I guess, said differently, like, is it enough of a prize in its own right? And how do you think about prioritizing your work between the physical and virtual worlds?
39:37 Speaker 00 Yes, so I actually think the virtual world and the physical world ultimately will just be different realities on a single axis. So let me give one example. So there is a technique called domain randomization. And how it works is that you train a robot in simulation, but you're training it in 10,000 different simulations in parallel. And for each simulation, they have slightly different physical parameters. Like, the gravity is different, the friction, the weight, everything's a bit different, right? So it's actually 10,000 different worlds. And let's assume if we have an agent that can master all the 10,000 different configurations of reality all at once,
40:22 Speaker 00 then our real physical world is just the 10,000 first. Virtual simulation.
40:29 Speaker 00 And in this way, we're able to generalize from sim to real directly. So that's actually exactly what we did in a follow up work to Eureka, where we're able to train agents using like all kinds of different randomizations in the simulation and then transfer zero shot to the real world without further fine tuning. So I do believe
40:50 Speaker 02 And that's Dr. Eureka.
40:51 Speaker 00 That's Dr. Eureka work. And I do believe that if we have all kinds of different virtual worlds, including from games, and if we have a single agent that can master all kinds of skills in all the worlds, then the real world just becomes part of this bigger distribution.
41:08 Speaker 02 Do you want to share a little bit about Dr. Eureka to ground the audience in that example?
41:11 Speaker 00 Oh yeah, absolutely. So for the Dr. Eureka work, we built upon Eureka and still use L O Ms as kind of a robot developer. So the L O M is writing code and the code is to specify the simulation parameters, like the domain randomization parameters. And after a while, after a few iterations, the policy that we train in a simulation, we'll be able to generalize to the real world. So one specific demo that we showed is that we can have a robot dog walk on a yoga ball. And it's able to stay balanced and also even walk forward. So one very funny comment that I saw was someone actually asked his real dog to do this task and his dog isn't able to do it.
41:56 Speaker 00 So in some sense, our neural network is super dog performance.
42:01 Speaker 02 I'm pretty sure my dog would not be able to.
42:03 Speaker 01 He called it A D I.
42:05 Speaker 00 Yeah. Artificial dog intelligence. Yeah, that's the next benchmark.
42:10 Speaker 01 In the virtual worlds sphere, I think there's been a lot of kind of just incredible models that have come out on both the 3D and the video side recently. All of them kind of transformer based. Do you think we're kind of there in terms of like, okay, this is the architecture that's going to take us to the promised land and let's let's scale it up? Or do you think there is kind of fundamental breakthroughs that are still required on the on the model side there?
42:34 Speaker 00 Yes, I think for robot foundation models, like we haven't pushed the limit of the architecture yet. So the data is a harder problem right now, and it's the bottleneck. Because as I mentioned earlier, we can't download those action data from the internet. They don't come with those motor control data. We have to collect it either in simulation or in the real on the real robots. And once we have that, we have a very mature data pipeline, then we'll just push the tokens to the transformers and have it compress those tokens, just like, you know, transformers predicting the next word on Wikipedia. And we're still testing these hypotheses, but I don't think we have pushed the transformers to their limit yet. There are also a lot
43:19 Speaker 00 of research going on right now on alternative architectures to transformers. I'm super interested in those personally, like there are Mamba, recently there's like test time training, there are a few alternatives. And some of them have very promising ideas. They haven't scaled really to the all the frontier model performance. But I'm looking forward to seeing alternatives to transformers.
43:44 Speaker 01 Have it even caught your eye in particular and why?
43:46 Speaker 00 Yeah, I think I mentioned the Mamba work and also test time training, like these models are more efficient at inference time. So instead of like transformers attending to all the past tokens, these models have inherently more efficient mechanisms. So I see them holding a lot of promise, but we need to scale them up to the size of the frontier models and really see like how they compare head on with the transformer.
44:15 Speaker 01 Awesome. Should we close it out with some rapid fire questions? Yeah.
44:18 Speaker 00 Oh, yeah.
44:18 Speaker 01 Um, okay, let's see. Number one, uh, what, uh, what outside the embodied AI world are you most interested in with an AI?
44:26 Speaker 00 Yeah, so um, I'm super excited about uh, video generation. Um, because um, I see video generation as a kind of world simulator. So we learn the physics and the rendering from data alone. Um, so uh, we have seen like OpenAI Sora and later there are like a lot of new models uh, catching up to Sora. So uh, this is uh, like a ongoing research topic and yeah.
44:49 Speaker 01 What does the world simulator get you?
44:51 Speaker 00 I think it's a, it's gonna get us a data-driven simulation in which we can train embodied AI. That would be amazing.
44:57 Speaker 01 Nice. What are you most excited about in AI on a longer-term horizon? Ten years or more?
45:02 Speaker 00 Yeah, so um, on a few fronts, like one is uh, for the, for the reasoning side. Uh, I'm super excited about models that code.
45:11 Speaker 01 Yeah.
45:11 Speaker 00 I think coding is uh, such a fundamental reasoning task that also has huge economic value. Um, I think maybe ten years from now, we'll have uh, coding agents uh, that are as good as human-level software engineers. And then we'll be able to accelerate a lot of development um, using the L O Ms themselves. And the second aspect is of course robotics.
45:34 Speaker 01 Hmm.
45:34 Speaker 00 I think ten years from now, we'll have humanoid robots uh, that are at the reliability and agility of humans or even beyond. And I hope at that time, project group will be a success. And then we'll able to have humanoids helping us um, in our daily lives. Um, I just want robots to do my laundry.
45:52 Speaker 01 Yeah.
45:53 Speaker 00 That's always been my dream.
45:55 Speaker 01 What year are robots gonna do our laundry?
45:56 Speaker 00 As soon as possible. I can't wait.
46:00 Speaker 01 Um, who do you admire most in the field of AI? And you've had the opportunity to work with some greats uh, dating back to, to your internship days. But who, who do you admire most these days?
46:10 Speaker 00 I have too many heroes in AI uh, to count. Um, so I admire uh, my PhD advisor Fei Fei. Uh, I think she taught me um, how to develop good research taste. So sometimes it's not about how to solve a problem, but identify what problems are worth solving.
46:28 Speaker 01 Hmm.
46:29 Speaker 00 And actually, the what problem uh, is much harder than a how problem. And uh, during my PhD years with Fei Fei, I transitioned to embodied AI. Um, and in retrospect, this is the right direction to work on. I believe the future of AI agents will be embodied uh, for robotics or for the virtual world. I also admire Andre Karparchi. He's a great educator. I think he writes code like poetry. So I look up to him. And then I admire Jensen a lot. I think Jensen, he cares a lot about AI research. And he also knows a lot about even the technical details of the models. And I'm super impressed. So I look up to him a lot.
47:10 Speaker 02 Pulling on the thread of having great research taste, what advice do you have for founders building in AI in terms of finding the right problems to solve?
47:17 Speaker 00 Yeah, I think read some research papers. I feel that the research papers these days are becoming more and more accessible. And they have some really good ideas. And they are more and more practical instead of just like theoretical machine learning. So I would recommend kind of keeping up with the latest literature. And also just try out like all the open source tools that people have built. So for example, at Nvidia, we built simulator tools that everyone can have access to. And just download it and try that out. And you can train your own robots in the simulations. Just get your hands dirty.
47:54 Speaker 02 And maybe pulling on the thread of Jensen as an icon, what do you think is some practical tactical advice you'd give to founders building in AI, what they could learn from him?
48:03 Speaker 00 Yeah, I think identifying the right problem to work on, right? So Nvidia bets on humanoid robotics because we believe like this is the future. And also like embodied AI. Because if we believe that, let's say 10 years from now, there will be as many intelligent robots in the world as iPhones, then we better start working on that today. So yeah, just like long term future visions.
48:29 Speaker 02 I think that's a great note to end on. Jim, thank you so much for joining us. We love learning about everything your group is doing. And we can't wait for the future of laundry folding robots.
48:38 Speaker 00 Awesome. Yeah, thank you so much for having me. Yeah.
48:40 Speaker 02 Thank you.
48:41 Speaker 01 Thank you.
48:42 Speaker 00 Thanks.
Previous Founder Eric Steinberger on Magic’s Counterintuitive Approach to Pursuing AGI Next Why Vlad Tenev and Tudor Achim of Harmonic Think AI Is About to Change Math—and Why It Matters Page updated October 12, 2026 · transcript and summary are machine-generated
