#331 Sergey Levine: The Robot Revolution Nobody Is Talking About

This episode is sponsored by Modulate. Most voice AI focuses on transcription. Velma takes it further by actually understanding conversations, analyzing tone, timing, stress, and intent using its Ensemble Listening Model architecture. Explore the live preview: https://preview.modulate.ai/ What does it actually mean to build a foundation model for robots? In this episode of Eye on AI, Craig Smith sits down with Sergey Levine, co-founder of Physical Intelligence and professor at UC Berkeley, to explore a fundamentally different approach to building robots, one inspired not by programming a single perfect machine, but by training AI on the broadest and most diverse data possible so robots can learn, adapt, and operate in the unpredictable real world. Sergey explains why the secret to general-purpose robots isn't perfecting one single machine, but training on massive, diverse data from all kinds of robots and even humans. The more variety the model sees, the better it gets. Just like ChatGPT learned from all the text on the internet, robotic foundation models learn from every robot that has ever moved, grabbed, or interacted with the real world. We also get into the big humanoid robot debate. Are they the future, or is it mostly hype? Sergey gives an honest and technical take on why the form factor conversation is changing now that foundation models exist, and why that actually opens the door for more creativity, not less. Finally, Sergey shares what he's most excited about next, building a true data flywheel where robots get smarter the more they are deployed, creating a continuous learning cycle that could change everything. Subscribe for more conversations with the people building the future of AI and emerging technology. Stay Updated: Craig Smith on X: https://x.com/craigss Eye on A. I. on X: https://x.com/EyeOn_AI (00:00) Introduction: What Are Foundation Models for Robots? (01:44) Meet Sergey Levine: Physical Intelligence and UC Berkeley (02:51) Breaking Down Foundation Models for Non-Technical People (06:46) Why Real World Data Beats Simulation (15:00) Building a Broad Robotics Foundation From Scratch (24:00) The Open World Problem in Robotics (40:00) Generalist vs Specialist Robots: Which Wins? (47:00) Humanoid Robots: Real Innovation or Just Hype? (55:10) The Future: Continuous Learning and the Data Flywheel (56:23) Guilty Pleasure: Sci Fi and Thinking Beyond the Limits

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Craig S. Smith?

  A lot of startups these days say they're building foundation models for robots. What does that actually mean for a non-technical listener?

• 0:08Speaker 01

  I think every time that people try to take robots out of the factory and into open world environments, they very quickly realize that in the real world, there's a huge range of things that can happen.

• 0:18Craig S. Smith?

  But since then, there's been this explosion of humanoids. And everyone's talking about humanoids. I mean, did I get it wrong? Do you think that humanoids are much closer to being in the world? Most AI is just speech to text, plus a language model. It's full for reading transcripts, not understanding conversations. Velma from Modulate, an AI built on ensemble listening model architecture, specializes in audio analysis. It orchestrates hundreds of smaller sub-modes, purpose-built to understand the nuances of voice, like tone, timing, stress, and intent.

• 1:00Craig S. Smith?

  Perfect for fraud defense, deep fake detection, agent attrition prevention, or customer service moderation. Check out the live Velma preview at preview dot modulate dot AI. That's preview dot modulate dot AI. To see how the model breaks down audio, providing timestamped, explainable signals. Stop transcribing, start listening with modulate dot AI.

• 1:34Speaker 01

  My name is Sergey Levin. I'm one of the founders of Physical Intelligence. I'm also a professor at UC Berkeley. And what I work on these days is algorithms for reinforcement learning, for optimal decision-making, as well as applications in robotics. And something that I've been very interested in lately, in particular, is robotic foundation models. These are general-purpose models that control any robot, in principle, to perform any task. And I think we've seen some like pretty sure dramatic transformations in the last few years in the capabilities of these kind of general robotic systems where we can use very diverse data sources for many different robotic platforms, performing a wide range of different tasks and acquire a kind of general physical understanding from these

• 2:18Speaker 01

  data sets that then make it much more feasible to rapidly acquire effective and robust and highly generalizable robotic skills. So this is something that I've been very interested in the last few years, and I think it's an area where we see a lot of progress.

• 2:30Craig S. Smith?

  Yeah, and a lot of startups these days say they're building foundation models for robots. What does that actually mean for a non-technical listener?

• 2:40Speaker 01

  Yeah, this is a it's actually a surprisingly nuanced question because after the success of ChatGPT, you know, the term foundation model became obviously very much a buzzword. So, you know, in some cases, it's almost synonymous to saying like, you know, I have a good model. It's a foundation model. But I think that insofar as there's a consistent definition, it's something like this, that the principle behind language models, vision language models, things like this, is that you can use very large and diverse data sources that are not necessarily of extremely high quality, like it might be just data harvested from the web. And you get a model that digests all this data and acquires a kind of broad

• 3:22Speaker 01

  and general understanding of how the world works. And this kind of understanding is not enough to be an expert, it's not enough to be extremely proficient, but it gives you that kind of basis of common sense. And that's why the term foundation model was coined by Percy Lang and his colleagues at Stanford for precisely this reason, because this kind of broad basis of knowledge gives you a foundation on top of which you can then put other things. And the key thing about the foundation is that in order to be useful, it needs to be broad. Insofar as there's a deep technical insight here, the insight is that if you If you need such a large amount of data, it's almost impossible to get in a single domain. But if you are willing to use data from many different sources, maybe all of the text data on the web, all of the image data you can get your hands on,

• 4:07Speaker 01

  or in our case, data from all of the robots that we've seen, that could be big enough, and that can give you that foundation. And on top of that foundation, now you can build up individual skills. In the case of language models, you can fine tune them for expert-level computer programming. In the case of our robots, we can fine tune them for, assembling things, or making coffee, or cleaning the kitchen with very high-quality data, but a much more limited amount of it. Because with that, once you can put on top of that foundation, you don't need a huge amount of data of very high quality for the downstream task. So that's the really important thing. Now, to come back to your question, you asked, well, there are many startups, many organizations that are building foundation models.

• 4:52Speaker 01

  I think one of the most important things about a robotic foundation model is to answer the question, where does all of that really broad and diverse data come from that can establish that foundation? And this is a place where the answer is very, very delicate. The strategy that I think will be most successful here is to not be too picky. In the same way that language models are trained on all of the text data that can be mined from the web, a robotic foundation model should be trained on all of the embodied data that we can get our hands on. And it's one of those things where once you cross a certain threshold of scale, it actually becomes easier to incorporate other data sources. So if you want really, really high-quality data

• 5:35Speaker 01

  of like one particular high-end humanoid robot, that's pretty challenging, because now you're very constrained. You have to have that system. You have to get good data for it. You have to figure out how to tell you operate and put it in the right environments and so on. But if you're willing to pull in everything, then you could pull in lots of robot data from many different kinds of robots. Some of them might be good. Some of them might be bad. Once your model understands diverse physical embodiments, you can also start adding in data from humans. Because to the model, the human body will look like yet another robot body. So this kind of diversity actually makes it easier to include other data sources. So.

• 6:12Craig S. Smith?

  And.

• 6:13Speaker 01

  No, go ahead.

• 6:14Craig S. Smith?

  No, no, no, go ahead.

• 6:15Speaker 01

  I was gonna bring this back to your question for the the final conclusion, which is out to to to actually answer what you asked me. I think a a big difference between how physical intelligence is approaching this and how most other research labs approach this question is that we are not being very picky about which robots we use for this. We're bringing in everything and trying to build this very broad foundation.

• 6:36Craig S. Smith?

  Yeah, and and and to build that model, I mean, to collect that data, there there are various strategies. Uh, one is simulation, and as I recall from our past conversations, you're not a great fan of simulation. Uh, but I just saw sixty minutes a little while back and they had, uh, uh, I'm trying to think which was a Boston, uh, uh, Dynamics in a Hyundai factory and they were showing the simulation of of sort of an endless army of, uh, of of Atlas Robots, uh, performing tasks. Uh, why are you not a fan of simulation?

• 7:22Craig S. Smith?

  That's that's one. And then, two, you're using, uh, vision language, uh, action models, is that right? Uh, and can you talk about how that differs from other strategies like, uh, world models or or, uh?

• 7:46Speaker 01

  Yep.

• 7:46Craig S. Smith?

  Other kinds of foundation models, you know?

• 7:49Speaker 01

  Yeah, for sure. So in regard to simulation, um, what I would say is this, that, um, simulation is a very appealing tool for, um, kind of, uh, very easily acquiring lots of data of a robot doing all sorts of different things. But it's not a very appealing tool for getting. Experience of very diverse environments and very diverse objects. So if we look at kind of the domains in AI where simulation has been successful and the domains where to struggle to get adoption. Computer vision is an area where simulation has actually been used very little, despite a lot of attempts.

• 8:26Craig S. Smith?

  Yeah.

• 8:27Speaker 01

  Why? Well, it's not actually because rendering images is hard. In fact, like computer graphics is very, very advanced. So we can render very realistic images. It's that getting real images is so much easier, right? Like you can just take a camera and go and photograph stuff and you get lots of real images. And I think that in robotics, a kind of a mental trap that people sometimes fall into is that they say, well, maybe in robotics it's hard to get data. It's not as easy as taking a camera, going out and taking pictures. And I think that's a little bit of a mistake because actually, if you're serious about building general purpose robots that'll go out into the world and do lots of things, the kind of boundary condition is in your favor. Meaning that the better you get at building generous robots, the more robots there are and the more data there should be coming in.

• 9:11Speaker 01

  There's a little bit of like an initial activation energy problem where you have to get over that hump to get enough systems out there. But that's like a transient period. Once you get over that, then you have lots of robots out there and lots of data coming in. So it actually to me makes a lot more sense to pay a little bit more of the upfront cost to kind of like force it over that threshold and then get lots of real world data. That doesn't mean that we shouldn't use simulation at all. It just means that we shouldn't worry so much about how hard it is to get robot data. For robot data, we should treat that as the industrial problem that it is. Get robots out there, get the data coming in. And then simulation can be very useful for addressing other edge cases. For example, you can simulate, you know, this is what the autonomous driving folks do all the time. You can simulate cases that you don't want to experience in the real world. Like you can simulate a car collision, but you don't want to experience a car collision in your car.

• 9:55Speaker 01

  So there's a lot of these kind of edge cases that you might want to take care of with simulation. But I don't think we should think of it as a substitute for real experience. Because in other areas where diversity has been critical, like computer vision and natural language processing, real data has been essential to get there.

• 10:11Craig S. Smith?

  Yeah.

• 10:11Speaker 01

  And further, once you have that real data, it's actually easier to incorporate other data sources.

• 10:16Craig S. Smith?

  Yeah, although couldn't you use simulation, as I mentioned, with the Boston Dynamics robot to get over the hump and then start collecting real, real life data?

• 10:28Speaker 01

  Yeah, that's a very good question. I think you could. I think in practice, in robotic manipulation, we found it to be a lot easier to do that with real data. Part of it has to do with the fact that simulation is very good for simulating the robot. It's much less good for like simulating everything else. Because the robot you only have to model once, whereas if you want to have like thousands of scenes, thousands of objects, then you have to go and model and simulate each one of those. I don't think that's impossible. I think it is attractable, problem potentially, but it's just more costly than just going out and getting lots of real stuff.

• 11:01Craig S. Smith?

  Yeah, and you know, there's been this explosion of humanoids. Yours are not necessarily humanoid. I mean, yours kind of platform agnostic, is that right? But on the humanoids, Joanna Stern at the Wall Street Journal last year had

• 11:24Craig S. Smith?

  Neo invited Neo into her home and it didn't perform very well. And apparently Neo is available for home use, but it comes along with a teleoperator that spends time in your house, like walking around doing something and that's to collect that data to train the robot. I mean, they need a lot of these deployments to collect enough data. That doesn't seem like a very scalable solution. So how are you guys approaching it? Is it, are you doing any teleoperation or?

• 12:08Speaker 01

  Yeah, so this is a very good question. And in fact, in some ways, this is like kind of the big question in modern robotic foundation models, which is what kind of data can you use? Now, I think it makes a lot of sense to start off building the initial foundation with teleoperation data. But as you pointed out, there are major challenges with this, like, you know, not the least of which is that if you actually want to collect data in deployment scenarios, whether it's somebody's home or a business or a factory or warehouse or whatever, like that's an additional kind of inconvenience that you have to deal with and it's a barrier to scale. I think that the right way to proceed with this is to Think of it as a mixture of different data sources where as your model gets better, it should be able to leverage more accessible and more scalable data sources.

• 12:52Speaker 01

  So initially, maybe when the model is not very good, we need data from teleoperation from humans that basically illustrates like this how the robot should act. But once the model gets better and the robot can be deployed with at least some degree of autonomy, then we can handle more accessible sources of supervision. One more accessible source of supervision is instructions. This is something that we actually found, not entirely intentional, like we were just kind of like, you know, trying out a few things in some of our research projects. But we found that we could actually get improvement in our policies by supervising the robot essentially through language. And this only started happening once the model became powerful enough that the low-level skills were already pretty good. Then you could correct the robot and say like, oh, maybe it's cleaning up the kitchen.

• 13:37Speaker 01

  It messed up. You could say like, oh, you needed to pick up the plate and make sure you put the plate in the sink. And the way the model works internally is it's very similar to how these modern reasoning models, LLMs work, where there are internal thoughts that are generated and then the final action is chosen based on those thoughts. So essentially, this kind of language feedback supervises the internal thoughts rather than the low-level actions. But once the low-level actions are good enough, actually supervising the internal thoughts already gives the robot a lot of learning signal and can improve a policy without direct teleoperation. But again, this only emerged with once the base model is strong enough. The other thing we can do is leverage autonomous experience where we can improve the system through reinforcement learning.

• 14:19Speaker 01

  So there was a research project that we, we actually published just a few months ago that describes a reinforcement learning system that we built on top of our foundation model. And again, it's the same story that you need the foundation model to be strong enough so that from there it can improve with autonomous experience and reinforcement.

• 14:37Craig S. Smith?

  And one time, as I recall, you had, I didn't see it, but you were setting up ranks of robotic arms and they were going to be sort of informing each other. So any arm that learns something, then that learning would be.

• 15:03Craig S. Smith?

  Transferred or to all of the the robots is is that kind of scale useful? I mean, are you doing that?

• 15:14Speaker 01

  Yeah.

• 15:14Craig S. Smith?

  With with, yeah.

• 15:16Speaker 01

  Yeah, that's exactly right. So the the the big benefit of of having robotic learning rather than human learning here is that you can have this fleet effect and you can do collective learning very effectively. So all of our robots, you know, share all the all the experience. And again, it comes back to this idea that the stronger the base foundation model is, the more readily it can incorporate experience from diverse robotic platforms. So the experiment that you're referring to, this was done at this point, maybe about eight years ago, the the Google Arm Farm project there. Every single robot platform and every single robot station was as close as possible to each other. They were virtually identical. And that worked very well with like the state of the art learning technology of, you know, 2017, 2018.

• 16:00Speaker 01

  But these days, when you have a foundation model that accommodate very diverse platforms, very diverse tasks, very diverse environments, the collective learning stuff becomes much easier because now we can pull in data from many of our own robots that are deployed. We can also work with other companies that have their own robotic platforms. And in fact, initially, when we started this, we thought that we would need to do something very special to accommodate this. So maybe we can somehow tell the model, like, here's the morphology or control. And here are the details of this robot platform. It actually turns out that very little cleverness is required in that respect, like the model can figure out from looking through the camera, the camera image, what kind of robot it's dealing with. A lot of careful engineering is still needed to make sure that training is efficient, to make sure the model is set up in the right way.

• 16:45Speaker 01

  But handling the cross embodiment aspect of this, it turns out to actually be pretty straightforward.

• 16:51Craig S. Smith?

  Yeah. And across form factors as well, is that right? Not only tasks.

• 16:58Speaker 01

  That's right. Other companies we've worked with have actually adapted our models for controlling multi-finger hands, humanoids, that sort of thing. In fact, some of them, we use them for mobile robots, things like agricultural equipment that we wouldn't conventionally think of as robots in the usual sense.

• 17:13Craig S. Smith?

  Yeah. Wasn't there a project that you were involved in that that brought together data from all around the world, from all various robotics labs?

• 17:25Craig S. Smith?

  And what was the outcome of that, or is that ongoing?

• 17:29Speaker 01

  Yeah, so this, what you're referring to, I think, is the RTX project, which in many ways was actually part of the the impetus for, for starting physical intelligence. So this was a project that we did in 2023, and myself and many of my colleagues that worked on this then went on to found physical intelligence. In the RTX project, this was very much an academic research project, but what we did is we contacted academic research labs, about thirty labs in total. And we asked them to basically send us the data from their robotic manipulation experiments. And we limited this to a single-arm robot manipulators with parallel geographers, just to pick kind of the most common form factor. And then what we did is we trained one model across all of these different data sets.

• 18:14Speaker 01

  And we sent back that model to some of the labs that had donated data and asked them to essentially evaluate it in comparison to whatever they were developing on their own robot for their own application. So each lab was doing a different research project with a different robot and a different task. And they had their own methods that they were developing. And we just said, like, whatever is the best you've got, just measure that against our generalist model. And what we found is that the generalist model, on average, was about fifty percent more successful than whatever each individual lab was developing. And that's really, really exciting, because this is kind of paralleling a lot of the development that we've seen in language models. With language models, the, the, the big result, scientifically, it wasn't actually ChatGPT.

• 18:56Speaker 01

  The scientific result that was so exciting is that the generalist model, the generalist language model, could outperform specialized models for machine translation, sentiment analysis, you know, all these NLP tasks that typically would require very specialized data sets and very specialized models could be done better with this more general. But what we saw with RTX was an early hint that something like that was actually happening in robotics. And I think that's actually really important.

• 19:19Craig S. Smith?

  Yeah, but in in RTX, you're you're using exclusively robotic arms. How does that data inform other form factors? Or or do you need if you're if you're training humanoids? Do you need to collect data across various different humanoid platforms?

• 19:43Speaker 01

  Yeah, so the current models we have at Physical Intelligence are trained on many more robots, and they do vary in morphology. Now, I should say that generalizing into an entirely new morphology that was not seen in the data, that is still kind of at the bleeding edge of current research. So we're not really that concerned with that. We're concerned with the case where someone has a robot, they have some data from that robot, but what they want is to benefit from transfer from other robotic platforms. You know, here's an anecdote that I can tell you about that maybe underscores this point. In the first year or so of the company, this was in 2024, we worked almost entirely with static arms. So these are not mobile robots. These were arms that were attached to a table, and they were doing manipulation tasks.

• 20:26Speaker 01

  And then in early 2025, we decided that we wanted to start experimenting with mobile robots. So these are basic arms on a wheeled base. And we had very few mobile robots, so we could only collect a little bit of data with them. And our first kind of publicly released research project on this, which came in April 2025, used a training set of which only 3% of the data was collected on mobile robots. So 97% was from these statically mounted arms. But we could get the mobile robots to actually generalize very broadly. They could go into a home that was never seen in the training data, clean up the kitchen, you know, put away the dishes, that sort of thing. And most of the knowledge in these models came from not the mobile robot, but these static arms bolted to a table.

• 21:10Speaker 01

  So that kind of underscores the power of this kind of cross embodiment learning, where we could use lower cost, more accessible platforms to get the bulk of your data and then adapt them to a downstream morphology.

• 21:20Craig S. Smith?

  Yeah, that's that's fascinating. Although I imagine when you move to bipedal bipedal for legged robots, you're you're gonna that data, it'll help in the robot arm manipulation, but not necessarily in in mobility. So in in this is.

• 21:47Craig S. Smith?

  A kind of an uninformed question, but when you see teleoperation, what is happening there? And what kinds of things are robots learning? Is that imitation learning? Is that something else?

• 22:11Speaker 01

  So the standard kind of default way to use teleoperation data is imitation learning. But this is, I think, a place where there's a lot of room for improvement in current research. And we've started studying this a little bit. There's other research groups that are studying this. Basically, what you would like to do with data ideally is not just copy it blindly, but actually understand dynamically which parts of what is being demonstrated are good and which parts are not so good. Basically, instead of using the data to answer the question of how should I do this? Use the data to answer the question of what is possible? And then among the things that are possible, pick like the best things. And that's basically where a lot of the ideas in offline reinforcement learning can come in. Roughly speaking, the way that this works is instead of supervising the model

• 22:55Speaker 01

  to produce the same actions that are in the data, what you do is you supervise the model to predict the outcomes. So you train the model so that it can predict like if I see this and I do this, will that be good or will that be bad? And if you can do a really good job predicting those outcomes, then you can tell the model, okay, now do whatever will lead to the good outcome. And that's actually potentially a very powerful tool because now you can bring in heterogeneous data of different quality. And now the variety of data quality actually becomes a blessing rather than a curse. Because if you see lots of good things and lots of bad things, then you can figure out how to distinguish good from bad and do better at test time. So this is kind of where a lot of the current like research is situated.

• 23:32Craig S. Smith?

  I see. How does, I mean, does teleoperation, is that using a VLA? A model in the background, is that collecting data for a vision and language action model?

• 23:47Speaker 01

  That's right. So our models are based on visual language action models. And this is, this is kind of become essentially like a de facto standard in robotic learning research. This is something that many of the folks on the team here pioneered back in like the early 2020s. Now it's basically what everybody uses. VLA's are kind of an interesting thing because initially, like the early, what I refer to as first generation VLA's, they were trained in a very straightforward way. Basically, Vision language models are models that answer questions and they can also take in an image, so they answer visual questions. Early VLA's were trained by basically taking this visual question answering paradigm and simply turning robotic control into like a visual question.

• 24:31Speaker 01

  So in robotic control, the question is the prompt, like pick up the socks, and the answer is the numerical value of the actions. So that's like a fairly straightforward naive way to cram robotic control into a format that vision language models can understand. But there's a lot more that we can do than just that. And there's probably speaking two big buckets where there is room for improvement with VLA's. One is dexterity and the other one is reasoning. So dexterity means go beyond treating actions as an answer to a visual question and actually develop a model design that handles dexterity first and foremost. So control is not a discrete thing, it's not, it's not an answer to a question, it's a continuous thing, it's a trajectory.

• 25:11Speaker 01

  So you can use models that are very well adapted to high dimensional, continuous, dynamical systems. Diffusion models are really good for this, or bringing diffusion models into vision language models can give you these kind of much more dexterous VLMs. The second thing is the knowledge that is learned by language models and visual language models from the web, a lot of that knowledge is semantic, it's not physical. So a big way to improve VLA's is to better hook into that semantic knowledge. Essentially, when the robot doesn't know what to do, what it should do, much like a person, is it should pause and think, and that thinking maybe taps into more semantic

• 25:55Speaker 01

  knowledge that is not yet fully grounded in the physical world, but can lead to reasonable inferences. So maybe it's trying to open a drawer to take out a knife to cut a vegetable, and the drawer is an opening. Now, maybe the robot experience is not enough to inform it what to do, but there's a reasonable semantic inference, you say, well, why isn't this thing opening? Maybe I should try a different one. Like it's kind of this common sense inference you can make, and that common sense, like if you ask like chat GPT, it can make that common sense inference and it will tell you something. And the trick then is to digest that inference into a format that the motor control component of this can understand. That's basically a thinking process. So that's where I think there's room for a lot of improvements for these models. And we've seen a little bit of that in some of our work on Shane and Ford. And I think it's where we'll see a lot of future developments.

• 26:37Craig S. Smith?

  Yeah, you know, I had Fei-Fei Li on recently talking about world models. And in the past, I've had Yan Le Cun on several times talking about world models. Where does world, where do world models fit into the stock? I mean, are you, because world models, as I understand it, certainly from Yan Le Cun's point of view, it's, it's building an internal representation of the world that, that then can be used to predict futures or reason through problems.

• 27:17Craig S. Smith?

  If, if you don't have that, it, it, it seems that it would be much harder to build a foundation, robot foundation model that can generalize and operate in, in the wild. I mean, right, right now, in your lab, in, in a lot of the industrial deployments, they're very controlled environments and, and the robot's training is very task specific. But if you want a more generalized model, where, where does, where do world models fit in?

• 27:55Speaker 01

  Yeah, it's a really interesting question. I think that some folks tend to present world, like, I guess we should, we should nail down what we mean by world models. Typically, what people mean when they say world models is some kind of predictive model that operates at the level of raw observations. It doesn't mean that it predicts for observations. It may be that it is like, you know, like Yan Le Cun, for example, I know he advocates for essentially a latent space world model, which predicts a sufficient statistic of observations. But roughly speaking, it's something that predicts something about your future observations. It's a very reasonable idea, but I think that something that we should keep in mind is that. For human behavior, like prediction definitely plays a role.

• 28:36Speaker 01

  But there are also things that we do that are not grounded entirely in prediction. There's a place where prediction is easy and there's a place where prediction is difficult. And the abstractions that we use are really critical to intelligent behavior. So to give you an example, like if I want to figure out how to get from where I am now in San Francisco to New York City. Maybe I'm going to imagine something about it. Maybe I imagine how I get my car keys and I get in my car. I might imagine that I'm taking an airplane. But the further out that I think about this, the more abstract that imagination becomes. I'm not imagining exactly what my seat in the airplane is going to look like. So abstractions are really key to actual effective world model.

• 29:18Speaker 01

  And I think at some level, what language models do, what vision language models do, and what video prediction models do and other kinds of world models is not actually that different. They're just operating with different abstractions. And I think for a real capable embodied intelligent system like a robot, we'll need many different abstractions. And I suspect that once we figure out how to use abstractions in general, like that that general part of the question is actually the important one. And then we'll use kind of the right kind of thing at each level and that'll be fine. So I guess what I would say is that like definitely I'm very sympathetic to the world model's view, but I also suspect that the dichotomy between language models and what people today call world models is not as large as some people may see.

• 29:56Craig S. Smith?

  Yeah, so so with a good world model,

• 30:01Craig S. Smith?

  there isn't anything that a robot couldn't do with being trained through

• 30:08Craig S. Smith?

  with a vision language action model. It's it's just, yeah, go ahead.

• 30:14Speaker 01

  I suspect the key is to have a system that correctly uses the right abstraction for the job. So if you are, you know, folding a piece of clothing, probably, you know, that's clearly not a semantic task. You're not thinking in words about like, oh, this fold is here, this fold is there. But you're probably also not imagining literally how all the particles of clothing move. At some level, especially if you're good at the skill, you're mostly kind of using your muscle memory a little bit, but it's it's reactive, it's using perception, but it's it's not as model based as just like imagining exactly how every particle will move. So the really cool thing about proficient human motor skills is that they blend prediction and this kind of model free behavior and semantic reasoning. It all kind of comes together with the right tool for the job at each level.

• 31:00Speaker 01

  And I think that kind of blending is really, really important.

• 31:02Craig S. Smith?

  So do your models actually imagine possible futures before acting?

• 31:09Speaker 01

  Right. So the way that our models work is they actually perform inference at different levels of abstraction. I think there's still a lot of research to be done to figure out exactly how that should be done and what abstraction should be used and where. Like, you know, right now, I would say that our choice of abstraction is quite naive. But you can reason semantically about higher level things. Like if you're cleaning a kitchen, do you pick up the plate or do you pick up the towel first? You can figure out where things go spatially and then you can produce the actions. So there's already a little bit of that multiple abstractions. But I think there's a lot more research that needs to be done to get the abstractions to emerge automatically and to automatically figure out the right abstraction for any given stage of task.

• 31:46Craig S. Smith?

  Yeah.

• 31:50Craig S. Smith?

  So world models, in your view, are not necessary for generalized robots that can operate in messy environments?

• 32:00Speaker 01

  I guess what I would say is not that it's necessary or not necessary, but that it's not as much of a dichotomy as I think that some folks tend to present. Like, I think that the view that I would disagree with is that there is like model-free policies, reinforcement learning, VLA's, and world models. And these are like totally separate things. I actually don't think they're separate things. And I think that the right answer will be a model that can do all of those things together, that can use world model-like prediction when it's necessary and can do the model-free stuff when that's more appropriate and smartly decides what kind of abstraction is the right abstraction to use for a given problem.

• 32:32Craig S. Smith?

  Yeah. Your models, the AI brain, so to speak, is not on the robots. It's communicating with them remotely. But when you send robots out into the world, you really can't afford, as with cars, autonomous vehicles, you can't afford for them to be waiting for instructions from the cloud. So how much of the models that you're building can be on-device?

• 33:12Speaker 01

  Yeah, that's a really interesting question. So, you know, so far, we're still very much in like the research and development phase.

• 33:19Craig S. Smith?

  Yeah.

• 33:19Speaker 01

  We haven't had to worry about this very much. And you're completely right that currently the models actually live on the cloud. They've actually run through an inference API that looks very similar to what someone might imagine using for an LLM. But in the long run, I think you're right that there needs to be an on-device component that is reliable, that is not vulnerable connectivity issues. Generally, I think that the way to move towards this, which I think is already reflected in current models that we and others have been developing, is to have a system that

• 33:52Speaker 01

  performs multiple types of inferences in parallel at different levels, where the highest levels are maybe more appropriate to offload to a remote inference server, and the lowest levels, the ones that are really doing motor control and closing a loop very tightly on perception, run locally. Now, the good news is that the lowest levels are probably also going to be the smallest ones in terms of the number of parameters, because they're not, you know, you can sort of think of these as like instinctual reactions, reflexes, that sort of thing. Like, these are things that are very important, they need to be very fast, but they also are not as cognitively demanding and not as complex, so they can run locally. Now, the trick, of course, is figuring out all that communication so that you still preserve the benefit of end-to-end training, so this whole thing is trained together to

• 34:35Speaker 01

  act in concert, but at inference time can be partitioned in this way with different size components running in different places. And it's kind of cool to imagine, like, you know, you have good internal connectivity, you get a lot of intelligence. Your internal connectivity starts to degrade. Okay, maybe the robot gets a little dumber, like it has to, you know, maybe pull down some nice inferences from the cloud, keep them locally and do some stuff, and then, okay, now it's time, let's stop and think again. Let's ask the cloud to think some more.

• 35:02Craig S. Smith?

  So, right now, you're not concerned about shrinking models to fit on-device? The first step is to get... A generalized model and then you'll do the distillation or whatever is required to get it on the device. Is that fair?

• 35:22Speaker 01

  So that's right, but even separately from any inference concerns, we do actually, we've sort of, I guess, conversion evolution in some sense. We already have models that have this kind of multi-scale property where the lowest level motor control components are already smaller. So while we're not so worried yet about running things on device, the natural trajectory of that technology is leading to a place that makes that actually reasonable straightforward.

• 35:46Craig S. Smith?

  Yeah, so I have some questions about generalization, but before I get to that, I get very confused about all the different kinds of ways that people are training robots. So there's vision language action models, but there, you know, there is one shot, two shot, few shot, vision, language, action, world models. What can you sort of give me a little primer on the different ways that that people are training robots?

• 36:28Craig S. Smith?

  And why you've settled on VLA's and RL?

• 36:33Speaker 01

  Yeah, yeah, let me think about how to best describe this.

• 36:39Speaker 01

  So I think it's, it's often hard to get like a complete picture of the robotic learning world. Just by looking at the kind of results that people present, there's kind of one general truism of the robotics demo, which is robotics demos can be set up in such a way that shows something really cool, but doesn't actually provide like a general solution to a problem. Because if you want to just like stage your robot demo, you kind of make things work in that one setting. So because of that, it's like a little hard to figure out what are like the big clusters of major effective techniques. But if I were to like very soberly look at the current robotic learning environment, I would say that there are actually like two big things that work very well.

• 37:17Speaker 01

  One thing which we've been discussing is vision language action models and more generally this idea of like learning manipulational skills and things like that from data. Typically it's with imitation learning, but it can also be with reinforcement learning insofar as it can leverage that data. And oftentimes actually the recipe is like use imitation learning to initialize and then maybe like fine tune it with RL. And then the other big cluster is Sim2real transfer using simulation to learn motor skills in a sufficiently randomized way and then run them in the real world. And when you see demos of like, you know, robots doing like, you know, dancing and acrobatics and all that stuff, that's typically doing sim2real. And it's kind of a funny unstable equilibrium of the current research world that these two

• 37:59Speaker 01

  types of techniques are very different and they are also used to attack very different domains. So the visual language action models, they're basically currently the dominant paradigm for robotic manipulation. Problems where robots need to interact with diverse environments and diverse options. The sim2real stuff is the method of choice for highly acrobatic and athletic movements, typically for humanoids. And these are situations where the task is physically extremely demanding, but the diversity of the environment is very low. So if you put the robot on stage and it's dancing, you're not really worried about the shape of the stage, you're just worried about the robot's body. So this maybe tells us actually a lot about the strengths and weaknesses of these two

• 38:39Speaker 01

  types of approaches, that the sim2real stuff is great for really understanding the physics of the robot, but not so great for generalization. The visual language action models are great for generalization, but of course, because they're dealing with the physical world, the real world data, they can't run these like giant RL loops that practice for billions of trials and really overfit to the particular body of the robot. Now, of course, there's a lot more to do in the future, but this hopefully gives you some sense for the layout.

• 39:06Craig S. Smith?

  And VLA, just walk me through the process of where the vision comes in, where the language comes in to get to the action.

• 39:17Speaker 01

  Yeah, so maybe one way I could describe this is to start with visual language models. These are also sometimes called multimodal LLMs. So if you use like Gemini or ChatGPT and you upload an image and you ask some question, that's basically a VLA. And the way that these models work today is you start with a language model. A language model is very, very simple. It's a transformer that takes in text and predicts future text. And to get these things to Process images. What we do is we train a vision encoder, basically another little piece of neural network that takes an image and puts it into the same space as the language tokens. It's sometimes called a vision encode. So now you're going to feed images to this thing the same way that you feed language,

• 40:01Speaker 01

  because this little like virtual visual cortex kind of takes the images and turns them into, you know, semantic like looking thingies that the language model knows how to process. So now with VLA's, well, with first generation VLA's, I mentioned, people basically just took exactly this model and just changed the output to literally like output numbers in text that represent actions. The second generation VLA's, which is what everyone's basically using today, they take inspiration from how VLMs add a visual cortex to the language model, and they also add a kind of a virtual motor cortex, a specialized little piece of circuitry whose job is to take the outputs from the language model backbone and decode them into continuous actions. And this is typically done with diffusion, basically the same kind of technology that's

• 40:45Speaker 01

  used to generate images and videos is now used to generate trajectories of robot joints. That makes a lot of sense because trajectories of robot joints are a continuous spatial object, the same way that images are continuous spatial objects. So it makes sense that the same technology would be applicable there. And I think this is really cool because almost like what we're doing is like building a brain piece by piece, like there is this, the language model backbone, I guess kind of like a prefrontal cortex almost, there's the little visual cortex part that encodes images, and now there's a little motor cortex part. Anyone familiar with biology would be very offended at this point because it's backwards, right? Evolutionarily, the motor cortex comes first, then vision, then prefrontal, but here it's the other way around.

• 41:24Craig S. Smith?

  Yeah, and then to get the continuous action, it's outputting a stream of numbers that are telling the actuators.

• 41:36Craig S. Smith?

  How far to move or or right and it's and the control theory comes in at at the end. Is that right? You're you're outputting.

• 41:47Speaker 01

  Well, and I should be very upfront about this. The amount of control theory in this is like pretty minimal. So currently, the way that most of these systems work, including ours, is that the model outputs a short trajectory of future target joint angles. So roughly like half a second of joint angles to to hit. And the actual actuation, the the motor commands to reach those joint angles are computed with a very simple like penis controller. This is kind of like, you know, the most basic type of controller you could put on a robot car, basically.

• 42:22Craig S. Smith?

  Yeah, you know, past robots were often built to do one thing really well, particularly, I mean, certainly before.

• 42:33Craig S. Smith?

  You know, they were, AI was involved. How realistic is is it that that one robot?

• 42:45Craig S. Smith?

  With with one of these models controlling it can can do many different things.

• 42:51Speaker 01

  Yeah.

• 42:52Craig S. Smith?

  And or or and I had a conversation with a guy, interesting guy, maybe you've run across him. Mike LeBlanc, I think his name is. He's got a startup called Foundation and he's doing humanoids for the military and and they're training them to do one thing like, you know, put a

• 43:18Craig S. Smith?

  explosive charge on a door, which is a very dangerous thing for a soldier to do. So they drop it from the Humvee, it walks, slaps a thing on the door and comes back, hopefully in one piece.

• 43:34Craig S. Smith?

  And that made a lot of sense that that rather than trying to build this generalized model, just train it to do one thing, but

• 43:46Craig S. Smith?

  but how much generalization is is developing from from yours and and other robots in the field?

• 43:54Speaker 01

  So here's maybe how I I can try to answer this question. I think every time that people try to take robots out of the factory and into open world environments.

• 44:06Speaker 01

  They very quickly realize that in the real world, there's a huge range of things that can happen. And it's very, very hard to get a system that is highly specialized and still robust enough to work in the real world. This lesson was probably learned very, like, I don't know if it's for the first time, but in an early instance of this lesson was actually autonomous driving. Like, in the very early days of autonomous driving, some people thought that, well, OK, driving is complicated. There's lots of stuff that could happen. But if we just like an instrument the road and like prevent other things from getting into like the autonomous car lane, maybe then things will work like well, still like magnetic sensors and all that stuff. And that really never took off because essentially the gap between a closed world and an open world is enormous.

• 44:50Speaker 01

  And you can't, it's like you can't be just like a little open world. Like, as soon as you're you're out in the wild, immediately stuff can happen. Maybe it happens rarely, but that's not, that doesn't save you. Like, even if it happens rarely, you have to deal with it. So here's an example from our work. We had a project on using our systems to assemble boxes. So you take like a flattened cardboard box and you have to like build it up, fold it. And it's like a little bit like an origami problem, basically. You think that's pretty structured, like it's one thing, just build a box. But sometimes you grab the boxes off the pile and you get two boxes instead of one. So you have to put one in the back and maybe there's someone, something is torn a little bit. So you have to discard it because it's torn. And maybe someone left like their phone on the table. So you have to put the phone away. So there's so many things that happen, even like, once you cross that boundary from the factory to the real world,

• 45:35Speaker 01

  that you can't actually have a narrow specialist. That even if what you want is to do one thing, you have to handle all these other things that can happen all around. And the generalist model, the one that could handle a wide variety of tasks, actually becomes a better specialist because it can deal with all that weird stuff that arises. So I think that generality is really essential, even if you really want to do one thing. And I think that's a lesson that's been learned time and time again in robotics.

• 45:58Craig S. Smith?

  And are you going to move into humanoids or have you?

• 46:03Speaker 01

  Yeah, that's a good question. So the main reason why we've stayed away from humanoids ourselves for the data collection that we're doing here is actually, it's not ideological. It's just because humanoids are expensive, complicated, and teleoperating humanoids is harder. So we've basically stuck to robotic platforms that can give us the most data, the most data diversity. But I think humanoids are really cool. Like, we've worked with other companies that have humanoids, the model runs on humanoids, and I think that, you know, it's something that I'm sure will want to explore more in the future. My own take when it comes to robot morphology, though, is that I think that, you know, this is maybe like a little bit idealistic, but I, I actually really hope that robots will kind of end up being a little bit like personal computers, where there's like general software,

• 46:48Speaker 01

  and the form factor of the device can be very different for different jobs. So when you get a computer, maybe you get a laptop, maybe your computer is your cellphone, basically, maybe it's a big desktop, maybe it's a big server machine, it's like whatever is right for your job. And I think robots will be the same way, that maybe like if you live in a small apartment in New York, maybe you have like a little home robot that kind of attaches to the ceiling and pivots around and cleans up the thing. And maybe if you, you know, if you live on a farm or something, you have a big mobile robot, you know, tractor with a bunch of arms attached that can drive around and do stuff. And maybe there will be one software stack that can accommodate these things with different applications and prompt engineering that people do for particular domains, and there will be this heterogeneity of platforms, like physical intelligence everywhere.

• 47:28Craig S. Smith?

  Yeah, well, how does, how do your models work on humanoids? Because your, your models have not been trained on operating legs and walking.

• 47:40Speaker 01

  Yeah, so typically, right now, when we work with folks that have humanoid robots, they have their own software stack that takes care of balance and all that, and all those things. And then the model drives basically the manipulation behaviors.

• 47:51Craig S. Smith?

  Yeah. And, and what you were just saying, do you think we'll end up with an iPhone of robots that everyone uses? And maybe they're specialized robot bodies for, for different tasks? And would there be, do you think there will be a dominant model?

• 48:18Speaker 01

  Yeah.

• 48:21Speaker 01

  I don't know, but but I think that one like one vision of this that's very appealing to me is that, you know, with robots, we get to build them, we can build them however we want. And I and I think that's one of the advantages of of robots is that they can be designed not to replace people, but to do to do things differently from people. And so you could design form factors that are suitable for particular domains. They can be much smaller, much bigger, they can have, you know, five arms, seven arms, one arm, whatever is like appropriate for the the cost point that you have, whatever is appropriate for your application. And people can be pretty creative about it. It's just the thing that's been preventing folks from experimenting with these things is that if you want a robot to do anything, you have to solve the intelligence problem. And if there isn't a ready made off the shelf thing that at least gives you like a rough prototype,

• 49:06Speaker 01

  you just can't get started. So what I really hope is that good robotic foundation models will provide that kind of middle layer. You know, in a computer, that layer is basically taken up by the operating system. The operating system is the thing on top of which you put applications. So if there's an intelligence layer that's pretty general on top of which somebody can essentially prompt engineer their task, you know, they can experiment with all the aspects of that application, really nail it correctly, design the right form factor, then we'll see a lot of experimentation, a lot of creativity. And then I think we'll actually see what robots should really look like instead of just thinking of them as like a metal version of a person.

• 49:39Craig S. Smith?

  Yeah, yeah. And on the humanoid question, I was asked to to sort of write a brief for somebody late last year or mid last year about humanoids. And I wrote this very bearish thing about how, you know, all of the different problems from, you know, dexterity in the hand to to models that can deal with unpredictable environments and things like that. But since then, there's been this explosion of humanoids. And everyone's talking about humanoids. I mean, did I get it wrong? Do you think that humanoids are much closer to being in the world than than I than I thought?

• 50:27Speaker 01

  It's a question. Um, I mean, I think it's it it I think there's a good reason to be excited about humanoids. And like in the sense that like just emotionally, this really kind of captures the imagination. It also makes it easier for people to think about because like we know what people can do. So if we kind of build a robot that does what people do, that like kind of makes sense. But I do think that it's a somewhat limited view to restrict ourselves just about.

• 50:52Craig S. Smith?

  Yeah.

• 50:52Speaker 01

  My my sense is that humanoids will will have a place. Like, I think there will be humanoids that do something that the people find useful. But I think there will also be a place for all sorts of other things. And, you know, as with any new technology, oftentimes, what's most important to figure out the right way for for for it to exist in the world is the tools and machinery and structures to allow people to experiment, to prototype, to try out all sorts of different things. And I think the trouble with robots is that just like the barrier to entry for serious, open world robotic systems is extremely high. Like, you have to solve open research problems just to get your prototype out the door. And I and I think that's kind of what's actually limiting a lot of creativity.

• 51:30Craig S. Smith?

  Yeah, and and for example, with Boston Dynamics, but they're they in the past, they were pure control theory. But now they've they have an AI partner and they're they're using AI models to control the robots. And as I said, there's, you know, there's been a lot of talk of them being used in factories. I think Hyundai is is deploying them.

• 52:02Craig S. Smith?

  And I understand the the argument that, well, it's the human form factor. The world is built for that. And if we can have robots operate in in that form factor, with that form factor, there's it can they can presumably eventually do everything that humans can do. Uh, how much of this do you think is hype? It's very hard for a layman when you when you watch videos or, as I said, the 60 Minutes piece to know how real it is and how much is our sort of fascinating demos that in

• 52:42Craig S. Smith?

  two years, people say, oh, yeah, yeah, we had that pilot, but it didn't work out.

• 52:48Speaker 01

  So I have two things I can say about this. The first is, I think in regard to demos, I mean, I think you're right that for for a demo, it's like, there's a big difference between setting up a demo and setting up something that works. And, you know, one thing that I've struggled with in my career, I think, quite a bit is that, you know, I work on robotic learning and robotic learning, the purpose of it is to get systems that can generalize and work in open world environments. But it's very hard to illustrate. Like, it's if you show somebody a demo in a particular setting and a robot does something cool, like, yeah, it's obvious, like, cool stuff going on. If you show it doing something fairly simple, but in like 100 different environments, well, then each of the videos of that is just a robot doing something simple. So the fact that it can do it in all these different

• 53:31Speaker 01

  settings is harder to convey. And that's kind of a science communication challenge that I think we have to be cognizant of when we look at the videos. But I think there's a deeper, more technical thing I want to say about humanoids and about robots in general, which is that conventionally, when somebody thinks about building a really cool robotic system, they naturally start with building the physical robot. So if you start from the premise, I'm going to build one robot, it makes sense, like, if it's going to do something general, it should be a very general body. And you should really get that right, because once you've committed to like that form factor, like, you're kind of stuck. So I think there's actually a technical point here, which is that the old way of thinking

• 54:12Speaker 01

  about robotic software as something that drives one robot, kind of naturally leads to that. Because if you want one very general robot, then you kind of have to like have it have everything. But if you accept the premise that we're going to have robotic foundation models that can drive lots of different robots to do lots of different things, now it kind of unchains your thinking. And now it's okay to experiment with different form factors, experiment with different applications, different ways of approaching things. It's just that you need this kind of general AI system to be able to do that. And without it, it actually kind of makes sense that you would go into this, you know, single ideal, perfect body plan. And maybe a humanoid is a good choice for that.

• 54:48Craig S. Smith?

  Yeah. And you've made tremendous progress since I first met you. What's on the horizon? What are you guys working?

• 54:59Speaker 01

  So for me, one of the things that I really want to figure out over the next year, or maybe the next few years, both in my academic work and in the work here of physical intelligence, is how to go from the foundation, which I think at this point, we have a pretty good understanding of, to something that is a true data flywheel, a true continual learning system, where the robot experiences more and more tasks, and the more it experiences them, the better it gets. And that involves, I think, a lot of autonomous learning, a lot of reinforcement learning, a lot of learning from other supervision signals like these language feedback that I mentioned before, and turning that into a continuous cycle, where the more the system is deployed, the more capable it gets. And I think there's some very deep technical challenges there that'll stress kind of all aspects of this. So that's what I'm most excited about.

• 55:42Craig S. Smith?

  Yeah. Okay. Sergey, I'm gonna ask one last question. I asked Fei-Fei Li this and David Ha and some others. What's your guilty pleasure? Do you have one?

• 55:56Speaker 01

  Ooh.

• 55:58Craig S. Smith?

  Something you do to relax that you know is kind of silly or a waste of time, but you enjoy it nonetheless. I remember you were a big video game guy.

• 56:13Speaker 01

  Yeah, I actually, back when I was in college, I actually thought that my career would be making video games. I think these days is probably science fiction. I'm a big science fiction nerd. I made sure to put like a little epigraph with quotes from science fiction stories that I like into a lot of the papers that we've been putting out.

• 56:30Speaker 01

  I like that very much because I think it's uh it's just it's very good, I think, especially for somebody working in science or engineering. And this is maybe the bit of advice that I'd have for folks listening to this too, is that it's good to not feel too constrained. And sometimes the rigor of doing serious engineering and scientific work forces into forces into very constrained thinking. So it's very important, I think, to try to get out of that mindset sometimes, not to get too carried away with it, but it's good to like, let yourself think things that are maybe not not good to do in polite circles in science and engineering, to think about things that are more out there that are more fanciful, more creative, or more improbable, let's say.

• 57:07Craig S. Smith?

  Yeah. Is there a favorite work of science fiction that you've read that you would advise people to read?

• 57:16Speaker 01

  When I was younger, I didn't read very much Robert Heinlein. So I think I only discovered Robert Heinlein's work in the last few years. So I've been really getting into it.

• 57:24Craig S. Smith?

  Yeah, that's wonderful.

• 57:25Speaker 01

  Classical American siphon, yeah.

• 57:28Craig S. Smith?

  That's right.

• 57:29Speaker 01

  That's very optimistic aspect of the of American culture that I that I think is very refreshing, especially in today's day and age.

• 57:35Craig S. Smith?

  Yeah. Okay, Sergey, well, I hope we have a chance to talk again. And I'll see you at the conferences. And this has been fascinating. Most AI is just speech to text plus a language model. It's full for reading transcripts, not understanding conversations. Velma from Modulate, an AI built on ensemble listening model architecture specializes in audio analysis. It orchestrates hundreds of smaller submodels, purpose built to understand the nuances of voice like tone, timing, stress, and intent. Perfect for fraud defense,

• 58:15Craig S. Smith?

  deep fake detection, agent attrition prevention, or customer service moderation. Check out the live Velma preview at preview dot modulate dot AI. That's preview dot modulate dot AI to see how the model breaks down audio, providing timestamped, explainable signals. Stop transcribing, start listening with modulate dot AI.