Ep 70: Karol Hausman and Danny Driess (Physical Intelligence) Unpack the Most Recent Breakthroughs & Path to Generalist Robots

In this episode, Jacob sits down with Karol Hausman (Co-Founder) and Danny Driess (Research Scientist) from Physical Intelligence, two of the minds behind some of the most exciting advances in robotics. They unpack the last decade of progress in AI robotics, from early skepticism to the breakthroughs powering today’s generalist robot models. The conversation covers everything from folding laundry with robots to building scalable data pipelines, the limits of simulation, and what it’ll take to bring robot assistants into everyday homes. It's a wide-ranging and thoughtful look at where robotics is headed, as well as how fast we might get there.

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Jacob Effron?

  Physical Intelligence is one of the leading AI robotics companies. They've raised over $400 million and released a slew of fascinating research. Today on Unsupervised Learning, we had Carl Hausman, their CEO and co-founder, and Danny Drace, one of their key researchers, talk about everything AI robotics. This is a particularly fun one to do with a red plane portfolio company, and we hit on a bunch of different things, including what works and what doesn't in the robotics world today. We talked about what the key challenges are ahead for scaling these models and when they'll be widely distributed. And then we also talked about what it takes to run an effective research lab. This was the best conversation you could have in robotics with two of the best minds in the space. I think folks are really going to enjoy it. Without further ado, here's our chat. Well, thanks you both so much for coming on the podcast. Really appreciate it.

• 0:41Speaker 01

  Thanks for having us.

• 0:42Jacob Effron?

  Thank you so much. Yeah, I've been looking forward to this. You know, I figured I'd start in the broadest possible way, which is that I feel like the two of you have been involved in like many of the seminal advances in AI robotics over the past decade. And obviously, you know, things like RT2, POMI, what you're doing now at Physical Intelligence. And I think our listeners would love if you could kind of just contextualize, what has this last decade been like in AI robotics and how you think about the progress and milestones?

• 1:07Speaker 01

  So I've been in robotics for something like 15 years or so.

• 1:10Jacob Effron?

  All right, you can do 15 years too. What was it like day one when you came?

• 1:14Speaker 01

  Were pretty boring. It was, it was really hard. There was lots of ups and downs, but I never felt that there was a light at the end of the tunnel. It always felt that the problem is way too hard. And there were all kinds of ways to improve it, maybe for a certain task or certain problem. But you never, I personally never saw a path to like actually solving it fully and having robots out there doing all kinds of different tasks. That changed recently, and I'm happy to talk about it. But I think basically if I look back, the problem I saw back then is that it seemed like the world is just so, so complex that it's not possible to write code to let the

• 1:57Speaker 01

  robot operate in that world. And at the very beginning, like when I started, that was the predominant path to solving robotics. We'll just write the code that will explain to the robot what it sees. It will control the robot and move its limbs to accomplish tasks. And we will do it task by task, and eventually we'll have big enough of a library of these primitives or of these tasks, and we'll be able to handle this diversity of the world. And then sometime, I don't know, maybe seven years ago or something like that, uh, maybe ten, we started uh seeing these new wave of approaches showing that you can't really code the robots to do these things. You can just like sit in your armchair and figure out all the different rules that apply.

• 2:41Speaker 01

  And then you need to learn it. They need to learn it the same way as as we do. You know, um, they need to learn it from their own experience. They need to experience the world and they need to get the understanding of the world through learning. And initially, when it when it was starting, a lot of people were were questioning that, you know, it's it's kind of uh it's unsatisfying in some ways because it's not on us anymore. Yeah. It's more on the on the algorithms on the robots to figure these things out. Uh, and there's a lot of things you lose. You don't understand exactly how the robot is doing the thing or why is it doing the thing? Um, so there was a lot of pushback initially, but then over time, I feel like we're constantly keep on moving towards making it more learning based and more end to end and more learn from experience.

• 3:25Speaker 01

  And then in the most recent history, which was which has been the most exciting to me, there's been many advancements over the past five years or so that that really started pushing the needle and showing that these methods can can really work. Some of these we were involved in, maybe, do you want to talk about Paul me?

• 3:40Jacob Effron?

  A good job. And I think also maybe contextualize like how many of these advances were directly tied to the advances that were happening in, you know, transformer based models and how many of them were maybe independent of that?

• 3:48Speaker 02

  I think one thing that is very different from what it how it was in the past. In the past, it was that people were trying to solve a certain robotics task, like one, let's say, like

• 3:58Jacob Effron?

  Like a Rubik's Cube maybe from a certain foundation.

• 4:00Speaker 02

  Exactly. And then you developed a machine learning algorithm specifically for that. And then you you've done the project, you achieved a certain amount of success rate and then you moved on and you developed a new thing, right? And then when large language models started, it seemed that you can train a single model on many, many things at the same time. So the the question is, can you replicate something like this in robotics? So can you train one model on many, many, many tasks at the same time? And so that you have one algorithm that you that, you know, the the more data you collect, the more tasks it can solve. And that is what I think really Has changed. And so maybe one of the first projects that we've been doing was this Palmi.

• 4:38Jacob Effron?

  Yeah, was it immediately obvious? You were both at Google at the time, right? Was it immediately obvious as this progress was being made, you know, on the language side that like this would be applicable to robotics? Or was there a lot of early data experimentation there?

• 4:50Speaker 02

  So if you just tried those models directly on robotics, they didn't work because they like language models didn't have any vision, right? They couldn't perceive the world. So that in in Secan, there was a way of grounding the model to the real world by having these controllers that could tell the model which can you run them or not?

• 5:10Jacob Effron?

  Right.

• 5:10Speaker 02

  And then in Palmi, we directly integrated vision into the model and trained it on robot data and also other data sources to such as the model directly can perceive the world and make make decisions in that. What is very, very exciting there was that, you know, these people always say these are large models, but and large models require so much data, but these models are very data efficient in certain ways because to learn a very hard task, if you have already good amount of pre-training, you actually don't need that much data, much less than if you had not a powerful backbone from which you can build upon. Yeah. So and I think that maybe was a little bit an exciting moment when we did.

• 5:50Jacob Effron?

  Yeah, I guess it was kind of unclear whether it would just require, you know, such an amount of data that would that would be completely inaccessible.

• 5:56Speaker 01

  Yeah. For for me, one other moment like this was this RT two moment that where we run this experiment. That the the specific version of the experiment that was really exciting that that people know about is when we asked the robot to move the coke can to.

• 6:12Jacob Effron?

  Yeah, yeah.

• 6:12Speaker 01

  Picture of Taylor Swift. And when you look at that demo, it's actually quite pathetic. I can see this robot with a single arm like barely grasping a coke can and moving it like barely on the picture of Taylor Swift. I think it tumbles over. It's just like, it's not very good. But I remember when we saw this, it was extremely eye opening. And that was one of those moments where it was like clear, like, holy shit, this is this is gonna work. And the the thing that was in particular really exciting about this is, as Danny was saying before, our idea was that to to make these models work the same way for robots, we would need to collect the same kind of data sets that people have for language. But there is no.

• 6:50Jacob Effron?

  You need the internet.

• 6:51Speaker 01

  You need the internet of of robot data. It doesn't exist. And that would take so much money and time and everything to to do that. It's basically not possible. And what that result showed is you can actually take a pre-trained vision language model that has been pre-trained on internet data, has this web-scale understanding of what's happening in the world, how the world works, and then add a little bit of robotics data, and these two connect.

• 7:15Jacob Effron?

  Yeah.

• 7:16Speaker 01

  And now the robot, even though it's never seen a picture of Taylor Swift before, it doesn't know what Taylor Swift is, it doesn't understand that concept, now it's somehow it can connect that concept that it got from the internet to the motion that brings the coke and to the picture of Taylor Swift. So no longer the robot needs to experience everything firsthand, they can get some of that knowledge from the internet.

• 7:36Jacob Effron?

  Yeah.

• 7:36Speaker 01

  And that was-

• 7:37Jacob Effron?

  Basically getting perception and reasoning like from the base model and then adding kind of the actuation side.

• 7:41Speaker 01

  That's right, and you can connect it to motion. You can make this like very sophisticated connection between these two different parts that are very, very different, and the model can do it implicitly. And that was super, super exciting.

• 7:52Jacob Effron?

  Yeah.

• 7:53Speaker 02

  I actually remember you, I think you were writing at the time, you know, like, oh, we will all remember that moment when this happened. The first thing was actually grasping a fish, because there was never a fish in the robot dataset. I think it was a fish, like a toy fish. This worked, like, the first time we tried it was like...

• 8:11Speaker 01

  You can see how pathetic the tasks are.

• 8:14Jacob Effron?

  Yeah, we've gone to cooler ones now. I mean, I'm curious, so obviously you were doing all this really interesting and innovative work at Google. When did you kind of decide, hey, this makes sense to, you know, do separately as a company or spin out?

• 8:25Speaker 01

  Yeah. There were a few moments throughout where like things start to work and I started thinking a little bit more about where that could lead and what would require to actually make it work. And one thought that I had is, you know, Google is an incredible organization, they do incredible research, all kinds of different things, but I think this problem is so large and so complicated that to truly solve it, you need an organization that is fully optimized just for that. Whose sole goal in the world, the reason to exist is to solve the problem of physical intelligence. And that became very, very clear to me, very apparent.

• 9:06Speaker 01

  And that was the moment where, you know, we still need to figure out if it's actually possible to start a company like this, that will have enough resources, can attract the right people and so on. But when it turned out that it is possible, it was very, very clear that this is necessary to solve the problem.

• 9:22Jacob Effron?

  Yeah, and obviously you've had quite the journey since then. You know, I think it maybe would be helpful for our listeners, can you contextualize, like, where are we today? So obviously, you've made a lot of advancements in the last nine, 12 months. How do you kind of categorize what works and what doesn't work today in the robotics foundation model world?

• 9:39Speaker 01

  Yeah, so when we were starting, we're thinking of three axes that we thought were really important for these models. One was capability. Can we get these models to do tasks that we've never seen robots do before, that are really dexterous, really long horizon, really complicated? Then we thought that the next step would be generalization. How can we get these models to truly generalize the same way as language models do, that you can ask them something, or put them in a situation they've never seen before and they could act correctly? And then the last one is performance. Can they perform as well as humans can, with very high success rates, really robustly? So we started with the first one, with capability. And within the first five or six months in the company existence, with the Pi Zero released,

• 10:20Speaker 01

  we showed that you can get these models to perform extremely dexterous tasks, like laundry folding, building boxes, bussing tables, all kinds of different things. You can do it across a wide range of robots. And it really works. You can get it to the point where if you can tell operate something, you can probably get a model to do it. Then next, we started tackling the generalization problem. So can we get these models to work in situations they haven't been before? And that was our most recent release, Pi 0.5, where we showed that you can bring the robot to a new home that it's never seen before. And we pick a home because homes are the most diverse environments, so it's kind of like the hardest, the hardest version of that challenge. And it kind of worked. You can bring it to a new home and it can do long horizon tasks, like clean your bedroom,

• 11:05Speaker 01

  or make your bed, or put dishes away, and things like that. It doesn't work every time, but it works some of the time and we kind of started getting better understanding of what it takes to get these models to generalize better. And now the biggest challenge is performance is the third one. And that one is, I think, a little bit more uncertain. I don't think there is like a very clear solution where you just get more data or you need more diversity or something like this and it's going to work. I think it will require some algorithmic work, some new ideas. But that's more or less where we're at today. We need to get these modes to perform much, much better. They still fail a lot. And they're more like demo ready than they are like actually deployment ready. But I think we'll get there too.

• 11:49Jacob Effron?

  Yeah, well, I will say, I mean, I think watching some of the videos, the performance on like shirt folding seems better than I can do already. So it's already, you're already off to a, to a good start. I mean, how do you think about the performance threshold you even need to hit? Cause obviously I think one of the promises of this is, you know, you could go to sleep and in the background like these things are happening. And yeah, maybe it takes them 10 times longer than it might take me, but like, that's actually a fine outcome.

• 12:08Speaker 01

  Yeah, I think it probably will depend on the task. There are some tasks where it really matters how fast you are, how accurate you are, things like that. There are certain industry where, you know, failure is not acceptable. And there are some other tasks where it's totally fine and you can, you know, you can take the whole night and, you know, if that's what it takes to fold your laundry, you'll still be happy in the morning. I remember I had this one moment in the office where we have bathrooms right next to robot stations. And there was robot folding a shirt and I was looking at it. And as I was walking to the bathroom, I was looking at it and kind of being mesmerized a little bit, how it moves and how it does everything. And I was a little disappointed that it's slow and it still makes mistakes. It was kind of like still the early version of the model. And I went to the bathroom, I come out and I come out and I see a full pile of folded clothes.

• 12:53Speaker 01

  And I was just like, wow, like, like it doesn't matter. Right? Like some labor was, was done, like it came, it went from this chaotic, you know, pile of shirts. That was just like in a, in a basket, you know, in all kinds of different configurations. So now like fully folded pile of laundry. And that was kind of mind blowing. That was kind of like the moment where I like internalized that. Yeah, like for some tasks, it doesn't matter. You just need to get it done.

• 13:20Jacob Effron?

  I think what's impressive watching so many of the demo videos is even when they do fail in some way, like they're actually like quite resilient in figuring out.

• 13:27Speaker 01

  They're much more human like than than the robotic things of the past where there if you make a mistake, you kind of you can't recover. Like you have to be extremely precise. You need to go to the exact point where you're supposed to go and you just go there as fast as possible. These models and the way that that robots look when they use these models is much more human. They make mistakes, but then they correct. They, you know, will redo the task if they don't think they're good enough. Sometimes it's frustrating. Yeah, they're it's a very different kind of of robotics.

• 13:58Jacob Effron?

  Yeah. Well, I'm gonna ask you what I think is probably the most common like layperson question about robotics, which is. Everyone loves to draw the comparison to self driving cars. And they're like, is this is this gonna be the same journey as the last bit, the hard part? You know, that comparison, what works about it, what maybe is is not correct?

• 14:14Speaker 02

  So both of them are very hard problem and both of them, if they are being solved, have a huge impact on humanity. I think like if we can make a little bit of thought experiments, if you removed all humans and all other cars in the world. Autonomous driving would be rather trivial. I'm saying it's a hundred percent trivial, but it would be rather simple. For robot manipulation, if we remove all humans or all other robots, the problem would be as hard. And so an autonomous vehicle has to make many decisions in real time when each decision could catastrophically influence the outcome. And the environment is changes quickly in unpredictable ways.

• 14:54Speaker 02

  But the an autonomous vehicle never has to interact with the environment. It never has to touch anything. I mean, hopefully.

• 15:01Jacob Effron?

  Hopefully isn't.

• 15:04Speaker 02

  So but a robot that manipulates something inherently has to touch things and in contact make things exponentially hard. But let's say you knew that an autonomous vehicle could go from position A to position B. So like, you know, like park there to make a right turn. So if you knew that this was safe, then executing this motion is very simple. We know how to do that. But for for for in robot manipulation, if you know that, oh, you just have to fold the shirt, that is still super hard. You you cannot do it. So like, it's it's fundamentally different in that regard. But of course, like, if you think about the long long tail performance, or like, one

• 15:47Speaker 02

  big issue that autonomous vehicles have is that the long tail of things that can happen in the world is very, very diverse and very large. And to nail ninety nine point nine nine nine or whatever accuracy, you the long tail is the problem. And similar similar thing can happen for for robot manipulation, where if we want robots to be in rolled out everywhere, there's a very large long tail. So I think these these are the the learnings and that that are that are similar.

• 16:15Speaker 01

  And I think that trajectory could be similar for this technology, too, where, you know, with with chat G P T with L L M's, it's kind of like, all of a sudden, it appeared. And like, everybody was surprised, and it's like, whoa, like, now I have access to this like crazy intelligent chatbot. And with with autonomous driving, that wasn't the case. It was like a very long road, though we were promised autonomous cars a decade ago or so. And it seemed like there was no progress being made, it's just going very slow. And then, all of a sudden, it's here. Like, Waymo's are here, you can take it anywhere you want in in San Francisco. And I think something similar might be happening with with robotics, too, that that will be a long road, maybe longer than people expect. But there will be a moment where at some point, like, they're here, and it will be

• 16:58Speaker 01

  a very like a holy shit moment, similar to what we experienced with Waymo.

• 17:02Jacob Effron?

  And does it feel like it's a similar, you know, obviously, for the self-driving companies, a lot of it is just getting enough of this like weird long tail data, like, you know, the ninety nine percent of the stuff that happens when they're driving now isn't that helpful to them. And then, like, it's the the really weird stuff that happens, and they have to, like, amplify that through simulation. Like, do you think that ends up being a similar problem in robotics, where, like, you know, that there's, it's that long tail data that really matters?

• 17:22Speaker 02

  So one thing that happens already is that on the scale that we are, we can collect a lot of data, and we can. Collect a lot of wrong data in some sense, so like.

• 17:32Jacob Effron?

  Yeah, I see you always messing with the robots in some of the demos.

• 17:36Speaker 02

  Yeah, but what I mean with this is basically if you just look at the number of data points that you have, this not necessarily gives you better policies. You really have to collect the correct data. And it's similar, I guess, like most autonomous, most vehicles that drive around, if you have this data, it doesn't, it's not very helpful anymore.

• 17:56Jacob Effron?

  There's lots of parts of what you're building that are fascinating. I'd love to dive into a bunch of them. Maybe to start with just like the hardware side, like obviously so much of the capabilities of what you're able to do with the models depend on the hardware. Even just like your data collection efforts depend on the hardware. What is like the state of hardware in robotics today? What's, what works well for you guys? Where like, what are some of the pain points?

• 18:16Speaker 01

  So preface it maybe with a misconception that I think is fairly common, which is that we're bottleneck on hardware, like in general as a field.

• 18:24Jacob Effron?

  Yeah.

• 18:24Speaker 01

  I don't think that's the case. And there's this famous demo from more than 10 years ago from Pier 1, which is this very old robot that is roaming around the home and tidying it up and doing an incredible job. It would be like state of the art humanoid demo today. And it does it just fine. And the caveat is that it's teleoperated. There is a person behind it that is using their own brain to control the robot. And I think that is a proof that hardware has been there for a long time.

• 18:55Jacob Effron?

  Totally, I mean, to your point, anything you can do via teleoperation shows the hardware is fine. And then it's just the.

• 18:59Speaker 01

  That's right. That's right. And we see it all the time, like we can tell our, tell operate fairly simple robots to do incredible things. So.

• 19:05Jacob Effron?

  Like a lot of other robotics companies, that's their demo stuff.

• 19:08Speaker 01

  So it's very, very clear that the bottleneck is intelligence. Like if we had the same intelligence that the person has when they tell operate a robot, we would have robots everywhere doing all kinds of different things. And I don't think there will be bottlenecked on hardware. So I think that's kind of like the the main point here. In terms of hardware in general, I think we see a ton of progress there. And you know, the the modern humanoid robots, all kinds of robots that are coming out now every day are extremely impressive. They're, you know, amazing pieces of mechatronics. I don't think they're through the bottleneck. I don't think that's something that, you know, is is really constraining us in in the moment. But eventually, probably it will. So I'm glad that these efforts are are starting and so on.

• 19:51Speaker 01

  But I think without without the intelligence, we'll still be bottlenecked on on that part.

• 19:56Jacob Effron?

  And as you think about like milestones and intelligence, obviously, it sounds like when you saw the the Coke can Taylor Swift thing, you were like, this is going to be a moment we'll we'll remember. You know, sometimes it's kind of, you know it when you see it and you'll make progress in the models and know something's a big moment. But how do you think about what like some of the other meaningful milestones are ahead? Where you might uh might jot that down in the future?

• 20:14Speaker 01

  The big one that we that we just achieved that to me was really significant was this generalization milestone, was bringing a robot to a new home. Because as Danny was saying, in in the past, basically every robotic demo, including our own.

• 20:28Jacob Effron?

  Very precise kitchen.

• 20:29Speaker 01

  Very precise thing. You have to collect that in the exact same environment and, you know, don't touch the light switch. Like, you know, if there's a different time of the day, it might not work and everybody's okay with that. And we really wanted to move away from that and evaluate it in a in a home that it's never seen before. And there are so many other things than, you know, lighting and and and things like this, like small distractor objects that are different. Like you come into a new home and, you know, it can be very confusing. You know, the tabletop is different, all the lighting and all the other stuff. The objects are in different places. There's so much understanding that you need to have to to kind of know how to operate in that space. And with with Pyo5 being able to do that, that was a really, really big moment for for us, I thought.

• 21:13Speaker 02

  Also, in machine learning, sometimes I think it's good to take a step back and think about what would be the best machine learning algorithm that achieves zero error on the training set. It's just a lookup table.

• 21:25Jacob Effron?

  Yeah.

• 21:25Speaker 02

  And in robotics, you fundamentally can't do that because you can never reset the scene to the same, the same way, right? But still, a lot of people for a very long time had the feeling that, or the thought that if you train on your test distribution, the performance must be strictly better.

• 21:44Jacob Effron?

  Yeah.

• 21:45Speaker 02

  And I think that one of the coolest things that happened in PyL5, if you look in our paper, we have a plot showing that, said if you train on a large or sufficiently diverse enough set of scenes, then you can achieve the same performance in an unseen kitchen that you could if you had data from that kitchen.

• 22:05Jacob Effron?

  Right.

• 22:06Speaker 02

  And I think that that's just a very magical moment, that you can achieve just the same performance, even though if you've not seen data compared to a model that has seen data in that, in that kitchen. And I think that, for me, was a big, a big moment.

• 22:20Jacob Effron?

  Totally. It mirrors a lot of the magic of all, right, being able to do things that I haven't seen before. And it sounds like the next set of milestones for you guys is really around performance. Like, is there something you think about, like, God, like, when are, when Py, you know, whatever it's gonna be called, Py1, Py.6, what is able to do, you know, this thing in performance? That'll be, you know, kind of a step change?

• 22:38Speaker 01

  Yeah, I, I think we kind of like go step by step, looking at what is the bottleneck for really deploying these models widely. And, you know, initially it was capability, then like this generalization aspects, because it would be really hard to deploy it if, you know, you change lighting and it stops working all of a sudden. Now it's clear that we are past those, or like, we at least understand them enough. And now we really need to get them to work, really need to get them to be really, really robust. We are making a lot of progress. So I think in the past, there's been a few indications of how this problem could be solved on small scales. There's a lot of research papers around, you know, using a rel for these things and, and stuff like that. I think the challenge is how do you create a recipe that you can apply to any task at scale?

• 23:20Jacob Effron?

  Yeah.

• 23:21Speaker 01

  Where you just, you know, have a Have a little recipe that you just apply one by one and every task is being solved one after the other, regardless of what hardware you're using and all the other stuff. And that is still unsolved. So it's a little hard to predict what it will look like, what the recipe will look like, what the final result will be. But you know, if the past is any indication of when this will happen, I hope it will happen soon.

• 23:44Jacob Effron?

  Yeah, I guess I'm curious, obviously, you know, of the three things to get right in these models that you rattled off. It seems like you've got two of them relatively early in the company. Is that what you'd expected? Like, how does this compare what's happened so far to what you thought when you incorporated?

• 23:58Speaker 01

  I personally didn't expect it at all. My timelines were way longer. And that's maybe something about me, I should be like way more optimistic about these things. I should change my model. But yeah, I thought it was gonna take at least like three, four years until we get to where we are today.

• 24:14Jacob Effron?

  What about you, Danny?

• 24:15Speaker 02

  I strongly believe in the large model plus large data scheme. Like there are many, still like many unsolved problems and it's not like, you know, we can just, this is it and we can work. But like, I knew that we haven't fully tried everything. So for me, it was just carefully trying the things that we always wanted to try. And I think what was very nice though at PyE, is like how quickly we could try these things that we always wanted to try. And yeah, so that, like, I always had this hope that these things work out, but then seeing them work out faster than maybe, than we thought, that's very nice.

• 25:00Speaker 02

  So, but like, this is very often interesting that, there's an old saying that ideas are cheap, but it is kind of is the way in machine learning, very often a case that many people have certain ideas, but then to actually implement them in such a complex system with so many moving pieces, like real data collection, real hardware, real, you know, like different model sizes, real time constraints, all these things are making it actually work, there's still a lot of uncertainty, but seeing it coming together is nice.

• 25:29Speaker 01

  Yeah, we sometimes talk about how easy it is for LLM people, how jealous we are.

• 25:34Jacob Effron?

  Because they have the internet or because?

• 25:36Speaker 01

  They have the internet, the evils are easy. You don't need to have any real robots. The data is just there. Yeah. While their objective is basically the evil objective, there's so many nice things about it that we don't have in robotics.

• 25:51Jacob Effron?

  Well, one thing I'm curious about, actually, on that line of just, we're on this crazy exponential slope of LLM improvement right now, and obviously, a lot of the LLM world is focused on, you know, getting models that are good enough at coding and ultimately AI research, and then accelerate AI research itself. I'm wondering like how that trajectory plays into the work you're doing, and if we continue on this like crazy exponential of LLM improvement, does that, you know, meaningfully change your timelines on the robotics side? Or is there still like the cold hard constraints of the real world and hardware and data gathering that, you know, we could have the most superhuman coders and still it takes time?

• 26:23Speaker 01

  Yeah. It's a little hard to say, to be honest. I think initially the hope was that in this moment with RT2, once we connected the real world data to VLMs, it would be just, we'll kind of attach ourselves to this bandwagon, that like, once VLMs get better, we'll keep on getting better. I don't think we've fully seen that yet. I don't think we're there yet, where we can just take benefit for, from these models getting better and better. I think there is many indirect ways how we benefit from them. You know, models are getting faster, cheaper, the inference is better, all of those things. But I don't think we're right now bottlenecked on the underlying LLM being better.

• 27:07Speaker 01

  I do think that we see quite a bit of visual capability being a problem with what we do. So even if we have models, our models that can perform activities fairly well, we still need to command them usually with language, what to do in the current situation. And that we usually, for that, we usually use VLMs, or we train our own VLMs or fine tune existing VLMs to do that. And if those were better, I think the overall system would work better, but I don't think that's the main bottleneck right now.

• 27:38Jacob Effron?

  It feels like every, you know... Few months, you're probably learning answers to research questions that folks want to know. And I feel like in the robotic space, you know, back in the day, there were all these questions around simulation data versus teleoperated data. What feel like the big outstanding questions right now that you're like, oh, if I could fast forward to two, three years and know the answers to that, it would be pretty helpful for our research today.

• 27:59Speaker 02

  I think for me, there are two. So the one is about generative video models. And the other one is maybe if the recipe is already sufficient, maybe I can expand a little bit.

• 28:07Jacob Effron?

  Please elaborate.

• 28:08Speaker 02

  So generative video models, I think they maybe.

• 28:12Jacob Effron?

  Your former employer has some pretty impressive ones.

• 28:15Speaker 02

  Yeah, I did not expect them to be that good that quickly, I have to say. But if you try them on robotics, they're like, they still don't work enough. And so I would like to know.

• 28:25Jacob Effron?

  Sorry, what does it mean to try a generative video model?

• 28:27Speaker 02

  So let's say you put in a scene, you condition a model on, let's say, this table here, and then prompted to move the glass to another location with a robot arm. The way the video will unfold very often looks highly unrealistic.

• 28:45Jacob Effron?

  Yeah.

• 28:46Speaker 02

  So the physics is very often not quite correct. These models make a lot of progress in physics, like in other scenes. But as soon as there's a robot in this, in the scene, it's just, it's just not there. And like unfortunately, if we want to execute actions in the real world, tiny things like a millimeter off or whatever can make a huge impact. So I would like to know if generative video models overcome that and then become very useful for robotics, or if they remain, if this remains a problem.

• 29:13Jacob Effron?

  And if they did overcome that, if physics were suddenly, you know, all working correctly in generative video models, would that like completely change the architectures you guys use?

• 29:21Speaker 02

  So at least like a lot of the concerns of how to achieve generalization could be solved, like to new tasks, because you could just use the generative video model as a simulator.

• 29:32Jacob Effron?

  Right.

• 29:33Speaker 02

  When you try to solve a new task in this particular scene that you are in, and try different action trajectories and see what, which one of the is the best. And if that generalized really, really well, I think that would, would not fundamentally change things, but it would provide a faster path towards generalization to new tasks in unseen environments.

• 29:53Jacob Effron?

  Yeah. And then what about the recipes?

• 29:55Speaker 02

  So. I think, I mean, I don't know if it's true for L L M's or something, but it might have been that L L M's worked earlier than the inventors of them thought. And so I also wonder, when is there a point in robotics in which the stuff that we have works without us realizing? And works in the sense of that it's not necessarily a research problem anymore, but like still a very hard, but an execution problem. Right now, I think it's, yeah, a scaling problem. Right now, I think it's still a research problem, but yeah, I would like to know.

• 30:32Speaker 01

  Yeah, so if someone could come from the future, like two years, and just give you the enough conviction, it's just like. Exactly.

• 30:38Jacob Effron?

  You got the recipe.

• 30:38Speaker 02

  Exactly.

• 30:39Speaker 01

  Keep doing what you're doing. That would be helpful.

• 30:42Speaker 02

  Yeah.

• 30:43Speaker 01

  So does the same questions for you, or what? Um, I, I agree with the answers here. I, I think one other one that I would really like to know the answer to is some version of scaling laws. But I think usually when we think of scaling laws, we think of it in terms of the L L M scaling laws and, you know, how to pick the right batch size and how much compute for certain data and, and things like this. I think the scaling law that we are actually after in robotics is very different. It's something, you know, as crude as, um, uh, dollars to model capability. Uh, and having some kind of monotonic scaling law of that and understanding the underlying recipe and how to translate one into the, the other. I don't think we are there yet.

• 31:25Speaker 01

  But I think if, if I had something like that, then again, it would turn just into an execution problem.

• 31:30Jacob Effron?

  Right, the dollars being allocated to data collection, compute.

• 31:34Speaker 01

  Whatever it takes, whether it's, yeah, compute, data collection, whether it's a compute for video models or, or something else. Um, but yeah, having that, that understanding would be really, really helpful.

• 31:44Jacob Effron?

  I'm curious, you're obviously, you know, one of the first companies to really scale up, you know, uh, building these models, data collection. I'm sure you're coming across all sorts of problems that you're just solving yourselves. And it feels like over time, you know, uh, as the people from the future come and tell us what the recipes are, or if you figure them out, uh, there'll probably be, you know, more and more folks going after this and, and building stuff. You know, is there a set of like infrastructure needs that you've come across where you're like, God, it would be really nice if there was someone else that was, uh, building this. Uh, it would make it easier for us and all future robotics companies.

• 32:14Speaker 02

  Yeah, so robotics, the way, because the type of data that we're dealing with is all time series data and multimodal has multiple different camera views, the time series. And you also have robot states and actions. And then you have language annotations. These language annotations are also still a research problem, so you iterate a lot of them. So the data infrastructure that we're building, we all built in-house. And it's a, it's a real challenge to make this fast enough to just store it well enough so that you also can still iterate on it. And we have some custom requirements there, which are different from like static data sets that you just collect once and then they, I mean, so because our data grows every day, right? Like it just grows every day.

• 32:57Speaker 02

  Yeah, so, so I guess that it might, for some people, maybe it might sound boring, but I, I, I would absolutely disagree with this. But like data, data storage, data pipelines and so on, that, that's a very important infra problem.

• 33:11Speaker 01

  Yeah, that's one thing maybe that I underappreciated coming into this, like how much data robots generate.

• 33:16Speaker 02

  Yeah.

• 33:17Speaker 01

  Like we knew it was gonna be a lot, but like it grows quick.

• 33:20Speaker 02

  Yeah.

• 33:21Speaker 01

  It's, it's a lot. And, and it's also like, everything they collect is grand truth in some way, right? Like it all tells you about how the world works. Even if you fail at folding laundry or something like this, it still tells you that if you, you know, move the arm this way, that's how the shirt is gonna deform. Uh, which is not the case with LLM, you know, there is some data that is just like wrong.

• 33:39Speaker 02

  Yeah, take that out of it.

• 33:40Speaker 01

  Yeah, it's just bad data. Um, so yeah, the, the amount of data we are dealing with is, is kind of crazy. I think the other aspect that would be really nice for someone to solve, that we are trying to solve in-house is, um, how do you decide what data to collect and build all the machinery around understanding that? Understanding the quality of the data you collected so far? Um, how do you do quality assurance at that kind of scale? How do you language annotations in the right, in the right way at that scale? Um, you know, how do you know how much more data is needed for the model?

• 34:13Speaker 02

  Yeah.

• 34:14Speaker 01

  Uh, how do you know what strategy to use to collect that data? Um, how do you train, uh, people who are actually collecting that data to, to do this at scale? So all the infrastructure around that, on, on managing the performance of this, on, on understanding that data, on being able to look through all of this all at once and have like a, uh, uh, you know, bird eye view, understanding of like, how, this is how my data set is, more or less, that would be really, really helpful.

• 34:39Speaker 02

  Yeah.

• 34:40Jacob Effron?

  How do you know what data to collect? Why start with folding shirts and picking out dishes? And how do you pick the tasks?

• 34:47Speaker 01

  Right now, we pick them just to move as fast as possible. So pick tasks that are really, really hard. That if you can fold laundry, you can probably assemble iPhones and do many other things. It's just like really, really complex tasks that we don't think people were able to do in the past. And then we pick tasks that are easy for us to understand, easy for us to deploy robots to collect data for and things like that. So every one of us has a home. We all folded laundry before. We know what a successful folded t-shirt looks like. We know what a clean kitchen looks like. It's much easier to do it that way than to work with a manufacturer, for instance, and understand their process, get under schedule, get all their parts over. That could be expensive. Understand exactly how it should be done.

• 35:32Speaker 01

  So we just want to be trading very, very fast. And because the technology is very general purpose, we think that it's going to then apply to all the other tasks as well.

• 35:40Speaker 02

  Maybe one thing to add here is that when we talk about the Taylor Swift moment in the beginning, like this work got criticized by a lot of people saying like, oh, you could just run an object detector and then like run. And so we could use a very similar technology now also to solve folding laundry. And it's very clear that you just cannot apply the same, like an object detector and a trajectory planner to fold laundry in the same way. So that's also one way of picking tasks which like push the boundary of the capabilities and still convince ourselves that we have the tools to still solve them.

• 36:19Jacob Effron?

  What's the intuition you use for like this task is going to be hard for a robot versus easy?

• 36:23Speaker 01

  I think it's hard to describe it. I don't think there's like a list, a checklist of like if there's a formidable object, hard. If you've never seen before, hard. I feel like there's just like a sense that you get over time as you work on the problem. For me, when we set goals and we talk about certain tasks, you can look at the faces of people in the room and like how terrified they are. I think it's all laundry.

• 36:52Speaker 01

  Yeah, there's some things that just robots are, they're quite counterintuitive, that robots are really bad at. These things usually involve things that are not the same every time. So for instance, if you pick laundry out of a laundry basket, you know, the shirt is always in a different configuration. You've never seen the shirt in the exact same configuration that you've seen before. And that makes it hard. That kind of, the initial state distribution, the variance of that makes the task really, really hard. If the task is very long, that makes it hard because these errors can compound and that could be hard. If it's very precise, or if there is like some bottleneck during the task that makes it, that is very precise. That's a hard task as well.

• 37:34Speaker 01

  But, you know, I could, I could try to list some of those, but I don't think I'll be able to capture all of those things.

• 37:38Jacob Effron?

  Yeah, intuition from having worked with it so closely. I mean, similarly, I'm curious whether it's intuition or science at this point, like evals. How do you know when these models are getting better? You know, I mean, there's no sweep inch, I guess, in robotics world.

• 37:50Speaker 02

  Yeah. Yeah, evaluations are very important and a very hard, hard problem. So the interesting thing is, I think I said this before, you can never reset the scene to exactly the same way, right? Like you just cannot repeat that. And so we just have to run enough evals, run statistical analysis if like the numbers that we get out are statistically significant. I mean, one thing that we also learned is that you, to really show that a policy is better than another, it's very often important to run it on many, many different tasks and many, many different environments. And then suddenly you might see, might see differences. But evaluations is maybe one of the hardest.

• 38:34Jacob Effron?

  You're just like, oh, that shirt looks pretty crisp when it's folded. How do you actually?

• 38:38Speaker 02

  Yeah, so we have, we have like guidelines of the operators that run evaluations to how they should rate it.

• 38:44Jacob Effron?

  Okay, so the operators themselves evaluate the?

• 38:46Speaker 02

  Yeah.

• 38:47Jacob Effron?

  Especially how crisp was the shirt?

• 38:48Speaker 02

  Yeah. And what we also do is we never really compare absolute performance on this day to absolute performance on a few weeks ago. But rather we, we always run the baseline and the new model at the same time, run with the same operator and then look at the relative ranking. I mean, there are some tasks that we have sufficiently understood for which we also look at the absolute performance, but yeah. Just to account for even just like lighting changes or like the robot has changed. And it's a different operator who might is much more picky about a shirt than another one.

• 39:24Speaker 01

  Yeah, it's also that the trends for for this are not very good. Because as these models get more and more capable, we need to evaluate them on more and more tasks. And on more and more robots too. So you're kind of need to run more and more evals to get to to an answer whether your model got better or not. So there's a problem. It takes a long time. It's operationally very heavy. There is a lot of variance in them. That's one of these other places where we're really jealous of L L M people. But but there are some things that we're exploring to to try to overcome that issue. Other than just operational excellence, try to do this really, really well. You know, you can start using simulation for these kind of use cases, video models, things like that. People have been investigating this for a while.

• 40:08Speaker 01

  We haven't seen them be robust enough yet to to reduce that variance to give us more confidence that this model is is good. But but there is some hope that they will get there.

• 40:18Jacob Effron?

  Yeah, well, talk a bit about like the role of simulations. I, you know, as I understand it, one of the early bets you guys made was there was, you know, some people have tried to go the pure simulation approach. You know, you guys have really leaned into the teleoperation side as being a necessary component of the data recipe. Maybe talk about like, you know, what drove you to that decision initially? And then how you think about the role simulation plays in this going forward?

• 40:38Speaker 01

  Yeah, I I personally spent quite a bit of time in on simulation in the past. And especially recently, simulation has been extremely useful for for robots, for especially for locomotion. And I think the reason for that is locomotion or like walking is is mostly around modeling your own body. If you can model your your legs correctly, then you can simulate them on all kinds of terrain. And then you'll be able to to transfer that to the real world and then walk properly. I think for manipulation, the difficulty is not in modeling your own arms. The difficulty is modeling everything that the arms are interacting with.

• 41:15Jacob Effron?

  Right, right.

• 41:15Speaker 01

  The problem is not how I move my arm from A to B. It's how do I manipulate the world around me? And that is much, much harder to get right and much less scalable. So I do think that simulation is going to play a role. I think the kind of the the entry point for it is for evaluations. If we are not, if simulation is not useful yet, is not like fully accepted as the evaluation tool for these models, then I don't think, you know, it has a ton of use cases beyond that.

• 41:44Jacob Effron?

  Yeah, that's like the first.

• 41:45Speaker 01

  Yeah, that's kind of like the first gate that you need to pass. And once you're past that, then I do think it might really, really help. I also suspect that it will be more helpful the more diverse your real data is. I think if you are just collecting, you know, small real data set on a single robot, maybe for a single task, and then you have a simulated data set that is, you know, very, very different than the data that you collected, it will be very hard for the model to kind of reconcile the two. But I think the more diverse your real data is, the more simulation feels just like another version of reality that the model can incorporate and kind of not separate them into two different corners of the of the space. So I I do think that collecting large scale real world data is going to help with that if we were to use simulation later on.

• 42:30Jacob Effron?

  Yeah.

• 42:31Speaker 02

  And I I also try to not be dogmatic about these things. Like so far, simulation has not been sufficiently good for as a data source. But maybe if that changes in the future, you know, like this is the beauty of of the type of models that we are dealing with is that they could just be trained on all of this at the same time.

• 42:48Jacob Effron?

  Yeah.

• 42:49Speaker 01

  Yeah, that's right. If at any point we we see that simulation is the way to go, we'll switch everything.

• 42:53Jacob Effron?

  I'm curious about this because I feel like in the air world, it's like, you know, all these strong opinions loosely held because the world changes so fast and all these things develop. I'm curious what that means for physical intelligence of the company. Like, how do you, you know, kind of allocate research resources or like, you know, and prioritize things based on, you know, I'm sure you have some more concentrated bets, but at the same time, exploring lots of things is probably really beneficial to furthering it forward. So how do you think about that from the company level?

• 43:19Speaker 01

  It's hard to do. It's hard work. It's, I think it's hard for any organization that does a lot of research. You know, how do you, how do you find the right trade-off between exploration, exploitation? How do you, you know, make enough bets but also have enough, you know, resources behind each bet for it to succeed? We do it in a few different ways. I think the main thing there is that I don't think there is a single recipe that is just like, this is how you run a research lab, and this is how you do it, and this is how you decide where you bet. I think it's mostly a matter of taste and belief and having just really, really good people, researchers that do have taste. And I think if you look at the past, that's often what happened where breakthroughs came through. There was just like researchers that had enough conviction, enough belief, and they

• 44:04Speaker 01

  just happened to be right. So that's what we try to do. We try to hire the best researchers in the world that work on this problem, researchers that have the right taste, the right, that can distinguish these ideas. And then we try to give them the resources they need to explore these things and try not to do too many of these things at all times so that we have enough resources behind each one.

• 44:26Jacob Effron?

  How do you tell if someone has taste? Obviously, like, you know, past performance can be indicative of the future, but you probably, I know you hire a lot of really young, talented people.

• 44:33Speaker 01

  Yeah.

• 44:33Jacob Effron?

  How do you get a sense of that?

• 44:35Speaker 01

  It's a good question. I think the past can be some kind of indicator. You know, if you, there's only so many papers or research results that made a big impact in the field. There's so much research happening, there's so many papers, no one can read them all these days. So the probability of you having an impactful research result is actually fairly small. So if you have people that have had that multiple times. Sitting across from two of them, yeah. If you if you encountering people like this, then that means that, you know, they they are doing something right. They're thinking about the problem right, the right way, or they are deep thinkers and they can change their opinions and and they have the right taste. So I I think past here could be some kind of indicator. That is really hard.

• 45:20Speaker 02

  So I think like on our our interview process, we really make sure that like a lot of different people in the company are exposed to a candidate. And like, and then I I think if you have multiple people that have different styles of of of interviewing, interaction with people, I guess, I think you get a good, good sense of the alignment and the way we discuss discuss it very openly and very fairly. I think you, yeah, I I think so far we we always, I don't think we made any mistake in hiring so far. So that feels, that's very, very good. We're all all very aligned.

• 46:00Speaker 01

  Yeah, there's also some soft skills, maybe there, you know, like people who are not super dogmatic, who can like take new evidence and change their opinions, who let science lead the way, who are like willing and who are very open minded and willing to try something that, you know, even though they don't believe in it, but they think that it could be something that that they want to try.

• 46:23Speaker 01

  Yeah, lots of lots of these things that are like, I think are difficult to put your finger on, but but you know it when you see it. Yeah.

• 46:29Speaker 02

  But one, the one thing that comes to mind here is that I I like interacting with people that are just willing to do, to do the work and not like say, oh, no, I don't want to do X, Y, Z because, I don't know, for whatever reason, but just like to do the, to do whatever that's needed to achieve a certain research goal and then, yeah, and nothing that that's, those are the people I like interacting with.

• 46:52Speaker 01

  Yeah.

• 46:52Speaker 02

  But even, you know, like, even, let's say you've never worked on a data pipeline, but now you, for your problem, you need to figure out the data pipeline, and you just go into it and do that and not saying, oh, no, it's not what I, it's not my role or something.

• 47:04Speaker 01

  Right.

• 47:04Speaker 02

  Yeah, that's, I think, nice, nice problem.

• 47:06Jacob Effron?

  Yeah, I mean, another part of your research strategy I'm curious about is you've obviously you've open sourced Pythero, you are, you know, published a lot of research. How do you think about like, you know, what you want to open source, what you want to like, kind of the insights you want to share with the world and obviously in the LM world, I feel like started kind of open and it's gotten, you know, much more hush, hush about what folks are doing. How do you see that?

• 47:27Speaker 01

  Yeah, it's, it's a very deliberate choice and I think if you're too long on Twitter these days, it might seem like, you know, robotics is about to be solved and like it's just a matter of collecting a little bit more data and like all of this is going to be out there and we'll have, you know, humanizing the home like today doing everything for us. And we think that we are still very, very early. There are still breakthroughs to be had. The technology is not there yet. And the biggest risk for us as a company and as for the entire field is not that, you know, someone is going to get there first before us, that we're going to get outcompeted by somebody else. The biggest risk is that we're just going to fail. Like, science is not going to cooperate and we're not going

• 48:10Speaker 01

  to solve this problem. And then it doesn't matter, you know, whether you shared your results or you didn't or what your strategy was, what kind of robot you were building. So we want to make sure that that doesn't happen. And to burn down that risk, you, you want to make sure that you expose these ideas to as many people as possible and kind of invite them in. Because we do believe that the path we're, we're taking is, is the right path to solve the problem. So the more people kind of subscribe to it, get to play with these models, get to understand where they are going, get to contribute to them, the more likely it is that we're going to succeed. Not as just as the company, but also as the entire field and the likelihood of solving the problem is, is higher. It might change at some point, you know, once it's very clear that the problem is getting solved. But I

• 48:55Speaker 01

  think so far it's been. Very deliberate, very deliberate and extremely successful for us, frankly. Like, it's also really helps with recruiting, for instance.

• 49:03Jacob Effron?

  I'm sure.

• 49:04Speaker 01

  People want to share their work. They want to make sure that, you know, they can talk to their friends about it and and get more people to help them with them. Also seeing just the impact of open sourcing these models, of PyZero being used in so many

• 49:17Jacob Effron?

  Yeah, what's your both like favorite way it's being used?

• 49:20Speaker 01

  Ah, I've seen so many, like, I've seen some weird, yeah, Brown was super crazy. Brown being pre-trained on PyZero and grasping things.

• 49:28Jacob Effron?

  Yes.

• 49:28Speaker 01

  That's kind of crazy. I've seen it on surgical robots. I've seen it on autonomous cars. I've seen it on so many different tasks that I didn't even think that PyZero would help on. So that was super cool to see. Yeah, that's really cool impact to be able to deliver.

• 49:45Speaker 02

  And also, I think like robotics very often has been, you show a video and then no one really knows what it is.

• 49:53Jacob Effron?

  Right, right.

• 49:53Speaker 02

  And like, just being like, putting it out, putting the model out for people to use, it's also a good way of learning how this could eventually be done as a real deployment and seeing what, how the models react with other players and so forth. So there's a tremendous value in that.

• 50:14Speaker 01

  Yeah, it kind of speaks to this problem of evaluations with robotics. It's very hard to compare, you know, is this model better that this team published on or this other one? Like, there's kind of no way of doing this well. There is no established benchmarks, and even if there were, you know, if anything changes, like, the variance is so high that you can't tell. So, yeah, I agree, just like putting it out there and letting people use it.

• 50:37Jacob Effron?

  The same thing has been true of the LLM space, right? In many ways, the evaluations are kind of only so helpful and, you know, turns out users will tell you which models they like better.

• 50:45Speaker 01

  That's right.

• 50:46Speaker 02

  And what I also like, I think when LLMs came out, people didn't really know what people all can come up with. And I'm looking forward to the same happening with the physical world also, like.

• 50:57Jacob Effron?

  Yeah.

• 50:57Speaker 02

  I want to be surprised what people, what applications people come up with, with the tools that we're building.

• 51:03Jacob Effron?

  Well, I wanted to get a little bit to some of the research you guys have been doing. I mean, obviously, you've had some really interesting work on kind of, you know, training these models faster, right? And getting it faster, you know, lowering the cost of it. Talk to our listeners a bit about like that problem space and, you know, some of the stuff you've been working on.

• 51:18Speaker 02

  Yeah. So when we build a robot foundation model, we usually start with a pre-trained vision language model. And I think we talked about this earlier, like, oh, can we just like, when the vision language models get better, do we benefit from it? And so far, it has been that the vision language model itself is not yet adapted to robotics. So the features that it has, the visual features, are not good enough yet for robotics. So we have to fine tune it on our robotics problem. But when you do this fine tuning, so you take a vision language model backbone, and then maybe you add certain new weights to it to make it output robot actions. And then if you fine tune this only on the robot data, what happens is that the model does lose some of its capabilities.

• 52:03Speaker 02

  So it's not a good VLM anymore. And this has multiple effects. On the one hand, it lowers the generalization capability of the models. And in some sense, the model forgets a lot what it has learned in the internet. But also, the training dynamics is not as good because it has to like, change so much that it might train very slowly. And so what we've been doing is figuring out how can we fix this? And one concept that we came up with, we call knowledge insulation, which is the idea of, it's a multifaceted concept. So we try to basically modify the training recipe to maintain most of the model's capabilities. And how we do this is we still train the model on web data,

• 52:47Speaker 02

  so non-robotics related data. And that helps, and that especially helps also for generalization. But that's not the only thing. The other important insight is that for the best performing recipe right now for building robot models is to use flow matching or diffusion as to produce robot actions. But it turns out that if you use this to train the model backbone, it trains Very slowly and the gradients that you get, they kind of destroy a little bit the backbone, even though you're using other sources. So the idea that we had is we can train the VLM backbone with tokenized actions. So we take robot actions, convert them into something like text and train it this way.

• 53:32Speaker 02

  And it turns out that this make adapts the VLM to robotics in a way that without losing its capabilities. And then we can bolt the this action experts with flow matching on top of it and let them not communicate that much with each other, stop the gradient and and then have very good action prediction while still like adapting the backbone. And the cool thing about this is we, I think we sped up training by 10x.

• 54:01Jacob Effron?

  Wow.

• 54:02Speaker 02

  And also and also generalization. So that that has been quite good. So we we cut our iteration speed much down because that's something I also maybe for you to be aware of. In robotics, you know, like you collect data, which takes a while. Then you train them all, which can takes multiple weeks. And then you have to run real robot eval. So until you get a signal of where my decisions in any of this pipeline sensible, it can take a very long time. So if you make any of these stages faster, your research is not only accelerated by this amount you change, but by a much larger factor because it can run, make your decisions much earlier.

• 54:36Speaker 01

  Especially what we were saying at the beginning regarding the data. If only we knew what data to collect.

• 54:41Speaker 02

  Right.

• 54:41Speaker 01

  Like right now, we only know. Yeah, right now, the only way we know is by training them all on that data, you know, waiting all this time, incorporating all of the data and then running an eval. And that also has some variance. So if there is some way of figuring that that out way earlier, maybe even without training the model, that would be super, super helpful.

• 54:58Jacob Effron?

  I mean, another part of the research that I think you did very recently was making these models work kind of faster in difference, right? Where like there's all these dynamic things that happen in the real world that you might need to on a split second, you know, make a make an adjustment or make a decision. I mean, talk about some of the latency issues that happen with these larger models, how you go about solving that. And is that all going to be algorithmic or is there, you know, actual hardware, you know, needs that you'll have there?

• 55:19Speaker 02

  Yeah, so we had a recent work from Kevin Black from our company who worked on this. So one thing that maybe to explain how these models are being queried. You get an observation and then you start querying the model and the inference can take a few hundred milliseconds. And during that time, the assumption is that the world stands still. At least, but this obviously is not how the world works. So the world could have moved on, right? So what you might think of doing is, let's say you query the model based on the current observation. And then you always, so we never predict only one action. We always predict an action, a whole action trajectory called an action chunk for the robot doing.

• 56:03Speaker 02

  So maybe you should just continue predicting the old action chunk. But while the inference of the model is happening. But then one issue that arises, it could be that if then the new action chunk is ready after the inference time delay. So that the world might have changed or like that this new predicted action chunk is not consistent with the old one. And so Kevin figured out a way of using techniques from image impainting. So like the diffusion models for generating images, you can like remove a patch and then they just fill in the patch again. It's impainting. Can apply a similar technique to basically impaint the rest of the trajectory, given the one that you already executed.

• 56:47Speaker 02

  And that is a purely algorithmic advancement. You don't even have to retrain the model for it, you can just afterwards apply it. And it worked really nicely. So to, basically what happens is that the model is still, the system is still executing an action. While it's now computing the new action for the future. And then they're kind of like fused together, I think, a way of thinking about it. But then. As you said, as an alternative, we would greatly benefit from advances if to make like models much, much faster at different time. That would would would also solve a lot of a lot of problems. This is something that we currently haven't really looked into because we've solved it in different ways, but yeah.

• 57:25Jacob Effron?

  That's fascinating. I mean, taking something from, you know, filling in images, it feels like definition of like research taste, like how did you figure that out? Like that's, that's a

• 57:33Speaker 01

  Kevin has very good research.

• 57:34Jacob Effron?

  Yeah, incredibly, incredibly impressive. I'm curious, you're obviously building these generalist models and maybe just paint the vision of, you know, I imagine in the future, you know, there'll be some sort of like task specific post-training that people are doing on these models. But like, what do you think that look like? This works. What does that look like in, you know, 10 years?

• 57:51Speaker 02

  So I hope at least if I'm successful is that there's no post-training.

• 57:55Jacob Effron?

  Oh, that sounds good.

• 57:56Speaker 02

  Okay, maybe I'm actually serious, but maybe I'm more towards your question. So I, my hope is really that we provide better and better base models that are more steerable and more promptable so that they could get a task to a certain amount of performance and a certain speed in whatever task. It's similar, like large language models nowadays, you don't need to really post them when you give them to a user. They mostly work on, at some point, there were like these APIs to fine-tune models.

• 58:26Jacob Effron?

  Right. Those are RFT, they're kind of back, but with like a dozen examples, like not like, not some massive amount of data that's required.

• 58:33Speaker 02

  Yeah, that's right. But so what I can see is that if you want to squeeze out the last percent of performance, speed, reliability, that will be very efficient fine-tuning that you might can do like on a weekly basis even, like collect like data for 10 minutes, feed this back in and yeah, and figuring this out. I could see that version. So the base model gets better and better and better, but there was always like this last percentage performance squeeze out.

• 58:58Speaker 01

  And we start seeing some of this in in our latest research. Initially, when we wanted to get these tasks to work, especially these complex tasks like like folding shirts, we always had to do very specific post-training procedure. And it required like quite a few people to like figure out exactly. Exactly what the procedure should look like and all the details in the procedure really, really matter. But now, as we get better and better pre-trained models, we realize that some of them do work zero-shot just as well as the old post-trained models. And that is super cool to see. That's like exactly what we saw from, you know, G P T three point five, three point five to, to four.

• 59:38Jacob Effron?

  Right.

• 59:38Speaker 01

  Where like all of a sudden this whole paradigm has changed. I, I do agree that I think post-training is, is there to stay. It's a little unclear whether it will be, you know, you collect a few more demonstrations or you just tell the robot a little bit more. You guide it in some way, how to, how to do the task, or maybe the robot explores on its own.

• 59:57Speaker 01

  I think it's still to be, to be determined, but I assume that there's going to be a little bit of, a little bit of guidance that you can provide that will lead to faster execution or better performance at the end.

• 1:00:06Jacob Effron?

  Well, this has been fascinating. We always like to end our interviews with a standard set of quickfire questions, which are overly broad and just stuff everything else into the end.

• 1:00:13Speaker 01

  We love these.

• 1:00:14Jacob Effron?

  Maybe to start, I mean, you guys are so close to the cutting edge of A I research. I'm wondering, what's one thing you've changed your mind on in the last year?

• 1:00:21Speaker 02

  I think for me, it was seeing this like environment scaling curve of you can achieve the same performance in unseen environment. Is if you had training data from it, I think it's really has changed my mind.

• 1:00:33Jacob Effron?

  Did you think it would just, you thought that would just take longer or did you, were you not actually sure?

• 1:00:36Speaker 02

  I thought that was, I, I thought that would, would have been a gap or like that you need more diversity. Yeah. So, I mean, we've talked about that the world is so incredibly diverse, but maybe it's actually like feasible. Maybe it's not that diverse, so it's feasible to kind of cover it.

• 1:00:52Speaker 01

  We, we had these conversations before we started the project, like how many homes would it take to see before you generalize to the next one? And like the numbers that were thrown were like a million, like, I don't know, like homes are so different, right? And then it turned out to be a hundred, which is like still kind of mind-blowing to me. Like it, it does seem to indicate that maybe we're giving too much credit to how diverse the world is. For me, the, I, I fully agree with that. The other one would be video models. I think they, they've made a ton of progress way faster than I expected. You know, I still remember playing with them a few years ago and seeing all kinds of different problems with them. I think that are getting very, very good, very, very fast.

• 1:01:32Jacob Effron?

  Yeah. And does that change like the, you know, some of the research directions within physical intelligence today? Or is it like, you're still waiting to see how much of a physics understanding?

• 1:01:40Speaker 01

  We are definitely like looking out. We are definitely starting to explore them. We're trying to see where they can help and what the, where did they change and how to make them more steerable and things like that. I don't think they got to the point yet where we are, you know, completely change our research agenda, but we definitely are paying attention.

• 1:01:59Jacob Effron?

  What's one thing that you feel like is overhyped in the discourse around AI and robotics? And one thing that's underhyped?

• 1:02:04Speaker 02

  I think hype in both directions is always good because it draws people into, I'm sorry that I give a non-answer. It draws people into like, the things that a lot of people work on should, a lot of people should work on because apparently they work so hyped, there's good. And things that are underhyped means also that people that like finding more niche problems are also working on. So I like, I don't want to even put names on these two things because I like things being over and underhyped.

• 1:02:33Jacob Effron?

  Yeah.

• 1:02:35Speaker 01

  Let's see. I think underhyped, generalist robotics models.

• 1:02:40Jacob Effron?

  Nice. I'd be worried if you said overhyped.

• 1:02:43Speaker 01

  That's right. Heavily overhyped. Yeah, I think if it works, and so far, you know, all the signs seem to indicate that there is something to that idea, that that will fundamentally change how we think about the physical world and intelligence in that world. Overhyped, the other thing, underhyped, Waymo?

• 1:03:02Jacob Effron?

  Yeah.

• 1:03:02Speaker 01

  Yeah.

• 1:03:03Jacob Effron?

  It really is. It's like the most visceral demo you can have.

• 1:03:05Speaker 01

  It's absolutely crazy. Like when you walk in, if you, when you sit in the Waymo, it's just like a very different experience than if you had, you know, the most intelligent chatbot. It's just like so visceral. Yeah, it's crazy. Overhyped, I don't think there are many people, many things overhyped actually these days. I think maybe the craze around humanoids is like a little, a little too far, I think. And to me, it mostly comes from this misunderstanding of like how AI is going to evolve, that for these models, it doesn't matter if it's, you know, this kind of robot or that kind of robot, it can take the data from all the different form factors and generalize across all of them. And it will be the most general purpose thing that you can apply to all the different specialists.

• 1:03:46Jacob Effron?

  You don't think the world's been optimized for the human form factor?

• 1:03:49Speaker 01

  Oh yeah, I've heard that. I think, I mean, there is, there is, you know, there is some, there is a point to that. But I think it's kind of like thinking that. It is true. So therefore, let's build a robot that looks like a human and that will be the robot that will see the most diverse data. But I think you can go even further and say, well, like, actually, to these models, it doesn't matter what robots the data come from. They can take data from all kinds of robots. It could be cars, it could be drones, you know, all kinds of things. And that would lead to even better capabilities of these models. So I think if you go with that argument just a little bit further, you end up in the world where you have a gigantic, big foundation model that powers all the robots, some of which are human ones.

• 1:04:30Jacob Effron?

  Yeah. All right. Ready for a really shameless podcast question?

• 1:04:32Speaker 01

  There we go.

• 1:04:33Jacob Effron?

  What year do you think we'll have a million Americans with robots in their home?

• 1:04:38Speaker 01

  I think we already have a million.

• 1:04:40Jacob Effron?

  Okay, with robots powered by seven, folding laundry and doing all these things.

• 1:04:45Speaker 01

  As I said, I should be more optimistic about my predictions, given the past progress, how mistaken I was. I think we'll see robots being deployed, doing things that we've never seen robots do, probably this year or next year, if we continue on this trajectory.

• 1:05:03Speaker 01

  When they will be deployed in a home, doing all kinds of different things. I think kind of the conversation has changed from never, or like, the problem is fundamentally unsolved, to like, somewhere between five to 10 years or something like that. And I think most of the community agrees that, you know, these are the timeframes now. I think if the progress continues and we're successful, and I think the progress has been speeding up, then I think it will be within the next five years.

• 1:05:28Jacob Effron?

  You must think a lot about the implications of this model improvement and, you know, what the world will look like when these are deployed. And obviously, as you mentioned earlier, we, in some ways, have no way of knowing, like, all the amazing ways these models will be used across the board. But is there some, you know, prediction for the future when these things do work that you have, that you think we're like, not thinking about enough today? Or is like, maybe a weird or under discussed implication of the, you know, of these models being, you know, available prolifically?

• 1:05:53Speaker 01

  I thought for, maybe to like turn it around a little bit, for software, I did not expect vibe coding to be a thing. And it's like, it's super cool.

• 1:06:02Jacob Effron?

  Yeah.

• 1:06:02Speaker 01

  You know, we have like non-technical people in the office doing incredible things, building tools for themselves, like being extremely productive.

• 1:06:08Jacob Effron?

  What are they vibe coding?

• 1:06:09Speaker 01

  Uh, all kinds of different dashboards, all kinds of different tools for operations. Yeah, it's, it's super cool. And I was thinking the other day, um, what would vibe coding look like for, for hardware? You know, where you can just, um, prompt the model, the intelligence is there for you. As soon as you create something, you kind of just like breathe intelligence into it. It's more the world that we know from cartoons where, you know, like the hard part was making the thing, but like as soon as you make it, like it's, it's alive. And I think, and I think that's going to be a very crazy transition.

• 1:06:42Jacob Effron?

  Yeah, that'll be nuts when you can create any piece of hardware and then infuse it with these models.

• 1:06:45Speaker 01

  That's right. And it just, you breathe intelligence into it.

• 1:06:48Speaker 02

  I think I would largely, largely agree. I guess, I, I'm, I'm wondering what people, like, I really look forward to a time when there are no chores anymore in a home. I think like a lot of people would look. That's also interesting, like, I think, if you think about, uh, GPT or the GPT moment or something, uh, I guess one aspect of this was that a lot of people have never heard what a language model is, and suddenly the, the whole world knew what a language model is. But if I talk to, if I show anyone a robot video, then they always say, oh, I want this instantly. So they all already have like a, a very clear conception of how it would change their life, which I think is, um, pretty, pretty interesting.

• 1:07:33Speaker 02

  But I'm very curious what then people will do if they have never have to unload their dishwasher anymore.

• 1:07:39Jacob Effron?

  Yeah.

• 1:07:39Speaker 02

  Like what, what like, what will this enable humans to focus on? Um, which I think could be very exciting to.

• 1:07:46Speaker 01

  Yeah, I, I have a one-year-old at home and I was just washing myself all kinds of stuff I need to do around the baby, just to spend more time with her.

• 1:07:54Jacob Effron?

  Yeah.

• 1:07:55Speaker 01

  And it's just crazy amount, you know, sanitizing balls, cleaning this, cleaning that, um, you know, loading and loading dishwasher, all of those things that like, I don't wanna do, I don't wanna spend my time on this, I just wanna have more time with her. So not not having to do that and just be have the home in the state that I wanted to be in at all times would be pretty cool.

• 1:08:16Jacob Effron?

  When she's older, you're gonna give her a chores list, or is this a moot point now that will uh?

• 1:08:19Speaker 01

  Yeah, that's a good point. I need to think about that. I need to slow it down.

• 1:08:25Jacob Effron?

  Well, this has been a fascinating conversation. I'm sure there's all sorts of things folks listening will want to learn more about you two about physical intelligence. So I want to leave the last word to you both. Where can folks go to learn more about what you guys are doing and any research or jobs pages or anything you want to point folks to? The the floor is yours.

• 1:08:41Speaker 01

  Yeah, you should check out our website. It's pi, pi dot website. There is.

• 1:08:46Jacob Effron?

  Us pi dot com.

• 1:08:48Speaker 01

  I think it's up for sale, but it's like super, super expensive. I'd rather spend that money on compute these days.

• 1:08:55Jacob Effron?

  Fair, fair.

• 1:08:56Speaker 01

  And you can see all our blog posts or all papers. We're fairly open. We not just publish videos, we publish the full write-ups. We also open source our models. You can find them there and we'll continue publishing research there. So keep following us there. Check us out on Twitter as well.

• 1:09:13Speaker 01

  And yeah, follow us. You should you should stay in touch.

• 1:09:15Speaker 02

  Also, you can really write us an email. There's research at physicalintelligence dot company. We we will answer and like we we always like hearing what people think about our work and how they think it could impact their lives and so forth.

• 1:09:31Jacob Effron?

  Amazing. Well, thank you both so much. This has been super fun.

• 1:09:33Speaker 01

  Thank you. I enjoyed it a lot.