The Robotics Revolution, with Physical Intelligence’s Cofounder Chelsea Finn

This week on No Priors, Elad speaks with Chelsea Finn, cofounder of Physical Intelligence and currently Associate Professor at Stanford, leading the Intelligence through Learning and Interaction Lab. They dive into how robots learn, the challenges of training AI models for the physical world, and the importance of diverse data in reaching generalizable intelligence. Chelsea explains the evolving landscape of open-source vs. closed-source robotics and where AI models are likely to have the biggest impact first. They also compare the development of robotics to self-driving cars, explore the future of humanoid and non-humanoid robots, and discuss what’s still missing for AI to function effectively in the real world. If you’re curious about the next phase of AI beyond the digital space, this episode is a must-listen. Sign up for new podcasts every week. Email feedback to [email protected] Follow us on Twitter: @NoPriorsPod | @Saranormous | @EladGil | @ChelseaFinn Show Notes: 0:00 Introduction 0:31 Chelsea’s background in robotics 3:10 Physical Intelligence 5:13 Defining their approach and model architecture 7:39 Reaching generalizability and diversifying robot data 9:46 Open source vs. closed source 12:32 Where will PI’s models integrate first? 14:34 Humanoid as a form factor 16:28 Embodied intelligence 17:36 Key turning points in robotics progress 20:05 Hierarchical interactive robot and decision-making 22:21 Choosing data inputs 26:25 Self driving vs robotics market 28:37 Advice to robotics founders 29:24 Observational data and data generation 31:57 Future robotic forms

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:05Speaker 00

  Hi listeners, welcome to No Priors. This week, we're speaking to Chelsea Finn, co-founder of Physical Intelligence, a company bringing general-purpose AI into the physical world. Chelsea co-founded Physical Intelligence alongside a team of leading researchers and minds in the field. She's an associate professor of computer science and electrical engineering at Stanford University, and prior to that, she worked at Google Brain and was at Berkeley. Chelsea's research focuses on how AI systems can acquire general-purpose skills through interactions with the world. So Chelsea, thank you so much for joining us today on No Priors.

• 0:33Speaker 01

  Yeah, thanks for having me.

• 0:34Speaker 00

  You've done a lot of really important storied work in robotics between your work at Google, at Stanford, et cetera. So I would just love to hear a little bit firsthand, your background in terms of your path in the world of robotics. What drew you to it initially and some of the work that you've done?

• 0:48Speaker 01

  Yeah, it's been a long road. At the beginning, I was really excited about the impact that robotics could have in the world. But at the same time, I was also really fascinated by this problem of developing perception and intelligence in machines. And robots embody all of that. And also, sometimes there's some cool math that you can do as well that keeps your brain active, makes you think. And so I think all of that is really fun about working in the field. I started working more seriously in robotics more than 10 years ago at this point, at the start of my PhD at Berkeley. And we were working on neural network control, trying to train neural networks that map from image pixels

• 1:31Speaker 01

  to directly, actually, to motor torques on a robot arm. At the time, this was not very popular, and we've come a long way. And it's a lot more accepted in robotics and also just generally something that a lot of people are excited about. Since that beginning point, it was very clear to me that we could train robots to do pretty cool things, but that getting the robot to do... One of those things in many scenarios with many objects was a major, major challenge. So 10 years ago, we were training robots to like screw a cap onto a bottle and use a spatula to lift an object into a bowl and kind of do a tight insertion or hang up like a hanger on a clothes rack. And so pretty cool stuff, but actually getting the robot to do that in many environments

• 2:16Speaker 01

  with many objects, that's where a big part of the challenge comes in. And I've been thinking about ways to make broader data sets, train on those broader data sets, and also different approaches for learning, whether it be reinforcement learning, video prediction, imitation learning, all those things. And so, yeah, moved from spent a year at Google Brain in between my PhD and joining Stanford, became a professor at Stanford, started a lab there, did a lot of work along all these lines. And then recently started physical intelligence almost a year ago at this point. So I've been on leave from Stanford for that. And it's been really exciting to be able to try to execute on the vision that the co-founders

• 3:01Speaker 01

  that we collectively have and do it with a lot of resources and so forth. And I'm also still advising students at Stanford as well.

• 3:09Speaker 00

  That's really cool. And I guess you started physical intelligence with four other co-founders and an incredibly impressive team. Could you tell us a little bit more about what physical intelligence is working on and the approach that you're taking? Because I think it's a pretty unique slant on the whole field and approach.

• 3:21Speaker 01

  Yeah, so we're trying to build a big neural network model that could ultimately control any robot to do anything in any scenario. And like a big part of our vision is that in the past, robotics is focused on like tried to go deep on one application and like developing a robot to do one thing and then ultimately gotten kind of stuck in that one application. It's really hard to like solve one thing and then try to get out of that and broaden. And instead, we're really in it for the long term to try to address this broader problem of physical intelligence in the real world. We're thinking a lot about generalization, generalists, and unlike other robotics companies,

• 4:06Speaker 01

  we think that Being able to leverage all of the possible data is very important. And this comes down to actually not just leveraging data from one robot, but from any robot platform that might have six joints or seven joints or two arms or one arm. We've seen a lot of evidence that you could actually transfer a lot of rich information across these different embodiments and allows you to use data. And also, if you iterate on your robot platform, you don't have to throw all your data away. I have faced a lot of pain in the past where we got a new version of the robot and then your policy doesn't work. And it's a really painful process to try to get back to where you were on the previous robot iteration. So yeah, trying to build generalist robots and essentially kind of develop foundation models

• 4:49Speaker 01

  that will power the next generation of robots in the real world.

• 4:55Speaker 00

  That's really cool because I mean, I guess there's a lot of sort of parallels to the large language model world where, you know, really a mixture of deep learning, the transformer architecture in scale has really proven out that you can get real generalizability and different forms of transfer between different areas. Could you tell us a little bit more about the architecture you're taking or the approach or, you know, how you're thinking about the basis for the foundation model that you're developing?

• 5:19Speaker 01

  At the beginning, we were just getting off the ground. We were trying to scale data collection. And a big part of that is unlike in language, we don't have like Wikipedia or an internet of robot motions. And we're really excited about scaling data on real robots in the real world. This is, this kind of real data is what has fueled machine learning advances in the past. And a big part of that is we actually need to collect that data and that looks like teleoperating robots in the physical world. We're also exploring other ways of scaling data as well, but the kind of bread and butter is scaling real robot data. We released something in late October where we showed some of our initial efforts around scaling data and how we can learn very complex tasks of folding laundry,

• 6:05Speaker 01

  cleaning tables, constructing a cardboard box. Now, where we are in that journey is really thinking a lot about language interaction and generalization to different environments. So. What we showed in October was the robot in one environment and it was trained, it had data in that environment. We did, we were able to see some amount of generalization. So it was able to fold shirts it had never seen before, fold shorts it has never seen before, but the degree of generalization was very limited and you also couldn't interact with it in any way. You couldn't prompt it and tell you what you want to do beyond kind of fairly basic things that it saw in the training data. And so being able to handle

• 6:46Speaker 01

  lots of different prompts in lots of different environments is a big focus right now. And in terms of the architecture, we're, we're using transformers and we are using pre-trained models, pre-trained vision language models. And that allows you to leverage all of the rich information in the internet. We had a research result a couple of years ago where we showed that if you leverage vision language models, then you could actually get the robot to do tasks that require concepts that were never in the robot's training data, but were in the internet. Like, one famous example is that you can pass a Coke can to Taylor Swift, or a picture of Taylor Swift, and the robot has never seen Taylor Swift in person, but the internet has lots of images of Taylor Swift in it. And you can leverage all of the information in that data and then the weights of the pre-trained model to kind

• 7:30Speaker 01

  of transfer that to the robot. So we're not starting from scratch and that helps a lot as well. So that's a little bit about the approach. Happy to dive deeper as well.

• 7:37Speaker 00

  That's really amazing. And then what do you think is the main basis then for really getting to generalizability? Is it scaling data further? Is it scaling compute? Is it a combination of the two? It's other forms of post-training? Is something like, I'm just sort of curious, like, as you think through the common pieces that people look at now, I'm sort of curious what you think needs to get filled in. Obviously, on the, again, the more language model world, people are spending a lot of time on reasoning modules and other things like that as well. So I'm curious, like, what are the components that you feel are missing right now?

• 8:05Speaker 01

  Yeah, so I think the number one thing, and this is kind of the boring thing, is just getting more diverse robot data. So for that release that we had in late October last year, we collected data in three buildings, technically. The internet, for example, and everything that has fueled language models and vision models is way, way more diverse than that, because the internet is pictures that are taken by lots of people and text written by lots of different people. And so just trying to collect data in many more diverse places and with many more objects, many more tasks. So scaling the diversity of the data, not just the quantity of the data, is very important. And that's a big thing that we're focusing on right now, actually bringing our robots into lots of different places and collecting data in

• 8:49Speaker 01

  it. As a side product of that, we also learn what it takes to actually get your robot to be operational and functional in lots of different places. And that is a really nice byproduct, because if you actually want to get robots to work in the real world, you need to be able to do that. So that's the number one thing. But then we're also exploring other things, leveraging videos of people, again, leveraging data from the web, leveraging pre-trained models, thinking about reasoning, although more basic forms of reasoning, in order to, for example, put a dirty shirt into a hamper. If you can recognize where the shirt is and where the hamper is and what you need to do to accomplish that task, that's useful. Or if you want to make a sandwich, and the user has a particular

• 9:34Speaker 01

  request in mind, you should reason through that request. If they're allergic to pickles, you probably shouldn't put pickles on the sandwich, things like that. So there's some basic things around there, although the number one thing is just more diverse robot data.

• 9:46Speaker 00

  And then I think a lot of the approach you've taken to data has really been an emphasis on releasing open source models and packages for robotics. Do you think that's the long-term path? Do you think it's open core? Do you think it's eventually proprietary models? Or how do you think about that? And in the context of the industry? Because it feels like there's a few different robotics companies now, each taking different approaches in terms of either hardware only, I mean, because I mean, hardware plus software, and they're focused on a specific hardware footprint. There's software and there's closed source versus open source, if you're just doing the software. So I'm sort of curious where in that spectrum, physical intelligence lies.

• 10:18Speaker 01

  Definitely. So we've actually been quite open. We've, not only have we open sourced some of the weights and release details and technical papers, we've actually also been working with hardware companies and giving designs of robots to hardware companies. And some people have actually, like, when I tell people this, sometimes they're actually really shocked that, like, like, what about the IP? What about, I don't know, confidentiality and stuff like that? And we've actually made this, made a very intentional choice around this. There's a couple of reasons for it. One is that we think that the field, it's really just the beginning and these models will be so, so much better and the robots should be so, so much better in a year, in three years. And we want to support the development of the research and we want to support the

• 11:02Speaker 01

  community, support the robots so that when we hopefully develop the technology of these generalist models, the world will be more ready for it. We'll have better, like more robust robots that are able to leverage those models, people who have the expertise and understand what it requires to use those models. And then the other thing is also like, we have a really fantastic team of researchers and engineers and really, really fantastic researchers and engineers want to work at companies that are, that are open, especially researchers where they can get kind of credit for their work and share their ideas, talk about their ideas. And we think that having the best researchers and engineers will be necessary for solving this problem. The last thing that I'll mention is that I think the biggest risk with this bet is

• 11:47Speaker 01

  that it won't work. Like, I'm not really worried about competitors, I'm more worried that no one will solve the problem.

• 11:54Speaker 00

  Oh, interesting. And why do you worry about that?

• 11:56Speaker 01

  I think robotics is, it's very hard and there have been many, many failures in the past. And unlike when you're like recognizing an object in an image, there's very little tolerance for error. You can miss a grasp on an object or like not make, like the difference between making contact and not making contact in an object is so small and it has a massive impact on the, on the outcome of whether you're, whether a robot can actually successfully manipulate the object. And I mean, that's just one example. There's challenges on the data side of collecting data. Well, just anything involving hardware is hard as well.

• 12:31Speaker 00

  I guess we have a number of examples now of robots in the physical world. You know, everything from autopilot on a jet on through to some forms of pick and pack and/or other types of robots in distribution centers. And there's obviously the different robots involved with manufacturing, particularly in automotive, right? So there's been a handful of more constrained environments where. People have been using them in different ways. Where do you think the impact of these models will first show up? Because to your point, there are certain things where you have very low tolerance for error. And then there's a lot of fields where actually it's okay, or maybe you can constrain the problem sufficiently relative to the capabilities of the model that it works fine. Where do you think physical intelligence will have the nearest term impact?

• 13:13Speaker 00

  Or in general, the field of robotics and these new approaches will substantiate themselves?

• 13:18Speaker 01

  Yeah, as a company, we're really focused on the long-term problem and not at like any one particular application because of the failure modes that can come up when you focus on one application. I don't know where the first applications will be. I think one thing that's actually challenging is that typically in machine learning, a lot of the successful applications of like recommender systems, language models, like image detection. A lot of the consumers of that, of the model outputs are actually humans who could actually check it. And the humans are good at the thing. A lot of the very natural applications of robots is actually the robot doing something autonomously on its own, where it's not like a human consuming the commanded arm position, for example, and then checking it and then validating it and so forth.

• 14:02Speaker 01

  And so I think we need to think about new ways of having some kind of tolerance for mistakes or scenarios where that's fine, or scenarios where humans and robots can work together. That's, I think, one big challenge that will come up when trying to actually deploy these. And some of the language interaction work that we've been doing is actually motivated by this challenge where we think it's really important for humans to be able to kind of provide input for how they want the robot to behave and what they want the robot to do, how they want the robot to help in a particular scenario.

• 14:31Speaker 00

  That makes sense. I guess the other form of generalizability, to some extent, at least in our current world, is the human form, right? And so some people are specifically focused on humanoid robots like Tesla and others under the assumption that the world is designed for people and therefore is the perfect form factor to coexist with people. And then other people have taken very different approaches in terms of saying, well, I need something that's more specialized for the home in certain ways or for factories or manufacturing, or you name it. What is your view on kind of humanoid versus not?

• 15:00Speaker 01

  On one hand, I think humanoids are really cool. And I have one in my lab at Stanford. On the other hand, I think that they're a little overrated. And one way to practically look at it is I think that we're generally fairly ball-necked on data right now. And some people argue that with humanoids, you can maybe collect data more easily because it matches the human form factor. And so maybe it'd be easier to mimic humans. And I've actually heard people make those arguments. But if you've ever actually tried to teleoperate a humanoid, it's actually a lot harder to teleoperate than that a static manipulator or a mobile manipulator with wheels. Optimizing for being able to collect data, I think, is very important. Because if we can get to the point where we have more data than we could ever want,

• 15:46Speaker 01

  then it just comes down to research and compute and evaluations. And so we're optimizing for, that's one of the things we're kind of optimizing for. And so we're using cheap robots. We're using robots that we can very easily develop teleoperation interfaces for, in which you can do teleoperation very quickly and collect diverse data, collect lots of data.

• 16:08Speaker 00

  Yeah, it's funny, there was that viral fake Kim Kardashian video of her going shopping with a robot following her around, carrying all of her shopping bags. When I saw that, I really wanted a humanoid robot to follow me around. If it were, that'd be really funny to do that. So I'm hopeful that someday I can use your software to cause a robot to follow me around to do things. So exciting future. How do you think about the embodied model of development versus not on some of these things in terms of, that's another sort of, I think, set of trade offs that some people are making or deciding between.

• 16:38Speaker 01

  Well, the AI community is very focused on just like language models, vision language models and so forth. And there's like a ton of hype around like reasoning and stuff like that. Oh, let's create like the most intelligent thing. I feel like actually people underestimate how much intelligence goes into motor control. Many, many years of evolution is what led to us being able to use our hands the way that we do. And there are many animals that they can't do it, even though they had so much, so many years of evolution. And so I think that there's actually so much complexity and intelligence that goes into being able to do something as basic as like make a bowl of cereal or pour a glass of water. And yeah, so in some ways, I think that actually like embodied intelligence or physical intelligence is very core to intelligence

• 17:23Speaker 01

  and maybe kind of underrated compared to some of the less embodied models.

• 17:28Speaker 00

  One of the papers that I really loved over the last couple of years in robotics was your Aloha paper. And I thought it was a very clever approach. What is some of the research over the last two or three years that you think has really caused this flurry of activity? Because I feel like there's been a number of people now starting companies in this area because a lot of people feel like now's the time to do it. And I'm a little bit curious what research you feel was the basis for that shift and people thinking this was a good place to work.

• 17:52Speaker 01

  At least for us, there were a few things that we felt like were turning points that felt like where it felt like the field was moving a lot faster compared to where it was before. One was the SAICAN work where we found that you can plan with language models as kind of the high level part and then kind of plug that in with a low level model to get a model to do long horizon tasks. One was the RT2 work, which showed that you could do the Taylor Swift example that I mentioned earlier and be able to plug in kind of the, a lot of the web data and get better generalization on robots. A third was our RTX work where we were, we actually were able to train models across

• 18:35Speaker 01

  robot embodiments and significantly, we basically took all the robot data that different research labs had, is a huge effort to aggregate that into a common format and train on it. And we also, when we trained on that, we actually found that we could take a checkpoint, send that model checkpoint to another lab halfway across the country and the grad student at that lab could run the checkpoint on the robot and it would actually, more often than not, do better than the model that they had specifically iterated on themselves in their own lab. And that was like another big sign that like this stuff is actually starting to work and that you can get benefit across, by pooling data across different robots. And then also, like you mentioned, I think the Aloha work and then later the mobile Aloha

• 19:18Speaker 01

  work was work that showed that you can teleoperate and get models to train pretty complicated, dexterous manipulation tasks. We also had a follow-up paper with the shoelace tying. That was a fun project because someone said that they would retire if they saw a robot tie shoe laces, so.

• 19:34Speaker 00

  Did they retire?

• 19:35Speaker 01

  They did not retire.

• 19:36Speaker 00

  Oh, that's awful. We need to force them into retirement, whoever that person is. We need to follow up on that.

• 19:41Speaker 01

  Yeah, so I think that those were a few examples. And, and so, yeah, I think we've seen a ton of progress in the field. I also. It seems like after we started Pi that that was also kind of a sign to others that if the experts are really willing to bet on this, then something, maybe something will happen.

• 19:59Speaker 00

  So one thing that you all came out with today from Pi was what you call a hierarchical interactive robot or high robot. Can you tell us a little bit more about that?

• 20:09Speaker 01

  So this was a really fun project. There's two things that we're trying to look at here. One is that if you need to do like a longer horizon task, meaning a task that might take minutes to do, then if you just train a single policy to like output actions based on images, like if you're trying to make a sandwich and you train a policy that's just outputting the next motor command, that might not do as well as something that's actually kind of thinking through the steps to accomplish that task. That was kind of the first component. And that's where the hierarchy comes in. And the second component is a lot of the times when we train robot policies, we're just saying like, we'll take our data, we'll annotate it and say like, this is picking up the sponge, this is putting the bowl in

• 20:53Speaker 01

  the bin, this segment is, I don't know, folding the shirt. And then you get a policy that can like follow those basic commands of like fold the shirt or pick up the cup, those sorts of things. But at the end of the day, we, we don't want robots just to be able to do that. We want them to be able to interact with us where we can say like, oh, I'm a vegetarian, can you make me a sandwich? Oh, and I like, I'm allergic to pickles. So like, maybe don't include those. And maybe also be able to interject in the middle and say like, oh, hold off on the tomatoes or something. It's actually kind of a big gap between something that can just follow like an instruction like pick up the cup and something that could be able to handle those kinds of prompts and those situated corrections and so forth. And so we developed a system that basically has one model that takes as

• 21:37Speaker 01

  input the prompt and kind of reason through is able to output like the next step that the robot should follow. And that might be, that's kind of like, it's going to tell it to, then the next thing will be pick up the tomato, for example. And then a lower level model that takes its input, pick up the tomato and outputs the sequence of motor commands for the next like half second. That's the gist of it. We, it was a lot of fun because we actually got the robot to make a vegetarian sandwich or a ham and cheese sandwich or whatever. We also did a grocery shopping example and a table cleaning example. And I was excited about it at first because it was just like cool to see the robot be able to respond to different prompts and do these challenging tasks. And second, because it actually seems like the right approach for solving the problem.

• 22:20Speaker 00

  On the technical capability side, one thing I was wondering about a little bit was, if I look at the world of self-driving, there's a few different approaches that are being taken. And one of the approaches that is the more Waymo-centric one is really incorporating a variety of other types of sensors besides just vision. So you have LiDAR and a few other things as ways to augment the self-driving capabilities of a vehicle. Where do you think we are in terms of the sensors that we use in the context of robots? Is there anything missing? Is there anything we should add? Are there types of inputs or feedback that we need to incorporate that haven't been incorporated yet?

• 22:53Speaker 01

  So we've gotten very far just with vision, with RGB images even. And we typically will have one or multiple external, what we call base cameras that are looking at the scene and also cameras mounted to each of the wrists of the robot. We can get very, very far with that. I would love if like skin, if we could give our robots skin. Unfortunately, a lot of the tactile sensors that are out there are either far less robust than skin, far more expensive, or very, very low resolution. So there's a lot of challenges on the hardware side there. And we found that actually that mounting RGB cameras to their wrists ends up being

• 23:36Speaker 01

  very, very helpful and probably giving you a lot of the same information that tactile sensors can give you.

• 23:41Speaker 00

  Because when I think about the set of sensors that are incorporated into a person, obviously to your point, there's the tactile sensors effectively, right? And then there's heat sensors. There's actually a variety of things that are incorporated that people usually don't really think about much.

• 23:54Speaker 01

  Absolutely.

• 23:55Speaker 00

  And I'm just sort of curious, like, how many of those are actually necessary in the context of robotics versus not? What are some of the things we should think about? Like, just if we extrapolate off of humans or animals or other, you know?

• 24:05Speaker 01

  It's a great question. I mean, for the sandwich making, you could argue that you'd want the robot to be able to taste the sandwich to know if it's good or not.

• 24:11Speaker 00

  Or smell it at least, you know?

• 24:13Speaker 01

  Yeah, I've made a lot of arguments for smell to Sergey in the past because it's, there's a lot of nice things about smell, although you've never actually attempted it before.

• 24:23Speaker 00

  Yeah.

• 24:23Speaker 01

  In some ways, the redundancy is nice. For example, and I think like audio, for example, like a human, if you hear something that's unexpected, it can actually kind of alert you to something. In many cases, it might actually be very, very redundant with your other sensors because you might be able to actually see something fall, for example. And that redundancy can lead to robustness. For us, it's not. Currently not a priority to to look into these sensors because we think that the bottleneck right now is elsewhere, is on the data front, is on kind of the architectures and so forth. The other thing I'll mention is actually right now, we're most like, our policies right now do not have any memory. They only look at the current image frame. They can't remember

• 25:06Speaker 01

  even half a second prior. And so I would much rather add memory to our models before we add other sensors. We can have commercially viable robots for a number of applications without other sensors.

• 25:20Speaker 00

  What do you think is a timeframe on that?

• 25:21Speaker 01

  I have no idea.

• 25:23Speaker 00

  Yeah.

• 25:23Speaker 01

  Some parts of robotics that make it easier than self-driving and some parts that make it harder. On one hand, it's harder because you're not just, like, it's just a much higher dimensional space. Like, even our static robots have 14 dimensions, seven for each arm. You need to be more precise in many scenarios than driving. We also don't have as much data right off the bat. On the other hand, with driving, I feel like you kind of need to solve the entire distribution to be have anything that's viable. You have to be able to handle an intersection at any time of day, or with any kind of possible pedestrian scenario, or other cars and all that. Whereas in robotics, I think that there's lots of commercial use cases where you don't have to handle this whole huge distribution.

• 26:08Speaker 01

  And you also don't have as much of a safety risk as well. That makes me optimistic. And I think that also, like, all the results in self-driving have been very encouraging, especially like the number of Waymos that I see in San Francisco.

• 26:21Speaker 00

  Yeah, it's been very impressive to watch them scale up usage. The thing I found striking about the self-driving world is, you know, there was two dozen startups started roughly, I don't know, 10 to 15 years ago around self-driving. And the industry is largely consolidated, at least in the US, and obviously the China market's a bit different, but it's consolidated into Waymo and Tesla, which effectively were two incumbents, right? Google and Tesla was an automaker. And then there's maybe one or two startups that either SPAC'd and went public, or are still kind of working in the area. And then most of it's kind of fallen off, right? And the set of players that existed at that starting moment 10, 15 years ago was kind of the same players that ended up actually winning, right? There hasn't been a lot of dynamism in the industry, other than just consolidation.

• 27:04Speaker 00

  Do you think that the main robotics players are the companies that exist today? And do you think there's any sort of incumbency bias that's likely?

• 27:11Speaker 01

  A year ago, like, it would be completely different. And I think that we've had so many new players recently. I think that the fact that self-driving was like that suggested that it might have been a bit too early 10 years ago. And I think that arguably it was like, I think deep learning has come a long, long way since then.

• 27:34Speaker 01

  And so I think that that's also part of it. And I think that the same with robotics, like if you were to ask me 10 years ago, or even even five years ago, honestly, I think it would be too early. I think the technology like wasn't there yet. We might still be too early for all we know. I mean, it's a very hard problem and like how hard self-driving has been is, I think, is a testament to how hard it is to to build intelligence in the physical world. In terms of like major players, there's a lot of things that I've really liked about the startup environment and a lot of things that were very hard to do when I was at Google and Google is an amazing place in many, many ways. But like, as one example, taking a robot off campus was like almost a nonstarter, just for code security reasons.

• 28:16Speaker 01

  And if you want to collect diverse data, taking robots off campus is is valuable. You can move a lot faster when you're a smaller company, where you don't have kind of restrictions, red tape, that sort of things. The really big companies, they have a ton of capital and so they can last longer. But I also think that there's, they're going to move slower too.

• 28:36Speaker 00

  If you were to give advice to somebody thinking about starting a robotics company today, what would you suggest they do or where would you point them in terms of what to focus on?

• 28:43Speaker 01

  I think the main advice that I would give someone trying to start a company would be to try to learn as much as possible quickly. And I think that actually like trying to deploy quickly and learn and iterate quickly. That's probably. The the main advice and try to, yeah, like actually get the robots out there, learn from that. I'm also not sure if I'm the the best person to be giving startup advice because I've only been an entrepreneur myself for 11 months. But yeah, that's probably the advice I give.

• 29:14Speaker 00

  That's cool. I mean, you're running an incredibly exciting startup. So I think you have a full ability to suggest this to people in that area, for sure. I've heard a number of different groups doing is really using observational data of people as part of the training set. So that could be YouTube videos, it could be things that they're recording specifically for the purpose. How do you think about that in the context of training robotic models?

• 29:36Speaker 01

  I think that data can have a lot of value, but I think that by itself, it won't get you very far. And I think that there's actually some really nice analogies you can make where, for example, if you watch like an Olympic swimmer swimmer race, even if you had their strength, just their practice at moving their own muscles to do the to accomplish what they're accomplishing is like essential for being able to do it. Or if you're trying to learn how to hit a tennis ball well, you won't be able to learn it by kind of watching the pros. Now, maybe these examples seem a little bit contrived because they're talking about like experts. The reason why I make those analogies is that we humans are experts at motor control, low level motor control already for a variety

• 30:18Speaker 01

  of things that our robots are not. And I think the robots actually need experience from their own body in order to learn. And so I think that it's really promising to be able to leverage that form of data, especially to expand on the robot's own experience. But it's really going to be essential to like actually have the data from the robot itself, too.

• 30:36Speaker 00

  In some of those cases, is that just general data that you're generating around that robot? Or would you actually have it mimic certain activities? Or how do you think about the data generation? Because you mentioned a little bit about the transfer and generalizability. It's interesting to ask, well, what what is generalizable or not? And what types of data are in orange and things like that?

• 30:54Speaker 01

  I mean, when we collect data, we have it's kind of like puppeteering, like the original Aloha work. And then you can record both the actual motor commands and the sensor, like the camera images. And so that is The like experience for the robot. And then I also think that autonomous experience will play a huge role, just like we've seen in language models. After you get an initial language model, if you can use reinforcement learning to have the robot, the, the language model bootstrapped on its own experience, that's extremely valuable. Yeah. And then in terms of what's generalizable versus not, I think it all comes down to the breadth of the distribution. It's really hard to quantify or measure how broad the robot's

• 31:37Speaker 01

  own experience is. And there's no way to categorize the breadth of the tasks, like how different one task is from another, how different one kitchen is from another, that sort of thing. But we can at least get a rough idea for that breadth by like looking at things like the number of buildings or the number of scenes, those sorts of things.

• 31:56Speaker 00

  And then I guess we talked a lot about humanoid robots and other sort of formats. If you think ahead in terms of the form factors that are likely to exist in N years as this sort of robotic feature comes into play, do you think there's sort of one singular form or there, there are a handful? Is it a rich ecosystem, just like in biology? Like, how do you think about what's going to come out of all this?

• 32:18Speaker 01

  I don't know exactly, but I think that my bet would be on something where there's actually a, a really wide range of different robot platforms. I think Sergey, my co-founder, likes to call it a Cambrian explosion of different robot hardware types and so forth. Once we actually can have the technology that can, the intelligence that can power all those different robots. And I think it's kind of similar to like, we have all these different devices in our kitchen, for example, that can do all these different things for us. And rather than just like one device that can, that cooks the whole meal for us. And so I think we can envision like a world where there's like one kind of robot arm that does things on, on the kitchen that has like some hardware that's optimized for

• 33:03Speaker 01

  that. And maybe also optimized for it to be cheap for that particular use case. And another Hardware that's kind of designed for for like folding clothes or something like that, dishwashing, those sorts of things. These are all like speculation, of course, but I think that a world like that is something where, um, yeah, it's, I think, different from what a lot of people think about.

• 33:23Speaker 00

  In the book The Diamond Age, there's, there's sort of this view of like matter pipes going into homes and you have these 3D printers that make everything for you. And in one case, you're like downloading schematics and then you, you 3D print the thing. And then people who are kind of bootlegging some of this stuff end up with almost evolutionarily base processes to build hardware and then select against certain functionality is the mechanism by which to optimize things. Do you think a future like that is at all likely? Or do you think it's more just, hey, you make the foundation model really good, you have a couple of form factors and, you know, you don't need that much specialization if you have enough generalizability in the actual underlying intelligence?

• 33:58Speaker 01

  I think a world like that is very possible. And I think that you can make a cheaper hardware, piece of hardware, if you are optimizing for a particular use case and maybe it'd be like also be a lot faster and so forth. Um, yeah, obviously very hard to predict.

• 34:12Speaker 00

  Yeah, it's super hard to predict because one of the arguments for a smaller number of hardware platforms is just supply chain, right? It's just going to be cheaper at scale to manufacture all the subcomponents and therefore you're going to collapse down to fewer things because unless there's a dramatic cost advantage, those fewer things will be more easily scalable, reproducible, cheap to make, et cetera. Right? If you look at sort of general hardware purchase. So it's an interesting question in terms of that trade off between those two tensions.

• 34:37Speaker 01

  Yeah, although maybe we'll have robots in the supply chain that can manufacture any customizable device that you want.

• 34:44Speaker 00

  It's robots all the way down. So that's our future. Yeah. Well, thanks so much for joining me today. It was a super interesting conversation and we covered a wide variety of things. So I really appreciate your time.

• 34:53Speaker 01

  Yeah, this is fun.

• 34:55Sarah Guo?

  Find us on Twitter at NoPriorsPod. Subscribe to our YouTube channel if you want to see our faces. Follow the show on Apple Podcasts, Spotify, or wherever you listen. That way you get a new episode every week. And sign up for emails or find transcripts for every episode at no-priors.com.