Why Robots Still Struggle With Simple Tasks (And What Might Finally Change That) | Karol Hausman, Co-Founder & CEO of Physical Intelligence

Karol Hausman https://x.com/hausman_k is the co-founder and CEO of Physical Intelligence https://www.pi.website/, a robotics company building a general-purpose “AI brain for the physical world.” The company has raised more than $1 billion in funding to develop foundation models that allow robots to operate across many machines, environments, and tasks rather than being programmed for a single purpose. The core thesis: the same scaling dynamics that transformed language models may also unlock robotic intelligence. But only if you resist every commercial pressure pushing you toward specialization. The central challenge isn’t mechanical design. It’s intelligence: how robots learn, generalize, and interact with a physical world that is far harder to simulate than it is to describe. Before launching Physical Intelligence, Karol worked at Google Brain and Stanford University, studying robot learning alongside researchers Sergey Levine https://x.com/svlevine and Chelsea Finn https://www.linkedin.com/in/cbfinn/, who later became his co-founders. In our conversation, we explore: How growing up in a small town in Poland and watching Star Wars sparked Karol’s fascination with robotsThe moment a lecture from Sergey Levine convinced him to abandon his PhD research direction and pivot fully to deep learningWhy robotics has historically lagged behind breakthroughs in language modelsThe case for building a general “AI brain” for the physical world rather than a single specialized robotThe role of real-world data in training robots, the limits of simulation, and how deployment could create a powerful data flywheelThe return of reinforcement learning and the parallels between human learning and robot trainingThe unique challenges of physical intelligence and why robots must operate with far higher reliability than language models— Thank you to the partners who make this possible Brex https://www.brex.com/mario: The intelligent finance platform. Granola https://granola.ai/mario: The app that might actually make you love meetings. — Transcript: https://www.generalist.com/p/karol-hausman-physical-intelligence — Timestamps (00:00) Intro (04:05) Karol’s early fascination with robots (07:38) How Karol relates to Fei-Fei Li’s biography (08:52) What inspired Karol to build better robots (11:19) Philosophical influences (15:33) Parallels between The Inner Game of Tennis and robotics (18:21) Karol’s entry point to robotics and PhD program (25:49) Combining robotics with LLMs: The Taylor Swift demo (30:48) The 1970s SHRDLU AI experiment (32:33) Founding Physical Intelligence (35:13) How Lachy Groom got involved (39:40) How research shapes what Physical Intelligence builds (45:22) The importance of real-world data (49:07) The return of reinforcement learning in robotics (53:31) The risk of commercializing too early (55:47) Finding the right partners for the business (57:13) Open research questions (1:00:00) NVIDIA’s simulation engines (1:01:57) The surprising speed of progress (1:04:16) Reliability in robotics (1:07:31) Compensating for missing senses (1:12:28) Book recommendation — Follow Karol Hausman LinkedIn: https://www.linkedin.com/in/karolhausman X: https://x.com/hausman_k — Resources and episode mentions: https://www.generalist.com/p/karol-hausman-physical-intelligence — Production and marketing by penname.co https://penname.co/. For inquiries about sponsoring the podcast, email [email protected].

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  The more robots you deploy, the better the models should get. Because I think the models will keep on getting better. They will be able to absorb more and more data. As I started looking into robotics and started studying it, it was very disappointing. It seemed like there are no intelligent robots out there, and no one was working on them. Every robot you would see would be this pre-programmed machine that just goes from A to B and does it repeatedly and has no intelligence whatsoever. That was maybe like a wake-up call. So many people excited about this future that we saw in the movies, that we read about in the books, and it seems like no one is working on it. We had this one particular experiment with a Coke can in front of a robot and three pictures of different celebrities in front of it. And the prompt we gave to the model we built was, "Put a Coke can on a picture of Taylor Swift." The robot picked it up and then slowly moved it towards Taylor Swift,

• 0:45Speaker 00

  all from internet data. That was the moment where it clicked for us, where you can bring in a lot of prior knowledge from LLMs from the internet and connect it to robot motions. And it felt like it opened another door. Maybe, just maybe, if we do everything right, if we combine it with internet knowledge, if we scale it up, if we do all of the pieces that need to be done, it might work. And at that point, it became clear that the way to accomplish this is to create an organization whose sole purpose is to solve physical intelligence.

• 1:19Mario Gabriele?

  We taught machines to beat the world's best chess players long before we taught them to reliably fold a towel or carry a coffee cup. That paradox is at the heart of the robotics industry, responsible for many of its false dawns over the years. Carol Hausmann believes that he's found the way forward. The CEO of Physical Intelligence has raised more than $1 billion in venture funding to build an AI brain for the real world, one that helps robots generalize accurately across different tasks and environments. He's doing so by taking a very different approach from many of his best-funded competitors, focusing on gathering data from actual interactions rather than through simulations. In today's episode,

• 2:04Mario Gabriele?

  Carol and I discuss lessons learned from the inner game of tennis, why reinforcement learning is making a comeback, and the challenges of creating true physical intelligence. I'm Mario, and this is The Generalist.

• 2:18Speaker 02

  This episode is brought to you by Brex. If you're a founder, the hardest part isn't the idea. It's scaling fast without getting buried in back office work. That's where Brex comes in. Brex is the intelligent finance platform for founders. With Brex, you get high-limit corporate cards, easy banking, and high-yield treasury, plus a team of AI agents that handle manual finance tasks for you. They take care of things like expenses, all according to your rules, so you can move faster while staying in full control. One in three startups in the US already runs on Brex. You can too, at brex.com/mario. I'm really excited about today's sponsor, Granola.

• 3:00Speaker 02

  Simply put, Granola is the AI notepad for people in back-to-back meetings. I've been using Granola for over a year now, and honestly, it's a tool that has transformed the way I work. Granola takes meeting notes for you without any intrusive bots joining your calls. You can jot down rough notes, like you always do, and in the background, Granola transcribes and turns those notes into clear, useful notes when the meeting ends. You can also chat with your notes, which is one of my favorite features. If someone says something on the call that you didn't quite catch or want to learn more about, Granola can help you out. It's an amazing way to be better informed during a conversation without having to interrupt everyone else's flow.

• 3:41Speaker 02

  You can also have Granola review all your recent conversations to pull out to-dos, write a weekly recap, or surface interesting ideas you might have forgotten, another thing I love. To get started with Granola, head to granola.ai/mario. And for new users, you can get three months free with the code MARIO. So go to granola.ai/mario and use code MARIO for three months free.

• 4:06Mario Gabriele?

  I'm so excited to have this conversation today. In studying what you're building with physical intelligence, naturally, much of the conversation is around the technology. But the more I looked into your story and the company, it feels like what you're building is really personal and connects to some of your background in philosophy and all these other things. So I'd love to start maybe with some of that first. When were your earliest memories of being excited by robots?

• 4:37Speaker 00

  Probably with watching movies as a kid, I think I was a big Star Wars fan. I was reading all the, all the books, watching all the movies multiple times. And I, I really loved the way that robots were portrayed in that world. The diversity of them, the different functionalities, the way they interacted with people, the way that there are good robots and bad robots. But I don't know if I can pinpoint a, a specific point. I remember just, just growing up being fascinated by that concept, that you can build something that is so human-like. And I always thought that if you were able to do that, it would allow us to understand ourselves a little bit better.

• 5:20Speaker 00

  And I was fascinated by kind of the things that I think people are just fascinated with as they're, as they're growing up, you know, asking deep questions about the nature of reality, how the brain works, what does it mean to, to think? And I thought that robots probably are going to have some answers to these questions. Because reading other kinds of books or studying philosophy didn't have good answers to these questions. Then, more importantly, as I started looking into robotics and started studying it, the, the first impression I had was that it was very disappointing. In that every robot you would see would, basically it seemed like there are no intelligent robots out there

• 6:04Speaker 00

  and no one was working on them. Every robot you would see would be this, this pre-programmed machine that just goes from A to B and does it repeatedly and has no intelligence whatsoever. So I think that was maybe the, the most defining moment for me when I saw that. Because that was like a, like a wake-up call of, you know, there are so many people excited about this future that we saw in the movies, that we read about in the books, and it seems like no one is working on it.

• 6:30Mario Gabriele?

  You said that, you know, in some ways you had, you know, that you're asking the, the classic questions of every child about the nature of the universe and the human mind. Uh, I don't know if that's as universal as, as you might imagine it to be. I think probably there are some kids who really become fascinated by that, but, but maybe not all. Were your parents scientists? Were there things that they did that sort of encouraged that natural curiosity that you must have had?

• 6:54Speaker 00

  Not really, no. Um, my dad was a mechanic, uh, is a mechanic and my mom uh was an entrepreneur. So not really, there wasn't a lot of science in, at, at, at home. I think it was mostly coming from, from watching movies and being fascinated by it. Um, I also went to, so I come from a small town in Poland. I went to high school that had very high talent density for that, for that area. And I remember that being the, an environment where a lot of these questions were being asked. And it was very encouraged. So I found a, a group of friends that were really fascinated by similar things.

• 7:35Speaker 00

  And I think that would really increase my interest in that area. This was just a thing that you would talk about in high school.

• 7:40Mario Gabriele?

  You had a tweet that I, I saw where you were talking about Fei Fei Li's biography and how it sort of spoke to you in some sense that, you know, the immigration story and also the fact of, you know, having teachers that believed in you along the way were, were sort of parallels to your own life. In high school, was that when you sort of had that first batch of those teachers who maybe saw some of that, you know, really remarkable early talent?

• 8:04Speaker 00

  I think it was throughout. It wasn't just high school. It was, it was throughout. There were multiple people that now looking back, I'm really grateful to, that either saw something or just encouraged me to, um, to dig deeper. Uh, or motivated me or give me some opportunities that I wouldn't have otherwise. And I think at, at every single stage, there was someone I could point to. And reading Fei Fei's book, um, just reminded me of that. And I remember after I read it, I, I thanked some of them and thanked them, um, for everything they did for me. So yeah, it was, it was a long journey from, from Poland, through Germany,

• 8:43Speaker 00

  to the US, going through multiple places in the US. And, um, it felt like at every single stage of the journey, there was someone, um, kind of rooting for me and, and helping me.

• 8:53Mario Gabriele?

  You know, before you realized that you wanted to work in robotics in particular, as an undergrad, you studied sort of computer science and philosophy. But I'm curious if there were interests before that, like as a kid, were you thinking, I'm going to be a astrophysicist? Or, uh, I don't know, a neuroscientist?

• 9:12Speaker 00

  Yeah, I was I was really into physics, actually. Right after high school, I applied to a lot of physics and math programs. And I thought I was gonna be an astrophysicist or a theoretical physicist. Um, I was really fascinated by it. I was reading a lot of books. I was really into physics in high school. But then a lot of my friends decided to go for engineering degrees and we're all gonna go to Warsaw. And I thought that I could maybe try to study both or I could start with engineering and then go back to physics if it turns out to be boring. It wasn't like a very intentional choice at that time. It was more like, these are really smart people. I really wanna stay with my friends. And and and robots are really, really cool. So let me try that and then see how it goes.

• 9:56Speaker 00

  So I went to I went to Warsaw to study. It was called mechatronics at the time, which was basically like a mixture of electronics, some mechanical engineering, computer science. Then I also studied philosophy. But I I think that's when, once I moved to Warsaw, that's when I started realizing that robotics is a thing, that it's a thing that is very disappointing. I remember studying robotics in Warsaw. The the kind of the dream job you could get was um uh being a roboticist at an Avon factory. Ha. Uh, that was right next to Warsaw. That was like the job that everybody was competing for. If you could work for Avon at the on the production line. That's like, that's how you made it in the robotics industry.

• 10:39Speaker 00

  And I remember I went there once and and saw that production line and it was the most uninspiring thing I've ever seen. With robots, as I said before, robots just moving from A to B, not being intelligent whatsoever. And I think that's when when it started clicking for me that I would really like to do something beyond that. And it seemed like no one was working on it, which sparked my interest even more. So that's where I started thinking, who is doing intelligent robots? And is it even a thing? And that's then what defined, I think, the rest of the journey of trying to find those people, uh, move from place to place as I try to, as I try to find them. And then eventually start working on it myself.

• 11:20Mario Gabriele?

  This sort of study of philosophy, as you've looked back on that time, you know, maybe it's a line on the CV at this point and you're like, you know, I, I don't reflect on it necessarily that much. But I'm curious if there are writers that you've ended up coming back to more and more as you start to do this sort of work around, I don't know, the nature of consciousness or the, you know, the certain senses that make us more or less human.

• 11:44Speaker 00

  It was a really interesting period. So I think there was a few realizations I had while studying it. I think one was that a lot of philosophy is the history of philosophy. It's not philosophy itself. It's talking about what others thought about years ago. From the outside looking in, it always, it always sounds like, you know, Plato or, or people like that. They're, they, they, they were these geniuses that kind of like figured everything out way before their time. But then when you actually read their works, and that was, you know, part of the, part of the, the deal of studying philosophy, you realize that there's a very small percentage of things that kind of sound right.

• 12:27Speaker 00

  And there are a lot of things that they said that are absolutely wrong.

• 12:30Mario Gabriele?

  Yes.

• 12:31Speaker 00

  And, and that was quite eye opening that, you know, the, the history of philosophy shows that there are people who made many, many mistakes along the way. And thought that the world worked certain way when it's now very clear that it doesn't. They had some really interesting observations and I think we, we often overindex on those. But we were wrong a lot. There are still some philosophers that I remember studying that I think were very relevant and I was a big fan of. One of them is Spinoza. That was, I think, my favorite philosopher when I, when I look back at, at what I studied. But then there were some subjects, very, very few, that were more about philosophizing today. Rather than looking at the history.

• 13:15Speaker 00

  And I think those were the ones that were the the most fun for me. There was this one subject called ontology, which was a study of of things of how they are. I think it's even difficult to to comprehend what this means. But it was one of those subjects where you just like sit and contemplate reality. It just was very fun, even though it was very unstructured and it was. It wasn't clear what is what is right and what isn't. It just seemed like it touched something real. And probably that's the subject I remember the most. That's where I had the most fun, much more so the history of philosophy.

• 13:55Mario Gabriele?

  So interesting and and these things I love to explore because there is, to me at least, such deep parallels between some of the things that you're trying to build to fruition. Why Spinoza? What was it about him that that seemed deep to you or true?

• 14:09Speaker 00

  The the one thing that that really resonated with me was this idea that, you know, at that time, a lot of philosophers were contemplating the nature of God and the nature of reality. And there was often this idea that God is this separate entity that either was the creator of everything or had some kind of way of controlling everything that is that is happening in the universe. And I think with with him, what was what was so interesting was that he said that everything around us, like all of reality, that is God, that is that is what it is. And there is some underlying structure to it, and that structure in it itself

• 14:52Speaker 00

  is where the beauty lies and where the where the intelligence lies. And I and I thought that was very thought provoking. And as you study physics and things like robotics or AI, I think more and more you realize that there's some underlying structure to things. And in some ways, that structure is quite counterintuitive, like why why it's there? Why is it so that you can describe all the complexity of the world in a simple equation rather than in a million different equations? And the more beautiful and short that equation is, the more real it seems or the more accurate it is. And I thought this was kind of the closest to what it feels like today, given the current state of science.

• 15:34Mario Gabriele?

  You had a tweet where you were talking about a book that I tried to read as much of it as I could in preparation for this called The Inner Game of Tennis. And you sort of mentioned how, I mean, you were really talking about tennis, but it seemed like some deep parallels to robotics. You sort of make the point that they talk about the fact in this book that if you really focus with your conscious mind on improving your stroke, you could sort of get to this local maximum where maybe you are doing a little bit better at hitting a forehand. But really, what you need to do to get to an optimal stroke is sort of allow the unconscious mind to take over in some respect.

• 16:16Mario Gabriele?

  How do you think about emulating this aspect of almost the unconscious and the role it plays in fluid motion and, I don't know, proprioception and all these sort of things when it comes to robotics? Is that, were you thinking about robotics when you were reading that book?

• 16:33Speaker 00

  A lot. Yeah, definitely I was. I don't think it just applies to motion. I think it's, there's a big difference between how we learn things and how we think we learn things or how we teach things to others. You know, even if you look at something like learning a language, the way we thought you're supposed to teach it or the way we would want to teach it to others in school is by defining all the rules. And, you know, you first learn about grammar and all the different concepts and how tenses work and what word follows what other word. So you learn all of that structure. So that the thinking is that if you know all of this structure, you can follow those rules and then speak like a native speaker would. But it's not how we learn language, right? Like, you're just immersed in it.

• 17:18Speaker 00

  You just hear everybody using the structure that they kind of intuitively know. And then all of a sudden, you emerge with a full understanding of how language works, even though you can't really pinpoint any of these rules. There are many native speakers that speak perfectly, follow all the rules exactly, but have no idea what these rules are. What underlying structure is, what declination is, what tenses are, they have no idea of any of that, but they still follow them perfectly. With that book, I think that was that was shown, but in the in the in the in the in the motion aspect of things, where you know, you could you could read all you want about how to play tennis, or you can try to describe, you know, where your elbow should be exactly where you hit the forehand,

• 18:03Speaker 00

  or how this how to adjust your swing. But the only way to really learn it is by just doing it and kind of immersing yourself yourself in it. And I think there was a lot of parallels to the robotics world and and to the world of AI, how we how we thought that the machines should think, or how they should learn things versus how they actually learn them.

• 18:22Mario Gabriele?

  You, I think in 2023, say that you had a moment where for the first time you really felt like you could see a bit of the future of the robotics industry and things were really clicking into place. But you had really been in that field for quite a while already, it strikes me. So I wondered what the period before that moment was like. Like, was that quite a lonely time to be working in a space where you don't know how it's going to play out?

• 18:50Speaker 00

  So maybe I can go back and tell you a little bit more about the the journey.

• 18:54Mario Gabriele?

  I would love that.

• 18:56Speaker 00

  So you understand that kind of the feeling. After I moved to Warsaw, as I as I mentioned, it was very disappointing to see the state of robotics. And for a long time, it felt like no one was working on the intelligent robots that I really wanted to work on. And I remember at the time, I was searching for for really anything, anybody who who is doing something that that is a little bit closer to what I imagined robots to be. And for a long time, it felt like no one was. Then I remember I finally found someone, I found a professor in Germany that was that was writing papers that seemed a little bit closer to what I would imagine robots to be. And there was this this master's program in in Munich.

• 19:38Speaker 00

  Called Robotics, Cognition and Intelligence. And I thought if I if I apply for this program and get in and they don't do intelligent robots, then I'm basically lost. Like if the program with with that title doesn't do anything that I that I imagined, then something is very, very wrong.

• 19:56Speaker 00

  So I applied and moved to Munich. And I remember the first day of classes, I went to one of the the one of the very first classes and I got stuck in a in a metro somewhere and was super late, so missed the entire class. And I and I showed up, no one was there. The class was over at that point, but I saw a janitor and I was looking for a for a bathroom. So I asked him where the bathroom is. He pointed me to the to the third floor of the building. I remember it very vividly. And as I went upstairs, I remember seeing these big, big doors and there was some mechanical sound coming from behind them. I don't know why, but I just decided to like open them and see what's going on there. The the sound was very intriguing.

• 20:39Speaker 00

  And as I opened these doors, I see these two, basically humanoid robots doing, one of them was making popcorn and the other one, I believe, was spreading butter on a toast. And that was the moment where, you know, this is something I've been searching for at this point for for years. So that was the moment that that was probably one of the most powerful moments I I had during that journey where I finally found it. There's robots that that look intelligent, that do something that is very human-like. There are people who work on this. So I immediately let the room knock, because there's basically no one there. And I knocked at the at the at the room next door and asked if I could work there. And basically, I was completely unqualified. I had no idea how to program these robots.

• 21:24Speaker 00

  I was not familiar with any of the tools. But the person who was there, it was another one of those people that gave me a chance and and asked me to just come in on Monday and see what I could do. Wow. I got involved, got a job at that at that lab. That's how I got into finally working on intelligent robots. So, yeah, I I think this is like yet another person that I'm really grateful to. And then afterwards, I I moved to the US, got into a PhD program studying studying intelligent robots. And at that point, I was so stoked that like I finally get to do this and I finally found a small group of people that work on those things. It didn't matter that, you know, it doesn't work that well. I was so excited that I finally get to, you know, meet the people that are, that are working on this.

• 22:08Speaker 00

  I can learn from them, I can, I can push it forward and, and I can finally work on the thing that I always wanted to work on. So I did this throughout my PhD and I had another moment like this during my PhD. This kind of defining moment where I was working on a certain sort of problems where it was referred to as, as active perception and I can, I can go a little bit deeper into this if you're interested. But um, at that point, it kind of felt like there are ways to write, to, to, to write papers, to, to progress in your PhD. But there isn't anything that felt like it could actually solve this problem. It, it all felt like the world is too complex.

• 22:49Speaker 00

  You can push it to some extent, this technology, but you, you won't really get to the finish line. They're not, they're not kind of adding up. You don't really see a path out of it. And there was another moment where a poster candidate stopped by our lab. And at that time, when a poster candidate, candidate stops by, you are supposed as a PhD student, show them your work and kind of talk with them as part of the interview. So I showed to that poster candidate stuff that I was working on. And he listened to all of it. And then at the end of it, he said that I should drop all of this and switch to deep learning. That's the way to, to solve the problem that I was actually working on. At that point, I just thought to myself that he had no idea what he's talking about and he probably didn't listen to me.

• 23:35Speaker 00

  Because at that point, I think I spent already two years or something like this, working on the topic I was, I was working on. But then later that day, I went to his lecture where um, he showed what, what he was working on. And that was another one of those moments where it was extremely eye opening because for the first time I saw something that like could actually work. Where it was less of like a, you know, heuristic, a little paper over there and a little paper over there, but they don't add up. But it's something that that kind of brought all the pieces together. And yeah, that was another one of those moments, like the similar one to that to the one I've experienced in Munich, where at that point I decided to to drop my PhD topic, change

• 24:17Speaker 00

  it completely, start collaborating with that person and do everything I can to push deep learning in robotics. And that person was Sergey Levin, who is now my co-founder at Physical Intelligence and one of the pioneers of of deep learning in robotics. And yeah, I never looked back after that. But to kind of, this is a long way of answering your question, but it didn't feel lonely in that, you know, I was just really excited to be working on that problem.

• 24:45Mario Gabriele?

  That's such good stories there. The first one is almost like, you know, when a wizard is accepted to Hogwarts or something. It's like you finally found your, you know, collection of of magical people and and the magic that they were doing. The Sergey story is also so interesting to me because you seem to have gone from skeptical to convinced very, very fast. Like, what was the cycle on on that? Was that like literally the same day, the same week?

• 25:13Speaker 00

  I think it was within that lecture. I think like in the middle of that talk, I was like, yep.

• 25:18Mario Gabriele?

  This makes sense.

• 25:19Speaker 00

  This is hundred percent that whatever I was doing is wrong. This is the right way. I want to do everything I can to work on on this topic. And I I had feelings like this. I think every researcher has has a feeling like this every now and then, where they just see something and it clicks. And at that point, it's a very powerful feeling. You kind of want to drop everything you've been doing and you just you just feel like you you found something that that is much closer to truth than what you thought before. I think we all had a similar moment afterwards when we're working on combining large language models with with robot learning. And that was, I think, one more of these, another of these powerful moments where like things start to click.

• 26:04Mario Gabriele?

  That was the Taylor Swift demo?

• 26:06Speaker 00

  That's right.

• 26:07Mario Gabriele?

  Yeah, maybe you could tell that story because that is a fascinating one.

• 26:11Speaker 00

  Yeah, so what happened afterwards is I started working with Sergey. There was a few of us who were really getting into that field. It kind of felt like we were a small group of renegades within the robotics community. Because there were a lot of problems with deep learning at the time. It was, and some of these problems remain. Like, it was not interpretable. There was no way of really making it modular. It seemed like nobody fully understood how it worked. It wasn't very sample efficient. And there were all of these problems. So like, at that time, it was extremely unpopular to write any deep learning in robotics paper. And it was kind of like between these two worlds of machine learning world, where any robotics paper seemed kind of like the wrong paper for that venue.

• 26:55Speaker 00

  The robotics world that never fully embraced or didn't embrace deep learning at that time. But it was still really cool to work with a small group of people and push these methods forward. So then afterwards, the only place that was really embracing that view was Google Brain. So I decided to join Google Brain right out of my PhD, continue working with Sergey and others on those set of methods, with the basic premise being that the way to really get it to work is to scale it up. So we're scaling it up. We're trying to figure out how to really get it to work at scale. But again, it started feeling like there is so much complexity in the world

• 27:36Speaker 00

  that if robots need to, the only way for robots to learn all of this complexity is to experience all of it firsthand, it would be very difficult to scale. It will be very difficult to have them learn about logic and how to break down a task if they had to do it all by themselves. Yes. So then, I think around 2022, I would need to look back to see what year it was exactly. We started combining it together with Brian Dichter, my other co-founder. We started combining these robotic methods with large language models. This was before the ChatGPT moment. This is where people just started experimenting with large language models and kind of understanding them better and better. There was this one particular demo.

• 28:20Speaker 00

  So we were really excited about this because there was one, we thought that this would be a path to bringing in a lot of prior knowledge that robots didn't experience firsthand, that we learned from the internet. Into the robotics world. And that would kind of solve this problem of you having to collect so much data to understand how the world works. Because that that understanding is already embedded in large language models. So if you figure out a way to combine these two, that should really, really help. And we had this one particular experiment where we we were testing how much transfer do you get from this internet scale knowledge to robotic behavior? And usually when you when you work on robots, you work on this very specific task and then you test that specific task.

• 29:05Speaker 00

  So you're very rarely surprised. It's mostly disappointing because like you really want this task to work. You've been working on this task for a very long time. And then it usually doesn't work, but if you work hard enough, eventually you get it to work. But very rarely you are in a situation where you didn't expect the task to work or like it's not the task that you're working on and it works. So very early or positively surprised. So we set up a few of these experiments trying to test how much knowledge transfers. And one of them was this experiment with a Coke can in front of a robot and three pictures of different celebrities in front of it. And the prompt we gave to the model we built that combined LLM and robot models was put a Coke can on picture of Taylor Swift. And one of the pictures was Taylor Swift. And it's you can see this video on the internet.

• 29:50Speaker 00

  It's a pretty pathetic demonstration of what robots could do at the time. But the robot picked it up and then slowly moved it towards Taylor Swift. And that was another one of these moments of huge, huge excitement, even though if you watch the video, it's totally unimpressive. Because this was the robot models had never had the chance or had never had any of Taylor Swift in their data. It had to understand the concept of Taylor Swift, connected to the image of Taylor Swift, and then connected to the right motion that would move Coke and to the picture of Taylor Swift, all from internet data. So that was the moment where it clicked for us that that actually worked. Where you can bring in a lot of prior knowledge from LLMs, from the internet and connected to robot motions.

• 30:35Speaker 00

  And even though the demonstration itself was very unimpressive, just the fact that you can combine these two knowledge sources, that was really, really, really impressive for us and it felt like it opened another door.

• 30:47Mario Gabriele?

  Yeah, it feels like, I don't know if you're familiar, I'm sure you are, I can't believe I'm asking if you're familiar. You know much more about this than I do. But there was some experiment in like the seventies, I think, called like Shard Lou or something like that, where the entire approach was to try and teach AI all the rules of the world programmatically. And to say, you know, here is the category of what a bird is. It has these characteristics and then here's the category of a bat. It has some overlapping character, you know, all of these sorts of things. But essentially, you know, people are familiar with this concept now because of the way we use LLMs. But it was almost as if the robot suddenly inherited all of these rules and, you know,

• 31:29Mario Gabriele?

  pieces of knowledge from tying them up with the LLMs that allows the Taylor Swift demo to happen.

• 31:36Speaker 00

  Yeah, that's exactly right. And I think there's maybe two points there. One is that we made the same mistake in robotics. We wanted to write all of these rules. We thought that if we, if only we had enough of these rules, the robots would be able to follow them and do the right thing. But kind of like we said about the inner game of tennis, you know, you can't just write all of the rules. You kind of have to do it. And there is some underlying structure, but you can't just fully put your finger on it of what it is. You need to learn it from data. And I think people thought about it similarly in language as well. They thought that, you know, if only we could write all of the rules.

• 32:12Mario Gabriele?

  Yes.

• 32:12Speaker 00

  That would be enough. But it turns out that there is, you know, trillions or billions of these rules. And sometimes we can't fully even express them in language. You just need to learn them. And if you do, then you would be able to follow them, even though you still don't understand what fully, what they are. So I think we're learning this lesson over and over again.

• 32:33Mario Gabriele?

  And so, you know, you have this almost third eureka moment for yourself where the technology has, you know, taken yet another jump. By that point, it sounds like you have most of the, the physical intelligence co-founders around the table. But how did you sort of pull the, the last few members aboard and decide to make that leap?

• 32:54Speaker 00

  Yeah, I think at that point, it started becoming clear that.

• 32:59Speaker 00

  It could be possible. And you know, if you worked on something for so long, at that point, that was basically my entire adult life, fifteen years or so. And for a long time, you thought that there was no solution to this problem. There were some times where it felt a little bit more tangible, but it never felt like it could actually be solved. And then you have this moment where, for the first time, you see the light at the end of the tunnel. Like, maybe, just maybe, if we do everything right, if we combine it with internet knowledge, if we scale it up, if we, if we do all of the pieces that need to be done, it might work. If you see that light at the end of the tunnel, you can't unsee it. You, you want to do something about it.

• 33:39Speaker 00

  And at that point, it became clear that the way to, to accomplish this is to create an organization whose sole purpose is to solve physical intelligence, is to solve this problem. It can be solved as like priority number twenty in another organization. The only reason to, to exist for this organization would be to solve physical intelligence. So then the next question is, how do we actually do it? And what are the, what are the, the conditions to, to make it happen? And what became clear immediately is you need to have truly the best people, truly the best researchers to make it happen. Because the second best team in those areas doesn't, doesn't really work with a company like that. You need to have the top top people in the field.

• 34:24Speaker 00

  So that one actually wasn't that hard because we were already working with each other for a very long time. So I happen to be friends with the best people in the field already. So it was just a matter of making sure that we all want to do it and, and we're ready for it. The other condition was that you need to get access to a lot of funding. It would require very long, very long term bet. And investors that are fully aligned with this, taking some time and this starting as a, as a research company. And not being oriented around revenue or around the short term revenue. So it was the second requirement. And the third requirement was just like build a very, very incredible, build an incredible company with the right focus, with the right people, with the expertise in.

• 35:08Speaker 00

  All the other areas that we didn't have expertise in like hardware operations and things like this. I spent most of my time figuring out these two points. How do we, how can we find the right investors that can, that can really help us in this, um, in this adventure? And they're fully aligned with how we want to do this. And then how can we find the right people to, to fill all the missing pieces?

• 35:32Mario Gabriele?

  How did, uh, Lachy Groom join the crew? Because, uh, he's obviously had a very impressive career as an operator, as an investor. But isn't someone who one would automatically think, you know, this is someone who's going to devote the next chapter of their life to robotics?

• 35:47Speaker 00

  So I didn't know Lucky before we started thinking about starting a company. But then, um, and he should tell the story rather than me. But I believe what was, what was happening is that he was in, he's been investing in that point for a few years. And as long as he's been investing, he always thought that he doesn't want to be investing forever. He wants to find something else that he can fully devote himself to, and he wants to build things. And the one area he was always fascinated by was robotics. And he's been, he was seeing over the past few years that there was, uh, there was a lot of innovations happening there. There was, there was some kind of moment, robotics was having its moment.

• 36:27Speaker 00

  And particularly, he was impressed, he was seeing over and over papers coming from Chelsea, Finn and Sergey Levin's lab. As well as our papers from Google. So he, he talked to a lot of, um, a lot of his friends that if ever Chelsea, Sergey, or Carl, or anybody from those teams are thinking of starting a company, please connect me to them. I would at least want to invest or at least talk to them. So that's when we got introduced by, um, by a friend. And we pitched to Lucky as a way to, um, to, to have him invest. And at the end of that pitch, it became clear that he would like to do much more than just invest. This is the opportunity he's been looking for, for all the years he's been investing. And he really wanted to, to, to dive in.

• 37:12Speaker 00

  So then the next step was us trying to get to know each other as quickly as possible, spend as much time together as, as we could. And, and then as we were doing that, it also became clear that this is the person that, that I've been looking for, that could help us on, on the, on the commercial side, on fundraising side, on operations side. Kind of one of the, the missing links that we really needed.

• 37:35Mario Gabriele?

  Were there certain? Philosophical alignments that you needed to find in someone who you hadn't worked with before? What were the sort of, you know, even beyond maybe the tactical pieces? The traits that you saw in him that made you feel like, yes, this is someone I can can bring into this very trusted group? Where, you know, we've already built all this history together.

• 37:57Speaker 00

  There were a few. I think maybe the biggest one was that he immediately got it. At that point, there were a lot of investors or other people that I talked to, other business people that I talked to. And it was very hard to get that idea across, that idea that you have to do research, right? You need to get, you need to build a technology first and you can't be distracted by short-term revenue. And if we do do this right, this is going to completely change the world and it's going to be most valuable business of all time. But you need to, you need to have the patience to let it, let us do it the right way, rather than short-circuit that path and and kind of cap the ceiling. And he got it, I think, within like the first minute.

• 38:42Speaker 00

  So that was very reassuring. And then on top of that, what, what I really liked about our conversations was that in many of those previous conversations with other people, I always felt that like I'm the one pushing the ambition of this project of how far it could go. If it's upright. And I think this was the first time where I had somebody else set up even higher ambitions for us. That was really cool to see because it felt like we're all pushing in the same direction. He will make us even better and more ambitious. So I was really glad to find someone who will, you know, make sure that we are not gonna, that we're not gonna short-circuit this. He's fully aligned in doing this in the biggest way possible.

• 39:26Speaker 00

  He makes us more ambitious and he's the person in the areas that we really need more expertise. And he'll make sure that those areas are fully aligned with how we want to develop this company. So it was, it was basically a no-brainer at that point.

• 39:41Mario Gabriele?

  You had this insight that this was the right moment to go and solve this problem. But at least from the outside, one could imagine many different form factors that might take. Like another version of this company might be saying, hey, we're going to try and build the best humanoid robot ourselves, or we're going to take this approach to getting the data we need to run these models very effectively versus another approach. How did you sort of land on the version of physical intelligence as it is today?

• 40:13Speaker 00

  I think from the get go, we had an idea of, we had the thesis for the for the company. And that thesis was around all the research that we've done up until this point, that similarly to what happened with language, it's not going to be specialist models that's really going to solve this problem. It's going to be generalist models that work across all kinds of different tasks, all kinds of different environments, and all kinds of different robots. And it was similar to what we've seen in language, where you would think that the best translator would be just specialized for translation, or the best coder would be just specialized or just trained on code. But it turned out that the way to be the best of all of those fields is to

• 40:56Speaker 00

  train one generalist, and a generalist that takes in poetry data, and the coding data, and translation data, and then it turns out to be much better than all of those specialists at those specialist tasks. And we started seeing something similar in robot learning, in robotics, that if only we could collect enough data, if that data was very, very diverse, and if we do it the right way, we should be able to build the best generalists. And if we solve that, that is really going to allow us to have this world of many diverse form factors and robots that can be very intelligent. So I think the two thoughts that we had from the beginning is that, one is, intelligence has always been the bottleneck for robotics.

• 41:39Speaker 00

  Rather than trying to start a robotics company that focuses on a specific robot, how can we tackle this problem head on and just focus on the intelligence? And then the second thought being that the way to solve intelligence is to take the lessons that we learned from vision and language and other fields, and really take the foundation model approach, which includes things like cross embodiment learning, large diversity of data, a lot of real world data, and do the research necessary to figure out how to build these models.

• 42:10Mario Gabriele?

  So you sort of land on this idea of, you know, the AI brain for these different robot types, to put it sort of simply. How did you start to think about the right way to gather the data? Because that's clearly such a big piece of it. And obviously, there are different pieces, different players that take different approaches there. So I imagine you must have had to reason through that in a million different ways to land on the one you have.

• 42:35Speaker 00

  I think we're still reasoning through it. I don't think we have all the answers yet. I think the important piece about how to think about physical intelligence is we are not very dogmatic. It's not that we sit down and think very, very hard and then come up with the solution and this is our bet. I think the better way to think about it is that we are really truth-seeking and we run experiments and we don't know the solution. And we know that we don't know the solution, so we want to follow the scientific method and really try to find the truth and really try to find what works and what doesn't. Because I think there is a true answer out there. We just need to be very humble in finding it.

• 43:18Speaker 00

  So that's, I think, how we arrive at the current set of answers. We run a lot of experiments, we try many different ideas, and we see which ones stick and then we double down on them. Now, in terms of, based on what we've seen, based on the evidence we've seen, how we think it's going to work, or how I think it's going to work. These models will need to be able to absorb very diverse data sets, where it's less about picking the right data or picking the right way of collecting data, more about building the engine that allows you to absorb all kinds of data, whether it's videos of people, whether it's teleoperation data from robots, data from handheld devices, video data really, or simulation data, really anything.

• 44:02Speaker 00

  And the more data they can absorb, the better, the better the model will be. And I think we're kind of at this stage of robot learning where we kind of try to throw anything we can at these models, build them in a way that they can absorb as much of it as possible, and get them to the threshold of being able, being deployable, where you can actually deploy them in the world and have robots out there collecting data for real, doing economically valuable tasks.

• 44:29Speaker 00

  And I think once you're at that stage, once you're at that threshold of they can actually work and do valuable valuable tasks, that's when you enter this next stage of now deploying robots at scale and deploying it in multiple verticals in diverse environments and actually delivering value. And I think that second stage is actually going to be the the stage where we get most data from, where the more robots you deploy, the better the models should get, the more you can deploy them, and there's a natural flywheel to it. And what's exciting about the moment today, this moment right now, is that I believe we're very close to this threshold, or already above that threshold. And that's something that is really, really exciting, because I think the models will keep on getting better, they will be able

• 45:11Speaker 00

  to absorb more and more data, but they will also have this sustainable source of data that is very, very valuable, because that's the data that is the most real, that is the closest to how you actually want to deploy the robots.

• 45:23Mario Gabriele?

  For folks that maybe haven't spent as much time digging into this, or sort of coming to it fresh, I would say that one of the experiments, as you think about this data collection, that physical intelligence has done maybe more than others, is this real-world data approach. Why is that so important and so valuable to to get and, you know, so valuable to enter that phase two that you talked about that can sort of start the the flywheel?

• 45:48Speaker 00

  We have a few theories why this is really important, but I think maybe the meta point here is that if there was a different path that worked much better, like maybe for simulation or for videos or something else, we would happily pick that path. So again, it's not that, you know, we we sat down and thought that this is the best path and this is the only thing we're going to do. We run a lot of experiments, we tried out a lot of ideas, and this is the one that seems to be working very, very well. Why do we think that that this is important? I think it's it's kind of difficult to describe it in the absolute. It's, I think, a little bit easier to compare it to to other alternatives. So one popular

• 46:28Speaker 00

  alternative is simulation. And this is in particular an area that that had a lot of success, and especially in locomotion use cases, where you have robots walking around or doing stunts like backflips and things like this. Most of these methods are training simulation first. And I think for those kind of tasks, the main complexity is about how you move your own body. It's less about interacting with the world and more about how do I move my legs correctly so that I can I can walk or I can run? And in that sense, as long as you model your own body accurately, you should be good. If you model it very, very well, your own particular robot, and that transfers from simulation to the real world, you should be able to to learn that behavior in simulation,

• 47:13Speaker 00

  and then that's good enough to them to work in the real world. Now, when it comes to the problem of manipulation, where you're manipulating the world around you, the difficulty is less about modeling your own body, like how you move your arm from A to B. But it's more about modeling how the world will react to it. Right? So the problem of manipulation is more about this interaction with the world that you have, or the interaction of the world that you're interacting with. And I think in this case, it's it's just much harder to simulate the world around you than it is to simulate your own body. It's no longer about just a single, a single robot that you need to simulate, you need to simulate everything. And we don't really know how to simulate everything at scale, how to do this accurately enough and scalably enough

• 47:59Speaker 00

  so that it would work for for anything, for every single one of of tasks or objects. It takes really long to get it exactly right, to get all the friction parameters right, to get the simulation behaviors right. And it's just not as scalable. That's been our finding so far.

• 48:14Mario Gabriele?

  And so it's sort of the case to to try and distill that, that, you know, with simulation, you can do some of these, I don't know, maybe this isn't the perfect word, but sort of coarser, larger actions that are more self-contained. But once you start trying to, yeah, implement picking up the coffee cup or the towel or whatever it might be, you're starting to rely on a simulation that would have to be so good, you'd effectively have to, you know, create a simulation that is perfectly similar to reality. And that's where the the real world data starts to become so important.

• 48:50Speaker 00

  Yeah, you would need to simulate all of the external world.

• 48:53Mario Gabriele?

  Yeah.

• 48:54Speaker 00

  Any one of those tasks. And that's just too, too costly, too difficult to do. But as long as you can get away with just simulating your own body, I think that works perfectly fine. And that's what we've seen in locomotion or, you know, backflips or dances or things like that.

• 49:08Mario Gabriele?

  There was a A moment, I think, about a year and a little bit ago, where you talked about reinforcement learning making a comeback. And that has since become a really interesting part of the way that physical intelligence seems to work with recap. What were you seeing at the time that made you think, this technique might have sort of a second or additional wind here? And why has that been so useful for you?

• 49:37Speaker 00

  Yeah, there's been a long history of applying reinforcement learning to robots. And the problem of reinforcement learning is such that you need to be learning from your own experience. And while gathering that experience, you need to encounter some successes. And then you want to increase the probability of good actions that lead to those successes and decrease probability of the actions that lead to failures. What this means is that you need to have, as you explore the world, as you collect your own experiences, you need to have some of these successes. Because if you don't see any of them, you basically don't really know where to go. You're kind of lost. And if you start from scratch, if you just try to command random commands to a robot,

• 50:20Speaker 00

  it's very, very unlikely that you'll encounter a success based on that. So let's say you're trying to teach a robot how to grasp an object. And you command basically random commands to all of the motors of the arms. It's very, very unlikely that some of these commands are going to lead to a single success of you grasping an object successfully. And that's what we often refer to as the exploration problem. But we don't really have a way of guiding the robot on how to explore so that you can encounter a success. And as soon as you encounter it, then you're on the flywheel. You can now increase the probabilities of those actions. It's just like to get to the very first one, very, very hard. And that's been the problem of reinforcement learning for a very long time.

• 51:02Speaker 00

  But now with the models that we've been putting together, this exploration problem became much easier. Because robots now don't start from scratch. They start from a foundational model like pi zero, pi five, or pi six, where they already have some intuitive understanding of how motions work and how you can explore the world around you. So even if you put a new object in front of a robot, the chances are that if you ask it to grasp it, it will either grasp it or it will fail in an interesting way. So the probability of you actually doing something useful starts to become very, very high, or at least much higher than it was before. So now you have a way to explore the world and that allows you to to apply reinforcement

• 51:45Speaker 00

  learning methods much more easily because they can encounter successes much quicker. So this is kind of how reinforcement learning has evolved over time. And that's what allowed us to actually kind of start or start applying it again and seeing if we can, if we can see the successes of it. Now, from the other perspective, the reason why we thought reinforcement learning would be important is that the way most of these models are trained today is mostly based on imitation. So you collect a lot of data, whether it's in simulation or from human videos or from real collaboration data. And then the objective for these models is to try to replicate the actions that you've seen in the data set. And the important caveat there is that it's not actually the the objective that we care about.

• 52:27Speaker 00

  We don't care about executing every single action exactly the same as it was presented to you. What you really care about is the success of the task, whether you, whether you accomplish the task or not. And we don't really have a way of codifying this objective with with these models. Right? They're all optimized for for imitation and therefore it's very difficult to drive the success rate of those models because it's not the objective that they care about. They only care about imitating the actions as closely as possible. And reinforcement learning provides you a way to to basically codify this objective, to have them now be optimizing for actually accomplishing the task successfully. Both of these things combined kind of led us to believe that we should, we should start

• 53:11Speaker 00

  applying reinforcement learning so that we can improve reliability of these methods and have models that work, you know, not 70% of the time, but 99.99% of the time because they actually optimize for the right objective. And at the same time, it became possible because now these models can explore in ways that are more intelligent than it was possible before.

• 53:32Mario Gabriele?

  To circle back to a topic you alluded to earlier, you mentioned that when you first started chatting with Locky, he understood the need for patience on commercialization. And you sort of referenced the fact that there's a version of this company that if you're too impatient, maybe you can short circuit the sort of long-term vision. What are the ways that that happens and how do you have to be alert to them?

• 53:57Speaker 00

  I think there's a long history of robotics companies doing this or this happening to them, where we're not the first company that starts with a big vision and broad vision for robotics. But if I analyze what happened in the past, what often happens is you start with that vision, you start developing the technology, and then, because there's usually some external pressure, you try to commercialize this technology at that point, when it's not fully ready yet. And you dive into a particular application, maybe the application that has the biggest TAM or something like that. As soon as you do that, you start cutting corners on the technology itself. And you short circuited that vision of it being a very generalist, general-purpose technology

• 54:42Speaker 00

  that would actually deliver the most value, but that you're just trying to deliver the most value for that customer or for that particular vertical that you chose. And that's perfectly reasonable, right? Like, all of your incentives are tied to that. Your revenue is tied to that, the customer satisfaction is tied to that, your valuation, how employees think about this. So you have all incentives in the world to start cutting corners and trying to make a more special-purpose solution. That's usually what happens. So you start with this broad vision of how general-purpose robots are going to be solved, but you very quickly end up becoming an application company, like a warehouse pick-and-place company. And this would be a heartbreaking outcome for me.

• 55:23Speaker 00

  I think we really have a chance to solve the big problem, the problem of general physical intelligence. And if we do that, you won't just enable that one application that you could have focused on initially, but I believe you'd be able to solve all of it. You'd be able to solve it for any robot to do any task. And counterintuitively, that would be the much- Much better commercial outcome as well.

• 55:48Mario Gabriele?

  It's all just about trying to have the the right time span in your head. How does that influence how you think about the right partners for the business? Like, are you optimizing for, I don't know, a range of use cases in that case? So that you're getting the diversity of data? Is it about scale and, you know, getting as much of that data from, you know, in-person deployments as you, not in-person, but, uh, real-world deployments as you can?

• 56:15Speaker 00

  Right now, we're optimizing for the rate of learning. So it's still quite early, you know, unlike maybe what you can see on Twitter today. It's not that, you know, robots are about to knock at your door and show up at your home and do everything.

• 56:28Mario Gabriele?

  Yes.

• 56:29Speaker 00

  I don't think everything is solved. The app is just a matter of scaling up the existing recipes. I think we can, we can push them much, much further than where there are today. But there's still a lot of research to be done and a lot of questions that are unanswered. So right now, we're mostly optimizing for speed and how we can learn as much about the problem as possible. A lot of it does come down to what kind of data to collect, how to integrate the data into the model, and how to have that loop be as tightly closed as possible. It's a long way of saying that it's a fairly nuanced question. But what we're optimizing for is learn as much as possible so that we, we can figure out the scalable recipe that then we can just scale as much as possible.

• 57:14Mario Gabriele?

  Do you have theses at this point on what gives you the, the sort of steepest rate of learning? Whether that's, you know, I don't know, very fine-tuned tasks or, or something totally different?

• 57:23Speaker 00

  I think there is, um, there is a few of those things that we, that we learned. We know that the diversity of data is really, really important. We know that, um, the quality of data is really important. We know that closing the loop with the models are very important. I just feel like these are fairly broad statements that, you know, if you had asked me this a year ago, I would probably say something similar. But we did learn a lot about what these terms actually mean. I think very often people talk about the diversity of data or quality of data or something like that. But we don't really have very good definitions of what quality of data actually is. What does it mean? Or what diversity of data actually means? Or how do you measure it?

• 58:07Speaker 00

  So I think these are fairly deep questions and we are we are getting more and more of a feel of what it actually is and how we can measure it and how we can optimize for it. But it's a constant push and pull on how much do you scale the existing thing? How much can you better understand that? How much can you increase the slope of improvement there? So I think we'll just continue on the trajectory as we scale it.

• 58:32Mario Gabriele?

  You mentioned that there is still a need for real research in this space to achieve the ultimate end goal, but that there's also plenty to optimize. In this sort of thought experiment where we get no new research, how far do you think that that sort of takes us? Do you have some rough heuristics of what kind of a robot that is able to produce?

• 58:54Speaker 00

  It's a question that we think about a lot. Because I think one powerful insight from the language model work is that for a long time, people thought that we still have a few ideas that are missing or we need a different architecture or we need different this or that. But it turned out that it was good enough. It was just a matter of scaling. And we just didn't have enough foresight to predict how scaling is going to resolve some of the issues that seem like the issues of today at small scale. So I think there is a non-trivial chance that the recipe is already there. The recipe that we have today would work and would just solve everything. And we just need to scale it in the right way and execute on it really, really well.

• 59:39Speaker 00

  Well, I'm not certain about this at all, but I think that's one question that is an important question that we're thinking about quite a bit. So what we're trying to do is verify it. So start scaling the existing recipe and see how it performs, how it scales. And in the same time, continue to do research that at the very least could improve the slope of that scale.

• 1:00:01Mario Gabriele?

  Nvidia is probably the best example of a company that is putting a ton of money into improving its simulation engines and getting that to higher and higher fidelity. How optimistic are you about that improving at a sufficiently high rate that it really closes the gap in some meaningful way or allows you to get the kind of data at scale that could improve things meaningfully?

• 1:00:28Speaker 00

  So as I mentioned before, I'm quite open minded about these things. I'm not very pragmatic about the path it's going to take to get there. I think simulation over the past decade plus has been improving like crazy. Simulation that we see today is much more realistic than what we've seen in the past and it's becoming more and more scalable. I believe that the first place where we're going to see the impact of it is not going to be as much data collection as it's going to be evaluation. And one thing we see here at Physical Intelligence is that as these models become more powerful, it takes more and more time to evaluate them. So now not only you need to evaluate them for longer, you need to take more samples to distinguish between the model being at 99% success rate

• 1:01:13Speaker 00

  versus 95% success rate than it was between 50% versus 70%. You just need more samples to be statistically significant, but the repertoire becomes broader and broader. So now you need to evaluate them on more tasks, across more robots, in more environments. And that trend, I think, will continue. As the models get stronger, you will need to evaluate them for much longer in much more sophisticated ways. So I think if simulation starts to work in the way that we all hope it will, I think that's where we'll see the first impact of it, where we can start evaluating these models in simulation across diverse scenarios. And if they're realistic enough, that should correlate with the real world evaluations.

• 1:01:58Mario Gabriele?

  We talked about some of the real inflection points for you over the course of your career. I'm sure that Physical Intelligence's biggest inflection points are to come, but over the past couple of years so far, what have been maybe the biggest surprises or the moments that have impressed you the most?

• 1:02:17Speaker 00

  I think the main thing is just how fast all of this has been moving. And how much it's picking up speed as we go. So before we started the company, we thought that it was gonna take us something like five years, maybe to start deploying robots. And we deployed our first robots, I think, eighteen months into the company. So it either means that I'm very bad at making those predictions, or it's been just moving much, much faster. There's been multiple of these moments. I think our first model, our first release Py0 was definitely one of those moments where I thought it would take a very long time before we have robots folding laundry, especially diverse pieces of laundry fully autonomously. This was one of these holy grail of robotics that I thought we're gonna,

• 1:03:02Speaker 00

  we're, it's gonna take many, many years before we get to. And we got there within the first seven months of the company. I think the next model was another one of these moments where I thought it was gonna take very, very long to get robots to perform in an environment that they've never seen before. And with our Py5 release, we got, we got robots to new homes that they've never seen before. And home is like the most diverse environment you can imagine. So it's the hardest version of that challenge. And that started working as well. Then with, with Py06, we started seeing that these robots can perform tasks at very high clip, at very high success rates. And now we have videos of robots doing extremely dexterous tasks, like making, making coffee on the proper espresso machines,

• 1:03:47Speaker 00

  a machine for like thirteen hours without any cuts, just like thirteen hours straight. The robot is doing coffee over and over again and cleaning after itself. So I think all of this is just moving very, very fast. And every single release so far has been quite surprising to me. I thought it was gonna take longer or that the problem is much more difficult than I, than I expected. But I think that's maybe like one thing about working with these models. I think probably people working on LLMs would have similar answers that they thought it was gonna take much longer to get to a certain level of capability.

• 1:04:17Mario Gabriele?

  You know, when you're running a robot for thirteen hours straight making espresso, what does a good failure rate look like in that, in that case? Or what are you excited by when, when you see that happening, if anything? And then, you know, something I was talking about with, with a friend who's in this industry is that a lot of the times, the failure for these kinds of robots is maybe less obvious and more like it almost doesn't realize it's stuck in some way and, and doesn't find a way out of, of being stuck in that loop. Is that, is that sort of? Something that you you see a fair amount in this case?

• 1:04:54Speaker 00

  I think in general, people don't pay enough attention to this problem of reliability. And that's maybe a big difference between something like a language model or a chatbot and a robot. When you talk to a chatbot and it makes a small mistake, or it just like puts not quite the right word. It's totally fine because you're right there and you can work all of this, you can correct it, you can ask it again. Or you can you can interpret it the right way. You're very forgiving. And the physical world is is the opposite of that. It's not very forgiving. If you make any small mistake, if you know, if you grab the portafilter just like a little bit off. Or you don't insert it just quite right. Then the whole task is going to fail in a catastrophic way. So I think the the bar on performance is just much, much higher.

• 1:05:40Speaker 00

  Because there isn't kind of someone to help you, someone to catch you. And I think this is a very big problem. Now, the the other thing you point out is more about the model being self-aware of where it's where it's not going well. And the power of reinforceable learning is that it very often needs to have an additional function that tries to predict how close you are to success. We call this a value function. And that value function should have an understanding of how well you're doing so that then you can you can improve upon it. And what we found so far is that you can train these value functions to be pretty good. You can train them to the to the extent where we don't fully know that the robot is about to fail yet, that everything is looking okay. But the value function already knows.

• 1:06:25Speaker 00

  And they start seeing that like the six, the the the expectation of of of success is starting to go down. This is actually something that we've also seen with with words like AlphaGo, where we had a computer program learn how to play Go. Where if you if you watch some of the games that this AI system had against the the Lisa Doll, the the world champion that that Go. What was quite intriguing is that in many of these games, you have world's experts at Go talking about and commentating on the game. They're all thinking that the game is basically head to head. It's like, you know, it's a very unclear who is winning, whether it's the AI or Lisa Doll. But then you look at the value function predictions and it's basically over. Like the the AI system knows hundred percent that the game is over.

• 1:07:10Speaker 00

  It's already won by AI and Lisa Doll has no chance. But the entire world, you know, all the experts, including probably Lisa Doll himself, think that. That this is a very, very close game. So I think we will probably get to similar models in robots where they will be able to predict how well they're doing or how likely the success is much better than we will.

• 1:07:31Mario Gabriele?

  One of my favorite activities to do with my, my son is he loves to look at my bookcase and just pull out whatever covers he finds interesting. He's, you know, 15 months old, so he's just doing it based on, you know, what interesting face is on the front or whatever. And he's gotten really interested in Oliver Sacks's biography called On the Move. Cause I don't know, it's a picture of Oliver Sacks. It's, you know, engaging. And I was just paging through it with him like last week, and I was already starting to think quite a lot about our conversation. And it lands on this page where Oliver Sacks talks about being obsessed with proprioception and how this is, you know, the sixth sense. And it's actually the most important sense because, you know, you can strip humans of,

• 1:08:15Mario Gabriele?

  of all the other five and they can more or less get, get along. But if you, you know, get rid of this sense of where your body is, life becomes extremely hard. And he talks about working with these patients who, you know, maybe have a virus and they lose this, this sense of, of their own, their own body. And he has this one case, which made me think a lot about what you're doing and, and maybe what the final state of it might look like, which is this guy gets a virus and he loses the sense of proprioception, but he compensates with visuals where, you know, if the lights go off in a room, he can no longer move, but as long as he can see where he is, he

• 1:08:58Mario Gabriele?

  can sort of walk, but not walk and talk, et cetera, et cetera. And so it's, it's basically the way that the, the brain compensates for the loss of this sense. Is that sort of the stage we're at at the moment with physical intelligence where we don't have true proprioception? And so we're sort of compensating with these other senses. And is there a point in the future, do you think, where? However, a computer is able to emulate that, we have something that really feels equivalent to it.

• 1:09:30Speaker 00

  Yeah, there are many newer scientific experiments showing something like this, how you can disable one sensing modality and then compensate for it with another one. I think in general, it's quite unclear what the right sensing modalities should be. I think our intuitions are often wrong about this because also these machine learning algorithms can extract the right level of information from signals that to us seem like insufficient. Like a common question I get is, you know, about touch sensing on our robots. And instead, we put wrist cameras and it turns out that they can compensate for the lack of touch just fine.

• 1:10:12Speaker 00

  And they probably find some kind of regularities in the data where maybe you see the deformation on the gripper or something like this that indicates to you how much pressure you're applying. And that's enough for it to fully compensate for the sense of touch. It might be that we'll find some password that's not the case and you really need some other sensing modality to tell you the, to give you that signal. But so far, I think it's been quite surprising to the entire field, how far you can push it with just simple cameras.

• 1:10:40Speaker 00

  I think in general, though, a lot about intelligence is about compensating. And what I mean by this is kind of like predicting what's going to happen and then trying to figure out if what actually happened, how different it is to what you predicted. And it doesn't just apply to sensing modalities. I think it also applies to precision. So for instance, as a human, you're very imprecise when it comes to movements, right? Like if I ask you to put your finger on a specific point 100 times in a row and I measure this exactly, there will be quite a bit of variance, especially compared to modern industrial robots that can go to sub-millimeter accuracy and very good repeatability.

• 1:11:24Speaker 00

  But if you compare our dexterity to dexterity of those robots, you know, it's night and day. We're way more dexterous, even though we're way less precise of a machine. And I think the reason for this is that we have physical intelligence that can compensate for it. We have enough feedback that if we just look where our fingers are or if we just feel it, we can compensate for it immediately. And I think we'll see that being translated to robot design as well, where maybe the the robots of the future don't need to be as precise as we had thought, or, you know, they they can they can accommodate for a lot of things, like a little bit of backlash in the motors or just noise. Because with the right intelligence, you can you can compensate for it. And that's what we've seen here at at Physical Intelligence.

• 1:12:08Speaker 00

  We show some of the most impressive demos ever on robots. But these robots are really bad robots. If you compare them to to any state of the art industrial machine, they're really, really bad. They're very imprecise, not very reliable, a lot of backlash and other problems. But that doesn't really matter because the right intelligence can compensate for it.

• 1:12:28Mario Gabriele?

  Well, you're that was a very interesting and gracious way of answering a very baggy question from me. So I'm glad I asked it because, uh, yeah, that was that was fascinating. As a final wrap up question, I always like to ask folks that if they had the chance to assign a book to everyone on earth to read and know that they would understand it, what is a book you would you would love to give everyone?

• 1:12:53Speaker 00

  I reread recently a book that I'm a big fan of. I'm not sure if this is the the book I would recommend to everyone, but this is the book I think is that is really, really good. It's Why Greatness Cannot Be Planned by by Ken Stanley.

• 1:13:09Mario Gabriele?

  Yes.

• 1:13:09Speaker 00

  Yeah, I think it's just a wonderful counterintuitive book that shows multiple examples of how we arrived at something really spectacular without planning for it. I find it quite inspiring and quite motivational

• 1:13:24Speaker 00

  in a way that is maybe a little less straightforward than than than usual. So I think it's just like a very insightful book that would that that would help a lot of people to to show shedding new light on how to think about innovation and achieving something spectacular.

• 1:13:41Mario Gabriele?

  Well, that's a perfect place to end. Thank you so much, Carl. Thank you.

• 1:13:46Speaker 03

  That's it. Thank you for listening to this episode of The Generalist Podcast. Please subscribe on Apple Podcasts, Spotify, or your preferred podcast app. Ratings and reviews help others discover these discussions, so if you enjoyed the conversation, I'd be grateful if you could take a moment to leave one. For all past episodes and more, visit us at thegeneralist.substack.com. See you next time as we continue to explore the future.