Pete Florence: When Robots Improvise

Pete Florence https://www.peteflorence.com/ and I began by agreeing that his company has an excellent name. Generalist https://generalistai.com/ builds AI for robots, including models that can begin learning a new task from a single demonstration. We talked about dexterity, the strange business of gathering physical experience for a machine, and what we might build if robots could do much more of the work. — Every timestamp 00:00 Intro 02:24 GEN-1.5 and one-shot learning 08:55 What makes a task difficult 10:53 Dexterity in robotics 14:53 Matt Mason’s definition of a hand 17:34 Giving robots a sense of touch 20:30 Growing up in a family of technologists and scientists 22:46 What drew Pete into robotics 27:38 Generalist’s founding thesis and what sets it apart 32:17 Protecting simplicity 35:01 How Generalist’s data collection has evolved 35:30 Generalist’s scaling philosophy 39:53 Generalist’s build-versus-buy philosophy 42:15 Balancing general intelligence with commercial deployment 44:24 Why Pete is betting on dexterity 47:33 The durability of the robotics application layer 49:09 Open-weight robotics models and data flywheels 50:52 Geopolitics, safety, and alignment 52:42 The parallels between teaching humans and robots 1:00:05 The timeline for robotics progress and adoption 1:03:37 Book recommendations — Pete’s complete reading list The Score Takes Care of Itself: My Philosophy of Leadership: https://www.amazon.com/Score-Takes-Care-Itself-Philosophy/dp/1591843472Wooden on Leadership: How to Create a Winning Organization: https://www.amazon.com/Wooden-Leadership-Create-Winning-Organization/dp/0071453393— Other resources Generalist: https://generalistai.com https://generalistai.com/GEN-1.5: https://generalistai.com/blog/gen-1.5Proprioception: https://en.wikipedia.org/wiki/ProprioceptionCutaneous receptor: https://en.wikipedia.org/wiki/Cutaneous_receptorFukushima nuclear accident: https://en.wikipedia.org/wiki/Fukushima_nuclear_accidentThe bitter lesson: https://www.cs.utexas.edu/~eunsol/courses/data/bitter_lesson.pdfGEN-1: https://generalistai.com/blog/gen-1— Where to find Pete Florence LinkedIn: https://www.linkedin.com/in/peteflorence X: https://x.com/peteflorence Website: https://www.peteflorence.com https://www.peteflorence.com/ — Production and marketing by penname.co https://penname.co/. For inquiries about sponsoring the podcast, email [email protected].

• 0:00Speaker 00

  2011 Fukushima nuclear disaster.

• 0:02Speaker 01

  The Fukushima number one nuclear power plant has been hit by its biggest earthquake.

• 0:06Speaker 02

  Simply heartbreaking.

• 0:09Speaker 00

  There wasn't a robot in the world that could walk into the nuclear facility after the meltdown and just turn a valve. And so we had to send people in for that. If you're not touching and moving the world and you're a robot, what are you doing? Any process in the world that is gated by physical progress, building houses, whatever it is, we could just do that way faster, way better. Gen 1.5, it has a couple of different capabilities where you can learn a physical task in just a single demonstration. The success rates of all these are modest today, but the fact that these success rates are anything above zero is the incredible thing.

• 0:40Speaker 03

  Sholto Douglas, well-known anthropic researcher, called this sort of a GPT-3 moment for robotics.

• 0:46Speaker 00

  The example that we highlighted is using a brush to sweep an object into a bowl. And then instead you take the brush away and you give it a dustpan.

• 0:53Speaker 03

  One of the stories of these past few years has been finding sort of the killer use case in coding. When it comes to physical intelligence, do you have an intuition of what like that killer use case might be?

• 1:03Speaker 00

  I think the best analogy would be dexterity. If we can really crack dexterity, then the robots can build the robots and the robots can start building the next generation of robots.

• 1:17Speaker 03

  Robots don't learn the same way that people do. Teaching a robot a new task can take months of programming, data collection and retraining. Generalist, an AI company for robotics, is compressing that process down to seconds. In one of its recent demonstrations, a robot was shown, just one time, how to take money out of a wallet. It then completed that task immediately without needing to be retrained. Today's guest, Pete Florence, is Generalist's co-founder and CEO. The company is building what it calls general intelligence for the physical world. AI models meant to work across different types of robots, from factory arms to humanoids.

• 1:59Speaker 03

  Since its founding a little over two years ago, Generalist has grown into a key player in the robotics race. In this episode, Pete and I discuss one-shot learning, why dexterity remains so hard, where physical AI follows the same scaling laws as digital AI, and which industries change first when robots learn new skills much faster. I'm Mario and this is The Generalist.

• 2:26Speaker 03

  Well, I think the only place to start is that I think you have the best name possible.

• 2:31Speaker 00

  Likewise, Mario. Yeah.

• 2:33Speaker 03

  Very good taste. Well, I'm really looking forward to this conversation and getting into what you're doing at Generalist. It feels like a particularly exciting time. Earlier this month, I think it was mid-August, you released Gen 1.5, which is your latest foundation model. And it seemed to attract a lot of excitement. You know, I saw Sholto Douglas, well-known anthropic researcher, called this sort of a GPT-3 moment for robotics. Maybe you can tell us a little bit about what Gen 1.5 is and why it matters.

• 3:04Speaker 00

  Sure. So Gen 1.5 is definitely in a few different ways, like a type of model that we had been kind of waiting for for a while, at least in terms of glimpses of types of capabilities that both are just exciting in and of themselves and then also point towards like what might be a future of what's coming next, right? So overall, Gen 1.5, it has a couple of different capabilities in what we call one-shot or a few-shot learning. And also some other things in what we think are really remarkable levels of zero-shot physical generalization. These like one-shot and few-shot broad types of learning capabilities,

• 3:49Speaker 00

  where you can learn a physical task in just a single demonstration. And mind you, just to be clear, like the success rates of all these are modest today, but the fact that these success rates are anything above zero

• 4:01Speaker 03

  Yes.

• 4:02Speaker 00

  is the incredible thing. And you know, if I dial back like a few years, one of my co-founders, Andy Zhang, and myself, like we've been talking for a while about this threshold of crossing into

• 4:18Speaker 00

  a very generalized one-shot and few-shot learning capability in robotics. And of course, like the rise of large language models around 2020, yes, of course, GPT-3. Like that was the kind of the main thing that, or at least arguably the main thing that people were especially excited about with the model GPT-3, was this broad one-shot and few-shot learning capability. They coined the term in context learning in that paper. It's been an incredible thing to see.

• 4:48Speaker 03

  And so one-shot and few-shot learning is sort of a necessary principle to have that sort of like generalizable intelligence in the same way that GPT-3 had, where you're able to say, hey, you know, make me a poem in the style of, you know, Emily Dickinson. And after a few tries, if you give it an example, it can do that. You're sort of seeing that similar style of behavior in the physical world. Is that sort of the parallel?

• 5:12Speaker 00

  Yeah. So you know, specifically one-shot is where you you give a single example of an input and an output. So you could say like, who wrote Romeo and Juliet? Answer, William Shakespeare. And that's like a question answer, an input and output. And then you give another input and you let the model do the output. So you give a single example and then the model then it knows what to do from that single example. That would be like one-shot. You can also do zero-shot, which is you just like say, you know, who wrote Romeo and Juliet? And you don't give any, you know, example of a full input output.

• 5:47Speaker 03

  Yes.

• 5:49Speaker 00

  And it was actually, if you chase back the history, it was actually GPT-2 started to show a little bit of these types of capabilities. But it was really GPT-3 that like brought broad one-shot, few-shot learning into the like, wow. Basically, a very wide variety of tasks, you could just give it a go. You could just ask the GPT-3 model, for those that remember, just like trying to prompt it with a few different examples. And it had a reasonable shot at doing something intelligent. And that's like the level that we feel like we're now seeing, where the success rates are not super high. There are limitations on the types of tasks that we can just prompt the model to do. But a very wide variety of tasks, you can just try to prompt it and see what happens. And there's a reasonable shot that something intelligent occurs.

• 6:35Speaker 00

  And again, this like, the fact that this happens at all is the incredible thing to us.

• 6:39Speaker 03

  And then, you know, the tasks are things like take these bills out of a wallet or, you know, sort these cups, which it's amazing to watch these things play out. Did you expect to see this capability as part of this model? Like where are things tracking versus how you were sort of imagining they might?

• 6:57Speaker 00

  It was surprising to us that this model started to show this particular type of one-shot, few-shot capability. In part, and it's something that we talk about a lot in what we released, describing some of the development behind the model. But it was not trained to do like one-shot or few-shot examples. And that is actually the same way that, at least as far as we know, as it is described in the GPT-3 paper, that happened as well, where you actually just train the model, in the case of GPT-3, on every single possible piece of text that OpenAI could find at the time. And it's later been studied and given a name that it's kind of, you know,

• 7:40Speaker 00

  the way that the data is just laid out, it has this property which is called burstiness, which is that when you see a single SAT question example, just to give an example here, it's pretty likely that your next, the next thing that the model sees is another SAT question. So like when you start to load the context of the model with similar types of things, it, oh, it's pretty likely that the data distribution it was trained on has a similar type of thing occurring after it. That's like kind of one of the ways that we can kind of try to hypothesize how this capability showed up in the model GPT-3. And similarly, like we never trained Gen 1.5 with like an example of a task and do it again.

• 8:22Speaker 00

  But the model just, you know, the physical world also has burstiness, where if people are like taking bills out of a wallet, it's pretty likely they'll take another bill out of a wallet, right? Is it like, it's literally just guessing what is most likely to happen next or what it should do next.

• 8:39Speaker 03

  Yeah.

• 8:40Speaker 00

  Based on a little bit of prompting in its context. It's been surprising for us to see this model start to have that. I would have thought it would have taken maybe a little bit more time. But at the same time, I do think it was inevitable at some point, maybe through different means, to start to have this capability.

• 8:56Speaker 03

  Do you have a sense of why it's more successful at some of these tasks than others? Like it seemed like, I think the wallet, you know, example is actually, it's one of its best tasks, right? Versus, you know, things that to a human look sort of equivalently difficult. Is that just a data availability question? Is it something else happening?

• 9:15Speaker 00

  That's, it's a great question. And the honest answer is it's really hard to trace exactly why. We can sort of build up some intuition from a lot of empirical experimentation. You know, in the pre-training distribution, you know, a lot of these tasks, there's no way that like every single task that you can think of is actually in your pre-training data. Like definitely, there are tasks that we can do that there was no pre-training data for this. And one of the easiest ways that we know this is we can just 3D print objects that we're pretty sure have never existed before. We can demonstrate a task with it and it can do it, right? But there are certain sort of underlying skills or behaviors that together comprise a task. And if you have more data supporting those tasks, then that's maybe one that would have higher success rates.

• 10:00Speaker 00

  But it's not only, it's not only about how much data is there. It's not very simple to describe what makes a task challenging in robotics, especially for dexterity or dexterous manipulation. But I think there's a few different kind of component features that add up to make things challenging. Like one is how precise, from a motion precision standpoint, does a task have to be? How precise from a force precision standpoint? So how like hard do you have to push on something or not hard do you have to push on something? How coordinated do all of your degrees of freedom have to be? So like a task that really requires a lot of bi-manual precise coordination between the two different hands, that's very challenging.

• 10:42Speaker 00

  Those are a few, but there are other aspects as well, where it's actually, it's a very rich space that we can try to kind of make sense of about what makes a task challenging in the physical world. Those all contribute here. Yeah.

• 10:53Speaker 03

  It's, robotics is so interesting. You know this, you know, a thousand times better than I do. But I find myself getting so excited about some of the stuff that you're doing because it touches on like what we think of, touches on biomechanics, it touches on like philosophy. You get into these evolutionary questions. Like when I was starting to do some reading about, you know, the Evolutionary development of our own hands. It seems like it's extreme. It traced extremely closely with the development of our cerebral intelligence as well as our manual intelligence. Like those two things just, you know, came together in a really interesting way. You've mentioned before that dexterity is something that's sort of like this, I don't know, holy grail or a big topic in robotics for a long time. More specifically, why is that something that seems to be just so important to get right?

• 11:38Speaker 03

  Is it just about the precision? Is it about those different factors that you sort of just mentioned?

• 11:43Speaker 00

  So one, for sure, you know, it's something that honestly, both like me personally and the company has driven a lot of inspiration from is exactly what you mentioned, which is that if you study evolutionary biology, um, you know, of course, there's evidence to suggest these things, but we can never know for sure. But, but it does seem that, and there's this, there's this term for this like area of the brain. My brother's a neuroscientist. It's called like Broca's area or something like that.

• 12:06Speaker 03

  Which is language, maybe?

• 12:07Speaker 00

  It's, it was believed to, there's, you know, all these different regions of the brain can be associated with many different things. But there is some evidence that I have read in published research, which suggests that like the development of dexterity and the evolutionary adaptiveness of like having better and better tool making skills then helped create the evolutionary pressure to have the right type of scaffolding in the brain, or even could have been the thing that helped just grow the brain in general.

• 12:38Speaker 03

  Yeah.

• 12:39Speaker 00

  That's me extrapolating a little bit from what I've read from the scientific research, but there's just the general connection between humans becoming tool builders and the drive to master more and more dexterity and, and some suggestion that that might've helped open up the brain for other types of cognition, like, like, uh, language and all these other things as well. The reality is that of course, in, in, uh, in AI, like all these connections are sometimes nominal at best, but are also like genuine sources of inspiration, right? Like the, the way that we have created AI today, um, you know, digital neurons are not the exact same, of course, as neurons in our brain, but, but there is, there's a, there's a definite inspiration that has led to, you know, genuine, a significant advancement.

• 13:23Speaker 00

  So anyway, I, I think that that's just fascinating. But then over in, in the case of more just like grounded in what has been the state of robotics, like over the past several decades, leading up into now, and why, why dexterity is such a thing that stands out in general, I would say. And just the state of the state has been that very generalized dexterity, AKA not just being able to pick up something that you have a predefined model of and you're doing it over and over again, but being able to touch and move the world and manipulate it, AKA like move it into any type of configuration that, that is the goal for the robot.

• 14:07Speaker 00

  That has been an incredibly challenging thing. And just the, the empirical, uh, state of research has been that it has been the hardest thing to master across a very wide range of tasks. Whereas other things like, you know, in an amazing way, it wasn't the case, uh, a couple of decades ago, but like walking robots worked pretty well, at least on like pretty flat ground. And there, there are edge cases that, you know, general case of what's called local manipulation or parkour type things are challenging, but walking around on flat ground has become a pretty well-developed capability. Meanwhile, putting a robot in front of a bunch of different objects and, and getting

• 14:49Speaker 00

  it to do anything with them, especially for challenging different types of objects, challenging different types of tasks, that has been, that has been a very elusive thing.

• 14:57Speaker 03

  I, I always like to ask a few people close to the interview subject, uh, what I should ask you. And one thing that someone suggested was you should ask him, you should ask Pete how he would describe a hand. What does he think a hand is fundamentally?

• 15:12Speaker 00

  There's this guy, Matt Mason, who's, um, arguably like the godfather of robotics manipulation. He has a nice definition, which I'll butcher a little bit. I haven't read this in years, but it's somewhat in my head, which is that robotic manipulation is the process or something like that of selectively choosing contact with the world to put the world into some desired configuration. And the, the operative part here is selectively making contact, right? And it is actually very carefully chosen because you can, uh, do things like, uh, not touch something and that can be your, that could be part of your goal. It's not just making contact, but it's also choosing to make contact or not.

• 15:55Speaker 00

  And there's this very particular word that just, um, for those that are maybe not acquainted with robotics, like contact, uh, you know, that's a word that in many different contexts has many different meanings.

• 16:05Speaker 03

  Yes.

• 16:06Speaker 00

  But contact is really everything in dexterity, or what's called manipulation in, in research. It's about touching something and being able to, you know, have things that are not touching, put them so that they are now touching. And when they are now touching, you can move them around. And it just turns out that when you have to actually touch and move the world, that is where the physics is really hard to predict.

• 16:32Speaker 00

  It's an incredibly rich, sort of at its core, mathematical problem that, uh, you know, for many decades, we've tried to like solve these types of things analytically. It's very challenging. And in practice, the thing that has made the most progress, just like in language and, you know, at this point, it's, it's, it's just such a well-known story, but just having models that are trained on a massive variety of data and, and very carefully, the training recipes are, you know, refined through tons of experimentation. That's been the thing that has given us the most progress. So sorry, back to hand.

• 17:05Speaker 03

  This is exactly the sort of rabbit hole I wanted to go down. So that's great.

• 17:09Speaker 00

  I think a hand, you could say is anything that helps you selectively make contact or not, is, is, is what I would say is the definition. So, you know, that's pretty broad.

• 17:21Speaker 03

  Yeah. Yeah. You can imagine the evolutionary pressures of our, of the hands of the future could be, could look very different.

• 17:27Speaker 00

  Yeah. And, and even like, you know, things like feet, we can consider hands.

• 17:31Speaker 03

  Yes.

• 17:32Speaker 00

  And of course we can use our feet as hands.

• 17:34Speaker 03

  How do you think about giving robots a real sense of touch? Like, is that pressure? Is it temperature? Like, what are the right ways to gather that kind of data? And what does it miss at the moment versus real human touch?

• 17:51Speaker 00

  Human touch is incredible. I haven't studied human touch, um, that much, but of course I have an intuitive sense of human touch. And I think in robotics, touch is certainly important. A simple exercise on how much though, like touch is limiting us is you can teleoperate a robot and you can have no touch feedback. And you'll realize that like you pretty quickly could teleop a robot with no touch feedback at a very like super AI level, right? Not like superhuman level of, because you are, you, you can do it much better than, than our best AI models can today.

• 18:33Speaker 03

  Yes.

• 18:33Speaker 00

  And what that shows you is that like, yes, there are certain cases where touch would, would really help us a lot more, but at the same time, human brains still are, are quite incredible compared to even our best models, even G1.5, et cetera, at generalized dexterity manipulation. That said, I strongly believe touch is important and will be part of like reaching the highest pinnacles of performance that we can with robots. Then there's actually quite a lot of different options for how robots can effectively get these types of like non-visual or non-audio types of sensation of the world. I said before, I haven't studied human touch that much, but I, I know a little bit.

• 19:13Speaker 00

  Like, so one is that like, there's this concept called proprioception, which is, um, sometimes people would say different than touch, but it's, it's very relevant here in that it's like, you know, non-visual, non-audio senses of the world. And we as humans have a bit of proprioception in that, like, we know how our arms are bent. We, we can close our eyes and we know how our arms are bent.

• 19:32Speaker 03

  Yes.

• 19:33Speaker 00

  And that one is, is pretty easy for robots to get. Basically every robot gets, gets that for free.

• 19:37Speaker 03

  Yeah, that makes sense.

• 19:38Speaker 00

  What's called encoders. You know, you know, your joint angles. Another type of proprioception is knowing how hard something is like pushing on the muscle at your joint. And that one, a lot of robots now also have either through what's called current sensing or torque sensing at the joints, but that one is also like in most robots already today. And those two alone, I think, provide a lot of what you might think of as touch sensing. But then there are other things where, um, I believe the term is called like cutaneous touch or like the ability to like, you have your fingertip and you can like scratch it and you, you're not feeling it like down in the rest of your arm, but you can feel it right there. That is the one where there's starting to be more and more sensors there. I don't think, um, it's not at least a widely deployed thing to have lots of those sensors

• 20:22Speaker 00

  all over, uh, robots today. I think over time we might see more of those. And yeah, I think that there's a lot of cases where those types of sensors will be useful.

• 20:29Speaker 03

  You, you grew up in the Bay Area. Were your parents in tech? Did you have models of entrepreneurship sort of around you all the time? Or was that not sort of the version of your, your upbringing?

• 20:39Speaker 00

  Both my parents worked in, uh, technology at different points. My mom's side of the family has been in the Bay Area for a very long time. So I, I think this is actually like, um, you know, living in San Francisco, a lot of people come and go. I, I love San Francisco, but the reality is a lot of people that you meet in tech, you know, uh, maybe moved here, uh, this generation or last generation. For my mom's side of the family, it traces back to like, I'm a sixth generation Bay area.

• 21:03Speaker 03

  Oh, wow.

• 21:03Speaker 00

  And it was my, uh, triple great grandfather moved from Lafayette, Indiana, settled in the Bay Area up in Petaluma. That's my mom's side of the family. My dad's side of the family, uh, my dad moved up from LA. But yeah, my, my mom worked in, um, in, in tech. And then my, my dad worked at a bunch of different startups. And my grandfather was the, was the scientist. He was a chemist that, you know, back in the middle of the 20th century, one, uh, frontier area of technology was, uh, rubber.

• 21:33Speaker 00

  And yeah, he was, he was a chemist and he worked at a, a rubber factory.

• 21:36Speaker 03

  And was he sort of, I know you studied chemistry. Was he a bit of an inspiration in picking your sort of first paths?

• 21:42Speaker 00

  I, I did study chemistry. That's what I did in, in college. A hundred percent. Yeah. I think my grandfather helped inspire that.

• 21:47Speaker 03

  Were you the sort of kid that was, you know, programming, uh, you know, in your early days or, you know, building, building things in, in that sense? Or did that come, you know, a little bit later?

• 21:57Speaker 00

  There was always building stuff. Uh, I was actually just talking with my wife about this the other day that like, she was talking to my brother and we were just talking about like Pete is a 12 or 13 year old and, uh, modifying Nerf guns to make them like, uh, you know, more powerful. Uh, we actually, me and my, one of my best friends, uh, we had a, a website we called a modification directory for modifying Nerf guns.

• 22:19Speaker 03

  Oh, I love it.

• 22:20Speaker 00

  The, the, the same thing with super soakers. Those are like the types of things that I would, would, would do at that age.

• 22:27Speaker 00

  And then, yeah, programming in, in college I did, you know, enough to sort of like support like, uh, scientific research that I was doing in chemistry or in, in physics. Um, but then graduate school was really when I like became a more full-blooded software engineer and, and, and, and everything else. And, uh, that you need to know to, you know, do AI and robotics.

• 22:47Speaker 03

  When did robotics sort of enter the picture for you? When did you start to think, you know, this was, you know, as, as rubber and chemistry was to your grandfather, maybe the most exciting challenge technically of, of our generation?

• 22:58Speaker 00

  So I, I had done undergraduate in chemistry and I did a, when your masters in physics, um, and that, everything I was working on was, was a fascinating field to be in. You know, before you choose to do your PhD, there is this opportunity to, to just think about like, what really would be a field of research to like, really dedicate a, you know, huge amount of years and potentially a lifetime to. There was a couple of things that just felt like, you know, this was 2014, that robotics might have just a massive impact on, on the future. There was a little bit of a, a direct sort of connection in what I had been working on before, which is that

• 23:41Speaker 00

  I was in the lab doing physics research and I would, um, come into the lab and I would just think about like, what is the thing I can do today that would most advance my research? And science requires a lot of different type of thinking. But I do think that like, if I just thought about the hours of the day and what I could do the most to advance my work, it was, a lot of it was just like doing all of the manual labor to push through a bunch more experiments. And that like, as, as simple as it is, like iteration speed, both in robotics and in AI, but also the rest of science, especially empirically driven science, which is a very massive fraction of, of all science.

• 24:23Speaker 00

  It, it felt like being able to rapidly accelerate that, uh, like we didn't have the machines to do that. And of course, like, there's been a long history of scientific automation, uh, for things like, uh, in physics and chemistry, different types of experiments. But, uh, there have always been very sort of narrow regimes in which those types of instruments have worked. And a very, um, broad capability to like, have a robot just run all of your experiments. That, that, that was something that felt very far off at that time, but felt like it, it could happen in, in my lifetime. And that was one, you know, yeah, very direct reason why I thought it might be very exciting to, to start to work in robotics.

• 25:02Speaker 03

  I love that that's the call to adventure for you because I feel like often the call to adventure for a robotics person is, hey, you know, there's these, the set of jobs that are dangerous or sort of, you know, drudgery that would be great if humans didn't have to do it. And that's, I suppose, this was, you know, a, a form of drudgery, but it's a very high-minded version of that where you're like, imagine if we had thousands of robots conducting scientific research, uh, at, at this massive scale. It's a really interesting vision.

• 25:31Speaker 00

  I, I think too, like, for me, there's, there's like a few levels here. So one is, yes, like there are the drudgery things. And, and actually like, you know, even people talk about this less today, but like when I was starting graduate school in robotics, like the example that everybody gave of the obvious thing that of course we want a robot to do instead of people to do, was the 2011 Fukushima nuclear disaster.

• 25:52Speaker 03

  Yes, right.

• 25:53Speaker 00

  And the, the story that, that, you know, I would, would always hear, and I, I just think, yes, it's something, everybody's just like, yes, that makes sense. Is that there wasn't a robot in the world that could walk into like the

• 26:05Speaker 03

  Wow.

• 26:06Speaker 00

  nuclear facility after the meltdown and just turn a valve. And so we had to send people in for that. So that's like, okay, that's certainly something where

• 26:13Speaker 03

  Everyone's on board. Yeah.

• 26:14Speaker 00

  We should have robots do that instead of people. Um, and then on, and I was just talking about the science example. You know, for me at least, it's, it's a very inspiring example that like, yes, we could have

• 26:24Speaker 00

  things that we already know today, but we could just do them way faster. Or like the same type of thing already exists, but it could be way better. Or we could make things, we just, you know, iterate way faster on, on how we build, design things. We, yes, we could run way more science experiments, any process in the world that is like basically gated by physical progress, building houses, whatever it is.

• 26:46Speaker 03

  Yeah.

• 26:46Speaker 00

  We could just do that way faster, way better. That is one layer. But then there is also this third layer of just like, things that we just can't even imagine really yet today. That will be possible when, you know, we could actually like, build the structures that might take like thousands of people.

• 27:06Speaker 03

  Yeah.

• 27:06Speaker 00

  But then now we can just have a whole bunch of robots do it. It would have been very hard to foresee all of the implications of the internet when the personal computer was coming around. Or like, you know, before fire existed, it's just like, before fire existed, it's hard to imagine all the implications of fire. I think the same thing will very much be true with the level of robotics that, that will be possible. It's just hard to even think of all the possibilities.

• 27:28Speaker 03

  Yeah, there'll be all these new, new use cases, new, uh, structures that suddenly become economically and, and sort of logistically viable in this way that, that weren't before. So this is sort of the moment you, you become, you know, increasingly excited about robotics. You know, you go to, to, to Google and spend some time there. At, at what point did you sort of realize this should really be its own standalone company? And I actually happen to know these two other folks that I really want to do it with.

• 27:55Speaker 00

  We just kind of talked through it in, in terms of like, why do we actually want general intelligence for the physical world? And I, I think like, we, we, we just talked through, I think, a few different like, you know, very much for me, examples of like, why is this something that I think it's just really important to have? Like an absolutely incredible team behind. And then in terms of why did, you know, why did we think we might have a shot to make it all happen? Robotics is just amazingly interdisciplinary. And the decisions that are made around like the interplay of what hardware are you using? What data are you, are you getting? How are you training the models?

• 28:38Speaker 00

  How are you closing the loop between the hardware and the data and the models? How are you, what's your plan for how you get these models to start to do something useful? And then close that loop back between the hardware and the data and the models. And like, all of that requires both like, from what is the plan, but then also from who is the, the team doing that? Like, it just requires an incredible set of expertise and

• 29:02Speaker 00

  yeah, between, uh, the few of us that, that started getting it going. And then very quickly the, the founding team we got in place. And, and everybody we've, we've hired, you know, and joined, has joined the team since. Um, I think we just have the best team in the world to, to go do this thing. So.

• 29:17Speaker 03

  When you think about what generalist is trying to do, there are obviously other companies in this space that think about, you know, this, this broad aspiration of creating, uh, you know, embodied intelligence, physical intelligence. What do you think are the things that you're doing particularly differently or the different bets that you're making versus, you know, uh, uh, other, other players?

• 29:39Speaker 00

  I think that we have this very deep commitment to like the philosophy of everything behind what we build. There's several aspects to that, but I would start with one, which is that it's something that we've learned from machine learning has worked really well, which is that simplicity scales really well. A huge fraction of the decisions, not just on the machine learning side, but also on the hardware and how we do data and how everything comes together and how we think about deployments. We want to keep everything as simple as possible and focus on like the core model training recipe and how all these things come together so that we can have the fastest possible iteration loop.

• 30:24Speaker 03

  Yeah.

• 30:24Speaker 00

  So we can like, figure out how to get all these things going together. And then, once we figure out the model intelligence recipe, that's something that we know will scale horizontally across all the other different types of things that are maybe less simple to start with, right? So I could talk about many different ways in which that's sort of embedded into the, the philosophy and, and strategy of the team, but that would be one place I would start with. I think also too, right? Like one thing that you've seen with the way that the best labs in the world that do like large language models and other types of, you know, uh, primarily focused in the digital world intelligence, the approaches come and go a little bit.

• 31:07Speaker 03

  Approaches like the different strategies that work at a given moment in time?

• 31:11Speaker 00

  The, the different like technical approaches.

• 31:13Speaker 03

  I see.

• 31:13Speaker 00

  To a certain extent, technical approaches will come and go over, over the course of years. But in setting up a team, which for a very long period of time will sustain like all of the innovations required, because like quite frankly, like what we have today is amazing.

• 31:36Speaker 00

  But to really achieve like the real goal of like have this very general intelligence for the physical world, which yes, can power all these robots to do all these different things, there's going to be a very significant amount of advancement that will happen. And I think that's setting up the team and the culture to have all the right emphasis on all the right pieces. I think that's something that that we think a ton about and we try to be very not precious about like any of the choices that we've made so far today. We try and always just see what works the best and be constantly both willing to and even excited to completely change our mind about what we think is is the right way to be to be focusing our time and efforts.

• 32:18Speaker 03

  This, you know, idea of of simplicity and, you know, that that tight iteration loop and having the right culture around it. Achieving simplicity is is so hard, right? I imagine there are so many temptations to convolute the problem in one way or to pursue something like when you talk about that as being sort of core to the company, what does that involve? What do you have to do to sort of like protect that simplicity and protect that that culture built around it?

• 32:44Speaker 00

  People who really like to get things to work has just like continued to attract all of the amazing people that that come and join the company. Again, it just kind of self replicates. We don't have to like consciously think about this part too much, but it is really important to us. It's just that being really focused on what are the goals that that we're actually trying to do. And if we're very focused on the goals and we're less precious about the methods that get us there, that's one thing that really helps conjure simplicity. And again, it comes from multiple different sort of expertises and disciplines that come together, right? It's it's yes, it's on the machine learning side, right? So just to call it out, right? There were so many other different possible ideas of how the level of artificial intelligence

• 33:27Speaker 00

  that we just take for granted today could come into existence. The bitter lesson, as the essay has well canonicalized, is there are some levers that matter in an outsized way relative to others. And it's simply stated, you know, things like compute, things like data. It's simply stated, having an incredibly talented team focus on just the things that actually matter is is not easy because there's a lot of, if you know the expression nerd sniping.

• 33:56Speaker 03

  Of course.

• 33:57Speaker 00

  It's very easy to get nerd sniped along the way and think, oh, we can make this other thing and we can make this other thing. But yeah, we try to just focus on whatever actually really makes the most progress towards towards our goals. And again, that was a more sort of machine learning inspired example of the bitter lesson. But I think, you know, we have some just incredible hardware, mechanical engineers, other types of folks that have, you know, been behind building many of the the robots at Boston Dynamics and other places that you know of that have like changed your mind about what a robot is is possible of, what it can look like, what it's capable of, rather. And, you know, we have people that several people that have built humanoids before, et cetera.

• 34:38Speaker 00

  And and humanoids are an awesome form factor, but for the purpose of maximal iteration speed on the core model intelligence recipe, which we think that's really been the bottleneck is is the intelligence piece rather than the hardware today. There's a lot of things on the hardware side, too, where we just we keep them as simple as possible for the purpose of maximizing the velocity of the model intelligence research.

• 35:02Speaker 03

  I think in 2025, you talked about gathering something like 10,000 hours a week of new, you know, lived robotics data. Where is that at today? If you can share directionally or specifically, depending on what you're comfortable with.

• 35:17Speaker 00

  Sure, we can say directionally, it's up.

• 35:19Speaker 03

  That's good to hear.

• 35:22Speaker 00

  Yeah, I think so. Definitely directionally, it's up. And then I'll also say that, right? I talked about the importance of things like the bitter lesson, but we are very passionate about scaling laws. We're very scale, we are very passionate about scaling. You can fundamentally shift the shift the curve, the shift the scaling law curve when you have just algorithmic advances and other things you can do across your training, infrastructure, architecture, et cetera, to just have better scaling. And so, yes, data is incredibly important and it is table stakes. But there's a lot of other things, too, that, you know, we have found over the past few years, just make you incredibly more efficient at at scaling.

• 36:04Speaker 00

  And I think between hardware and how you do data collection and how you do data processing and how you distribute your training infrastructure and how you fold all these things together and how you think about deployment, et cetera, like you can have efficiency gains that are easily 10, maybe even 100, maybe even 1,000x relative to somebody who's just like buying expensive stuff and seeing what happens with the robot.

• 36:33Speaker 03

  I heard that, you know, in the early days, this is another person I talked to in preparation, to get the data sort of flowing, you have to really, you know, do the sort of classic YC thing of, you know, do things that don't scale of, you know, lugging suitcases full of hard drives and digging trenches for fiber and all of these sorts of things. Can you talk me through a little bit of, yeah, what were the the craziest things you had to do to to get that up and running?

• 36:59Speaker 00

  Those are a couple of great examples. I mean, it's crazy, but like putting a bunch of hard drives in a backpack and getting on a plane. Turns out the the bandwidth of that is pretty hard to beat compared to the internet.

• 37:10Speaker 03

  Wow.

• 37:11Speaker 00

  Especially when you don't have, you know, good fiber lines in remote places of the world. We've also, I think we've shared about a little bit about this publicly. We have built like entire custom server machines that just simultaneously upload data from huge numbers of memory cards, like, you know, basically, I mean, it's all built on Linux, but I would say something roughly the equivalent of kind of like an operating system for for that type of, not not fully an operating system, but like significant amounts of software to to support effective scaling.

• 37:46Speaker 03

  And as you think about the data collection piece today, you know, some of this I'm sure is is maybe, you know, not for public ears, but to the extent that you can share, like, what does that operation look like? Like, what are you trying to run on a weekly, monthly basis? Is that sort of a portfolio of different things? Do you think of it almost like a portfolio manager of, you know, making sort of bets here and bets there?

• 38:09Speaker 00

  I think about it more from, I'd say, like, a scientific perspective rather than a portfolio management perspective, although of course there's parallels, you know, trying different experiments and seeing what happens. That is something that like we view as as very core is like, how do we close the loop, not between just like collecting a lot of data and then later seeing what happens, but collecting a lot of collecting a lot of data and then, you know, iterating very quickly on on what data is giving us the biggest gains in in model performance. And yeah, we try a lot of different things, right? Like, an hour of data for training a robotics model can be wildly different compared to another hour of data. Like, an hour of data is not a fungible unit, you know?

• 38:52Speaker 00

  There's a huge amount of factors that can really massively change how valuable that data is, depending on what you're trying to do. So like, the, yes, the types of tasks people are doing, the types of objects that they're interacting with, but also all the different metadata that is recorded for that session and how you guide which new types of data to collect relative to what you already have. And again, like, what are what is actually driving capabilities? These are actually all like pretty challenging questions to do right in in a, you know, quick data rate way. And that's something that we were always trying to do better on.

• 39:28Speaker 03

  In very sort of, you know, order of magnitude, broad strokes, can one hour of data be 1,000 times more valuable than another 10,000? Like, what are the sort of rough heuristics here?

• 39:37Speaker 00

  I think it's it's actually literally, I would say, infinite. Nobody would try to collect an hour of data that's useless, but you can have an hour of data that's completely useless. Like, if you if you don't move the robot at all and you're not, you're looking at a blank wall, it's useless. So I would value that at zero. And then anything is infinitely higher than zero.

• 39:53Speaker 03

  It sounds like, given, you know, how much you you've mentioned hardware already, that like vertically integrating this process matters a lot to you. What what parts of it do you not care about owning?

• 40:03Speaker 00

  You know, I and we as a company don't really think about it as like, what do we want to own? But rather like, just what do we think will help us make the most progress? Right? And so there's things that we feel like we don't have to reinvent the wheel when it's when there are, you know, the equivalents of wheels that already exist that are great. So turns out industrial and cobot arms are, there's been a lot of good work that has gone into, you know, robot arms over the many decades of developing robot arms. We could probably make robot arms that would be a little bit better for our different specs that we prefer, but that's an area where at least, you know, so far today, as demonstrated by everything that we've shown publicly, we haven't haven't decided to to build our own arms today.

• 40:48Speaker 00

  Hands, we have made our own hands, but we also have, as we recently announced, we've started scaling data on lots of other hands as well. We don't make camera sensors, you know, like there's, there's a lot of different parts of hardware where like there is good things that exist. And again, our philosophy is always what will maximally accelerate the model intelligence research? And we'll we'll put stuff together when it exists. And as we think about with everything across the company, certainly in engineering and elsewhere, too, we're very like no fear mentality on we'll build whatever we think needs to get built.

• 41:24Speaker 03

  What is it that building your own hands does for that model loop? Is it just higher fidelity data gathering? Is it sort of giving you more control so that you can iterate better? Both of those things?

• 41:33Speaker 00

  It's all about the iteration speed is is the most important thing. There's also other things in terms of just like the the capability envelope that is unlocked from just like the physics of a hand. But the most important thing for us is really, yeah, what what lets us go as fast as we can on the model intelligence research? So, yeah, there's, there's a lot of benefits of of having a bit of full stack control over the physical hardware itself, how the data is materialized off of that hardware and how the model controls it and all these things. There's a lot of advantages there. But again, also other people make hands that that we've been scaling on as well, and there's a lot of good options.

• 42:15Speaker 03

  You know, I've seen a lot of different robotics founders think about the distribution of their models in different ways. Some folks, you know, want to get out, you know, deploy it across customers as quickly as possible. You know, some folks are maybe spending more time on sort of their own data gathering. Why wasn't the right strategy to say, hey, even if it's not that generalizable, if we can achieve mastery in these two commercial applications, we should scale that massively and just gather tons of data? Is it just not diverse enough data? Is it not sort of useful enough for that core model loop?

• 42:50Speaker 00

  There's a delicate balance here of what type of data is most useful.

• 42:55Speaker 00

  If you have something deployed commercially, and it is really, really working, you know, with, let's just put a number on it. Let's say it's got like five nines of reliability, six nines of reliability. It's really, really working. Then you already have the intelligence to solve that problem. As you're climbing nines up to five or six, or however high you need to go, you'll find lots of interesting edge cases along the way. But like, if you have robots deployed that are autonomous, and you can already solve the problem, then you can already solve the problem. And then there's the diversity argument too. As you just said, like, if you're going to deploy robots autonomously to solve something, like, just, we don't have, like, we haven't previously at least had the capabilities

• 43:39Speaker 00

  to do that on a wide variety of tasks. So then you have a very narrow distribution of data. What you really want is a mix, right? You want as diverse as possible data as you possibly can find. We know that from everywhere in machine learning. And certainly now we've started to see that, at least with our own models, really well supported, that like diverse data is very important in having a very broad base of intelligence. But then you also, you know, to actually deploy things that are useful, you need to be able to get through these levels of performance that actually unlock commercial value. And certainly deploying is useful, but ideally, of course, we're deploying a very general intelligence that can also achieve very deep levels of mastering a particular task, rather than just a very narrow intelligence that only masters like one specific task.

• 44:24Speaker 03

  You know, it's tempting, and perhaps in some ways helpful to analogize to sort of LLMs and the difference between embodied intelligence and, you know, what we think of as sort of these traditional foundation models. You know, when we think about Claude Code and OpenAI and Fable and all these models, one of the stories of these past few years has been finding sort of the killer use case in coding. And coding has sort of, you know, taken things to this new level and driven this amazing feedback loop. When it comes to physical intelligence, do you think, do you have an intuition of what like that killer use case might be? Is there an analogy there that works? Where you think folding things or, you know,

• 45:09Speaker 03

  some task will sort of be the equivalent, sort of first driver of a lot of this demand?

• 45:15Speaker 00

  I think the best analogy would be dexterity. Right? And so, you know, you were talking about dexterity earlier and like kind of giving some definitions on it. It's very broad, but so is coding, right? I mean, like, you can code many different types of things. And similarly, dexterity, you know, moving, touching and moving the world using your hands. It's very, very broad, but it is quite different than things like, you know, navigation, locomotion, et cetera. And there's also like this connection, which is part of why coding has been a thing that has taken off as well, which is there is a little bit of an actual sort of self-improvement story.

• 45:58Speaker 00

  And that if we can really crack dexterity, then the robots can build the robots and the robots can start building next generation of robots, et cetera. But I also just think intuitively, like, you know, software creates a lot of value. And that's why coding is very useful. And similarly, if you're not touching and moving the world and you're a robot, what are you doing? If you don't touch the world at all, what can you do? And I would say, for all intents and purposes, if you don't touch and move the world, you're basically just moving a sensor around.

• 46:32Speaker 00

  Okay?

• 46:32Speaker 03

  Yep.

• 46:32Speaker 00

  And then let's, no, let's keep thinking about this. So if you're moving a sensor around and you're not touching the world, well, you could use a drone. And drones work pretty well today. So like, you know, there you go. There's, in a lot of cases, your solution.

• 46:45Speaker 03

  Yeah.

• 46:46Speaker 00

  But then in a lot of other cases, you could often just like tape that sensor to a wall.

• 46:51Speaker 03

  Mm hmm. And you could put one on a different wall and that would be a lot of a cheaper solution to cover that area.

• 46:58Speaker 00

  So like, there's applications and, you know, deploying robots to go inspect valves. And I always wonder myself like, why don't you just set up a tripod with a camera to look at that valve all day? But inspection is a great use case in certain cases. But, you know, there's just one facetious example. But touching and moving the world is kind of everything. It's a huge fraction of actual tasks that you can think of where it's like physically creating value.

• 47:24Speaker 03

  Yeah.

• 47:25Speaker 00

  So dexterity is really big. Yeah, I think dexterity is the best analog of coding that I can think of.

• 47:34Speaker 03

  When we think about, you know, the analogies a little bit, you're sort of building the foundation model layer. And then there will be these application layers almost in the physical world where you have construction or data center cabling or whatever it might be. I think there's a question when it comes to LLM application layer of whether the foundation model eventually sort of erodes the value of those businesses, whether that's, you know, legal businesses or website generation. I think, you know, you could make the argument either way. When it comes to the physical world, it strikes me that the application layer would be much more durable because you're sort of building, I don't know, these distribution relationships. You're sort of physically installing things. How do you, how do you think about that?

• 48:20Speaker 00

  I think that making robots do useful things in the world is such, as we were talking about before, from like a team perspective, but also from like a making it happen perspective. There's just so many different things that need to come together, right? And like, yeah, there's so much value in better hardware compared to less good hardware. And there's a ton of knowledge about workflows and industries that is required to like, take a robot, even a very generally capable robot, and make it useful for some particular type of application. As you said, I think that there's a lot of durable value in all the different pieces that need to come together to make robots massively useful. Is the models that we make or, you know, models in general

• 49:05Speaker 00

  to make robots do these things, those will be very useful. But there's all these other things that are needed too, for sure.

• 49:10Speaker 03

  How do you think about how the sort of open source, open weights world might play out in your space? Do you expect it to sort of follow a similar pattern?

• 49:20Speaker 00

  We talked a little bit before about, like, the value of different types of data. And having robots actually deployed and having you help climb this like, it works kind of okay, and then it starts to work better. And you're climbing and you're finding all these edge cases to find all these like things that, you know, bring you all the way up to the levels of performance, levels of nines of reliability, et cetera, that you need. That type of deployment data is definitely valuable for the right types of applications. And the thing that is not clear about the open source route is that, I mean, certainly there should be great open source models.

• 50:00Speaker 00

  But then it will be interesting to see if there is actual like, you know, people overuse this word, but it's a real thing, like, you know, data flywheels, that sort of, you can actually get more value from deploying more and bring that data back in and improve the model. It's not clear just in the open source case, like, if there's enough ability to accrue all of the data back to the open source model creator. Because there's not really an obligation to necessarily like give that data back. We'll see how it plays out. But I think that is one of the, at its core, biggest questions for how open source develops. But I think that there will definitely be, you know, better and better open source models.

• 50:43Speaker 00

  And there will be better and better models that are not open. And, you know, I very much think of it as like a, you know, rising tides, raise elbows type of situation.

• 50:53Speaker 03

  How do you think about it from sort of the geopolitical perspective? Because I could imagine a version where you would have Chinese models dominate an open weights ecosystem for robotics models as well. Clearly, we've already seen sort of, and the administration is concerned about, you know, protecting American companies in this space. Would you imagine a world in which, you know, American companies can only use American models? How do you sort of think about the next few years that play out here, maybe informed by what we've seen to date when we look at LLMs?

• 51:27Speaker 00

  In terms of the more policy side to it, I think there's a lot to be thought through with many different types of people on how we can think more about it. There's a huge amount of advancement in the field. I think it's also, you know, calling a spade a spade, like, you know, quite early days still. So I think that there is a good amount of time to start to have these conversations that need to happen to figure out, how do we think about? You know, these models from a very global sense and the open source and close source implications of all this stuff.

• 52:01Speaker 03

  You know, I think the other sort of maybe adjacent conversation is around safety and alignment, which is obviously something that these foundation model companies spend a lot of time thinking about because robotics is a little earlier. As you said, you know, we sort of hopefully have more time to think through that stuff. Does it start making sense to invest in, you know, researching those things now? Is it still a few years to come before, you know, you have to really look into some of those questions?

• 52:30Speaker 00

  It's definitely something that we're thinking about. If you look back at the the gen one blog post, we have, we have a short section in there talking about alignment. It's definitely something that we're starting to think about and of course, yeah.

• 52:42Speaker 03

  When I look back at your career, it seems like you've been a teacher or a coach many, many times in your life, sort of teaching English or tutoring people. When you think about training a robot and teaching a child, what are sort of the parallels? How similar do those things end up being?

• 53:01Speaker 00

  Oh, definitely. And I, I think there's a huge amount of parallels and especially one of the amazing things that we've started to see as the base models get stronger and stronger is that the, the parallels feel like even more there. And so, you know, we, we talked a little bit about like some of the like gen 1.5 few shot, like new task learning capabilities that we've been seeing. But even for a while now, even before gen 1.5, the, the most compelling like first glimpses for me of like really rapid learning, like with just a few examples that, that we had started seeing was that we could have a model that we had adapted to a particular task

• 53:44Speaker 00

  and it could start to do, do pretty well the task, but it would have like a certain type of failure mode that, that it would, that it would have a few times. Like maybe the robot arms just kind of, you know, meandered off to the side in a way that they shouldn't. Or they're just like stupid types of failure modes that were just like very non-intelligent behavior. You know, if you don't have a strong base model in place, like you're constantly running into these. You know, it feels like you're playing whack a mole with, with all the stupid mistakes that, that, that a model can make. But as the base models get stronger, and we've seen this for a while now, you have to only give them like a very small amount of guidance on how to not do the thing that it's causing, that's causing it to fail. And it just like, won't do it again.

• 54:27Speaker 00

  And we saw this like a year ago of like, we're starting to have models that would, would mess up a certain way. And we would show them, don't do that anymore. And it would just like, never do it again. And, and I, I think that that, I don't know, it, it feels more and more like there is this kind of coaching type mentality that, that goes into helping these models be successful at, at all the different tasks that we care about. So that, that's at least, you know, one, one very direct connection. But if, if anything, I think more about the sort of like different things I've done in teaching and coaching in my life, more just about, you know, the, the actual team that we have rather than the robots. But, um, yeah, for the, for the robots too, for sure.

• 55:06Speaker 03

  You know, I think sometimes it's helpful to think of some of the LLMs almost as, you know, a, a forgetful or inconsistent PhD that works with you, right? You're like, this is, this model is so intelligent, but every once in a while it sort of goes off the rails or, you know, forgets everything I've told it. When you think about, you know, gen 1.5, what is the analogy? Is this sort of like an extremely strong five-year-old? Or, uh, you know, how, how do you sort of analogize it?

• 55:35Speaker 00

  The comparisons to children as they grow up in terms of capability level equivalent, I, I think is always very interesting. Working on robotics for a while, we've always had to be very humble with, uh, you know, how capable are our robots relative to, um, you know, baby children? I do think, you know, even as amazing as a model of gen 1.5, there's still things that like a very young toddler is just much better than gen 1.5 out today. Yet at the same time, you know, as we started to get a model, like even previous generation gen one, we started to show that, um, you know, you can take a general model and you can pretty quickly adapt it to do a task hundreds of times in a row.

• 56:19Speaker 00

  I don't think you can get any toddler to do any task hundreds of times in a row. Right? So in that sense, that's more of in sort of like the ability to take a general model and specialize it. I think that, you know, I won't try and pick a specific age number comparison there, but

• 56:37Speaker 00

  there's, there's certainly some type of comparison to be made. And then I also think that these like one-shot, few-shot type learning capabilities that we're starting to see in gen 1.5, that, that feels much more of the type of behavior that you might expect, you know, sort of toddler level intelligence to start to be able to do. If anything too, though, I mean, I think one thing that we, that we hadn't talked before about with gen 1.5 is this like novel tool use improvisation, I think is the best thing that highlights it in terms of this like very physical generalization. And that, for me, actually,

• 57:15Speaker 00

  you know, the one-shot and few-shot capability is, is incredible. And probably the thing that is the most directly tied in a very clear way to sort of like development of language models. But it's also this like very zero-shot physical generalization that is, I would say perhaps like

• 57:33Speaker 00

  even more feels like we're on the right path. And to give like more of a language model analogy there, the zero-shot physical generalization that we're seeing, that's more of the equivalent of like

• 57:45Speaker 00

  asking a language model to just write a poem that has never existed before. There's, there's, there's no example that's being given. It's zero-shot, but, but it's also a task that requires a ton of creativity. You know, like write a, a limerick about, uh, you know, who's going to win the world series this year or whatever it is, right? There's several different ways that we have started to see gen 1.5 do that.

• 58:10Speaker 00

  But I think the, the most like very crystal clear to describe is this novel tool use improvisation type thing. We're like, we can teach the model to do a task with a certain tool. And then we just give it a different tool. And it knows how to use a tool with incredibly different physical properties to still accomplish the task. The, the example that we, we highlighted in, in the blog post in the video was, you know, using a brush to sweep an object into a bowl. And then instead you take the brush away and you give it a dustpan. We just take it so for granted, but like the physical properties of a brush sweeping something compared to taking a dustpan and lifting something up off the ground with a dustpan. Like those, those are the, the levels of physical generalization that, yeah, it's, it's just incredible to start to be able to see these models have.

• 58:55Speaker 03

  I loved that part of the blog post as well, because it's also corresponds to, you know, so much of what we know of intelligence psychologically of like, you know, not having sort of a flexible sense of functional fixedness of an object. You know, a brick can be a doorstop, it can be a hammer, it can be, you know, those are all, you know, in so many different ways are forms of intelligence, right? And so it's sort of incredible that you, yeah, literally see it do that.

• 59:20Speaker 00

  That, that's a beautiful example. The, the brick can be a door, yeah, a brick can be a doorstop. The physical intuition to like improvise with how things in the world can be used to accomplish whatever you want to do. I do think it also like, it all connects back to like some of what we were saying earlier of just like this like tool-making capability has been such a thing that has defined, uh, you know, human intelligence. We haven't been asking the robots to make tools yet, but the fact that like we can give, give them different tools and they can just figure out how they start to use them.

• 59:56Speaker 00

  And, and again, like just glimpses of this today. But the, the fact that we have glimpses of this at all is, is, is the thing that is, um, yeah, really remarkable.

• 1:00:05Speaker 03

  What do you hope your robots or, you know, the, the model is capable of empowering a year from now? What will you be excited to see it do that it can't do today?

• 1:00:18Speaker 00

  You know, having worked on AI research for a while now, a, a year is an interesting amount of time where this is very, uh, I'm sort of, uh, I'm getting to, to answer your question. But first I just think it's interesting that in very fast-moving research in general, but especially at least in my experience and in AI research in particular, a year is an interesting timeline because. In my opinion, it's hard to think of things that you think are likely to happen about a year from now, but are not possible of happening within the next month. Right? It's a little bit easier to think like, well, what's probably going to take five years or 10 years to get to? Meanwhile, anything that I would say, you know, would be a goal for a year from now,

• 1:01:02Speaker 00

  it's possible that we actually, we get there in the next couple of weeks. And, and I would literally say that like, novel to use improvisation. If, if we hadn't already seen that, that would have maybe been something that would sit here and say like, that would be amazing to start to see in a, in a pretty compelling way a year from now. Yeah. Even just about a month ago, I probably would have said that.

• 1:01:21Speaker 03

  What's your benchmark for when you think, you know, you or, or someone like you has a robot in their home that's like doing meaningful labor?

• 1:01:31Speaker 00

  Well, depending on your definition, you could say robot vacuum is already doing that today. So, but then.

• 1:01:37Speaker 03

  Let's say a dexterous one with our first use case.

• 1:01:39Speaker 00

  Yep. And then I think, you know, I, you know, this is, we'll see how far this goes, but there are these, you know, you see them at CES, these like robot vacuums that are starting to have these little arms on them, which clearly are capable of like picking up small things off the floor, not much else than that. But that's a certain type of dexterity. It's a very primitive type of dexterity.

• 1:02:02Speaker 00

  But then a much more higher levels of dexterous intelligence robot in, in your home. I think that if people haven't launched them already, I think there are at small scale products like this, but like, you know, a robot that can fold your clothes. I, I think that people will start to have them, at least in small scale, the question of whether or not they're like actually adding value, or if they're just sort of more there for, for novelty in a very cool sense. I mean, I think it's, I think it's awesome, but you know, it's a separate question of whether or not they're like, you know, really actually meaningfully net useful to you. I, I think that there, that there will definitely start to be that. I at least think, you know, we, we at generalist think that like the more industrial type spaces

• 1:02:42Speaker 00

  are more likely to take off in a bigger way, faster over the next couple of years. And then the data and other types of learning that happens from like a more of a takeoff in a, in industrial spaces, then like the models and other things that power that, it's then likely to be able to help transfer those benefits over to like making the home, you know, other consumer spaces happen more as well. One of the things about a home environment that is just, you know, we, we have to figure out more things before we just deploy tons of, you know, large, heavy robots is, is, is the safety thing is, is of course, you know, something to think about for sure in,

• 1:03:26Speaker 00

  in the home. And then in more industrial spaces, there's just other ways around that. Either robots can be, you know, just cordoned off in, in, in safe areas. There's just a lot of existing technology in more industrial spaces that make a lot of sense for that problem.

• 1:03:37Speaker 03

  Last question, and a bit of a lighter one. If you had the chance to assign a book to everyone on earth to read and know they'd understand it, what would you want to assign to people?

• 1:03:47Speaker 00

  I love these two books that

• 1:03:52Speaker 00

  basically like put together in a much better way than I probably could myself. But it's sort of like, you know, sometimes you read a book and it's like, I've already believed all these things. I just didn't realize that somebody else had written them down.

• 1:04:02Speaker 03

  Yes.

• 1:04:03Speaker 00

  And also not everything, but like, like the core messages are things like, oh my God, I love this. I'm so glad somebody wrote it down. And then you also learn, learn a ton from is Bill Walsh, one of the coaches of the 49ers, you know, amazing football coach, has this book called The Score Takes Care of Itself. And it just describes a lot of his like coaching philosophy. And then similarly, and then there's actually a lot of overlaps. I don't even what sort of a level of studying one another there would have been, but John Wooden, you know, amazing college basketball coach at UCLA for a long time. They have these two books that basically just describe like, yeah, a lot of their philosophy on teams and, and achieving, you know, greatness together.

• 1:04:43Speaker 00

  And, and I think, you know, we talked about it a little bit earlier, but I think robotics is such a challenging and interdisciplinary thing that like, it really requires and just an absolutely incredible set of people with an incredible diverse set of expertise. And I think, you know, yeah, the, the, those teachings are, are very useful, both for robotics and, and, you know, a lot of different things too.

• 1:05:12Speaker 03

  Amazing. Well, Pete, thank you so much. Such a pleasure.

• 1:05:15Speaker 00

  Thanks for having me, Mur.

• 1:05:17Speaker 03

  That's it. Thank you for listening to this episode of the Generalist Podcast. Please subscribe on Apple Podcasts, Spotify, or your preferred podcast app. Ratings and reviews help others discover these discussions. So if you enjoyed the conversation, I'd be grateful if you could take a moment to leave one. For all past episodes and more, visit us at thegeneralist.substack.com. See you next time as we continue to explore the future.