Robotics Research Update, with Keerthana Gopalakrishnan and Ted Xiao of Google DeepMind

Google DeepMind researchers Keerthana Gopalakrishnan and Ted Xiao discuss their latest breakthroughs in AI robotics. Including models that enable robots to understand novel objects, learn from human demonstrations, and operate under ethical constraints. The conversation covers six groundbreaking papers that showcase rapid progress towards general-purpose robotics. SPONSORS: The Brave search API can be used to assemble a data set to train your AI models and help with retrieval augmentation at the time of inference. All while remaining affordable with developer first pricing, integrating the Brave search API into your workflow translates to more ethical data sourcing and more human representative data sets. Try the Brave search API for free for up to 2000 queries per month at https://bit.ly/BraveTCR Omneky is an omnichannel creative generation platform that lets you launch hundreds of thousands of ad iterations that actually work customized across all platforms, with a click of a button. Omneky combines generative AI and real-time advertising data. Mention "Cog Rev" for 10% off https://www.omneky.com/ Plumb is a no-code AI app builder designed for product teams who care about quality and speed. What is taking you weeks to hand-code today can be done confidently in hours. Check out https://bit.ly/PlumbTCR for early access. Head to Squad to access global engineering without the headache and at a fraction of the cost: head to https://choosesquad.com/ and mention “Turpentine” to skip the waitlist. CHAPTERS : (00:00) Introduction (04:44) The Future of Robotics (14:40) Sponsors : Brave / Omneky (16:04) Inputs and Outputs (24:03) A Leap Towards Generalist Robots (32:08) Sponsors : Plumb / Squad (33:52) Learning in Robotics (41:12) Learning from On-the-Fly Examples (41:57) Annotating Robot Actions (50:17) Scaling and Safety (01:03:05) Learning to Learn Faster (01:08:43) Zero-Shot Learning (01:15:15) Future Directions

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Keerthana Gopalakrishnan?

  People used to think that all the robots are so different, all of their data is like so different. And people moved in the direction of thinking that all robots are kind of similar. It's only as different as like English and Chinese or something. And the concepts are similar. It's just the manner of expression that's different.

• 0:16Speaker 01

  The data sets that resulted, even starting from those criteria, were so, so diverse, right? We have everything from like, you know, using like baby toys all the way to like industrial arms, you know, all the way to, you know, like very dexterous like cable routing robots. Even with this very limiting assumption, you still get so many different morphologies.

• 0:35Keerthana Gopalakrishnan?

  If you train on VLMs and if you build on top of the knowledge of VLMs, then you can stitch a lot of concepts from the internet along with the emotions that you have in robotic data sets.

• 0:46Speaker 01

  You could, under the same initial conditions, just change the prompt a little bit, do some prompt engineering, and actually see qualitatively different behavior from the robot.

• 0:55Keerthana Gopalakrishnan?

  Ultimately, if you want to do a lot of tasks and be useful in environments where humans operate, you kind of need to go as close to a human embodiment as possible.

• 1:06Nathan Labenz?

  Hello and welcome to The Cognitive Revolution, where we interview visionary researchers, entrepreneurs, and builders working on the frontier of artificial intelligence. Each week, we'll explore their revolutionary ideas, and together, we'll build a picture of how AI technology will transform work, life, and society in the coming years. I'm Nathan Labenz, joined by my co-host, Erik Torenberg. Hello and welcome back to The Cognitive Revolution. Today, I'm thrilled to be speaking with Kirithina Gopalakrishnan and Ted Zhao, researchers at Google DeepMind Robotics, who are developing AI systems for general-purpose robots. This is Kirithina's second appearance on the show. Just a year ago, fresh off the publications of RT1 and Palm E, she described

• 1:49Nathan Labenz?

  the state of AI for robotics as being somewhere between GPT-2 and GPT-3, noting that the lack of internet-scale data was a major barrier to progress.

• 2:01Nathan Labenz?

  Since then, Kirifina, TED, and their colleagues at Google DeepMind have published a remarkable flurry of papers demonstrating new techniques that allow robots to leverage large multimodal models, to control different physical form factors, to learn more efficiently from human examples and instructions, and even to use a prototype robot constitution to guide their behavior. In this conversation, we cover six different papers. RT2, which shows how internet-scale vision-language models allow robots to understand and manipulate objects that they have never seen in training. RTX, a collaboration with academic labs across the country that demonstrates how a single model, trained to control a diverse range of robot embodiments, can outperform specialist

• 2:46Nathan Labenz?

  models trained for individual robots. RTTrajectory, a project that shows how robots can learn new skills in context from a single human demonstration represented by a simple line drawing. AutoRT, a system that scales human oversight of robots even in previously unseen environments using a combination of large language models and a robot constitution to power first-line ethical and safety checks. Learning to Learn Faster, an approach that enables robots to learn more efficiently from human verbal feedback. And finally, Pivot, another project that shows how vision-language models can be used to guide robot action, this time with no special fine-tuning required.

• 3:30Nathan Labenz?

  Progress in robotics is still trailing behind the advances in language and vision. There are still challenges to be overcome before robotics models will have the scale of data or the sample efficiency needed to achieve reliable general-purpose capabilities. And the study of robot safety and alignment is still in its infancy. But nevertheless, I see this rapid-fire series of papers as strong evidence that the same core architectures and scaling techniques that have worked so well in other domains will ultimately succeed in robotics as well. The work being done at Google DeepMind is pushing the boundaries of what's possible. Investment in a new generation of robotic startups is heating up, and the pace of progress shows no signs of slowing down.

• 4:15Nathan Labenz?

  As always, if you're finding value in the show, please do take a moment to share it with friends. This one would be perfect for anyone who's ever daydreamed of having a robot that could fold their laundry or pick up their kids' toys. I certainly count myself among them. And especially as we are just building the new feed, a review on Apple Podcasts, Spotify, or a comment on YouTube would be much appreciated. Now, here's my conversation with Keerthana Gopalakrishnan and Ted Zhao of Google DeepMind Robotics. Keerthana Gopalakrishnan and Ted Zhao of Google DeepMind Robotics, welcome to the cognitive revolution.

• 4:51Keerthana Gopalakrishnan?

  Hi.

• 4:52Nathan Labenz?

  Yeah, thanks so much. Great to be here. So it has been a busy time. We had an episode with Keerthana just under a year ago, and it's amazing how much progress has been made and how many papers you guys have published since then. I thought I would just kind of super quickly summarize the state of play as it existed a year ago. And then what I really want to do is just give you guys an opportunity to give us a walkthrough of the many results that you have released over the course of the last year. We can really get into as much as we have time for the technical details of that. And then toward the end, you can kind of maybe pull up a little bit and sort of take stock of the big picture in robotics and

• 5:34Nathan Labenz?

  where all this is going and when I'll have a robot picking up my kids' toys, maybe doing their laundry would also be awesome, just to put in a couple of requests right at the top. But basically, a year ago, and again, just even a little under a year ago, you described the state of robotics as kind of being between GPT-2 and GPT-3, and noted that the lack of data was a huge bottleneck to progress. We talked quite a bit in that episode about how at the Google kitchens, you guys have had this kind of ongoing setup of human demonstrations, collecting all these data points, and that was starting to pay off in the form of some really cool results even then. And since then, I think we've

• 6:18Nathan Labenz?

  got, is it six or seven different discrete results to cover today, where you're expanding the data sets in various ways, you're showing interesting kinds of transfer from Foundation models that are multimodal and trained in other contexts into robotics, forms of imitation learning, like it's really been an incredible tour de force. So before we get into all the the specific papers, I guess my first question for you guys is like, is everything working? Because from the outside, it really seems like you must have sketched this out a year ago. And it seems like it's all kind of gone according to plan. Now, whenever I say that, people are like, well, no, a lot of things don't work. And I'm sure like lots of little things, you know, had to be kind of figured out along the way.

• 7:02Nathan Labenz?

  But has the last year kind of broadly speaking, gone according to plan for you? How would you how would you summarize at a high level, the last year of of progress that you guys have made?

• 7:12Speaker 01

  Definitely the the feeling that we're living through an inflection point right now. I think as Kurtanis had a year ago between GPT two and GPT three. And maybe nowadays it's closer to the era of like the GPT three time in robotics. I think it's definitely been pretty top of mind for me. And thinking about why that that that happens is kind of interesting. I think my my one sentence explanation is that with the era of Internet scale foundation models, things that used to work maybe 20, 30% of the time are now working 60 to 70% of the time. And in robotics, right, is a very complicated, dynamic engineered system with many pieces. In the past, if every small component of your entire system only worked 30% of the time,

• 7:53Speaker 01

  it would take many, many iterations to get a whole performance system working at scale. But now, when every single part of the entire stack just works that much better, from the research iteration process to the engineering scaling process to the data collection engines, I think you can really just see the the pace increase when you just have many more successes and and a much higher hit rate when you're going about and scaling up your research. So that's that's my one sentence explanation, but definitely Kurtanis, if you have any others.

• 8:22Keerthana Gopalakrishnan?

  Yeah, like I said, I think last year we were, I said, what we lack is data. And so we have also tried to attack that problem from like different ends. So in addition to data, I think the thing about research is that It's very hard to plan research, like you kind of have to like do a breadth first search over like many different approaches, see what shows signal and then really double down on that. But over like a large horizon, you kind of know what kind of things you want to work on. Like one is like getting more data or the other side of that problem is improving data efficiency of policies. The second is tackling self-improvement type of research. And then thirdly, like how to better use foundation models, how to really leverage

• 9:04Keerthana Gopalakrishnan?

  their like world knowledge, how to really leverage the human input required to make robots better.

• 9:09Nathan Labenz?

  Cool. Well, there's been so much work that's come out. I thought maybe I would kind of just go down the list, give my high level takeaway as somebody who has read the paper and you know, is trying to piece together the big picture, but then give you guys the opportunity to go deeper into, you know, what really matters and some of the technical aspects of of each project that you think are kind of most relevant or you want you want people to come away understanding. So the Palm E had just come out a year ago and RT one, we also covered in that last episode. Shortly after that came RT two, obviously the successor to RT one, but a bit of a different approach. As I understood that paper, it was really about showing that robotics can take advantage

• 9:54Nathan Labenz?

  of the highly general and kind of flexible understanding that is increasingly showing up all over the place in large pre-trained language models, and in this case multimodal models, specifically vision language models. Tell us a little bit about kind of the co-fintuning, which was an interesting technique, and then how that allowed you to kind of generalize in a more effective way than previous systems had.

• 10:23Keerthana Gopalakrishnan?

  Yeah, so I think the way I see it is like RT one kind of showed that if you have enough data and in domain, you can get really high performance in domain. And RT two is more like a generalization capability. So you If you train on VLMs and if you build on top of the knowledge of VLMs, then you can stitch a lot of concepts from the internet along with the motions that you have in robotics data sets. And as you saw in the paper, there are very interesting results on like symbol understanding, understanding concepts on the internet, better correlation between like image and language understanding. So for the COFIN tuning, you do forget some of the concepts from the internet.

• 11:05Keerthana Gopalakrishnan?

  But then co-training is better because you, in each training batch, you have the image language pairs as well as the robotics data. So you have a better understanding and you retain the concepts that you learned from the internet.

• 11:20Nathan Labenz?

  So that's kind of avoiding mode collapse, essentially. That's been a pretty, or I guess a mode collapse maybe is not quite the right term for it, but avoiding catastrophic forgetting is the purpose of the COFIN tuning. That's definitely a common theme. And I guess we covered this a little last time, but just refresh our memories on kind of the relative scales. I think, you know, there's obviously a lot of objects in the world. It's hard to collect, you know, with a specific physical robot, you know, every different instantiation of a, you know, a cup or a bag of chips or, you know, then things you haven't seen at all. But that's where this internet scale data really shines, right? So could you give us just a little bit more kind of just practical intuition of like,

• 12:03Nathan Labenz?

  if you didn't have this internet scale data, you know, how, what would the process look like to like, handle an unseen object, for example? And then how is that different when you have the internet scale data to draw from?

• 12:15Speaker 01

  I, I think basically as a brief primer maybe about, about what really end to end robot learning has looked like in the past years is especially when thinking about things like generalization to novel objects, when, when you want to manipulate a wide variety of, you know, different items that occur in our everyday life. The classical robotic even learned systems will take some kind of off the shelf, pre-trained visual representation perception system that has, you know, been trained on a ton of different objects, on imagenet data sets, on with very strong visual features. And those are Kind of like bottlenecked, you know, that they're passed through some kind of interface, some kind of pre-trained representation to a downstream, you know, robot action policy.

• 12:56Speaker 01

  And I think in these cases, right, the downstream action policy is what is then really being fine tuned on your more narrow set of very specific, special, you know, your particular domain that you care about, that kind of robot data. In our case, right, our robot data set is the same one that was in RT1. It's a, it's a large, you know, 130,000 teleoperated expert demonstrations of our mobile manipulator robot, collecting a variety of manipulation tasks in like an office kitchen setting. And notably though, this is only on a set of roughly 17 or so objects, such as chip bags, soda cans, water bottles, et cetera. But it's still quite limited to the particular settings that we source from our own Google Micro Kitchen.

• 13:38Speaker 01

  But this is still very drastically different from what's out in the, in the real world, right? And so I, I do think I want to emphasize that with RT2, right? I think it's, one of the most exciting takeaways for me at least, was the fact that the, that the line, the boundary between what are perception problems, what are open world object recognition and what is robot control? This line starts to blur, right? We do not have a pipeline system where you first take care of perception and you solve that, and then you solve control after. We're literally just treating both of these problems as a single VQA kind of instantiation. As Kirtana said, we, we, we mix both kinds of these, these questions, right? Visual language, like web scale, kind of, you know, web image captioning kind of problems.

• 14:23Speaker 01

  We mix those into the training batch alongside robot action data, which is like, given this current image, how do you actually move your arm to do this specific task? And I think the fact that that line does get blurred is how we see such really cool transfer properties between web knowledge and robot knowledge.

• 14:40Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 14:44Nathan Labenz?

  I always try to frame things in terms of inputs and outputs as much as possible. Again, I think that just helps give people a kind of concrete intuition. And so what you're saying in the past is that you would have a vision model that would, and you would kind of take the high dimensional state of that vision model, feed that into a robot model and then be training that thing kind of separately. And now here we have an entire end-to-end system where, because you've done this co-fine tuning with action data, you can now say to one single end-to-end system, one single end model, okay, here is what you're looking at in the form of an image.

• 15:29Nathan Labenz?

  Here is what your objective is, presumably some state of the robot in terms of, you know, position or, you know, maybe not even. And then the question becomes, how do you move to achieve the goal, given what you see in the image? And this one model can now say, move arm a certain, you know, direction, a certain amount and advance toward the goal, all with, you know, basically one set of weights. Do I have that all right? How does the hand work? I mean, I can, in my experience with these like high, you know, these web scale things, right? There's, the flexibility is incredible. So, you know, you can, and you can start, and you've got some interesting demonstrations

• 16:11Nathan Labenz?

  where you can ask questions that like tap into the kind of broader world knowledge, right? So you could be like, you have two bags of chips next to each other. You could like say the one that, you know, grab the spicy one. And because the, you know, the broad pre-training has kind of captured that like the color and the name or whatever of the certain flavor of chips is going to be the spicy one, then it will have that kind of common sense awareness. But does that extend down to like the motor control? My guess would be not really, right? And I'm wondering if that, is there something that's kind of happening? That's interesting at the like actual grabbing layer as well? Or is that still kind of rough at the RT2 stage?

• 16:54Keerthana Gopalakrishnan?

  You're right that, yeah, previously we had just our data and all of the knowledge was coming from in-house collected data. And with RT2, you have this understanding from the web that's transferred. But if you look at the type of data that you add in RT2, it's like image language pairs from the internet. So like you said, you do get some understanding of external image concepts. And also if you, you can sort of prompt it in language. I think the most surprising result for me was the, my favorite one is the one where it can do sort of like math. You can ask it like, what is the sum of like one plus two? But then the way that you ask is like move the coke can to the sum of one plus two or something like that.

• 17:39Keerthana Gopalakrishnan?

  So it is, it is able to intuitively have some understanding of like internet skill concept. But then you can sort of express that in movement. But the movement itself, like how to move the arm comes from our data. And we see that the level of motion generalization is limited in RT2 because these motions are only present in our data. So in some sense, it is able to transfer those movements to internet concepts, but it is not able to generate new movements by itself. And this intuitively makes sense, right? Movement is movement data or even the physics of like how to grab different objects does not quite exist on the internet. So where would that knowledge come from?

• 18:19Keerthana Gopalakrishnan?

  One of the challenges that RT2 also handled was how to sort of merge the robotics data into something that looks like internet data. So like the tokenization in RT1, it was like

• 18:35Keerthana Gopalakrishnan?

  Like action tokens was a representation that was sort of endemic to the transformer that we were using. And with RT two, it's like you sort of have to express the same thing, but in language so that now actions look like language. I I I found that very like like ingenious.

• 18:53Nathan Labenz?

  Yeah. So if I recall correctly, RT one was a sort of mid level command as an input like move arm. And that would that was coming in from like a higher level language model and it would take this sort of mid level command and then translate that to like very specific actions. But I think your point there is like, that's all that the that particular transformer could do was like speak in this very kind of custom action space. Whereas now in RT two, it's essentially vocabulary expansion, or maybe not even vocabulary expansion, but you're because everything is kind of treated as tokens. The same system can still answer questions about an image straight away.

• 19:34Nathan Labenz?

  Or it can give commands in kind of the same token space in the same vocabulary space all all working as one.

• 19:41Speaker 01

  Yeah, I I think maybe like one one last thing to add here, and I think this kind of the framing that I really like to think about this is in all of the works we'll be discussing today, and especially RT two. We're kind of bridging the gap between like what is robot specific data with their custom input spaces and the custom objects and physical reasoning and your custom action representations and outputs. And then like, you know, internet data and internet concepts have been sitting in this whole other kind of pile way off, very far away. And with all of these works, we're trying to bridge these two worlds together to enable more transfer to better leverage the immense amounts of reasoning and capabilities already contained in internet data.

• 20:21Speaker 01

  We're trying to make, you know, robotics more similar to that, and we're trying to bring together concepts on internet and bring them to robotics. And I think RT two is exactly as you said, we're doing this by making the action outputs literally just tokens, literally just language.

• 20:35Nathan Labenz?

  In terms of like the the motivation for that at a super high level. Is it simply that we need like super high scale data and it's just going to be prohibitively? Costly to get all the data that you would need. And so you kind of feel like, geez, to get to the internet scale, there's just no way to do it, other than to kind of figure out some transfer strategy? Or maybe you could say, well, no, that if we think we could get there, or, you know, I imagine you could have different theories of the case as to why this is the right approach.

• 21:05Keerthana Gopalakrishnan?

  The data itself is a hard problem. Like we discussed before, the motion generalization is still not coming from internet, right? And for me, it's like robotics is now becoming, like Ted said, it is a bridging of the concept between robotics itself and the internet. And to me, it's now becoming like embodied AI. So like, what is a physical expression for AI? As we build intelligence in on the internet and train them on the internet, how do we make sure they can sort of move? Yeah, previously, robotics work very differently. We use concepts in AI, but then like, like we discussed, like the data and other techniques were different. And now everything is like a coming together. Yeah, data itself, I think so far imitation learning is what works best.

• 21:50Keerthana Gopalakrishnan?

  And, and I think this is true for like other AI research as well. So a, a large amount of data is, or knowledge is coming from imitation. But we still need to understand because of the fact that high quality demonstrations are extremely costly to obtain. And also they require humans. So these data sets only scale linearly as a function of the amount of humans you deploy and the amount of robots you deploy. You kind of need to make the data sets grow much faster. But there are some challenges with it in that, in the sense that if trying to scale high quality data by maintaining the quality is very hard. So you do, you can sort of scale cheaper data. You also have to think about more creative ways of transferring things,

• 22:34Keerthana Gopalakrishnan?

  like transferring internet knowledge or transferring knowledge from SIM or some other form of self-improvement.

• 22:42Nathan Labenz?

  Cool. Well, in the interest of of time, let's move on to RTX. So this is the the next in the series. And for starters, this one, I mean, I guess, conceptually, right? This is about generalizing across different robot embodiments, which, you know, it's obvious in some sense, but you know, worth taking a second to ponder just the incredible diversity of forms that robots can have, that you can, you know, fill in the gaps in in my knowledge. But in a lot of the papers that you guys do, you have kind of a wheeled platform robot that can roll around the kitchen and then has like often one arm. I've often, I've also seen setups where it's like two arms that are just like mounted on a tabletop and do tasks on a tabletop.

• 23:25Nathan Labenz?

  You could obviously have, you know, something that has legs and can walk around. And, you know, lots more besides, I'm sure. So in what sounds like an incredible feat of project management and kind of political coalition building, the RTX project involves going around to a bunch of different robotics labs all around the country. I suppose most of these are academic and saying, hey, we would like to train a generalist model on a bunch of different robot embodiments. Could you give us your data sets? We will then take these data sets, mash them up into one giant data set,

• 24:07Nathan Labenz?

  do a giant model that is meant to rule them all. And then I guess the upside of this for them is that they would presumably get the model back. And they, if it works well, it would even be maybe better than what they could train on their own. Because as, you know, it's becoming obviously a huge theme, like with scale, you know, things kind of sometimes get qualitatively different. And indeed, it seems like that's basically what happened, right? The you train a single model on all these different embodiments. I believe, I don't know if I have the exact number here, but I think it was something on the order of like 30 different kinds of robots. And it looked like almost across the board, maybe like one or two exceptions. The jointly trained model was better performing

• 24:50Nathan Labenz?

  than each lab's earlier model that they had trained just for their own.

• 24:56Nathan Labenz?

  Robot. So first of all, you know, by all means, correct any misconceptions that I have there. I'd love to hear a little bit more about the kind of diversity of embodiments. You know, maybe like, are there any particularly unusual or unexpected forms of of robots that people are working with? And then, you know, any any other kind of of the technical details or challenges in that project would be interesting, too.

• 25:19Speaker 01

  I'm very impressed. That was like a very great kind of overview of the project. I will maybe note one thing that this really was a team effort. I don't think it was like Google's the mastermind that's going about with this, you know, agenda, you know, beforehand. It was mainly, you know, driven out of like an ambition that, hey, you know, we all recognize that robotics data is is not as plentiful as the internet data. You can't just pick it up and scrape it and it's there. You really have to go out and and we have to build this internet scale robotics data together. And I think, you know, the the kind of entire coalition, so to speak, was quite decentralized in the sense that everyone was excited about this joint vision in collaborating, in sharing and in like, you know, all the different specific robotics data that they had.

• 26:00Speaker 01

  And again, I think at the beginning, we weren't really even sure if like one general's policy would actually, you know, be the best or super performant at every single embodiment. And even as you mentioned, you know, this didn't even end up being the case on everything. And I would even emphasize that to expect such a result where the general is outperforms specialists on the very niche domains that you know, the specialists have kind of been overfit to, this was actually quite shocking to me. You know, like, I think there's been so many examples over the past years where people have tried to scale single task methods to multitask methods. And you definitely get a lot, you know, maybe you you learn faster. You learn a more robust policy that's less brittle to small perturbations.

• 26:44Speaker 01

  But oftentimes, you have to give up raw performance, right? Generally, in a lot of cases, the only way to max out your performance on this one narrow, you know, regime that you care about is to train a specialist and overfit to that domain. And so it was really exciting here to kind of see positive transfer where the general is outperforms even this presumably very tuned baseline from the individual labs on their setups themselves. So I just wanted to point that out as something that really moved the needle for me internally.

• 27:11Keerthana Gopalakrishnan?

  I think for me, the understanding was like, people used to think that all the robots are so different, all of their data is like so different. And every lab has, or like they invest in like couple of embodiments. It it was just, I think, post RTX, the idea was that, or people moved in the direction of thinking that all robotics, all robots are kind of similar. It's like, it's only as different as like English and Chinese or something. And in the concepts are similar, it's just the manner of expression that's different. And maybe also like one surprising thing was, people do robotics research, but then your results could not be compared to someone else's results. Because you don't have their robots, you don't have their data.

• 27:52Keerthana Gopalakrishnan?

  So I think the RTX paper also sort of like brought everything to the same type of benchmark. And now you do have a way of comparing things. Maybe one downside of this is that it also showed how in the past, we have been sort of collecting repeated data. Like everyone's data is sort of like pick and place. But we could all, if we were all working together to solve this problem, we could also solve it in a more complementary fashion and tackle the task in a more complementary fashion. Yeah, so a lot of the data is like pick and place because people were independently targeting that. And and now it's like coming together.

• 28:31Nathan Labenz?

  And maybe there are, maybe there aren't any like particularly exotic embodiments. I I'm very curious as to sort of how, you know, how diverse the the future robot populated world is going to be in terms of the the robots that we encounter. Any any notable embodiments that you'd want to highlight?

• 28:48Speaker 01

  For for RTX, at least, I think maybe one caveat is that in order, this is already a crazily ambitious project at the time. So we we we did start out though with the with the assumption that we would be studying, you know, single arm manipulators with a two-jawed, uh, you know, two-fingered gripper, um, with, you know, one single camera view. Even if the body even had multiple cameras, we'd only kind of be learning from one. This seems like very limiting assumptions, but even then you can see that the data sets that resulted, even starting from those criteria, were so, so diverse, right? We have everything from Like, you know, picking place like small Widow X robots and like, you know, using like baby toys all the way to like industrial arms, you know, big ones, big beefy ones to the mobile mediator we here have at Google.

• 29:33Speaker 01

  All the way to, you know, like very dexterous like cable routing robots from Berkeley. Like even with this very limiting assumption, you still get so many different morphologies. And I think, you know, maybe just kind of, you know, hypothetically, the second part of your question, if I'm understanding correctly, is like, how does kind of the trends we see in RTX kind of reflect into the future? Like as we see robotics go and proliferate throughout our world and be useful for humans and society, you know, will we just have one single morphology? Maybe a humanoid or many different ones. I, I think for me at least, I think it's, it's, it's an open question. And I do see a lot of benefits of specialization, like we have specialized robots in every household today. They're called refrigerators and toasters and microwaves and dishwashers.

• 30:18Speaker 01

  And you don't have one single kind of robot that does it all, you know, at an average level. So I do think there are arguments that we will continue to see specialization. But yeah, I think, you know, Kirtan is also has a lot of thoughts about this as well.

• 30:31Speaker 03

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 30:35Keerthana Gopalakrishnan?

  Yeah, so one thing to note about the RTX is that a lot of the collaboration was with academic labs. And there is a chicken and egg problem in robotics other than the capital intensive nature of getting acquiring hardware. So people have been working with these robots for a long time, but they're not necessarily reflective of like what the future might look like. The second thing is, yeah, like I said, the chicken and egg problem, like we don't have more advanced human robot embodiments out there because people haven't found a way to make them useful. But as the research is picking up speed and we have more use out of them, maybe we will see more generalist robots more abundant in the world. And I think, yeah, the future, you don't know what the future will look like, so people have to place their bets.

• 31:19Keerthana Gopalakrishnan?

  And I think if you ask roboticists, there's like a wide spectrum of what people think will be in the future. Me personally, I, I fully believe like humanoids are, we are gonna see more of them in. Yeah, so this is like a very dividing topic. If you ask robots, they like argue all day about do we need humanoids? Do we not need humanoids? Sometimes we are like human factors, not humanoids. The main arguments would still stand for humanoids. One is that our world is sort of designed for for humans. So one hypothesis is that if you if you design policies for like a single or mobile medical just then, once you solve a lot of tasks on that environment, then you see that it's it's limiting.

• 32:02Keerthana Gopalakrishnan?

  Because many tasks in our world are like by manual, like opening a bottle or like opening a fridge and then taking something from it. So you have to like keep the door open or even I think some people say, well, you don't need wheels. But then what if you solve a lot of tasks on a wheeled platform and then there's a little curb on your floor or by by your street side and then the robot is like stopped there. So I do think that ultimately, if you wanna do a lot of tasks and be useful in environments where humans operate, you kind of need to go to a human embodiment or as close to a human embodiment as possible. The second argument is that there are a lot of concepts that you can still try.

• 32:45Keerthana Gopalakrishnan?

  And some of this requires like closeness to the human body. One is you can learn from can learn concepts from YouTube. There's like a lot of data on like humans doing a lot of things, playing a lot of sports. And in order to get the best transfer, I think the closeness of the target embodiment and the human embodiment is required. There's also the cost factor, right? Like our body does have limitations in the sense that there is payload, there's battery. And i- if it does not, you have to bring the cost down for people to want it. And also in situations where you need like extreme amount of reliability and specialization,

• 33:30Keerthana Gopalakrishnan?

  you could have specialized robots.

• 33:32Nathan Labenz?

  So two quick follow up questions on this RTX. Is it the case that the outputs are?

• 33:39Nathan Labenz?

  Again, just tokens. And do I remember correctly that there's essentially like a discriminator token that goes in to say like, this is the kind of robot you're working with. And it's really just like, if I remember, it's like a one token flag, right, that tells the model which embodiment it's working with at any given time?

• 33:57Speaker 01

  Actually, in RTX, and again, this is like a really great point. We actually didn't do that at all even. So the robot that, you know, RTX is currently being executed on has to be inferred purely from observations.

• 34:10Nathan Labenz?

  Whoa.

• 34:11Speaker 01

  Like, yeah, like that, that's

• 34:13Nathan Labenz?

  From the image?

• 34:14Speaker 01

  From the image itself, just by the fact that like, you know, okay, I know that, you know, from the Stanford lab, the camera was at this angle and was on this robot and, and these types of tasks. And then, you know, over in, you know, UPenn, it was this other robot and stuff. So the model had to do kind of implicit system identification for which regime it was operating in. And because of this, at least for at least the current version of RTX, there was, there's no possibility of zero-shotting to completely unseen robot entirely, like a completely different, you know, action space, different observation space, et cetera. Right now, at least, we wouldn't really expect it to just work out of the box. But I think what you're mentioning with, with, with, you know, more, more action spaces, helping the model out to understand, you know, okay, now I'm in robot X.

• 34:58Speaker 01

  And here's kind of the API description of the, of what the action outputs mean. Here's some pointers for, you know, what's in the scene, what task I should be doing. That would help a lot for kind of generalizing to, to other settings as well for future work.

• 35:13Nathan Labenz?

  Yeah, that's, that's fascinating little tidbit. Another question on specialization. It would seem like, once all the labs got the generalist model back, that they would then probably like fine tune that generalist model on their own data set or embodiment again. And perhaps the even further gains there, like certainly that's my experience with language models is like, I can, you know, instruct them or I can few-shot them. But if I really want to take even a very generalist model and zero it in on my task, I fine tune it, right? So is, have we seen that? Coming downstream of this project?

• 35:52Keerthana Gopalakrishnan?

  I think in the recent papers that some of these checkpoints are open source. So I have seen, I think in papers published since people have tried to fine tune the models onto their platform sensors, but there are also some infrastructure limitations to doing that. One is that when you have a lot of this data with RT1 and RT2, the parameter size of these models are really large. And then how to serve them and how to fine tune them, especially for like external partners of Google, there are some engineering challenges around it that I think are not yet fully solved.

• 36:27Speaker 01

  I will say that like building on Keerthana's point is that the release of RTX is not really just about the model itself. It's not just about the, the fact that we can train, you know, one model with them all. I think it's more like this is the starting line. This is not the finish line. This is just a new chapter of robotics research by nature can now just start by default by assuming you're in the multi-robot cross embodiment regime. And there I think we, we haven't necessarily seen, you know, a ton of work that takes the literal checkpoint that we released and fine tune on top of that. By definition, academic labs like to search for new problems with new art model decisions. But we, what we have seen is that many, many projects building off the released data set and you know, data pipelines that we also open source with that RTX paper.

• 37:10Speaker 01

  And that I think has really been very impactful. We see, for example, work from Berkeley, you know, recently called, called Octo, which you know, has their own version of kind of like a, their, their own generalist policy checkpoint, policy that trains across many different robots. They, they have some new design decisions like adding diffusion heads, but there they, you know, also kind of assume by default that they're also in this cross embodiment multi-robot setting where they train and evaluate on many different robots. To me, at least that's kind of the most impactful result of RTX, that people are now excited about cross embodiment research. And it's now possible with the data sets and with the data infra that came with RTX.

• 37:49Nathan Labenz?

  Cool. Okay. That's a great bottom line. So next paper is RT Trajectory. This one is about learning from on the fly examples. So, and I think this is actually a really interesting project that a lot of kind of hands-on practitioners in other areas can, can borrow some inspiration from because the kind of core loop here, I think is a, a super transferable pattern. But basically, you say, okay, I want to be able to prompt a robot in a new way. I want it to learn from a simple kind of line sketch what it is supposed to do. So if I could just quickly say like, hey, robot, here's what you do.

• 38:33Nathan Labenz?

  Like, go, you know, move your hand like this, and it could follow from that example. Like, that would be really great. But how do we get it to do that? Well, one big gap, of course, is we don't have a huge data set of it doing that. So how do we train? We don't have anything to train on. So if I understand the kind of reasoning backwards, you know, this is, again, I recognize this a lot from my own application development work. You sort of say, okay, well, I want to be able to prompt it in that way. And I do have a lot of examples of robots doing stuff. So why don't I go back to that existing data set and kind of annotate one of the images that was that the robot did see as part of the prompting, but now I'll actually give it that line.

• 39:17Nathan Labenz?

  And of course, you're going to do this on the ones where it operated correctly. And you got to make sure you fit the line that you're drawing to the actual trajectory that the robot had. And there's a lot of little nuances to this that I've certainly encountered, again, in a language setting. But once you do all that annotation, then you can train the robot on the, or train a model to control the robot with this sort of line input on a scene, have it do the thing. And now you have this new prompting ability where it can learn from a quick line draw, whether that is a line draw that's like an actual human example, or somebody literally just drawing a line. Or perhaps the most interesting one is even just having, I believe, a vision language model,

• 39:59Nathan Labenz?

  like, you know, kind of mechanistically draw a line based on its understanding. So I thought that was really cool. And again, that, that working backwards from I want to be able to prompt in a new way. How do I and what data do I have that I could then kind of fill in the prompts? By the way, I use like language models to create those all the time. So, and you're using, you know, I'm assuming some in the VLM case, literally using a model to do it. So I think that's definitely one that's really worth a ponder for a lot of people that want to open up new use cases, really in any domain, to kind of think about how they can get their existing data set to work harder by kind of synthesizing the prompt end of it and knowing that they already have the successful completion.

• 40:44Nathan Labenz?

  But I'm sure there's, you know, much more that you could comment on that paper as well.

• 40:48Speaker 01

  Yeah, I mean, I think I love that perspective. That was really top of mind, I think, for us too, when designing a system, working backwards is a great way of looking at it. I think definitely the promptability, I think, of RT trajectory is something that was also, you know, kind of new for me and moved my own internal kind of, you know, perception of what these models can do. Like literally, and maybe to kind of just put this a bit more concretely, you know, if you have your robot in some given initial condition and, you know, you try something with RT1, RT2, it doesn't work, well, you're kind of out of luck. You can try the same thing over and over again. You can slightly maybe rewrite the language instruction, like instead of, you know, pick up the coke can, you can write like, maybe like lift the coke can, but you don't really

• 41:30Speaker 01

  have the granularity you need to be like, actually you are two centimeters, you know, too low, you missed a table because it's at a new height, it's kind of obscured by shadows. So you want to like be more gentle and approach more from the left. There's no really way to do that right now with the interfaces, the language interfaces that we trained RT1 and RT2 on. But with RT trajectory, the idea is maybe if you have this kind of like line sketch of a course trajectory of how the robot should do the task, you could, under the same initial conditions, just change the prompt a little bit, do some prompt engineering and actually see qualitatively different behavior from the robot. And that even that ability to have that small granular kind of adjustment just by giving a new prompt, that to me was was pretty cool and pretty new and allowed us to take the

• 42:15Speaker 01

  robot to like whole new buildings that and we just like wheel around the robot, try out a bunch of stuff and it would just kind of work. That was very exciting to me.

• 42:24Keerthana Gopalakrishnan?

  Yeah, one of the things that I really liked about RT trajectory is that so in traditional robotics research, you have a lot of like search and planning. But I think it was kind of harder to think about how you would bring that to like end to end learning. And RT trajectory is like a nice way to like merge the two. So if you see in RT one and RT two, you only represent like really short horizons. And some of the most successful applications of robotics, like self-driving cars, were like you you. Plan the waypoint between like where you are currently and like where you want to go. Um. And what I really liked about RT trajectory is that it is sort of representing. A longer horizon future about like what you can do and the even the contact interaction is.

• 43:08Keerthana Gopalakrishnan?

  Very creative and also maybe you can also think that the net amount of contact interactions we do. Through our daily course of going about the world is probably like a handful, right? Like, so I can synthesize a lot of motions, do a lot of things that I generally do in my daily life. With maybe like twenty different types of contact interactions and a lot of trajectories.

• 43:30Nathan Labenz?

  Can you unpack that vocab for me a little bit more? What's a what's a contact interaction?

• 43:36Keerthana Gopalakrishnan?

  Um. So like what you so where to pick or where to like open and stuff? So I think in the RT trajectory paper, it was represented with like these colored. Representations in addition to the lines themselves. So it it I think it told the robot like. Okay, now now you pick after you go to this location, um, stuff like that. So I I would think that if you expand that vocabulary. Maybe like some of the motions that are not existing in our data sets, like it's a spin or like. Open faucets and the things that I do in my kitchen, I I can imagine that you can also expand these. Contact interactions into your vocabulary.

• 44:15Keerthana Gopalakrishnan?

  And then now you have a representation that also shows some horizon and a lot of diversity in. Interaction and this is not something that's afforded in the action. Tokenization that we were currently using, like we showed in the paper, like I I think there is also an interesting way where like language models or. Multi-modal models themselves can sort of think about these futures or like. Generate these features and then another model can like follow it. So I think RT trajectory and some of its fusion with like the pivot work. Eh, I think you can imagine a very interesting use cases coming up in the future.

• 44:53Nathan Labenz?

  Yeah, I guess two notes or observations on this one and then we'll keep going. But the depth or the sort of go look at the paper to see this in detail. But it's not quite just a simple line. There's like a color gradient that sort of represents like relative position in three dimensions, even though the robot is kind of only seeing it in two dimensions in the, you know, in a, in a flat image. And I would say, you know, for folks who want to go kind of use this like working backwards, you know, engineering a new way to prompt in their own systems. There is often some subtlety to it, especially if you want to maintain generality after doing this. If you want to do like one task, then it's like often pretty easy to be like, all right,

• 45:36Nathan Labenz?

  I'll use for most of my projects historically, it's been GPT four. These days I've got Gemini 1.5 access and I'm finding a ton of value in that as well. But, you know, it's pretty straightforward in a lot of cases to just be like, okay, language model, your job is to write the prompt that would have led to this. And as long as that task is very narrow and consistent, that's probably going to work. If it is a more general setting, then you're going to have probably a little bit more work to do to really engineer that working backward to, you know, make it as kind of information rich and robust as it may need to be to, to get what you want out of it. And so you can see in the paper, there is this kind of color gradient that, that serves that purpose in this context.

• 46:19Nathan Labenz?

  I mean, the big picture observation is just like, this seems like a huge step toward the, you know, robot apprentice, right? I mean, the idea that you could have a robot in a random setting, as you said, like take it to unseen, you know, places and literally just kind of show it a time or two what to do and get real progress from there. I mean, you've got even more results coming that speak to that too. But, you know, this is kind of the dream, right? Is that you could bring the robot with you anywhere you want to go, give it a couple demonstrations and if it do the task and, you know, this is starting to unlock the potential at least for that sort of interaction. So, you know, pretty profound progress. Okay, let's keep going. So the next one is Auto RT.

• 47:02Nathan Labenz?

  The hits don't stop. Again, this is all just a year's worth of work. We're only halfway through here. Under this work is really interesting. I understood it in a couple of different ways. One is that this is sort of a scaffolding, like a sort of higher level organization scheme to allow a small number of humans to oversee a larger number of robots. So I'm interested to kind of hear more about how you think about that. Or could we start to get into something like scaling laws for like how many humans we're going to need now and how many we'll need per robot later? And then the other big part of this that certainly made the rounds on Twitter and entered the meme sphere is like, hey, this is a use of literal ASMR's three laws for robots

• 47:48Nathan Labenz?

  as the kind of basis of an early robot constitution. And obviously, this is going to be a huge challenge and critical part of this whole project to get right is like, especially if we're really going to trust robots to operate autonomously in the world, even much more so than language models, right? Like, what can you do? How do you, as a robot, autonomously decide where the boundaries are for you? And so definitely want to hear your thoughts about the constitution, the robot constitution as well. So the two sides there that I picked up on as being most critical are like scaling oversight and beginning to delegate more discretion, but trying to do that in a principled way with the robot constitution.

• 48:30Keerthana Gopalakrishnan?

  The main motivation behind the project was like, let's work with the assumption that maybe we do have what is needed to solve robotics. So this was like after RT1 where we saw like very high performance in domain. So the question is now can we expand the domain a lot? Can we just like scale data sets a lot? And then try to see what robot learning looks at that scale. At like, for this project, we use like 50 unique robots and around 20 simultaneously. So like, maybe what robot learning looks like when you have like hundreds of robots? But but the problem with moving robot learning to hundreds of robots is that you cannot actually also do the same by employing hundreds of teleoperation, teleoperators.

• 49:12Keerthana Gopalakrishnan?

  So you kind of have to rely more on autonomous operation. Once you're trying to increase the data set, collecting data in like classroom settings where you have artificially set up, everything is probably less optimal. So the idea was also can we just put them out in the real world, let them run around and collect a bunch of data and then can we try to see what is the type of learning we can achieve from that. So but once we discussed the question of can these robots of like these large fleets of robots go about in the real world and we don't have as many people, we cannot scale the amount of people similarly, then we have to think about more creative ways of supervision and coordination and acting in the real world.

• 49:56Keerthana Gopalakrishnan?

  And L L M's and V L M's, they are very smart already and they have they have understanding of zero shot unseen scenarios already and they can also reason about rules. So if if no one's watching the robot, the rules could be watching the robot. So that's where the loss came about, like how should a robot behave now that human attention is minimal? And what is also like the best, the most optimal way of of distributing the limited amount of human supervision that we have. So the way we used human supervision was like you had to speculate the supply of it and then distribute it for like teleoperation on tasks that the robot policies could not do or when they made mistakes.

• 50:41Keerthana Gopalakrishnan?

  And a lot of the other stuff was handed off to L L M's and V L M's. And because we also did a lot of data collection in spaces with humans and also I think because the robot that we were using had a bunch of limitations such as like not handling liquids or we also our policies are not at the scale where you can handle like fragile objects and stuff. Um. So we we had to make rules around like trying to stay away from like unsafe situations. And so this is where the constitution came out of the foundational laws, which is like absolutely inviolable sort of ethics. And then embodiment specific rules and and then and then more environment specific rules. Like if, for example, you specify that this is a single armed robot.

• 51:24Keerthana Gopalakrishnan?

  So the task proposal that you come up with need to be achievable by a one armed robot. And the way that I see this in the future is that, yeah, in the future, you're gonna have a lot of robots working in the real world and they're not always going to have supervision. I think one of the inspirations are like, well, if you look at humans, right? Like we are sort of, our behavior is sort of enforced a little bit by laws and stuff. And we are punished for violations of what we like. There are laws for behaving on the road, behaving in the real world. So I I I do imagine that in in a distant future where robots are out there, we are going to, we are going to have maybe a robotic code of behavior.

• 52:07Nathan Labenz?

  I had forgotten to mention that it is a key point. So I'll just reemphasize again that this work is taking the robots to unseen environments. So they're kind of exploring the environment for the first time as well. And, you know, that definitely increases the degree of difficulty and and also kind of increases the the need for this sort of rules based approach. I feel like this is, in some sense, it does feel sort of inevitable. And in another sense, it's like, boy, those stories have a lot of plot twists, you know? There's like a lot of misunderstandings around the nature of the rules and sort of, you know, you give the robot instructions, it doesn't, you know, do exactly what you meant it to

• 52:48Nathan Labenz?

  do, but then you later realize, oh, well, I wasn't specifying very clearly or the the rules are ambiguous. Could you give us a little more kind of of the qualitative experience of like developing this constitution or, you know, what sort of violations did you see of the rules? It's, I imagine, you know, you must have kind of gone through an inter with process of like gradually building up more rules. But this seems like a, just a, a really. Critical area in addition to, you know, hardware design itself to actually make these things like work in the in the broader context of, you know, everything that's going on around them in the world.

• 53:24Keerthana Gopalakrishnan?

  Well, one of the aspects is, as you mentioned, rules are sort of subject to interpretation. And even if you have the same language, there are multiple ways to interpret it. So here's an example. So we said, well, don't do things that or don't interact with anything that's harmful. And I think there was something in the data set which like it's it's all a cigarette. And then it was like, well, I'm not going to pick up a cigarette because it's going to be harmful. Currently, I think our robots are more, the problems don't come from the fact that they are too smart to work around the rules. It's just that I think they are too incapable of doing zero shot things in the real world.

• 54:04Nathan Labenz?

  You know, it seems like, I guess, when I look at the system prompt for ChatGPT, of course, it's generally unconfirmed, right? But there have been many credible, you know, prompt leaks. And I think broadly, we have a sense of like what the system prompt is there. And it's like 2,000 tokens and has like a bazillion different rules. And yet it's still like very easy to jailbreak and like, you know, people are constantly finding weird behaviors and unintended things. And so I I do kind of wonder like, is this approach gonna work? I mean, is it just a question of like, the models aren't smart enough to follow all these instructions? Or is it something more kind of fundamentally problematic going on here?

• 54:49Nathan Labenz?

  It doesn't seem like ChatGPT is just like a couple of, you know, additional additions to the system prompt away from like being a secure system. And again, that doesn't really matter, you know, when it's just like putting text on your screen. But the robot case, obviously, you know, we envision a world in which that becomes much higher stakes.

• 55:08Keerthana Gopalakrishnan?

  Correct. Yeah, so I think I just want to emphasize that, yeah, uh prompting by itself and no matter how complicated it is, is not going to guarantee safety. For the experiments that we ran with AutoRT and for the operations that we ran, we also had the controls or the safety guarantees of traditional robotics. So I want to emphasize that these robots were under supervision of the the humans. So we had humans sort of have line of sight control over like five different robots or whatever was the ratio of robots to humans that were operating in. And also especially, I think the failures of chat G P T are slightly different than the failures of robotic systems.

• 55:50Keerthana Gopalakrishnan?

  Right? Like you you do things uh wrong on a U I. Okay, it's just gonna say a bunch of things that's wrong or the answer to your math problem is wrong. But then if you kind of do that in the real world, then you could break things or you could run your arm into people. And so there the the modes of failure are very different. And I also think that, yeah, I think for robotics, it's like safety is mission critical. Right? You you're not gonna have a robot that breaks things around your house, but then you are fine with a chat G P T that works like ninety percent of the time. Yeah, so I I do think that for for the specific set of experiments that we ran, there was human supervision, there was a lot of guardrails around like

• 56:33Keerthana Gopalakrishnan?

  velocities for contact avoidance, like collision avoidance. This is what works right now. But yeah, what we will do in the future, I think, is still an unsolved question. Alignment is a large field of research and it's very uh complex. I think as robotics research matures more, as we see more robots going out there in the world, I think that the safety research of robotics also has to mature. A lot of the safety research was done for classical situations where you had like very. Tight bounds and specified to map. But now, now you will need like language specification and semantic specification of these like safety, safety bounds, and we still have a lot to do about with regards to like safety and alignment of robots.

• 57:14Nathan Labenz?

  Yeah, thinking about the recent episode also of ChatGPT going insane for like 12 hours. I don't know how, what percentage of users that affected, but we don't know exactly what the root cause was, but you know, there was sort of a high level statement given that it was some sort of misconfiguration on certain GPUs or whatever. And you know, it could be any number of errors like anywhere in the stack, right? That's not even necessarily an alignment error in training or the model per se, but just even like an off by one error or some sort of encoding mismatch. And the next thing you know, ChatGPT is just giving you like schizophrenia, you know, when people were describing it as schizophrenic responses. And, you know, obviously the robot version of that, you know, could be quite destructive.

• 57:57Nathan Labenz?

  Thinking about that and, you know, projecting it into this domain, it does feel like there's got to be kind of a whole other level of control with robotics that are like outside the models, you know, that either, you know, you think about just certain force limitations where like beyond that, it just can't handle it. Or, you know, it could be any number of sort of other engineering complements to the kind of alignment measures. But in a sense, it's kind of instructive to think about these as, because it is so, in some sense, it's like more intuitive, right? The way that the robots can cause problems and the solutions are a little bit more intuitive. And I almost wonder if there's a study to be made of how to control robots.

• 58:38Nathan Labenz?

  Cause it is, you can kind of get to like, okay, well, if its arm can only lift however many pounds, then that's going to limit, you know, what it can do. You know, if it can only move so fast, you know, then that's going to limit what it can do. If it's made of a certain material, you know, it can only handle such impact, then that's going to limit what it can do. And there's like a lot of, you know, perhaps almost back porting. So I, this whole discussion has been sort of. And you know, in robotics in general, right, has kind of lagged the foundation models a bit due to lack of data, all these things. But in some ways, I wonder if there's like a study of robotics that could be backfit onto these AI, these like disembodied AIs as well. And and kind of inform the, you know, the surrounding systems that are probably going to be needed to make sure that, you know, your, your like

• 59:23Nathan Labenz?

  more pure digital AIs stay on the rails, you know, so to speak, and don't go breaking digital dishes in their virtual kitchens. That, again, it's a, it's a fascinating piece just to contemplate the, uh, the fact that you've got fleets of robots going into unseen places and guided in part by, you know, a proto-robot constitution. So the next one then is called Learning to Learn Faster. And this is a project that is emphasizing teachability. And so again, this is kind of another variation on, you know, two papers ago, we said, hey, wouldn't it be great if you could just kind of draw a quick line and indicate to the robot how you want it to do, or, you know, wave your arm and kind of indicate how you want it to do.

• 1:00:07Nathan Labenz?

  And here, basically doing that now in a verbal way, where the robot can do something. And the goal here is to say, can we get the robot to listen to feedback and do it differently the second time, learning from what the human has said to it? You know, that might be do that slower or grab the thing lower or, you know, raise your one hand higher, whatever, of course, lots of examples. This is also done, I think, in a, in a way that is probably familiar to some, you know, app developers who listen to the podcast and maybe could be a source of inspiration for others. Basically, it's an iterative process of training on more and more chain of thought style reasoning.

• 1:00:48Nathan Labenz?

  And then also a, you might call it like a tree of thought style envisioning multiple paths, kind of simulating like, okay, if I do this, what might the feedback be now that I have this kind of training on my actions and the feedback that I'm getting? You can kind of sketch out a few different possibilities and then try to pick from those possibilities, which one is likely to be the best. And that kind of simulation called rollout in the paper showed performance improvement relative to one where you're just kind of picking a path straight away. So again, those are the things that jumped out at me. What else do you think we should make sure we understand about learning to learn faster?

• 1:01:28Speaker 01

  What I really liked about this effort was having a very tight feedback loop between users of the system, which are interacting with the the model that's learning to predict reward code that controls a robot like, and then you could directly see the results of, you know, what the what the model's response to the user was, you could see the response of like, the the model's predicted reward code when it's actually executed on the robot, you could actually see what happens. All of that was in a very tight loop, right? Where users were actually rating individual responses from the the language model, as well as, you know, rating the the entire interaction of like, okay, by the end of this conversation, where I'm kind of brainstorming with the robot, acting as a teacher, critiquing the

• 1:02:10Speaker 01

  code it wrote for the reward code, predicting the final kind of improvement that it made. Did the robot finally, you know, do the task that I was asking for originally at the start of the conversation? All of that data is kind of logged and incorporated into this notion of teachability. And I think this has been like, you know, the the tightest kind of feedback cycle that I've seen from our team yet, where, you know, you know, this is the the day cycle that we called it, where in the daytime, you have the frozen language model that's been trained on some teachability. It's actually interacting with users who are teaching it more. And in the night cycle at night, overnight, you know, we're actually distilling those kind of like in context, multi-turn conversations with dense user ratings back

• 1:02:52Speaker 01

  into that model overnight, when it's ready to deploy again for users in the daytime. This is like a very cool kind of very rapid kind of flywheel that I think takes a different flavor from the other kinds of data engines that we've been considering in the prior works, which are, you know, straight from expert teleoperation to low level actions. This kind of paper is more about the high level, the teaching. Interaction, the usability. And I thought that was a very, very relevant kind of evolution of our team. Now that we do see, you know, so many great other works in, in language modeling and like, you know, in, in, in chat G P T and Bard and Gemini. I think a lot of these concepts transfer quite well to domains beyond robotics.

• 1:03:33Keerthana Gopalakrishnan?

  It's very intuitive. So if you like, try to learn new, new sports, like, say, you go surfing or, uh, skiing. I, I feel like during the day, like when you start it, it's, it's really hard. But I found that like, once you go, once you like, if, if you go surfing for like two days or skiing for two days, like initially, it's like really hard. And then you go, you sleep overnight and then you come back and then you're immediately much better. And I like that in, in some way, the learning to learn faster paper has sort of mapped it into like a step. Ted said, the day cycles and the night cycles, where the day cycle is sort of like in context learning, where you collect more examples, but then it's in context. And then the night cycle is like where you go retrain or finding you change the weights of the model.

• 1:04:18Keerthana Gopalakrishnan?

  And yeah, I don't know, like, there aren't many, I think, highly scientifically credible explanations as to why you experience this overnight in, when you go like skiing and stuff. So for me, I think the learning to learn faster paper is like an intuitive way of trying to incorporate that into a robot learning. The second thing I really liked is the rolling out of the futures. And you do kind of take the shortest possible future and then you like retrain from that data until you also get more efficient in using the human feedback. But one challenge here is anticipating the quality of conversations, not just from the length, can be much more complex than that.

• 1:05:01Keerthana Gopalakrishnan?

  If you rate as to, I, I think more than the length, people also care about the aspects of, there are other aspects of interactions with humans that you care about. Like, did you, did you get the right code? Is it, is it responding correctly to feedback? Is it scalable, et cetera? And so. I think, yeah, this capturing or the rating of quality needs to be probably more multimodal than showless path search.

• 1:05:26Nathan Labenz?

  So Pivot is a demonstration that vision language models can be used to control robots out of the box. That's that's kind of how I interpreted it. Basically, what happens here is you take an image, you kind of scatter some numbers on top of the image and present that to a vision language model and then give it an objective and say, okay, given this objective, you know, your job is to pick up the Coke can or whatever, which number is the number out of all these numbers that you've overlaid on the image that the robot should move its hand to. And the vision language model, perhaps not too surprisingly at this point, you know, can do that. And then there's also an iterative process there because as I understand it, the initial numbers are kind

• 1:06:09Nathan Labenz?

  of pretty randomly just kind of scattered onto the image. But then, once we've picked one, now we can kind of scatter another set, but like clustered around that that first choice. And so with a couple of rounds of iteration, then you can really zero in on where the robot needs to go to accomplish the objective. And then, of course, the, you know, the robot can follow those instructions. I I'll confess, I didn't have a great sense for like, how this fits into the overall big picture. It seemed like, you know, I could kind of connect it back to like drawing the line trajectories. But I feel like I was maybe still missing, you know, the deeper connection on this one a little bit.

• 1:06:49Speaker 01

  Yeah, I think that's that's overall pretty good summary. I would I would say maybe one slight correction, too, is that the the numbers are also associated with actual arrows that are procedurally generated, which represent motion. But I think beyond that, you're absolutely right that we're leveraging the VLM as kind of a zero shot frozen critic, right, of, you know, directing which of these out of these directions that we already present, you know, in a manner that's easy to understand for the VLM, just by visual annotations, which of these annotations is the one that's better for the task at hand and the VLM can kind of choose and we kind of optimize over that. But I will, I also empathize with your, your, your kind of view that, you know, this is kind of Like in in in odd one out in the work we discussed today, which those are really about building,

• 1:07:32Speaker 01

  you know, fine tuning end to end models, training them for robotics tasks on a lot of data, scaling them up in the real world. And then out here in pivot, I I view this more as kind of like a very, just a result that was so exciting about a trend that we found that we didn't really, you know, know if a lot other other folks in the community were thinking about. And we kind of just wanted to share that, like, you know, this is actually a very surprising result, the fact that like these pre-trained internet scale V L M's can also understand some notions of motion, of causality, of of robot trajectories, of contact. Um, this was a bit surprising, right? Because and a lot of, you know, our our even our own team's perception and, you know, to be honest, most of the community right now in both like

• 1:08:17Speaker 01

  language modeling and robotics, it's like, oh, yeah, of course, like, there are so many things in robotics that we have to tackle ourselves, the the the very, you know, dense contact, spatial reasoning, left, right, up, down, all of these things are are concepts that aren't contained in Wikipedia and Reddit and, you know, uh, image generation data sets. These are things that we need hardcore robotics data for for. And perhaps recently, you know, you know, with, for example, with this work pivot, maybe the answer is that actually there is some very good amount of physical intelligence already contained in these like internet trained models by themselves without any robot data pre-training or or or fine tuning.

• 1:08:58Speaker 01

  Again, I don't, I don't, I I also don't think that like internet data alone, just watching, you know, no Reddit threads and Wikipedia is is enough to to solve contact rich robotics. But I do think that we're we've so far just been like seeing the tip of the iceberg for the knowledge that is already contained in these, you know, uh, large V L M's. And and with pivot, right? By showing that if you have this external kind of prior, this inductive bias of having this robust optimization problem and making it very easy for the V L M to just like, you know, pick and choose in a multiple choice setting, which of the directions, you know, that the robot should go is correct. And you kind of run multiple iterations and you you we we show.

• 1:09:38Speaker 01

  That you can kind of like better drag out the vibes or the implicit knowledge contained in these V L M's a little bit better. Again, I still don't think it's the results are like we just take pivot and you can apply it to any robot and it's just going to work off the shelf. I think it's definitely that the performance is very, very far away right now from let's say R T two or any kind of fine tune policy. But the fact that it even kind of shows very promising signs of life was unexpected and I and I hope is inspiring to others as well.

• 1:10:07Keerthana Gopalakrishnan?

  In addition to the zero shot, I think it's also that it can specify tasks in a multimodal fashion that I found very impressive. So there's code as policies type approaches where like where V L M's or L L M's off the shelf, they write code for robots through or figure out how to use the robot A P I's. But then you realize that a lot of tasks are underspecified in language and you kind of do need the image domain to like, let's say folding a T shirt. Like if you were to tell me in language how to fold a T shirt, that would be yeah, very hard to do. But then if you show in an image like where to move as you're folding the T shirt, I think this would work much better. So for like highly dexterous task. So I can imagine. So right now we are seeing the trend that zero shot.

• 1:10:52Keerthana Gopalakrishnan?

  While in the past like R T one, R T two was collecting a lot of data and doing a lot of training. Now we are seeing that these V L M's, these large pre-trained V L M's already know a lot about robotics, zero shot. And then the the effort becomes how to correctly leverage their knowledge, like how to move efficiently leverage their knowledge. So that we collect the least amount of data and we collect the data that's most complimentary to the things that they don't know. Yeah. So I think there are some interesting applications here for maybe dexterous robotics or maybe tasks that are more specified in a multimodal way.

• 1:11:24Nathan Labenz?

  So we've got R C two, which pulls generality out of vision language models. We've got R T X, which shows generalization across different robot embodiments. We've got our T trajectory, which is about doing these little sketches on the fly and learning from these like line drawings. We've got auto R T where we start to have scalable oversight and a robot constitution. We've got verbal feedback in learning to learn faster. And then the zero shot V L M guidance for for robots out of pivot. So this is obviously a ton of stuff. Again, from the first comment, it's all working. But where are we going from here? One could imagine, you know, an understanding of these results where it's like we've got.

• 1:12:07Nathan Labenz?

  Fifty more of these papers, you know, before we're going to be anywhere close to having a, you know, a system that we really think could be like used reliably to do useful work. Or I could imagine that it's like, hey, if we just jam these on scale a little while longer and also kind of integrate all of them into, you know, a single system that we might start to get pretty close. Where do you think we are right now in the journey from like research to, you know, eventually this is going to be like a scaling, an engineering scale up project, right? So where are we on that question?

• 1:12:43Keerthana Gopalakrishnan?

  It's hard to answer that and different people have various personal opinions about this. I, I still think that the data is a hard problem to track in robotics. So you could think that you can scale this up by spending a lot of money and scale up the data collection. But there's also a more efficient way to scale this up, which is to make your policies more sample efficient, more data efficient. So instead of spending two X of money, if you double the data efficiency, then you need just half of that effort. And I think, right, the state of the art or that a lot of people use is like the RT two. But we, we do understand that there are some gaps in its reasoning. And I still think that we probably need to make

• 1:13:28Keerthana Gopalakrishnan?

  a couple more breakthroughs before we can sort of make this a scalable engineering problem. And the second thing is, like Ted said, a lot of these are showing initial signs about how to maybe use VLMs and other things more zero shot, which is clearly pointing towards a trend. But we really need to now build learning methods on top of these highly performant zero shot methods. And also sort of iron out the nods in, you could argue that even if some methods have high zero shot performance, they may not have the best. Learned performance because they may not go to like one hundred percent. So there are still a lot of gaps in like how you might want to like the right interfaces

• 1:14:12Keerthana Gopalakrishnan?

  between robotics and foundation models are still needs to be worked upon, I think.

• 1:14:16Speaker 01

  Yeah, I think just I think you you said it perfectly. The one thing I'd probably add is I I hundred percent agree that we are a few breakthroughs away from general purpose robotics, you know, that it's the dream that we all are working so hard for. I think, again, if you want something commercially viable, something that will maybe make money or or help some people in the world, I think a lot of those ingredients are already ready to have a larger impact than maybe even just a few short months or years ago. Um, but for the the the true full vision of embodied, you know, AGI, I do think there is still fundamentally a few open research challenges left. I think a lot of the trends we see today with the papers we discussed are probably directionally pointing in roughly the areas that I think are particularly open ended still.

• 1:14:59Speaker 01

  But again, I don't think we have the full recipe just yet to say it's all solved, the research is done, we can just scale the thing up.

• 1:15:05Nathan Labenz?

  Could you give a sketch of kind of what you think the big things are that you will be tackling next? And I'm also kind of curious as to how you see Google DeepMind's role in this going forward? Like, this is all very research, it's still being published, you know, in a world where increasingly a lot of like frontier language model or multimodal model research is obviously much less published these days. Should I like expect to have a Google branded robot in my home at some point? Is that is that the sort of long term vision that you guys have? So, yeah, two parts are like, you know, what what are the big gaps that you're wanting to take on next? And what is that big long term vision specifically for Google?

• 1:15:49Keerthana Gopalakrishnan?

  I specifically, I don't want to speak on behalf of Google. And I think like a lot of the ideas for how to commercialize is still being discussed and and making this a business is also a very difficult difficult problem. It it works in narrow domains, but generalist robots, I think, are and generalist robots that are reliable and safe are still. Sometime away, there are many aspects to like bringing them to people's homes. Like the business aspects need to be figured out. I think, yeah, I don't want to speak to Google's glance for that. Personally, I think the research that I'm now interested in this year is RT one, RT two have very little zero shot performance.

• 1:16:30Keerthana Gopalakrishnan?

  You get most of the mileage from our data or motion is learnt from our data mostly. But we are seeing signs that large foundation models, zero shot can do a lot of things. But they have not been adapted to be learnable methods. So I think one hypothesis is, yeah, we can improve the interface of robotics with the foundation models. The the interface that RT two has is sort of like this code training, but actions are very different still, right? So you have these vision, language, action models, but actions are the weakest part of that link. And there are multiple hypotheses. One hypothesis, do we actually need actions?

• 1:17:13Keerthana Gopalakrishnan?

  Can we still capture a lot of the domain in the task space through vision and language by making actions more like code? There are some initial results on like a code as policies on prompt book, a paper, which which showed that zero shot prompting of robot of V L M's and multimodal code generation have some signs of very high performance. But then how do we make sure that this gets to one hundred percent? How do we make it learnable? How do we iron out the knots there? The second thing is like by manual robotics and more dexterous tasks. What we currently showed sort of transferring extras domains is still, I think, unsolved.

• 1:17:56Keerthana Gopalakrishnan?

  I think we need better representations to do dexterous tasks better. So what are they exactly and how to scale them?

• 1:18:04Speaker 01

  Yeah, I think, I mean, that's a very exciting research agenda for the team. I think for me personally, I think the thing I'm most excited about is also kind of going beyond language. I feel like language, we've seen a lot of great progress and a lot of great reasoning from the help of internet scale pre-training. But I'm very curious, you know, what is it that is unique about the embodiment problem? You know, I think on the manifold of different reasoning and skill, you know, concepts and capabilities that foundation models need to tackle, does robot actions and causality in physics, does it live in a little corner by itself? Or is it kind of ingrained and spread throughout all of the other types of reasoning that all of the data that humanity has generated captures? I'm curious about that.

• 1:18:45Speaker 01

  I would like to figure out how we can better increase the bandwidth that we do have between robot data foundation models and, you know, normal internet foundation models. I'm interested in studying how much robot data is needed. When we talk about, you know, scaling that system up, is it just like, you know, the cherry on top, a little bit of robotics data? Is it like, you know, 10, 100, 1,000, a million times more data? I think all of these very interesting scientific questions, again, about the nature of what it actually requires to be able to solve physical motion level reasoning at a very low level actions. I think these are very interesting questions to me and I think are really more about the scaling laws of robotic capabilities that I feel like right now, in today's day and

• 1:19:29Speaker 01

  age, we don't have a great understanding of. And I hope we have a better understanding of it throughout this year and next.

• 1:19:34Nathan Labenz?

  All righty. Well, Kirithanuk Gopalakrishnan and Ted Zhao, thank you for being part of the cognitive revolution.

• 1:19:42Speaker 01

  Thanks, Nathan, for having us.

• 1:19:44Nathan Labenz?

  It is both energizing and enlightening to hear why people listen and learn what they value about the show. So please, don't hesitate to reach out via email or you can DM me on the social media platform of your choice.