One Brain, Any Body: Google DeepMind's Keerthana on Gemini Robotics 2, Cross-Embodiment & Humanoids

Keerthana Gopalakrishnan, Research Lead for Gemini Robotics at Google DeepMind, returns to discuss the release of Gemini Robotics 2, whole-body intelligence, and the pursuit of generalist humanoid robots. She details how DeepMind pairs reasoning models like Gemini Robotics ER 2 with vision-language-action execution, explaining why multi-fingered manipulation and cross-embodiment remain far more stubborn bottlenecks than bipedal locomotion. Together, they examine the real-world stakes of bringing physical AI into human environments, addressing crucial trade-offs across inference latency, sensor failures, and operational safety where errors carry immediate physical consequences. For full show notes, links, and references, read the episode page:https://www.cognitiverevolution.ai/one-brain-any-body-google-deepmind-s-keerthana-on-gemini-robotics-2-cross-embodiment-humanoids/ Sponsors: Athena: Athena matches you with a dedicated, top 1% executive assistant to handle your inbox, calendar, and daily workflows so you can save an average of 15 hours a week. Get matched with your EA today at https://athena.com/cognitive Parallel: Parallel provides enterprise-grade web search APIs for AI agents, offering the optimal balance of quality, speed, and cost. Get started for free at https://parallel.ai/tcr Deepgram Flux TTS: Deepgram Flux TTS is a streaming text-to-speech model built for voice agents with natural tone, context awareness, and interruption handling. Try it free until September 12 at https://deepgram.com/keep-talking OutSystems: OutSystems is the leading agentic systems platform, helping enterprises build, modernize, and operate mission-critical applications at the speed of AI. Learn more and start owning your agentic future at https://outsystems.com/tcr Claude: Claude is the AI collaborator for problem solvers, helping with writing, coding, financial models, strategy, and more. Get started with Claude and explore Claude Pro at https://claude.ai/tcr CHAPTERS: (00:00) About the Episode (04:22) Humanoid Olympics and simulation (11:11) Generalization versus task mastery (21:22) Gemini Robotics 2 architecture (Part 1) (21:28) Sponsors: Athena | Parallel (24:19) Gemini Robotics 2 architecture (Part 2) (Part 1) (36:02) Sponsors: Deepgram Flux TTS | OutSystems | Claude (40:00) Gemini Robotics 2 architecture (Part 2) (Part 2) (48:28) Whole body humanoid control (57:03) Progress in robotic hands (01:03:26) Safety and robot interaction (01:12:11) Future of robotics data (01:18:47) Interacting with robot fleets (01:26:56) Episode Outro (01:30:05) Outro PRODUCED BY: https://aipodcast.ing

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Nathan Labenz?

  Hello and welcome back to the Cognitive Revolution. Today, I'm excited to welcome Keerthana Gopalakrishnan, Staff Research Scientist at Google DeepMind and Research Lead for Gemini Robotics, back for her fourth annual appearance on the show. One of the biggest questions in AI today is how soon will general-purpose robots become broadly useful? AI is already affecting the world in major ways, even in its purely digital form, but the most sci-fi forecasts for the AI future predict that robotics will soon hit key tipping points. AI 2027, for example, predicts that humanoid robots will become useful sometime in mid

• 0:40Nathan Labenz?

  2027, and that by 2028, a whole robot economy could take shape, where robots build more robot factories, which in turn produce more robots, leading to unprecedented exponential economic growth, but also creating serious risk of AI takeover. So, how does that vision line up with the reality of robotics research today? We begin with a discussion of the extremely viral Robot Olympics held in China this summer, as I was really curious to get Keerthana's take on what it means that humanoid robots can now run faster than the fastest humans. Her point of view, as you'll hear, inverts the usual US-China dichotomy in AI. She says that while the videos are obviously impressive, skills like running on a flat

• 1:25Nathan Labenz?

  track are actually relatively easy to train in simulation. And more to the point, foot speed is not really a limiting factor in the utility that robots can provide. As such, her team at Google is more focused on practical value. From there, we get into Gemini Robotics 2, a suite of three models that Google released this summer. Gemini Robotics ER 2, for embodied reasoning, which Keerthana describes as a system 2 for robotics control, is based on Gemini Flash and is available via the API, allowing developers to define the affordances available to the models as tools, just like we do with digital agents. I played around with it and found it remarkably accessible.

• 2:10Nathan Labenz?

  The other two models, Gemini Robotics 2 and Gemini Robotics On Device 2, translate these higher-level tool calls down to the level of robot action, and are now capable of controlling the whole robot from fingertips to feet on a wide range of form factors, but are currently available only to trusted testers. Having understood how these models work, we again zoom out, and I ask Keerthana how she understands progress in robotics overall. On the one hand, like so many other AI researchers recently, she says that she has been surprised by the pace of progress. But at the same time, she still feels that robotics remains in its GPT-2 era. We have seen a number of recent demos of robots learning new tasks from just a few

• 2:55Nathan Labenz?

  or even just a single human demonstration. But Keerthana argues that the range of tasks you can teach this way, and the generalization profile of robotics models across different robot bodies, still isn't strong enough to deliver the versatility and reliability that real-world use cases demand. Toward the end, we move on to talk about recent progress in hardware, where robots are likely to be deployed at scale first, whether capabilities or safety, alignment, and robustness will ultimately be the limiting factor for the consumer market, and we consider the future of the field, including whether we can continue to run LLM-derived robotics models in a loop, as today's systems mostly do, or will need some sort of predictive

• 3:38Nathan Labenz?

  world modeling, as we humans use, to create systems that can anticipate and smoothly interact with the world. On that question, and on the timeline for key utility milestones, Keerthana isn't one to speculate too wildly. For her, these are all empirical questions to be resolved by research and experimentation. But what is clear to me coming out of this conversation is that robotics will either need to hit key tipping points soon, or some of the most aggressive timelines for AI transformation will have to be pushed back. With that, I hope you enjoy this very well-grounded update on the state of robotics with Keerthana Gopalakrishnan from Google DeepMind.

• 4:21Nathan Labenz?

  Keerthana Gopalakrishnan, Staff Research Scientist at DeepMind and Research Lead for Gemini Robotics. Welcome back to the Cognitive Revolution.

• 4:30Keerthana Gopalakrishnan?

  Hey, nice to be here.

• 4:33Nathan Labenz?

  Great to see you again. It's been about a year. Obviously, a lot has happened. I wanted to start off with something fun. There was recently a Robotics Olympics in China, and clips were obviously flying all over the internet, and people were having a laugh and at times, I think, a little empathy for robots, which is something we've talked about in the past. I'd love to just start by getting your observations and takeaways, given the depth of knowledge that you have on the subject. What stood out to you in watching clips of the Robot Olympics?

• 5:06Keerthana Gopalakrishnan?

  I hadn't until my dad was talking about it, and he was like, wow, did you see those humanoids running? And then there were all these videos about the humanoids running and catching fire. And then I also saw my friends in India who have now entered politics, and they are doing like various student protests there. And they were talking about, wow, look at that, in China, they're building the robots that can run faster than the fastest humans. So I think, in a way, that competition, even though from a researchal perspective, it's not very surprising, but I think it has really captured the public imagination in a very emphatic way. And also, I think it's like a benchmark, right? Running as a benchmark, and trying to see where humans are at, and then where robots

• 5:51Keerthana Gopalakrishnan?

  are at. And I think if you look at the videos from, people had made these collages where the DARPA challenge back in the, I think it was now 20 years ago, I think, those robots would just walk, open a door, and then like really fall down. And now, they are, here we are. So it has been amazing progress, and it's also progress that's very much in the public's imagination.

• 6:14Nathan Labenz?

  Yeah, the fact that a robot can run faster than Usain Bolt, on the one hand, it's like crazy progress, it's crazy impressive. But one question I did have for you about it, is it valuable? Are these Robot Olympics, is it just a, you know, is its main purpose to capture the public imagination and just have something cool to show off? Do we want robots that run 20 miles an hour in humanoid form? In your work, do you think, oh, I wish my robot could run faster, I need to work on that, or is that just a sideshow?

• 6:44Keerthana Gopalakrishnan?

  So I think, one is that there are people working on research, and then there are people trying to push how fast that research can go. And for better or for worse, locomotion is the research, there's locomotion and manipulation, and locomotion has advanced quite a bit, and especially, it is the type of field where simulation can make a lot of difference. And a lot of these models are like trained in simulation, and because of that, locomotion has advanced quite far. So it's like a display of where the state of the art of humanoid locomotion, like bipedal locomotion, is. I think, so there are many ways of looking at it. I'm pretty sure, maybe for some defense applications, it's useful, but I think we

• 7:26Keerthana Gopalakrishnan?

  are very much focused on doing, robots help people and do useful things in the physical world. And I think I'm a very productive human, a lot of my friends are also very productively employed, but we don't run faster than Usain Bolt. I think it's useful, but the frontier is so vast, right? Different people are trying to push in very different directions.

• 7:44Nathan Labenz?

  Tell me more about, it's like running the wall at their sprint and smash into pieces. Clearly, they are not training them like that on an ongoing basis, because they'd run out of physical robots to do the training on awfully quickly. So from that alone, I was like, okay, clearly, they must be doing a lot of training in simulation. Tell me more about why these kinds of tasks are amenable to training in simulation, and what kinds of tasks are still difficult to train in simulation?

• 8:15Keerthana Gopalakrishnan?

  Yeah, so, sim to real is the thing to notice, and sim to real, in robotics context, means the gap between, if I train something in simulation, can I replicate that in real? And as we know, especially with agents and stuff, the things that we can learn in simulation are probably the first ones to get solved, in that sense. So in locomotion, if you look at it, you can often model the ground or the world as like a, you can model it, and there is contact with that, and often, these are like very flat surfaces. And so that makes this task a very good task for in simulation, and then you can do society of the entire robot.

• 8:58Keerthana Gopalakrishnan?

  But for manipulation, though, you really, what you care about is contact, right? And when contact happens, and how objects behave, that's where current simulations start to break a little bit. It's harder to do cloth folding, because the cloth have friction, but even in simulation, a lot of pick and place can still be like very solved in simulation, because there also the contact dynamics are often somewhat predictable.

• 9:22Nathan Labenz?

  So basically, the more rigid the body is, whether it's the floor, or like, here's a wooden cube, the less deformation those things are going to have, the easier it's going to be to do simulation?

• 9:34Keerthana Gopalakrishnan?

  Yeah, contacts are basically, the physics is harder to model in simulation, so where the physics is easier to model, simulation gets it.

• 9:44Nathan Labenz?

  In terms of the AI side of these robots, how, what would you infer from what we saw? Obviously, I don't know about obviously, but it sure seems like there's not reasoning components to what's going on. I don't have a clear sense of what architectures were and were not allowed. And were all the models that were used to control the running robots? Were they on the device? Were maybe some of them off the device? How would you imagine that they were typically doing that? And how high up the stack do you think they were working? Clearly, they're doing a lot of stuff with stabilization, making sure it's not keeling over at any given point in time. But is that it?

• 10:25Nathan Labenz?

  How would you kind of, because we're going to get into Gemini Robotics coming up, and there's a much higher level part of that architecture, too, reasoning and understanding scenes. I'm assuming just all of that is kind of skipped for the purposes of these Need for Speed demos.

• 10:41Keerthana Gopalakrishnan?

  I didn't work on these demos, so I'm only speculating. But there is very widely published body of work around how to do locomotion and simulation. And I would imagine that it's like very good RL controllers that are probably trained from like motion imitation and then fine tuned with like good RL that suits the robot's body and for that specific task where it's targeted towards running really fast. And then maybe they have some navigation, although I would think that just running a race is quite easy.

• 11:12Nathan Labenz?

  So you mentioned the goal of having robots do useful things for people in the world. And obviously, foot speed is not typically the limiting factor on one's ability to do that.

• 11:24Nathan Labenz?

  In today's world, where you live and work around robots all the time, is there anything now that you actually delegate to robots on a practical level where you're just like, they are actually good enough at this, that it is abstracting away from my research interest and all these other things, but just straight up convenience? Robots win on these tasks? Are there any such tasks?

• 11:50Keerthana Gopalakrishnan?

  Yeah, the weird thing is that I'm actually a researcher, right? So if the robots win in some task, then I'm going to work on the things that robots cannot. The frontier is moving. And I remember back in the day when I started working on manipulation, even pick and place was very hard to do. And at the time, we didn't have the right algorithms. But now with the foundation models, this sort of stuff is quite easy to do. And therefore, like the research frontier has moved into like, humanoids are a newer thing. If you look at it back in the day, our demos were just like group of robots. And now it's with humanoids and hands. So the frontier is constantly moving because the things that used to be easy, the things that seem easy kind of get solved,

• 12:35Keerthana Gopalakrishnan?

  and then the frontier moves into harder stuff. So if you look at the robotics two release, right? Like it's whole body control, like whole body manipulation. This was not something that, that anyone had shown before, and especially whole body manipulation with generalization. So it's not just running really fast, but being able to take steps, being able to squat, and manipulating objects while doing that in a closed loop sort of way. And then also generalizing and reasoning about different types of objects. I would say like, maybe a year ago, if you had told me we would do this by now, I would be a bit surprised. So you can see over time, I do feel a bit proud about it because we started working on humanoids like two years ago.

• 13:20Keerthana Gopalakrishnan?

  And at the time, it was like just beginning, right? And they are like trying to just grab an apple, and even that was really hard. And now you can prompt it. And I put out this video on Twitter where like, I'm like playing with it with a watering can, and it's squatting in different locations and trying to pick it up. So yeah, things have quite a bit.

• 13:41Nathan Labenz?

  Maybe another way to frame the question is in terms of the meter curve. I'm sure you're familiar with the famous meter curve of task length. And the, it's typically plotted as over time, the models can do increasing task length as measured by how long it takes a human to do the task. But then there's also this factor that's like, the AI in that case can do it at 50% success, or they have an 80% success version. Obviously, one big difference between LLM usage and robotics deployments is 50% success rate in the real world is probably not going to cut it unless you have a really failure tolerant task. Whereas on my computer, I can be like, eh, you kind of missed some stuff, whatever.

• 14:26Nathan Labenz?

  But the robot drops something, it breaks, we have a problem. So maybe if you tried to describe where we recently were and where we are now on a meter like curve of how hard are the tasks that robots can do well enough that they could actually be deployed? Or what are the things that they can maybe do 99% plus of the time? Such that, hey, for these tasks, they can actually do the job. And then what would be where they're at like 80%? What would be where they're at like 50% today?

• 14:57Keerthana Gopalakrishnan?

  I think a lot of pick and place is close to getting very good signal and close to deployment. I do think though that, so there is like a balance that we need to strike in robotics and generally AI in general, right? You could build very narrow AI, just very good at one thing. But then you lose, then you're not working on generalization. So you can see generalization and mastery as somewhat orthogonal axes. And working on generalization is like lifting the boat, lifting the wave for all the boats. So if you imagine like the tasks are like sitting along this manifold, using very general common sense, very general understanding,

• 15:37Keerthana Gopalakrishnan?

  is making it much easier to get to mastery on kind of all the tasks. And, and this is also like very much part of our pieces in thinking about how to approach this. And you can build like, even today, you can build very narrow AI that's just like very good at one or two things. But then the cost of doing the Nth thing becomes exactly the same as the cost of doing the first or the second thing. But building a very general baseline then makes it much easier to quickly take it, to tune it towards mastery. I think we have seen this even in LLMs, right? Like initially, people had specialized models that do one thing. But now you know that the GPT and Gemini are like very good at a broad range of tasks.

• 16:22Keerthana Gopalakrishnan?

  And then from those general baselines, then you can climb really high. So I think the same thesis might hold true for robotics as well.

• 16:30Nathan Labenz?

  Yeah, that's interesting. So last year, when I saw your talk at the curve, the, one of the interesting points, as I recall, was taking the base model that you had at the time and then doing some task specific fine tuning and showing gains from task specific fine tuning. Recently, there's been, I don't know, at least a couple, probably several demos that I certainly haven't been able to interrogate myself in a hands on way, but you see videos online of different robotics companies saying, hey, look, we've got few shot generalization, or we've even got one shot generalization. Here's a person doing the task one time, demoing it to the robot, then they can go do it.

• 17:15Nathan Labenz?

  Obviously, that's great if you can get there. But what do you think is the actual likely path that people will follow to deploy robots? Which, are they going to do 100 demos of a task? Are they going to have to do 10? Could it just, do you think we are on the verge of getting to a lot of things working on just a one shot basis? What sort of investment should people be prepared to make to get their humanoid dialed in on whatever they most care about?

• 17:44Keerthana Gopalakrishnan?

  Yeah, I think it's going to be a spectrum, definitely. Being able to ICL or in context show a robot how to do a task, and then it doing it definitely reduces the time to deployment and the time to pick up the task, right? And you don't need any specific fine tuning. You don't basically need to train it, you can do it in context. And so that definitely is a very exciting development. Although you can think of it as video prompting, right? Even the older models, like when I say, let's say, pick up the object, right? And it is a very unseen object. What am I doing? I'm sending in language, and then I'm getting a very general behavior out of it that I did not show it before. So that's generalization.

• 18:27Keerthana Gopalakrishnan?

  Now, instead of prompting with text, you can prompt it with an image. You can say, here's the thing, and then draw a circle around the thing that you want manipulated. And that's prompting with image. And this is prompting with video in some sense. And so I would imagine, I think the ICL results currently are on the spectrum about of how do you prompt a foundation model? And you can prompt it with video, you can prompt it with language, you can prompt it with image. And then the question is, so now what is the generalization that you can get? Now, these models are also very subject to kind of memorization in some sense. So if you really show it exactly, this is what to do, they will copy it. But the question is, can they generalize, right?

• 19:10Keerthana Gopalakrishnan?

  Can you now, if you change the scene and stuff, can they do, can they now adapt? And then the second question is, so for the same task? And the second question is, what is the level of difficulty of the task that you're prompting for? There's a lot of pick and place, like the models have a lot of data, and it's also like easier to comprehend. But can you show a robot to tie a trash bag? And then can the robot tie a trash bag after you show it? So I think it's still fairly early, and we need to see how that evolves.

• 19:39Nathan Labenz?

  If you had to score, this is a silly question, but it might be useful for just calibrating. If you had to score robotics today on one to six GPTs, are we, like in earlier conversations, I think we were, eh, we're maybe hitting like a GPT two kind of moment. I think of GPT three as being really synonymous with in context learning. But we're following a somewhat different path with robotics, obviously, where we have like instruction following built in, maybe in many cases even before like good in context learning happened. So. That just exposes the flaw in my question. But if you have to analogize to how many GPTs we are along the way in robotics,

• 20:23Nathan Labenz?

  where would you score the field today?

• 20:25Keerthana Gopalakrishnan?

  I think still GPT two. And here's why. I think for GPT three, we need firstly, few-shot learning to work really well for a lot of different tasks. And secondly, also robotics weirdly is still very subject to cross-embodiment, right? If it just works on your robot with your specific setup, is it really a generic brain? Now, can I put that brain on my humanoid or my another robot or some other robot that I just bring in? It should be. And if it's completely helpless in that setting, is that a generic brain? I think GPT never had this problem, right? Like we, my phone or your phone, my computer, Mac, Linux, it doesn't matter where you run it. It kind of behaves, you can expect this very similar behavior.

• 21:08Keerthana Gopalakrishnan?

  But here I think we are very subject to which robots that you act on. And so I think there is a lot of work still needed to be done to make very generic brains that are, that can count, discount those factors out.

• 21:23Nathan Labenz?

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 21:28Nathan Labenz?

  Today's episode is brought to you by Athena, the executive assistant company on a mission to improve how people work and live. If you want to increase your impact, you have to free up your time. And that's what Athena does best. They match you with a dedicated, full-time, top 1% executive assistant, who can take over your inbox, calendar, travel, and everything else that's quietly eating up your week. Athena is SOC 2, Type 2 certified, so you can rest easy knowing that your sensitive data is in good hands. And as a former AI advisor to the company, I can personally vouch for how much they've invested in AI tools and training.

• 22:13Nathan Labenz?

  In fact, one of the very best AI users I've ever met is an Athena client who delegated the exploration of AI tools and the development of AI workflows to his EA. Athena clients report saving an average of 15 hours a week, and the average client refers more than two friends a year. That, to me, checks out. I was an Athena client while running my startup, and to this day, I continue to refer friends. Go to athena.com/cognitive right now and get matched with your EA. That's athena.com/cognitive. Give yourself back a few hours this week.

• 22:53Nathan Labenz?

  Go to athena.com/cognitive and see who they'd pair you with this month. Today's episode is sponsored by Parallel, where agents find answers. Most engineers today closely follow new model releases, but don't pay nearly as much attention to their agent's most important tool, web search. If you're like me, your agents are often conducting hundreds of searches per day. And while the unit cost is small, over time, this does start to add up. Before starting to use Parallel, I calculated that my agent's search bill would total roughly $500 this year. The good news is that I just had my agents conduct a systematic test, and I found that Parallel's fast mode, which costs just $1 per 1,000 queries,

• 23:38Nathan Labenz?

  worked just as well as my previous default provider. And switching to it will save me roughly 80% of my search bill going forward. Parallel's infrastructure is enterprise-grade, and their suite of APIs offers a range of Pareto-optimal options that allow you to choose the right balance of quality, cost, and speed for your needs. So whether you're building voice agents that need 200 millisecond latency or long-horizon agents that need deep research, adding Parallel to your agent's toolkit is a no-brainer. Get started for free at parallel.ai/tcr. That's parallel.ai/tcr.

• 24:19Nathan Labenz?

  Well, maybe that's a perfect transition to Gemini Robotics 2. How about, for starters, just give us an overview of the architecture. There's like, when you go to the webpage, there's three models released. There's Gemini Robotics ER 2, there's Gemini Robotics 2, and then there's the on-edge, on-device model.

• 24:40Nathan Labenz?

  Describe how those relate to one another. Just give me kind of a broad label and then I'll dig in with some more specific questions.

• 24:46Keerthana Gopalakrishnan?

  Yeah, so Gemini Robotics ER, you can think of it as like a system 2 brain that can do like very generic reasoning. It's based on the Flash line of models, but maybe more tuned towards robotics. It can do a lot of robotics tasks, and it's also widely available that you can query for like general image and video understanding and semantic understanding. Now, you can think of Gemini Robotics as the VLA that is like controlling the actions model, right? How the robot moves and what it should do, given a certain intention from either a user or another system 2 model. And you can think of the Gemini Robotics on-device model as a much smaller version of that

• 25:30Keerthana Gopalakrishnan?

  that kind of fits on the robot's computer that doesn't live in the cloud and do. It is comparable. It's essentially doing what Gemini Robotics can do, but is a smaller model that fits on device.

• 25:44Nathan Labenz?

  So one question I have around just how this is made. So it's based on Gemini 3.5 Flash. And with that comes a lot, right? Like you have chain of thought reasoning, you have the multimodal inputs, all that kind of stuff. Is this the long-term architecture that you expect robotics to work around? One obvious challenge is just language models, especially with their reasoning traces. They take a while to respond, right? So the, wait, I did a little, it was actually pretty fun. So this model's in the API, spoiler, that's another topic to touch on.

• 26:26Nathan Labenz?

  But because it's in the API, I was able to have a coding agent go in and use it and expose some tools to it. And I created little, just 2D demos, actually, where I like feed in a little image of a scene, but it's actually like a little canvas in the browser. It's amazing how far I was able to get just with a few prompts in terms of helping me understand what the model can do, how easy it is to use. But I was also getting timing traces back from that. And so some of the times it would be like, from prompt of move the banana to the plate to the tool call is six seconds or something. That seems like a long time for the real world. So that makes me wonder if this is a base of convenience that you're building on now

• 27:09Nathan Labenz?

  or if this is like the paradigm that you ultimately think we will see, perhaps just sped up or otherwise somehow modified, that we'll actually have the robots working in a responsive way in our environments?

• 27:23Keerthana Gopalakrishnan?

  Yeah, I think it all depends on the application specifically, right? Like certain applications, like if you're on an assembly line, you can't loiter, and now you need to do things really fast. But let's say if I'm at home or at some point and I'm like, hey, robot, can you go fold my laundry? I don't really care about how fast it is folding laundry. I care about how well it is doing the task I can wait. And then maybe there are also, scaling is a very important factor in LLMs and in robotics. And I would imagine that a lot of the very frontier capabilities are going to appear in the very large models. And those models are going to be naturally slower than the smaller models. So I think there is definitely a trade-off and a spectrum to be made.

• 28:06Keerthana Gopalakrishnan?

  And I think I would imagine that the Flash and the Pro paradigms would also exist in robotics. And even in robotics, maybe you could even see like, maybe like an on-device paradigm. Like even the Flash models, they still run on cloud, right? And they, and you need very good network connectivity to talk to them. And robotics deployment could be like very distributed, right? You might want to run it on some remote robot in like Michigan and stuff. So I think the paradigms will evolve and, but I imagine all scales to be relevant and useful for different types of applications.

• 28:39Nathan Labenz?

  It's interesting that you contrast, because typically I think robots will go first to warehouses, factories, commercial spaces where there's a lot of control and there's a lot of consistency to the tasks that they will need to do. And there's, of course, professionals who can maintain large numbers of them. All of those things historically have made me think, these are gonna, they're not going to come to my home before they're like pretty well deployed in these commercial settings. But what you said just there was a little bit pushing me the other way, where it's like, they may not be fast enough for the factory, but they could definitely be fast enough to do all your chores overnight. And that could be an inversion.

• 29:21Nathan Labenz?

  Do you still, I think last time we talked about this, you had said you would expect commercial settings to be the first places. Is that still the thinking? And how does that relate to the responsiveness question?

• 29:32Keerthana Gopalakrishnan?

  I definitely think that commercial use cases are probably more controllable therefore to try out different new intelligence compared to like people's homes, which are more complex and where you cannot exert control. And even there is higher bar that you need to meet for safety, given toddlers and stuff. So that still, I still believe that. However, the speed question, I, it's, I don't think it's going to be like we make the slower models first and then make them faster. I think we are going to simultaneously make the fast and the slow models and then distill the end one. And although it is very potentially possible that some capabilities are going to just emerge in the larger models, and then we'll need to figure out how to bring it to the models in the smaller scale.

• 30:18Nathan Labenz?

  So just the difference between the Gemini Robotics ER2 embodied reasoning model and the base model, the Gemini 3.5 Flash that it's trained on. One of the things that my coding agent did without even me asking, actually, was just compared the two and gave both of them, like, here's a scene, here's the tools that you have. The tools were like fairly high level in the little test that I did of grab at and it would grab the item and then place at. And it even did one where the, inspired by famous videos, like, the thing was moved right before this. It tried to grab and it failed. And then it got another image, but the thing's in a different place. But this was all like relatively simple stuff.

• 30:59Nathan Labenz?

  And on the tests that I ran, Flash and ER2 basically performed the same. They actually just both did fine. What would, how would you describe the frontier of embodied reasoning tasks specifically that ER2 can do that Flash can't do?

• 31:19Keerthana Gopalakrishnan?

  Yeah, so I think a lot of the benchmarks that we are tracking are like instrument reading. It's like the way that we think about is like ER is working much more closely with, for especially for robotics use cases. And, and that is like inspection, instrument reading, a lot of pointing. But it is a very like, it's a collaboration, right? It's not like ER and Gemini are at competition. There's a lot of data upstreaming, so it is a very collaborative effort to how to make the Gemini models really good at robotics generally. And if it's not, and but the way to appear, the way for the capabilities to progress would be like, even if the Gemini models are more generally intelligent, you can still create specialized models that were, let's say, if a specific robotics use case

• 32:04Keerthana Gopalakrishnan?

  needs like more specific engineering or fine tuning, then you have like specialized models that are slightly better in that use case. So I think it's a question of, it's a question of what you want and what you want to change.

• 32:16Nathan Labenz?

  The model has 128,000 context window. How much is that in the context of robotics? I can, I've done enough that I have an intuition for how many pages of text that is and whatever. But I don't really have a great sense for how the history of an episode gets encoded and how many images per second. How should we think about that in terms of time? Like, what's, what does 128K translate to in terms of the sort of maximum length of an episode for an ER2 powered robot?

• 32:55Keerthana Gopalakrishnan?

  It's about, I think about three minutes of memory. And it depends on how you tokenize and like how many like tokens you represent and what is the other information that you've given. However, I think regarding the memory, you would see that like the pro models are much larger context length and they have, they are also like, they are getting a lot better at like fitting much larger things. I would think ER right now, you can't fit like a very long episode of a robot doing a lot of things and try to get an output out of it. But as, working on context length is like a continual problem. So I imagine that especially for these like offline analysis use cases, the pro models are going to be more useful there.

• 33:40Nathan Labenz?

  So three minutes would be like, if you're basically packing context very densely, if you were to offer any advice to API users who are like, okay, I want to aim for higher than that. I can obviously down sample video and have fewer frames and do all kinds of things, right? I've got, I'm context engineering for robotics. Maybe the question I really want to ask is, what have you learned about context engineering for robotics that people should know? How should they think about like what parts of the history of a rollout they need to keep in order to have coherence? And maybe it's similar to LLMs, but I imagine there's probably some things that are distinct about it.

• 34:22Keerthana Gopalakrishnan?

  Yeah, so I think very dense memory. So there is like, I don't know, I think Daniel Kahneman or someone, the fast and slow. And but maybe you can think of it even for like memory aspects, right? Especially for like something that I did in the long past. Is it worthwhile to keep it in a highly dense modality like image? Or can I also do a lot of text summarization of like what I'm doing as I'm doing it? So let's say if I'm cooking, right? It's not sure in my, in my brain, like if, if I want, I can go back to the time when I was, I don't know, flipping my egg or something. But also a lot of the time, I, the information or the narrative of how I'm doing things

• 35:02Keerthana Gopalakrishnan?

  and, and summarization even in text is fairly useful or fairly informative about like what I, I did. So I think context is, it depends on what you want to maintain is the relevancy of the information for the task. And obviously, if you can remember everything, then that's great. But as practically, you need to be more efficient with how you manage your like RAMs, right? And you kind of want to have the most important information represented. And sometimes that can take the form of like also representing information in like text, which is a little bit more compressible. And this is also now related to the like, the hardness between like ER and the VLA.

• 35:43Keerthana Gopalakrishnan?

  Like the ER tries to parse the intentions from a human and via text and other modalities kind of talks to the VLA. And so there, like the bandwidth there is much more interpretable modalities.

• 35:57Nathan Labenz?

  Hey, we'll continue our interview in a moment after a word from our sponsors.

• 36:01Speaker 03

  I've got a text.

• 36:03Speaker 04

  The summer might be almost over, but a new bombshell has entered the villa. Meet the voices of Deepgram Flux TTS. I'm Drew, low key casual.

• 36:12Speaker 05

  I'm Alexis, an upbeat morning person. I'm Haley, always cheerful.

• 36:17Speaker 04

  Flux TTS has some real personalities that are ready to speak. You can interrupt, pause and keep talking without losing the plot.

• 36:24Speaker 03

  Fancy a chat? Come try all the voices free through September 12th at deepgram.com/keeptalking. Terms apply.

• 36:32Nathan Labenz?

  This episode of The Cognitive Revolution is brought to you by Outsystems, the leading agentic systems platform. Outsystems is helping their customers build, modernize and operate enterprise systems, starting from any coding tool in a governed agentic engineering model. It's time to innovate at the speed of AI without compromising quality or control, which is why thousands of enterprises worldwide trust Outsystems for their mission critical apps. And we're not talking about theory. We're talking about massive operational wins. Take for example, a leading global retailer that was struggling to keep up with their rapid expansion. They used Outsystems to build AI agents that automated their invoicing and translation processes,

• 37:16Nathan Labenz?

  shifting 95% of their daily store operations to a new digital app, saving thousands of hours of manual work. Or look at a major global logistics giant that was drowning in technical debt. Their engineers were spending their weeks manually firefighting by reviewing platform logs. Using Outsystems, they built an AI log review agent in just five days. And that one agent now saves them over 1,600 engineering hours every single year, letting their team focus on real innovation instead of reactive maintenance. Even in modern digital banking, companies are using this platform to standardize and automate complex financial workflows,

• 37:58Nathan Labenz?

  delivering secure, personalized experiences at a pace their competitors can't match. Outsystems gives you the flexibility to deploy how you want and the governance to do it safely. Stop chasing the hype and start owning your agentic future. Learn more at outsystems.com/tcr. That's outsystems.com/tcr. Today's episode is brought to you by Anthropic. By now, you know my story. Claude drafts my intro essays and I rewrite them. Not because the drafts are bad, but so I can stand behind everything I publish. Well, I have an important update.

• 38:40Nathan Labenz?

  Claude Fable 5 is the first model to have me rethinking my rule. Today, I now think co-authorship, not sole ownership, should often be the goal. Where the model excels, rewriting its work can be more about vanity or a misplaced sense of duty than integrity. I feel it most in songwriting. I'm no lyricist, but I'm good with a song concept and Fable writes some amazing verses. I give it feedback on its misses and I push it to aim for higher inspiration, add layers of meaning, optimize syllable density, and above all, write a hit song. These days, I get compliments on just about every song we write together.

• 39:23Nathan Labenz?

  Claude is the AI for problem solvers. It's the collaborator that understands your entire workflow and thinks with you, not for you. Whether you're debugging code at midnight, building a financial model, or strategizing your next business move, Claude extends your thinking to tackle the problems that matter. For problems worth solving, get started with Claude at claude.ai/tcr. That's claude.ai/tcr. And check out Claude Pro, which includes access to all of the features mentioned in today's episode. Once more, that's claude.ai/tcr.

• 40:00Nathan Labenz?

  Just before getting deeper on the VLA, why did you and DeepMind as a whole decide that this was the first robotics model to put out via the API? And what have people built with it?

• 40:14Keerthana Gopalakrishnan?

  Yeah, I think it's definitely ER is much closer to Gemini itself and in terms of the capability, so it is more mature. Actions, I think, is like farther out into the frontier and so it is still like very much, there's a lot of research to be done about how to make them very useful, how to serve them more widely and so on. And I think it has been quite surprising to see how people use it. I've seen it in so many, it has such wide appeal and the team has done a really good job there. So I think maybe one thing we learned was like, there was a lot of testing that we were doing with trusted tester partners and stuff, but it was once we put it more widely available on AI Studio and stuff, then the usage is on a very different scale than whatever

• 40:59Keerthana Gopalakrishnan?

  the information that we were finding from deeply working with trusted tester partners. That, that was surprising, like how much like things blow up once you put on a more widely platform. But second is, I think obviously there are like robotics companies using it, but then also I think for me as a researcher, I was quite surprised by how academics, people in academia use it. And I think some of the things that I've seen is on like definitely benchmarks where it was either using VLA or even trying to do even low level control now, there is a lot of conversation going on around like how agents will impact robotics. So I think how ER has been used in that context has been quite, it was quite nice to see.

• 41:40Keerthana Gopalakrishnan?

  And maybe thirdly, now that ER is more widely available, it also makes it very easy to be benchmarked. So it's a kind of like free information for us about what the model is doing well, what it's not doing well, what could be like the next frontiers that we may need to push into.

• 41:54Nathan Labenz?

  Are there any consumer products or any just users you would particularly want to spotlight that are using ER2?

• 42:03Keerthana Gopalakrishnan?

  So there are a lot of demos out there. I think I saw the Boston Dynamic Spot Robot going around reading instruments and like handing out snacks, the dog film, and I thought that was something. I like that.

• 42:18Nathan Labenz?

  Is it, I know that you can, because I did it, or my coding agent did it for me, you can define any tools for ER2 to use in a similar way that you can define any tools for a regular Gemini model to use. What, what kind of additional guidance would you give on what kinds of tools it can use? Well, maybe you could describe like what the Gemini Robotics 2 VLA model exposes? And then how people can go toward more abstract or more low level than that? And what works and doesn't work so well?

• 42:53Keerthana Gopalakrishnan?

  You mean how the, is the question about how ER controls the VLA?

• 42:59Nathan Labenz?

  Yes, but also my understanding of using ER2 is that I can give it user-defined tools with a just a little tool description, right? And then it chooses what tools to call based on its general purpose reasoning ability. But I'm assuming that some tools it can use better than others. And so I'm wondering what tools does, do you expose to it in the form of the, what are the sort of ways that it interacts with Gemini Robotics 2, the VLA? I'm guessing you probably have kind of the sweet spot of its ability, but then people could go toward higher level concepts or toward much lower level concepts. And I'm wondering what does it do well and what does it not do well as you depart from

• 43:44Nathan Labenz?

  Gemini Robotics 2 as kind of the canonical VLA that it presumably was trained most directly to use?

• 43:51Keerthana Gopalakrishnan?

  So I think one definitely is, the Gemini Robotics ER is built on top of the Gemini tool use and other capabilities. And as we are very much working on how to improve its relationship to the VLA and how to instruct that better, maybe one is, I would imagine that there are, the robots are like very different and each one has very different APIs and stuff. I would imagine that it's much more friendlier to the APIs of the robots that we commonly use that are more widely known, but then if you probably subject it, and here I am conjecturing, I think we would need to test to see how these things are. And the, like, maybe there is no standard for robotics APIs and the market protocol equivalent.

• 44:34Keerthana Gopalakrishnan?

  So I think, so that would be something where I would imagine that more widely using it could face some challenges. And definitely it's a very good model that's very useful for, for language instructing VLA. But you can think of VLA itself as like a, it's a mapping between joint space, language to joint space, language image space to joint space. But there are a lot of demos now and people are thinking about, can these like higher level models directly control in the joint space? And so I think there's a lot of work still to be done. I think current results are still that specific robot foundation models are still better, although the larger brains are showing impressive capabilities in orchestrating these robots. So I think that would be something where I would imagine a lot of push in the future.

• 45:20Nathan Labenz?

  So when I did my little demo, the tool that my coding agent decided to expose to ER2 was a grab at, place at, and it had to just, based on the image itself, pick X, Y coordinates of exactly where to grab and place. It's obviously been trained pretty well on that because it did a pretty good job, again in 2D. How, what, what sort of, with Gemini Robotics 2, what kinds of commands do you have ER2 giving to the VLA? Are they like, it sounds like not so low level as move hand to X, Y, Z coordinates. It sounds like that you were saying is like more experimental.

• 46:03Nathan Labenz?

  What you're, the interface that you're actually working with is more pick up this item or pick up the yellow, like in my demo, there's a lemon or a banana. So there, you could say, pick up the yellow item, but that would be ambiguous on banana or lemon. How would the ER2 model disambiguate for the VLA model? Would it give coordinates? Would it like use the term banana? Would it say, describe the shape of like oval versus crescent shape? How, how many concepts does the VLA have to work with? And conversely, the opposite side of that is like, how specific does the ER have to be in its commands?

• 46:39Keerthana Gopalakrishnan?

  I would tell you an answer, but it's only true at this point, right? It is like an evolving, so this, this field of work is called sterability. And people are looking at various ways of steering, so VLA sterability and then the ER is essentially using whatever modes that you can steer the VLA, right? And VLA's right now are steerable in language, you can point to objects. And so here, it's not like an API that I'm giving where I'm like saying, put this pixel location. I'm literally just doing pointing and then I'm pointing to an object and then it's, can the VLA do it? And then the ICL results are like, can I prompt it with video and stuff? So I think it is like a spectrum and eventually you would imagine that the way that you want

• 47:24Keerthana Gopalakrishnan?

  to interact with robots generally is very wide, right? You want to say, if the modality between the ER and the VLA is just text, a lot of things can be lost in there. So we want that contact interface to be also as rich as possible over time.

• 47:42Nathan Labenz?

  Yeah, so the VLA itself is multimodal in the sense that it can take all these different domains or all these different modalities.

• 47:49Keerthana Gopalakrishnan?

  Yeah, and I think like the ER kind of, you can think of it like as orchestrating or giving feedback to the VLA to do the task. And there is a range of things that like instructions, like even while we don't generally train it to do a lot of like low level stuff, I think like it says things like turn around, look to your, put your head there or look to your left and you can get all that information and the VLA responds to it. And sometimes, often, the thing about foundation models is that like, sometimes you can be quite surprised at what they do and because like even you didn't know that they could do that. And then sometimes you can see that coming out of just the harness as well.

• 48:28Nathan Labenz?

  So has it been disclosed or can you tell? It sounds like the Gemini Robotics 2 model is also created from a VLM base. But then, and obviously for... So the interesting, you mentioned earlier that it has whole body control. Tell me, what does whole body control mean exactly? Does it include? Because stabilization typically needs to run on like a pretty high cycle time, right? A pretty high frequency. And if this is a post-trained, especially post-trained VLM that has such high level concepts as you're describing, it seems hard to imagine that it wouldn't be able to run on the frequency

• 49:13Nathan Labenz?

  to do lower level stabilization. So maybe the robots themselves bring that kind of stuff? And that's not part of whole body control? Like what is in whole body control? And is there some stuff that's left for kind of even lower level systems?

• 49:29Keerthana Gopalakrishnan?

  Yeah, so I think you can think of as like the Chinese humanoid race as an example, right? Where there are definitely controllers that can control the body. But then eventually what you need to make sure is like, can I do the task that the person is asking to? And the very low level controllers often don't have a semantic brain there. And now you need to like control like, and in, I doubt a lot of these things cannot be abstracted away to, oh, this system will just do this or the system will do that. I think, so here the VLA is controlling the whole humanoid body from like fingertips to the feet. And only when you reason entirely about how the body is and also all the sensory information

• 50:12Keerthana Gopalakrishnan?

  about how the environment is, can you do that? I think that is the right design to do a lot of different tasks. So the VLA is controlling the whole body. And but there are more decisions being made about how to stabilize and how to achieve the targets of the VLA.

• 50:29Nathan Labenz?

  So this one is not available via the API, but you do have a program. What, I guess that's, you mentioned mostly because it's just more frontier, less reliable technology as it stands today. Does that mean that if we were to break down failures, the majority of the failures would be at the VLA level still as opposed to the reasoning level? Like the reasoning is working and the VLA is where failures, it can't quite actually do the thing?

• 51:01Keerthana Gopalakrishnan?

  We are working with partners to develop this VLA. So we have partnerships with Etronic, Agile Robots, Boston Dynamics. And so that's where we built or how we are building the VLA together with them. I do think in general robotics has this embodiment problem, right? VLA's are also still training on data from robots. And while we see large leap being made in cross embodiment, I think that taking a new embodiment and completely zero-shotting it to a lot of tasks with a very high reliability is still something that is, I don't think there is a very strong precedent for that. And there is also like a trusted tester program for the Gemini Robotics on-device model,

• 51:43Keerthana Gopalakrishnan?

  where we work with a lot of different labs and they have their specific robots. And I think the on-device release showed like it controlling. This was one of my favorite parts of it. You can, with very, very small number of examples, with 200 examples, you can learn a lot of different tasks on a lot of different bodies because you already have the foundation to do a very generic understanding of the physical world and also like how to control it from, and the data for how to control it with many different embodiments. I would think, I think a lot of the challenges with how the orchestration works is still split. There is surprisingly, there is the latency component, right? Because when you have two models, like one model is like doing the task and another

• 52:23Keerthana Gopalakrishnan?

  model is also like looking at how to switch the task, like when does it done and how, now how do you progress to the next task? I think a lot of failure cases often also come from how to do the orchestration really well. And yeah, so I think, and it is also a compounding of errors, right? Like now, if you have 10 tasks in sequence and now you have the ER model, a certain X success rate and the VLA model, which has a Y success rate. Now, the, if you have a sequencing task, you are basically compounding the error. Does that make sense?

• 52:55Nathan Labenz?

  So if I ask a, if I ask an AI robot to go fry an egg, how far does it typically get? Can, you know, are we getting some successes on eggs fried, plated and served to the table? Or would that be beyond what, what of your robots today could realistically do?

• 53:18Keerthana Gopalakrishnan?

  It depends on whether you have data or not. I think that if you, there are ways to make, like if you specifically just want to fry an egg and how to do that reliably, there are algorithms that can make it very reliable just for that task. But if it's like a very generic model, which has not seen how to fry eggs and now you bring it into your home, which is, to be fair, I think models have made a lot of progress on visual generalization. They are now not distracted by like lighting conditions or the different setups of different homes and stuff. So I would, but I think it's like the task semantics and the context, right? And the ability to make mistakes. I think especially for frying, it's like you can overcook an egg and no one would like that. And there, the room for error then becomes small, like.

• 54:00Nathan Labenz?

  Well, I overcooked my own usually too, so that would, we could count that perhaps as a success for me.

• 54:06Keerthana Gopalakrishnan?

  Tasks that allow retries are much easier. Like if you are doing like Lego assembly, it's fine if you make a mistake, you can go back and fix it.

• 54:14Nathan Labenz?

  Yeah, yeah, that's a great distinction.

• 54:16Keerthana Gopalakrishnan?

  If you crack an egg and then it's on the floor, then now you're in a mess.

• 54:19Nathan Labenz?

  Now you've got a mess. And you might slith in it too. What are the, so one of the value propositions of the Gemini Robotics 2 model, if I understand correctly, is it can work with any embodiment? What are like the most exotic embodiments that have come your way via the trusted tester program?

• 54:39Keerthana Gopalakrishnan?

  Primarily, not a lot of exotic embodiments. I think maybe the weirdest thing I've seen is people putting arms on drones and stuff. I think it's just that once you have, if you have data, you can learn anything. I think that is the principle. So I do believe that you can fit like very weird embodiments to it. But in general, I think the embodiments can look very like a normal distribution, right? Arms are very, if you look at like the eigenvectors in the embodiment space, like there are a lot of like biomedical arms, there are a lot of like hands, there are like humanoids. But, and then like, everything weird is like a long tail in that sense.

• 55:20Nathan Labenz?

  Any kids toys that you've seen that might be in demand this holiday season?

• 55:26Keerthana Gopalakrishnan?

  It's toys, like for robots, you mean, or?

• 55:29Nathan Labenz?

  Yeah, I don't know. I just can imagine, I'm surprised honestly, that I haven't heard more demand from my kids just based on what their friends have and whatever, for some kind of AI-enabled little robotic toy that might, I don't know, honestly walk around the house, but move around on wheels even and talk. And I'm not sure I want them to want this because I'm not sure I want it to be their life, but I'm surprised that I haven't seen more. So I'm just wondering if the.

• 55:55Keerthana Gopalakrishnan?

  I'm also surprised.

• 55:56Nathan Labenz?

  R and D pipeline is there.

• 55:58Keerthana Gopalakrishnan?

  I think it's probably because the market is a little bit burnt. I remember when I was in school, there were a lot of companies trying to do my tabletop, like a gaming robot. And I think, I remember when I was in CMU, there was a lot of conversations around these. But maybe what a lot of people learned from that was like, people buy these things for novelty reasons, but they don't go back to using them. I don't know how that is. None of those robots were machine learned robots, which was connected to the large models. And now things are very different. I have, I speak to Alexa and Siri and stuff. And sometimes I wonder, it would have been a lot more fun if they were much smarter.

• 56:39Nathan Labenz?

  I'm still waiting for a good Siri, for God's sake.

• 56:42Keerthana Gopalakrishnan?

  Yeah, I've been talking to the Gemini, the live user interface when I'm doing tasks or just talking about like my personal life problems and trying to get its perspective, like I think you shared. And yeah, it has been very useful in having an expert go over your problems where you're probably not very skilled to solve.

• 57:03Nathan Labenz?

  Yeah, I'm a big voice mode fan too, actually. In a previous conversation that we had, I asked you what you most needed from hardware makers. And you said, I need good hands. How are hands coming along? How is hardware coming along generally? Are you pleased with the progress? Is it going to become a bottleneck for you relative to how fast the models are improving? Where do we stand?

• 57:29Keerthana Gopalakrishnan?

  When I last spoke to you was probably in the May, March of last year, right?

• 57:34Keerthana Gopalakrishnan?

  So here's the interesting thing, right? We had in the March of last year, we came up with Gemini Robotics 1. It showed that a lot of dexterity with grippers. And in the summer of this year, and it's only like a year and four, four months, right? Three months, a year and a quarter. We showed Gemini Robotics 2, which does multi-finger dexterity, which can tie trash bags and stuff and do very fine control of multi-fingered hands. So it is, here's where my internal model is also updating, right? All the tasks that the gripper robots could do and that they were at the frontier of dexterity, the hand robots can do now. And the hand robots can do more things that the gripper robots could not do.

• 58:18Keerthana Gopalakrishnan?

  So in a way, hands are now at the frontier of dexterity and grippers are maybe not. And I think this has been, I think, I was quite surprised at how much the hardware has come along. There are a lot of hands on the market that are quite good, that people are doing a lot of research with. There is still more work need to be done about how to make them very reliable, repeatable, and also make them cheaper.

• 58:47Nathan Labenz?

  How strong are they? If my wife needs me to open a jar, would the robot be able to substitute for me and open the jar for her? Or would it be like, she would need to open jars that the robot can't open? Like, where does it fit on the kind of the, in addition to the dexterity, the ability to really torque something?

• 59:08Keerthana Gopalakrishnan?

  I think there is a spectrum and it depends on which hands that you use. I think especially even off the shelf, there are a lot of variations. The woojee hand is closer to, I think, maybe like a 10-year-old, I would say. It's, it's, it's a bit weaker and the sharper hand is, I think it can lift like 20 kg. I don't know how much the torque stuff, but I've seen it open jars, so I'm pretty sure that's possible. Though maybe not a very tight one, I don't know. So there is a lot of spectrum and some hands are built for lifting more. The sharper hand is also much bigger than my hand. And the woojee hand is maybe close, more closer and human-sized.

• 59:45Nathan Labenz?

  It'd be funny if the last job for humans is opening stuck jars that the robots don't quite have the wrist strength for. Random aside, it's fine if the answer is nothing or it's all fake. But what is soft robotics? What do you know about soft robotics? What should I be paying attention to? I recently talked to a founder in this space and honestly, I came away more confused than I went into that conversation.

• 1:00:10Keerthana Gopalakrishnan?

  Where they're building?

• 1:00:12Nathan Labenz?

  Basically, it was funny. They basically wouldn't really think. I was like, why did you schedule an interview if you don't want to say anything at all about what you're doing? So it was a random, kind of strange experiment or experience. But I did nevertheless come away thinking, what should I be paying attention to, if anything, in soft robotics?

• 1:00:29Keerthana Gopalakrishnan?

  Regarding soft robotics, there is a wide spectrum. And I have seen things from people building soft bodies to people building very soft hands. And what's relevant for AI-driven robotics is what's repeatable and what's durable. I think one trend that I'm very interested in is like the Umi type of effort for a lot of gloves and stuff and for a lot of like tactile modality. I think that type of how to do them really well and how to build that really well is still an area of research. And part of it is like soft robotics. I think, yeah, I'm not, I'm not, I'm also not fully, roboticists are of different kind. There are also like a lot of roboticists who are like very, like very mechanical engineering focused

• 1:01:14Keerthana Gopalakrishnan?

  and then who see the work that I'm doing, you're not a real roboticist. And I'm like, sure, okay. Computer science roboticists and the mechanical engineering, oh, you should go to a robotics conference. It's all kinds of people. ICA is a good example. There's every kind of roboticist there.

• 1:01:32Nathan Labenz?

  I did, that sounds fun. I did have a chance to go to WAIC in Shanghai this summer. And there were a ton of robots there. I mostly felt like the demos seemed highly staged. I didn't feel like they were really demonstrating a lot of generalization ability in most of the things that they were showing off. It's like a CES scale show with multiple giant venues and every AI company in China basically there. It was a cool experience. The funniest robotic, specifically the experience I had there was sitting down at the Tencent booth. They had a sort of half humanoid, I don't think it had legs, it was just mounted, upper body. And it was giving massages.

• 1:02:14Nathan Labenz?

  So I got a very brief robot massage, some sort of like pressure point thing. It was too short to really score. It didn't hurt me. It did apply some like non-trivial pressure. And I walked away with a smile on my face. So I guess it wasn't too bad, but I'm not sure. I think I still would be highly confident that a human massage would be better. They're letting robots touch people over there now. It was, that's something. Speaking of robots touching people and potentially crossing other lines. Everybody, of course, this summer is talking about OpenFace, the scandal, the incident, the, I sometimes call it a lab leak out of OpenAI.

• 1:02:57Nathan Labenz?

  And it just has me thinking like, maybe the research is all going to work and the alignment and the safety questions will ultimately be the bottleneck on robotics deployments. You could analyze that in multiple ways. One way to think about it is just like, when do you think if we put the alignment and safety questions aside, I might get my domestic servant robot? And then people can think for themselves, oh, I think that's like before or after I would expect alignment to be good enough to actually want one.

• 1:03:27Keerthana Gopalakrishnan?

  Yeah, I think this is definitely an area that we think very carefully about. And I think of safety as like a capability, right? Like I, there is a lot of discussion around like safety and capabilities being at odds with each other. But people are not going to use an unsafe robot and unsafe agents. I think the most useful agents are going to be ones that are safe, that know what they need to do. And they're not overriding human instructions and causing chaos or breaking laws and stuff. So I think safety needs to be front and center in the design for AI and also robotics. And especially in robotics, like in robotics, though, I think there are different types of safety.

• 1:04:09Keerthana Gopalakrishnan?

  Maybe one thing that is not relevant to AI safety is operational safety, right? Like, how do you make sure the robot is not like injuring people? And this is not injuring people because you're malicious, but just not injuring people because you're dumb. And a humanoid being stable enough, like there, it's not doing a vile plot to take over your house. But it's just if it falls, it can still be quite dangerous. So there are like a lot of different safety. So that's maybe like one type of safety research that's like very specific to robotics, but not yet. I don't see it being a big thing in the AI space. But then there is also things like, what should I do if there are like, obviously like the higher level goal stuff, right?

• 1:04:53Keerthana Gopalakrishnan?

  Like, how do you make sure you do the task that the human is asking, but without breaking guardrails? And then there is also the stuff about, and maybe this is also specific to robotics, like what happens like when some senses go wrong? And how should I react in those situations? This one very funny demo that we showed was like, the humanoid is doing something and then someone goes in and puts a basket on top of its head. And now what should you do? You can continue doing the task with your eyes covered, but the right thing to do there would be like really seeing that, hey, like something is obstructing my vision, can I, can you please remove it? So I think safety, and safety is like a full system thing, right? Like, it goes all the way from like the design, the mechanical design of the robot's system safety

• 1:05:36Keerthana Gopalakrishnan?

  to all the way to like the ER and the high-level brain. That's how I think about it. I do think as AI becomes more capable, like we need to think, spend more flops thinking about safety.

• 1:05:49Nathan Labenz?

  Yeah, your example of somebody messing with the robot is an interesting one. And we've seen this with Waymos and stuff, obviously, right? Where people like go out of their way to cause trouble for AI systems.

• 1:06:02Keerthana Gopalakrishnan?

  In fact, it's something that I was very surprised of. It's like maybe because you, and it's, it's hard, right? You have two brains in you. One brain is the researcher brain who knows that they are machines. And now you, you don't really treat them as human-like. But then there's the other brain, which is like, because they look so human-like, they're, it's a bit duplicitous, right? Because when it's talking and stuff, you feel, you feel kind of very human-like around them. Now, so as a researcher, I'm like very careful a little bit. But I think when we see new people interacting with the robots, they're very, people need to be more aware that these are machines. And I think sometimes when new people interact with the robots, because it looks very human-like, it's very, it's, that, that boundary can blur.

• 1:06:47Keerthana Gopalakrishnan?

  I also think this also creates a higher bar for humanoids. If you have a gripper-based robots, like it makes a mistake, like people don't expect it to be smart. But a humanoid just like grappling around and creating a lot of failures would be judged much harshly. Because even if you say, okay, this is the state of the manipulation research or whatever, people still expect humanoids to act smarter than robots that look anything unlike it.

• 1:07:12Nathan Labenz?

  Yeah, that's interesting. I don't want to go down too much of a rabbit hole on consciousness, but I.

• 1:07:20Nathan Labenz?

  Do think it's incredibly striking to me over the last year, how much analogous structure has been found in LLMs. And personally, my willingness to entertain the possibility that AIs actually are somewhat conscious or sentient or something is up a lot, just based on how much analogous structure we can now point to with things like functional emotions and functional wellbeing and the J-Space and so on and so forth. If you have any comments on that, I'm interested to hear them. If you want to pass on that, you can just pass on AI consciousness for the moment.

• 1:08:00Keerthana Gopalakrishnan?

  Well, I want to highlight one thing. So something that really happened for Gemini Robotics 2 that was not in our previous releases is the human-robot interaction component. So this is where it creates like very natural gestures. And it's not like pre-programmed. And previously, if you look at our videos from our last releases, it was, it would do all these nods and stuff where a bit more like pre-programmed HRI. And now it's like very natural HRI. And there now you see like very funny things happening because the robot itself is deciding what gesture should I use as I talk to the person. And something that was very interesting was as we were like filming for Gemini Robotics 2, there was, there are, if you look at the videos, there are also interviews with the robot.

• 1:08:44Keerthana Gopalakrishnan?

  So people ask the robot like, hey, how do you think you did the thing? And then the robot would be like, yeah, it was great, but maybe I made a mistake on this part of the task. And so it's, and also we had the filming crew and they were shooting the robot. And initially, the filming crew, they hadn't spent that much time around the robots. But at the end of it, they were like, action robot. And then the robot will do the thing. I think, I do think as we become more comfortable around humanoids and also as these models become very smart, I think, yeah, the lines are going to blur. I'll say one funny incident. So this was like the robot working in the basement and the garage and then trying to pack things and stuff. And then there was an interview about what was the hardest part of you doing that task?

• 1:09:27Keerthana Gopalakrishnan?

  And then the robot says, the videotape, which was my favorite object, I think it was very hard to put it in the, in the basket instead of holding on to it. And then we're all sitting in the back and we are all like, like laughing at the robot's response. And yeah, it's, I don't know, would a robot like a videotape?

• 1:09:47Nathan Labenz?

  I guess so. Self-report, self-report is at least we're seeing somewhat reliable, although not always.

• 1:09:54Keerthana Gopalakrishnan?

  There have been some very, very adorable moments. Yeah.

• 1:09:58Nathan Labenz?

  Where are the gestures coming from? If they're not pre-programmed, what is it that is causing the model to be inclined to gesture to humans?

• 1:10:10Keerthana Gopalakrishnan?

  It is being prompted. There is a model that is using HRI to come up gestures.

• 1:10:15Nathan Labenz?

  Gotcha. So like ER 2 is saying to the VLA model, raise your hand in a gesture as I give this response, or something along those lines.

• 1:10:25Keerthana Gopalakrishnan?

  No, I think it's the, it's like following along in the conversation. Like you can, it's not very explicitly prompted to do this or this, but it's like coming up with the gestures on the, so just like how the VLA is, right? No one is asking the VLA, put your hand out and then go grab the thing. So it's more natural than that.

• 1:10:43Nathan Labenz?

  Fascinating. Would you call that emergent? Not quite emergent. It sounds like you designed for it somewhat, but not super specifically, so it's semi-emergent, maybe.

• 1:10:53Keerthana Gopalakrishnan?

  Yeah. And this is also a field of research that we started thinking more about as you work on humanoids. I think in the outside world, there's a lot of conversations around why use humanoids for, why not just do armed robots and stuff? I think there are certain fields, entire fields of research that you would not touch because you would not be exposed to those problems until you look at what is the peak form factor that I want to target. And I really like that maybe DeepMind is probably the only lab here in North America where you can study humanoid intelligence in a cross-embodied way. It's not, we're not specifically building a brain for any one body, but it's very general. And there are multiple humanoids in our lab.

• 1:11:38Keerthana Gopalakrishnan?

  So it's thinking about like, how do you think about the intelligence substrate? But then also abstract away the body a little bit. And intelligence is, in some ways, a little bit abstracted away from the final form factor. And there are multiple fields that only arise that when you do different humanoid bodies. One is human-robot interaction becomes a much bigger thing. Whole body control becomes a bigger thing. Multi-finger dexterity becomes a new frontier. And to get exposed to all these three frontiers and research problems, you need to work on it.

• 1:12:11Nathan Labenz?

  So one big question I have about kind of the future of the field. Shout out to Dr. Jim Phan from Nvidia for an outstanding talk he gave that inspired a couple of big questions for me. He makes the case that basically model, the robotics models in particular, need to think ahead, right? They need some sort of forward-looking simulation, which is not exactly synonymous with world modeling, but certainly highly related to world modeling. But the key idea, I think, that he has, going back to my earlier question on the response time of a classic reasoning LLM backbone system,

• 1:12:53Nathan Labenz?

  is you can't, in his telling, you can't just have the model look at the present all the time, reason about the present, take action and run that cycle over and over again. You need to have some way of kind of projecting into the future how things are going to go so that I can move and measure the delta between my expectation and what's happening. And that's like the path. Certainly, that seems to be, from what I understand of how our brains work, that seems to be what we are doing. Do you think, I think this is, you could look at LLMs and say, they've come awfully damn far on a token-by-token prediction without explicit world models. Maybe robotics can do the same. Maybe it's different. Where do you come down on that? I'm sure a pretty central debate.

• 1:13:35Keerthana Gopalakrishnan?

  Yeah, I think the jury is still out there, right? Ultimately, the thing about working as a roboticist is that you are not emotionally attached to one method or the other method. You're emotionally attached to the problem itself. And whatever way you solve the problem, you will pick that. And yeah, there are many schools of thoughts. Definitely world modeling is a very interesting area for robotics. And not just for as policies, but also simulation and other things. And I think the, I say the jury is out there still because there are obviously the VLA's, the VAMs, but also look at how the agent robotics are going. And I think we will see. And this is maybe one of the most exciting parts of working in robotics

• 1:14:17Keerthana Gopalakrishnan?

  is that it is still very early that the recipes haven't stabilized. Different people can have very different schools of thoughts about how I'm going to go about and solve robotics. And we'll see who's right, and all of them could be right. And it might be the future, until we have solved the problem, it's very hard to very definitively say this is the solution or this other thing is the solution.

• 1:14:39Nathan Labenz?

  Yeah. Interesting. Okay. So it sounds like you're open-minded and obviously Google DeepMind has world modeling efforts that are pretty well-established as well. So to the degree that you need to pull a world model off the shelf and to start simulating into the future, you have a pretty good foundation right at hand to do that kind of work. He had a couple of other interesting points. I'd be interested in your take on. And one was basically, he thinks that we are going to go to egocentric video as kind of the main data used for training. He thinks that like teleop will go to a tiny scale, a tiny portion of data.

• 1:15:21Nathan Labenz?

  And even the sort of hand, we've seen a lot of innovation. I think it's been pretty cooled around just like wearable robot hands that you can use. But he thinks even that will ultimately just give way to first-person POV videos. And then beyond that, it's just going to be simulation. And he thinks basically like, all the problems of simulation will just get solved and that will really be the way that we'll get to enough robotics data to solve robotics. Any thoughts on how that data, that vision for the future of data compares to yours?

• 1:15:56Keerthana Gopalakrishnan?

  I think everything needs to be backed by results, right? And it's, it's, I don't think there is much point in making a prediction. And usually you have a prediction, you run an experiment to validate that prediction. And then now you know that is the right prediction. And these things can also change over time. And I think, as Jim suggested, it's great that he has very strong views on certain types of data. I think it might also very much look like it's a mixture over time. Like you look at LLMs, they're trained on all kinds of data. And different data has different purposes. Like the teleop data is definitely useful in grounding into the robot's controls.

• 1:16:38Keerthana Gopalakrishnan?

  UMI data is definitely very useful in learning much scaling, scaling without robot in the loop, so much cheaper scaling. And human data is also very, like the egocentric human data is also very useful in getting broad semantic understanding about how, what to do and stuff. But they also have their failure modes, right? If you imagine that you're controlling very fine hands, you don't get very fine signals about where the hands are unless you're getting, you're estimating all that from vision. And vision has a lot of errors. And so like each data modality has different strengths and different drawbacks. Like the UMI data, like with sensors, you are getting the action labels, not from vision, but from sensors. So it's a bit more accurate.

• 1:17:23Keerthana Gopalakrishnan?

  So there is, I would imagine each data source is on, on the exact as you can plot like scale and the precision. And maybe like teleop, not very scalable, but highly precise because it's exactly the robot's data. But it is also not very like future-proof, right? Because as your robot evolves, your teleop data starts becoming a little less useful, even if you have very good cross-embodiment. Then there is UMI data, which is like more scalable, but because of all these sensors, it's more precise than human data, but it's less scalable because of all these sensors. And then there's human data, which is like very scalable, but can also be very noisy because you don't have very precise end-effectual actions. And you also need to like now put that on the robot.

• 1:18:06Keerthana Gopalakrishnan?

  So there are errors that come from, and each human is very different. Some humans are small, some are large. And due to that, how they do the end-effectual tasks is very different. So I think it's going to be like probably closer to a mixture. And the thing is, hardware is changing each of these paradigms differently, right? Like as very good UMI hardware comes on the market, like that, how to scale that now goes up. So I don't think it's wise to take a very principled view about this because it

• 1:18:40Keerthana Gopalakrishnan?

  All of this will change as new evidence emerges and better hardware emerges. And you probably need all of all types of data.

• 1:18:47Nathan Labenz?

  Then maybe one other aspect of his talk that I thought was interesting, it wasn't a main focus, but it got me thinking a little bit, was he used the term physical API. And it got me thinking like, how do I want to interact with robots in a world where I'm interacting with like multiple different robots? Do I want them each to present as a entity unto themselves? Or maybe it turns out to be more of like a her model where I have one entity that I like interface with the most and it kind of deals with all of the robots as a fleet? And therefore the robots are not like individual entities in my life, but they're all sort

• 1:19:30Nathan Labenz?

  of extensions of this like single personal superintelligence or whatever you want to call it that can run all these other things. Do you have an intuition for how you would want to interact with many robots in your life?

• 1:19:46Keerthana Gopalakrishnan?

  Yeah, I don't have the context of this physical API conversation, but it also ties a little bit into what we were discussing before, right? I think the surface area of our interaction with robots needs to be highly multimodal. And I, the user, needs to have both high level and low level access to robots. I think that is probably the only way that we can build towards a very safe and flexible future. And anyway, from language and if robots code in, if a task requires multiple robots, then I need to be able to hand it off and then it can handle the collaboration. I think the Gemini Robotics too kind of had this where both the Duo robot and the Apollo

• 1:20:30Keerthana Gopalakrishnan?

  was doing the task, one was doing one and then it stops and the other one takes over. So there is, I think, yeah, maybe, I don't know, this is a boring answer, which I've said before, it's probably going to be a spectrum, right? If there are tasks which require multi-robot collaboration and a lot of agentic swarms, and then there I need to be able to hand off and they need to figure out how to do things. And then there are tasks where I'm only interacting with one robot and I either want to give it text instruction because I'm just talking to it, or I want to give it more fine-grained, specific feedback and there I need to go even more lower level. I think for some, a lot of safety and interpretability reasons, maybe we need to still maintain much lower level access to robots.

• 1:21:13Nathan Labenz?

  Yeah, what you said there reminds me of what I'm increasingly thinking of as like the universal UI for software products, which is like, there's the product itself and I can click on it and edit fields and drag stuff around. And then there's an agent or assistant that kind of sits alongside that and has basically all the same affordances that I do. And most of the time, I probably just want to tell that thing to do it and I don't really want to concern myself with the details. But I also have found that it's really annoying sometimes if the app doesn't let you get to that level of detail, because sometimes the AI is just like not getting it on its own. So I can see a way that paradigm ports over to robotics.

• 1:21:56Keerthana Gopalakrishnan?

  Yeah, especially let's say, like, I cannot tell a robot to clean my desk because I like my desk done in a certain specific way. So I need, I would like the ability to give it more feedback and even personalize it to my own like.

• 1:22:14Nathan Labenz?

  What's up with these days with, for lack of a better term, I'll call it circuit breakers on robots? What I mean by that is I saw one example where the company had made a sort of inflated airbag in the torso of the robot. And the idea was like, if you poke that thing, it is immediately disabled. The pressure of the bag that you could easily puncture as a human immediately disables it. I thought it was just interesting, new idea in terms of like how you could, worst case scenario, have an off button locally. But more broadly, there's like, what happens when it encounters resistance it's not expecting or whatever?

• 1:22:58Nathan Labenz?

  How has that evolved over the last year plus?

• 1:23:02Keerthana Gopalakrishnan?

  Yeah, so I think for the longest time, this air bubble that you break, there has been a version of that, it's called e-stop or kill switch. So robots, most robots have a button that you can press and then it will depower. And there are many phases of it, right? There is the hard e-stop where you stop all the kind of power to the body and then it can just collapse, which can then be dangerous on its own. There is also the soft e-stop where you just freeze the robot. And this is often safe for bipedal and other platforms which are not stable by themselves. And in recently, I think especially in manipulation, there are a lot of new features kind of emerging to control forces at the end effector, right? Imagine that you want to stack two chips together.

• 1:23:46Keerthana Gopalakrishnan?

  You can, if you press too hard, you crumble the thing. And or if you're handling very delicate material, you need force control and back drive ability at the end, compliance there. So I think sensing is improving quite a bit that you can have very, you can read what the end effector forces are and then the models can take much better decisions than they would if they didn't have that information.

• 1:24:10Nathan Labenz?

  Last question, kind of a galaxy brain or maybe a fly-brained one. So there's been all these demos, I don't know if you've seen recently of people using the fruit fly brain to do all kinds of things, including seemingly to control some small robots.

• 1:24:26Keerthana Gopalakrishnan?

  Actually not seeing them. Maybe our like Twitter feeds are still fairly personalized.

• 1:24:31Nathan Labenz?

  There's a, yeah, there's one, I'll send it to you. There's one where somebody has trained this thing to control a little, I think it was called a bee strand, one of those like beachwalker robots, but a real small one. The connectome of the fruit fly brain was fully mapped and put online in digital form and people have just started downloading it and training it to do different things. I'm not 100% sure how real it all is, but there's enough of it going on that I think some of it's real.

• 1:25:00Nathan Labenz?

  It sounds like you don't have necessarily take on that. I'd welcome any comments, but maybe zooming out, I wonder, far future, do you envision yourself with some sort of robot body? Is there like a neural brain computer interface where you imagine having a hybrid form yourself?

• 1:25:20Keerthana Gopalakrishnan?

  Yes, yes. And it already exists in some sense, right? Like when people are teleoperating, their physical body is somewhere, but their robot body is the one, just teleop itself is like that. And then there is people who are building remote teleop that works across the Atlantic and stuff. So the person is sitting in Europe, but the robot is in the United States like stacking, I don't know, stocking shelves or something. So definitely this is very interesting. I would imagine someday that you get to explore Mars or something with robots because you are teleoperating them. Yeah, I think the future is exciting. One of my friends, the thing that he wants is he wants to walk around, he wants, you know the guards in India, they have like all these arms popping out from their behind?

• 1:26:05Keerthana Gopalakrishnan?

  But now you can build them, right? You can put those truss and arms on your back and then those arms can go pick things and you can just walk around and assess with those. So that's the thing that he wants to build as a hobby. Sometimes I also think about like my dog, I looked up how many parameters she has in a neural network, apparently it's 25 trillion. So she has more parameters than a lot of the models out there, even though her language skills are not that developed. Yeah, I think the future is very interesting.

• 1:26:34Nathan Labenz?

  Yeah, that is a modest way to put it, I'd say, and that might be a great note for us to end on. As always, this has been a fantastic conversation and I appreciate your help in catching up with everything that's going on in robotics. Kirtan Agopla Krsnan, thank you for being part of the Cognitive Revolution.

• 1:26:53Keerthana Gopalakrishnan?

  Thank you so much for having me.

• 1:30:05Nathan Labenz?

  If you're finding value in the show, we'd appreciate it if you'd take a moment to share it with friends, post online, write a review on Apple Podcasts or Spotify, or just leave us a comment on YouTube. Of course, we always welcome your feedback, guest and topic suggestions, and sponsorship inquiries, either via our website, cognitiverevolution.ai, or by DMing me on your favorite social network. The Cognitive Revolution is part of the Turpentine Network, a network of podcasts, which is now part of A16Z, where experts talk technology, business, economics, geopolitics, culture, and more. We're produced by AI Podcasting. If you're looking for podcast production help for everything from the moment you stop recording to the moment your audience starts listening, check them out and see my endorsement at ai podcast dot I N G.

• 1:30:49Nathan Labenz?

  And thank you to everyone who listens for being part of the Cognitive Revolution.