Autonomous Vehicle Research at Waymo

Waymo’s VP of Research, Drago Anguelov, joins Practical AI to explore how advances in autonomy, vision models, and large-scale testing are shaping the future of driverless technology. The conversation dives into the dual challenges of building an onboard driver and testing that driver (via large scale simulation). Drago also gives us an update on what Waymo is doing to achieve intelligent, real-time performance while ensuring proven safety and reliability. Featuring: Drago Anguelov – LinkedIn https://www.linkedin.com/in/dragomiranguelov/Chris Benson – Website https://chrisbenson.com/, LinkedIn https://www.linkedin.com/in/chrisbenson, Bluesky https://bsky.app/profile/chrisbenson.bsky.social, GitHub https://github.com/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaLinks: Waymo Research https://waymo.com/research/New Insights for Scaling Laws in Autonomous Driving https://waymo.com/blog/2025/06/scaling-laws-in-autonomous-drivingAI in Motion https://www.youtube.com/watch?v=11WEL7sR1GASponsors: Outshift by Cisco - The open source collective building the Internet of Agents. Backed by Outshift by Cisco, AGNTCY gives developers the tools to build and deploy multi-agent software at scale. Identity, communication protocols, and modular workflows—all in one global collaboration layer. Start building at AGNTCY.org http://agntcy.org/. Shopify – The commerce platform trusted by millions. From idea to checkout, Shopify gives you everything you need to launch and scale your business—no matter your level of experience. Build beautiful storefronts, market with built-in AI tools, and tap into the platform powering 10% of all U. S. eCommerce. Start your one-dollar trial at shopify.com/practicalai http://shopify.com/practicalaiFabi.ai http://fabi.ai/ - The all-in-one data analysis platform for modern teams. From ad hoc queries to advanced analytics, Fabi lets you explore data wherever it lives—spreadsheets, Postgres, Snowflake, Airtable and more. Built-in Python and AI assistance help you move fast, then publish interactive dashboards or automate insights delivered straight to Slack, email, spreadsheets or wherever you need to share it. Learn more and get started for free at fabi.ai http://fabi.ai/Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

• 0:03Speaker 00

  Welcome to the Practical AI Podcast, where we break down the real-world applications of artificial intelligence and how it's shaping the way we live, work, and create. Our goal is to help make AI technology practical, productive, and accessible to everyone. Whether you're a developer, business leader, or just curious about the tech behind the buzz, you're in the right place. Be sure to connect with us on LinkedIn, X, or Blue Sky to stay up to date with episode drops, behind-the-scenes content, and AI insights. You can learn more at practicalai.fm. Now, onto the show.

• 0:40Speaker 01

  Well, friends, when you're building and shipping AI products at scale, there's one constant: complexity. Yes, your rating models, data pipelines, deployment infrastructure, and then someone says, "Let's turn this into a business." Cue the chaos. That's where Shopify steps in. Whether you're spinning up a storefront for your AI-powered app or launching a brand around the tools you've built, Shopify is the commerce platform trusted by millions of businesses and 10% of all US e-commerce, from names like Mattel, Jimm Shark, to founders just like you. With literally hundreds of ready-to-use templates, powerful built-in marketing tools, and AI that writes product descriptions for you, headlines, even polishes your product photography, Shopify doesn't just get

• 1:25Speaker 01

  you selling, it makes you look good doing it. And we love it. We use it here at ChangeLog. Check us out, merch.changelog.com. That's our storefront. And it handles the heavy lifting, too: payments, inventory, returns, shipping, even global logistics. It's like having an ops team built into your stack to help you sell. So if you're ready to sell, you are ready for Shopify. Sign up now for your $1 per month trial and start selling today at Shopify.com/PracticalAI. Again, that is Shopify.com/PracticalAI.

• 2:17Speaker 02

  Welcome to another episode of the Practical AI Podcast. This is Daniel Whiteknack. I am CEO at Prediction Guard and I'm joined as always by my co-host, Chris Benson, who is a Principal AI Research Engineer at Lockheed Martin. How are you doing, Chris?

• 2:32Speaker 03

  Doing great today, Daniel. Lots of, as always, lots of AI and autonomy to talk about. And you know what? We have Waymo to talk about as well.

• 2:42Speaker 02

  We have Waymo to talk about, yeah. Speaking of Waymo, we're very excited to welcome back Drago Engelov, who is the Vice President and Head of the AI Foundations Team at Waymo. Welcome, Drago.

• 2:56Speaker 04

  Thank you, guys. It's great to be back after five years or so, right?

• 3:00Speaker 02

  After five years, yeah. We were commenting before we started the recording that the last episode with Drago was on September 1st of 2020. So that was episode 103. So a few things have changed in the world generally, but certainly in relation to AI. I'm wondering if you could maybe just catch us up at a high level, Drago, in terms of driverless cars, autonomous vehicles. Like, how do you see the world differently now than you did in 2020?

• 3:36Speaker 04

  So, one thing I would say is, in October of 2020, we opened our Waymo One service in Phoenix East Valley to everybody. So, you know, just one month after we talked. But since then, we have launched and scaled quite dramatically in now five major metros. And this is San Francisco, Los Angeles, Phoenix, Atlanta, and Austin. And we are also serving hundreds of thousands of rides a week to paying customers.

• 4:15Speaker 04

  We are expanding, we announced expansion to at least half a dozen or no more cities. That will be going on through next year and we may announce yet more. In the cities we were at, we continue reporting the safety performance of our autonomous driver. And we are over one hundred million autonomous miles driven on the road at this point. So it's fairly statistically significant. And in those miles, our safety study at close to a hundred million miles showed that we are five times less likely to get into accidents with critical injuries.

• 4:57Speaker 04

  And over ten times, I think twelve potentially, less likely to get into collisions or injure pedestrians. So that has been happening. And we are on to doing more and more right now. I think we work on improving the driver further. We have a sixth generation vehicle coming up. We have started partnering with different companies. For example, we're partnering with Uber in Austin, in Atlanta. So our vehicles show up on their app in those cities. We have partnered actually with Lyft in Nashville, if I believe. And we partnered with DoorDash to explore delivery.

• 5:42Speaker 04

  So we are exploring and expanding the scope and the partnerships that we are doing as well. But I think in twenty five, I would say a lot more people have had and continue having the opportunity to try away more. I'm quite a convert myself. To me, probably the aha moment, the big aha moment was in twenty two when I got riding in San Francisco by myself, fully autonomously. And so since then, it took some time for more people to get exposed. But now I think the phenomenon is out there. And I think also the autonomous vehicle industry went through like cycles. There was certainly around twenty two, twenty three time of pessimism.

• 6:30Speaker 04

  In autonomous vehicles, but I think through our success, through generative AI, and I think there is other companies now, it's again a very lively space. There is others that are also trying to push what's possible with autonomous driving and robotics. So it's again very, very happening place. And yeah, we are contributing probably, I would like to think the most advanced version of an embodied physical AI today that you can do without.

• 6:58Speaker 03

  That's fantastic. I gotta say, as a native Atlantan, I'm so happy that you guys are in my city. And we're a very, very car-centric city as well. You know, you really have to have a vehicle to get around. And I noticed, you know, as you were naming the cities that you guys are in, that tended to be the case in terms of very auto. Does that play into any of the way that you guys think about testing in terms of being, you know, like Atlanta traffic for its size is notoriously bad. And I would love to see ever more Waymos and other autonomous vehicles here because I am terrified of all the drivers around me with our daily collage of traffic accidents and stuff like that.

• 7:40Speaker 03

  So I keep telling everyone, just wait, autonomous vehicles are coming. I'm kind of curious how you pick these different testing cities that you guys engage in? And what are some of the things that you're testing for that maybe those locations are particularly apt for helping out on?

• 7:58Speaker 04

  So I mean, it's a bit a combination of both technical and business reasons. I think we are trying to do large metros where, you know, autonomous vehicles can be a big market and help a lot of people. So that's one. And also we've intentionally been growing our ODD, so to speak, operational design domain. Our first service, Waymo One in Phoenix East Valley, Chandler, that's maybe a bit suburban with up to 45 hour arterial roads.

• 8:28Speaker 04

  And we learned to master it and then went to San Francisco that is denser burn with fog and some rain and hills and windy roads and narrow roads and you know tons of pedestrians downtown. So we dealt with that. And then we started expanding. I think some of this is Atlanta is a big city, also different state. Uh, there's some differences across the various states and uh both how people instrument the roadway and how people drive, right? So we're spreading geographically more and more. I think also we're spreading to other domains, a few, a few that are really top of mind is uh highways. We have been uh working on highways for a long time. Uh, we've gotten to a certain point with highways.

• 9:13Speaker 04

  Uh generally to have a good taxi service, you need highways, right? And turns out that's a very fascinating, interesting problem. They're difficult because you, whenever you move at high enough speeds, like sixty-five miles an hour or so, right, the consequences of any mistake are really high. And many things can happen. And so it pushes your, I mean, robustness and safety capability there. So we've been doing highways, but one thing I did do is I rode, uh, now we, we can give uh highway rides to employees. And uh I rode one to Millbrae Station to get to the airport. And it's fantastic. So hope to be able to bring it in the future to more and more people.

• 9:56Speaker 04

  I think that will make the service a lot more useful. Also, we announced that we will drive in other cities like uh that have snow, so potentially even in twenty-six, right? So our sixth generation platform is designed uh after the Jaguar, right? It's a Zicar vehicle and uh Geely Zicar, it's a Zicar. And that Zicar is designed with a hardware suite to be uh able to handle snow. And we are also heading out to other countries. So we announced that we intend to launch driverless capabilities in London next year. London is left side driving city.

• 10:40Speaker 04

  And so it's Tokyo where we are currently have vehicles and we're testing, right? So you can see, we're trying to cover little by little the operational design domain of most large metros with all of their properties. We're of course also in Texas, that's its own unique state. But we started with most southern states, large metros. So you don't have to worry about snow at least. Like you want to tackle these challenges in some order, not just try to do everything at the same time. It's very difficult to validate your ability to do well in everything all at the same time, right? So we're just taking, we, we're kind of mixing what is, what makes business sense with actually expanding the capabilities to become truly global driver.

• 11:23Speaker 02

  And you started, you, you've mentioned the driver, the car. I'm wondering if for those out there, those listening, which this is kind of maybe hard to do just from an audio standpoint, but if, if you kind of imagine the, the driverless car as, as a system in 2025, how would you kind of describe that architecture, that system? What are the kind of main components? Like I imagine, you know, the sensors, the actual car, the computer, like how, what does that system look like in 2025? Just at a high level. And then of course, I'm sure we'll get into some of the, the modeling things and foundation models and all of those things.

• 12:03Speaker 04

  I mean, the car is, you know, ultimately it's a robot on wheels, right? The main distinguishing capabilities are that it has a set of sensors. In our case, camera, lidar, radar and microphones. And microphones are quite helpful for many things, including listening to sirens, right? And occasional instructions. Then you have compute on the car. It's a non-trivial amount of compute. It's more than you can put in a phone, right? And all our vehicles are electric. That was a explicit choice of the company. I'm personally quite proud of this choice. I think. That's good for the environment to actually have such cars.

• 12:44Speaker 04

  And I think can accelerate, I think, transition to more electric vehicles, which I think is good personally. And so they have this robot on wheels with computer and sensors. And then you have actuators, right? And you need to, then there is a lot of system design engineering to make sure, you know, steering and brakes and all these things. They need redundancy and robustness to make sure that if any system goes wrong. Or we need to think also if compute parts of it can go down, that you have contingencies. It needs to be designed with redundancy. You need to think of what if, you know, steering wheel, column, like the, the, there can be also issues with steering. What is the redundancy? So for autonomous vehicle, I need to think additionally and build these things into the hardware.

• 13:29Speaker 04

  So it's a, it's a robot designed for safe transportation from the ground up. Even though we're built using, uh, we're just extending existing platform. Uh, and we work with the various software makers for this.

• 13:42Speaker 03

  As you're doing this and you guys have progressed over these five years, you know, since we last talked. You know, one of the, one of the challenges is not, I probably not every person out there is, is a Chris or a Daniel. Who's, uh, who's very invested in this kind of technology, you know, going forward. You have a lot of people out there here in the South. We joke that, you know, every, every other driver thinks they're a NASCAR driver and stuff. And, and that, that notion of control and safety and, and, you know, the general population may not have as much confidence in some of these technologies because they're not following it closely and living it the way you do all the time. How do you, how do you approach that?

• 14:24Speaker 03

  And how has that changed over these last five years since we talked to you last in terms of getting buy-in from the public? And getting them feeling, you know, like it's, as you talk about the, the safety statistics, which are amazing. But getting them to really feel that deep down, you know, inside that they can, that they know they can trust uh, and believe in this mode and that it is in fact much, much safer than what they are typically doing on a day-to-day basis.

• 14:50Speaker 04

  So there is, you know, people do not feel statistics. It's hard, right? Because they're product of many, many rides. Doing ten or even a hundred safely is not enough. I think what people feel is when they get into the vehicles and this worked for me in my moment, even though even before and also my wife and friends of mine, people get comfortable really, really fast. You need to pass a certain bar where they feel, okay, this thing actually is a really, really good driver. My, my mother-in-law sat in it, uh, just a few weeks ago for the first time. And she, she rode around, she's like, this car drives much better than me, right? And once she thinks this way, she's, she's immediately at ease, I think.

• 15:31Speaker 04

  And I think people relax after the first several minutes are very exciting. And then they relax and enjoy the experience and mind whatever they like to mind, either the environment or their phone or other things. People get really used to it if you cross this threshold of can, can I trust you? And I think your driving immediately shows this. Now, uh, us in the industry also understand that, you know, coming back to statistics, you need to back it up. And so with regards to backing up, Waymo, right, has, we believe in transparency and we, we're quite open with the incidents that happen.

• 16:11Speaker 04

  We file the details and we, uh, also track the statistics and do our best estimate. We have a great safety team. They publish these reports in them. We evaluate and try to estimate how are we doing compared to a fleet of human taxi drivers or human drivers driving in our, uh, area that we are handling. And this is both by us, but also like there are studies done by insurance companies who of course want to quantify this very well. And so there's a Swiss Re study also proving our numbers. They also believe we significantly can decrease, uh, claims of different kinds for injuries, for accidents and so on as well. So that's another external validation for the kind of thing we provide.

• 16:56Speaker 04

  So that's what I would say to people now, you know, it's a process. You need to work with the local communities. You need to work with police. You need to work with, you know, the various city stewards, officers. We train a lot of people. We engage with them. We work overtime. I think, I think you can see that in the cities we have been over time. I believe generally the trust in us increases. And I think that's a satisfaction of Waymost by the users. If you look at the apps like on in the stores. So I think they on the App Store, we had a five zero five star rating. Right. So that there is a lot of a bit of a almost like a people that would just use Waymost now. It's it's if they could.

• 17:41Speaker 04

  And that's a testament to the value that people see in the rights. And but it drives, of course, to safety. And, you know, ultimately engaging these people, getting them comfortable. Often when people experiences, many of them become converts. So I encourage people, try it. You may be the next convert if you have not yet. I personally love it. I I take it as much as I can. And it's it's always a pleasure working on a product you you enjoy yourself. So I I feel blessed that way.

• 18:28Speaker 01

  Well, friends, it is time to let go of the old way of exploring your data. It's holding you back. But what exactly is the old way? Well, I'm here with Mark Dupuy, co-founder and CEO of Fabi, a collaborative analytics platform designed to help data explorers like yourself. So Mark, tell me about this old way.

• 18:46Speaker 05

  So the old way, Adam, if you're a product manager or a founder and you're trying to get insights from your data, you're wrestling with your Postgres instance or Snowflake or your spreadsheets. Or if you are, and you don't maybe even have the support of a data analyst or data scientist to to help you with that work. Or if you are, for example, a data scientist or engineer or analyst, you're wrestling with a bunch of different tools, local Jupyter Notebooks, Google Colab, or even your legacy BI to try to build these dashboards that, you know, someone may or may not go and look at. And in this new way that we're building at Fabi. We are creating this all in one environment where product managers and founders can very quickly go and explore data regardless of where it is, right? So it can be in a spreadsheet, it can be in Airtable, it can be in Postgres.

• 19:31Speaker 05

  It's like really easy to do everything from an ad hoc analysis to much more advanced analysis if, again, you're more experienced. So with Python built in, you know, Python built in right there and our AI assistant, you can move very quickly through advanced analysis. And the really cool part is that you can go from ad hoc analysis and data science to publishing these as interactive data apps and dashboards, or better yet, delivering insights as automated workflows to meet your stakeholders where they are in, say, Slack or email or spreadsheet. So, you know, if this is something that you're experiencing, if you're a founder or product manager trying to get more from your data or for your data team today, you're just underwater and feel like you're wrestling with your legacy BI tools and notebooks,

• 20:16Speaker 05

  come check out the new way and come try out Fabi.

• 20:18Speaker 01

  There you go. Well, friends, if you're trying to get more insights from your data, stop resting with it, start exploring it the new way with Fabi. Learn more and get started for free at fabi.ai. That's F A B I dot A I. Again, fabi.ai.

• 20:39Speaker 02

  Well, Drago, I understand that every driverless car company is going to have a different, you know, approach to modeling and all of those sorts of things. You've talked a little bit about the hardware in the car, but we just, I think it would be good for people to understand, we talk about this driver or you, you mentioned the driver, people might have in their mind, because we do talk a lot about models now after the generative AI boom, that there's this model that can reason and blah, blah, blah. And so people might have this view of like, there is a model that drives the car. Could you help us really break down like in 2025, is this a system of models, models

• 21:20Speaker 02

  that do different things, a kind of combination of different types of models in even non-AI pieces? Could you just help us kind of generally understand how that, how that works?

• 21:35Speaker 04

  So when you think of the stack, right? Let's talk first about what it needs to do. It needs to perceive the environment using the sensors. It needs to build some representation of this environment. It needs to use this representation of the environment to make a set of decisions. And so traditionally, I mean, autonomous vehicles are around a long time. Waymo is around over fifteen years already, right? So it's a rapidly developing technology space. But traditionally, you can think of, there's this historically, people thought, okay, there are these models, there's a perception model that builds a representation of the world that can be useful to certain things. And then there is some kind of behavior prediction and planning module

• 22:18Speaker 04

  that reasons what we could do and potentially some people like also reason what others could do to cross-reference our behavior with the other folks. And then based on all this information, eventually select promising decisions, right? So that's what the stack normally does. Now, there's different ways to implement it. Generally, the trend has been to have few, and in some cases, people claim they have one, large models, AI models on the car. And you could say ML or AI. For a while, it was called ML. When the models became big enough, people called it AI, right?

• 22:59Speaker 04

  So you have these large AI models on the car, a few or one, depending on the various companies. And they're connected in certain ways. You can train them end-to-end or not. That's also an option. Different companies can choose. The two are orthogonal concepts. Whether you have modules and whether you can train them end-to-end is different concepts, right? Right? So it can be structured end-to-end. So essentially have models and train end-to-end. These are two. And so different companies on this very coarse taxonomy fall somewhere in this bucket, right? And I think Waymo, I mean, always has used AI or ML since I've been there. It's been the backbone of our tech.

• 23:41Speaker 04

  I think over the, over time, our models have streamlined and become fewer and fewer. I can say that. I think offboard, what my team does is build this large foundation models for Waymo that are not limited by how much compute or latency constraints you have. And they can be quite helpful in to essentially curate data or teach the models that actually run on the car in the simulator. We can get to simulators later. So we have experience with most aspects of these options, whether it's end to end and whether they're structured or not, right? I think offboard, I can definitely tell we've explored a lot with large vision language models. That's one of the latest technologies that's relevant to us.

• 24:26Speaker 04

  I think in the field of robotics, people talk also about vision language action models because you can tie in one model, you know, both understanding vision and language inputs and potentially ask for certain actions as outputs, right? Which is ultimately what the robot needs to generate. So that's an exciting area that has developed in 25. I think in our foundation model, Waymo foundation model, we combine like benefits of these vision language models, but also combine it with some bespoke Waymo architecture innovations. I think in areas such as, you know, fusing these new modalities that vision language

• 25:07Speaker 04

  models typically are not trained on, like LiDAR and radar is one. Another one is modeling the evolution, future potential evolutions of the world. There is some interesting Waymo technology and how to do this well that we also use. But we, we fuse all of this and VLM technology, world knowledge from also other bases, whether it's a world model or vision language model into something that then is able to do well on autonomous driving tasks. So that's offboard. Onboard, we don't typically talk exactly what is there, but I think we're trying to get state of the art, the best architectures that we believe solve the problem and put

• 25:52Speaker 04

  them together on the car. I think where it's a really, really high bar to have a model perform.

• 26:01Speaker 04

  In all the conditions and all the situations we needed to, right? And so we also have some notion of, as you know, VLMs also have this weakness of hallucination. So we have the safety harness around them to prevent hallucination, to double check what they are predicting, right? So we also have that aspect in our style as well, which we have worked on historically. So that's what I can say on a high level. I hope that's not too scattered. Maybe you guys, if you want anything specific, we can discuss that in a little bit more detail.

• 26:35Speaker 03

  So I do have a follow up to that. And recognizing that you're not able to get into the specifics of how of this of the architectural decisions and model decisions that Waymo is engaged in. If you could abstract it a little bit and maybe just talk about the space a little bit, you know? I'm curious, as you talk about world models, you know, and having representation of the environment that brings in not only AI, but the notion of simulation as, as, you know, one of the tools in the tool chest, if you will. I suspect like we have a lot of listeners that that are hearing lots of different AI use cases in general, but may not have as much expertise in autonomy.

• 27:18Speaker 03

  And so as you talk about that, that notion of of representation of that environment, could you talk a little bit about like what that problem looks like? And what are different things that you might think of to solve it without having to get into how you guys have done it? But just kind of like, what is that? What is that juxtaposition of simulation, AI and representation of the world and the environment around you look like?

• 27:43Speaker 04

  So maybe, I mean, simulation, if we're gonna go there, maybe I can just juxtapose two things there. I like saying this historically, I've been doing this for a while. There are two main problems in autonomy. One is to build this onboard driver.

• 27:59Speaker 04

  And another one is to test and validate this onboard driver. And both are really, really hard problems. And people usually talk about the first one. But I think, imagine there is some collection of models, and you need to prove that it's safe enough to put them out in the real world. That's in itself a really challenging problem, arguably no simpler than putting the model, the first model together. And that one, ultimately, because you need to be a bit more exhaustive, potentially takes even longer time to build the full recipe to validate things properly, right? So these are the two problems. Now, in autonomy, what is different maybe than

• 28:44Speaker 04

  this standard AI models is there's a few things. One is, I mean, ultimately output actions, that are commands to a robot, that are a different type of data than traditionally, say, text and images, right? I think that's one. Another one is, we operate under strict latency constraints. You need to react quickly. For us, what is also interesting in AVs, this is probably the first serious domain where we had to really learn how to interact with humans in the same environment. So it's highly interactive, multi-agent setup, right? And then we have additionally, if you choose to add additional sensors and cameras, we have a lot more modalities coming in and we have a ton of data.

• 29:27Speaker 04

  So essentially, the way to think of it is, imagine you get maybe billions of sensor reading per second, or even tens of billion, a lot. And you need to make decision, or you need to have a context of many seconds of this sensor inputs, maybe a dozen cameras, half a dozen light and radar. And so you need to collect, you know, maybe five to 10 seconds, some can argue 20, 30, of context to make a decision.

• 30:00Speaker 04

  Right, so and the decision is fairly low dimensional. It's like, okay, steering or, you know, acceleration. But the inputs are incredibly bulky. And so you need to somehow learn the mapping from this extremely high dimensional space, a representational space to decisions. That's very hard, right? Under latency constraints, under safety critical constraints. That's what makes our domain interesting. Now, a lot of the things that work in machine learning in one domain transfer to the other, right? So, yes, there is, for example, very similar scaling law findings that if you have cutting edge architectures and you do proper, you know, studies and scaling and you have a lot

• 30:45Speaker 04

  more data and compute and you feed it to these architectures. And now for every class of algorithms, there's a bit different scaling laws. But even the simpler imitative algorithms that people also did in language, predict next token, we can predict next action, right? There is these direct parallels. You can do reinforcement learning in language. We can do reinforcement learning in our simulator, right? These are the parallels. But how exactly things translate is interesting. The ideas translate, the implementation is a little more creative than the usual, just staying on the internet because there is a bit of a domain jump to the real world, right? So that's interesting. The other part is compared to same language, LLMs.

• 31:26Speaker 04

  You can actually, we have a paper, Motion LLM, from two, three years ago, where it was, the idea was, hey, why don't we tokenize motions to make them like language? It turns out it's a very effective idea. Now, it models that architecture, which is very LLM inspired. It models future interactions of agents in the environment very well. Like you can think of agents talk to each other with these motions they execute simultaneously in an environment. And now you can leverage the machinery. We have this paper. It's quite effective, right? So that's an example of this. Now, one other interesting point, though, is text is its own simulator.

• 32:08Speaker 04

  Essentially, you know, you speak text to each other, that's the full environment. You spit out text tokens, text tokens, and text tokens. In our case, we, well... We predict actions, we execute actions. Imagine now, but you need a simulator because now based on these actions, you need to envision what the whole environment looks like and how your whatever hundreds of millions to billions of sensor points look like. So now you need something that generates them as you act so you can test yourself how you behave over time. As you make decisions at a fairly high frequency, then there is a known problem which is called covariant shift. Essentially, decisions can take you to places you may not have seen before in the data.

• 32:52Speaker 04

  And there you may have particularly failure things that you may not observe unless you push yourself and drive on policy to those places in the data. But to drive there now you need the simulator. The simulator needs to be realistic enough where, you know, you don't go somewhere else entirely as opposed to the actual place you will end up with decision making. So that's another very interesting point, like simulation is hard. If you want robust testing, simply having drivers on the road is not a particularly scalable solution if you want to keep it a righty on your stack because some of the events happen once in a million miles or more. And you would much rather test them in the simulator, but for the simulator now you have to solve this problem, which is interesting and challenging. So that's unique in our domain.

• 33:50Speaker 01

  What if AI agents could work together just like developers do? That's exactly what AGENCY is making possible. Spelled A-G-N-T-C-Y, AGENCY is now an open source collective under the Linux Foundation building the internet of agents. This is a global collaboration layer where the AI agents can discover each other, connect, and execute multi-agent workflows across any framework. Everything engineers need to build and deploy multi-agent software is now available to anyone building on AGENCY, including trusted identity and access management, open standards for agent discovery, agent-to-agent Communication protocols and modular pieces you can remix for scalable systems.

• 34:35Speaker 01

  This is a true collaboration from Cisco, Dell, Google Cloud, Red Hat, Oracle, and more than 75 other companies, all contributing to the next gen AI stack. The code, the specs, the services, they're dropping, no strings attached. Visit agntcy.org, that's A G N T C Y dot org to learn more and get involved. Again, that's agency, A G N T C Y dot org.

• 35:07Speaker 02

  Well, Drago, I'm really intrigued by how you kind of helped me form a mental model for the types of problems that you're, that are part of the research in this area. I would definitely encourage our listeners to go check out waymo.com slash research. There's a bunch of papers there that people can find and, you know, read. But also there's even Waymo Open Data Set, which supports research and autonomous driving. So that's really cool to see. It's amazing. I'm wondering, Drago, as you look at this kind of, I see all sorts of things from, you know, scene editing to forecasting and planning to, you know.

• 35:50Speaker 04

  Did I mention you need to embody the agents in the simulator too? They're not deterministic.

• 35:55Speaker 02

  Oh, yeah.

• 35:56Speaker 04

  If you start doing different things, you need to, well, guide the agents to react to you in reasonable ways as well. Otherwise, you know, they'll be reacting to an empty spot where you're no longer, even if you collected the situation with your sensors, as you start deviating, committing the simulator, you still need the agents to do reasonable things, right?

• 36:15Speaker 02

  Yeah, yeah. Yeah, yeah, that makes sense. And I guess that really kind of gets to my question a little bit, which is, I assume over the last five years, as we haven't chatted, there's been. A lot of progress in certain areas and maybe certain challenges that are kind of holdouts that remain very, very challenging and maybe not as much progress is made. So in this kind of autonomous driving research world, can you paint in broad strokes kind of where there has been very rapid progress as things have advanced and maybe some of those of the like the hardest problems to solve that still remain kind of at arm's length, if you will?

• 36:59Speaker 04

  I mean, I would say one thing for folks that especially are closer to robotics, they will see, just like the field of AI is going through some crazy inflection point of, I mean, both methods, people develop and popularity. I think the same is true in robotics and the same is true in AV. I've been in the space over 10 years now, just doing AVs. And I would say every couple of years, our capabilities with AI and machine learning dramatically expand due to innovations. And this innovation train has not stopped. So like where we are five years later compared to five years before in terms of modeling,

• 37:41Speaker 04

  I think is still a huge improvements possible. I think, I think we're moving more and more to machine learn power stacks. And I think ultimately, understanding how to leverage, I mean, data-driven, right, elegantly, scalably handle this problem with data-driven solutions. And so that's been generally an evolution. And I think we understand how models behave better. I think these latest architectures and the scaling that we mentioned is a really interesting domain. We started studying it, for example, for a while back. So there's this paper, we have, for example, of scaling laws of motion L M architectures. It's an

• 38:25Speaker 04

  L L M-like architecture. So you say, oh, what are its scaling laws? How does it compare to L L Ms? We have a tech report on this, for example, still similar kind of learnings transfer as L L Ms. But there's some bespoke, really interesting things. For example, for that architecture, improving what's called open loop prediction performance seems to correlate to improving closed loop performance. That's not always true. Right? And we see different scaling factors compared to language like our motion space is nowhere nearest diverse as language tokens. Right? So we need actually for the same set of parameters model, we need a lot more data of examples of how the world behaves to scale. These are interesting findings generally, right?

• 39:09Speaker 04

  But so that's one. I think, I think now as the architectures keep evolving, now there's diffusion and auto-regressive models and now how do each compare and how do they compare in open loop and closed loop? These are all very interesting areas people are studying. I think generally there's this question lately as well of how do you build the best simulator with machine learning and what kind of models are there? And you know, most recently, there's some groundbreaking work like the Gini model by Google. I don't know if you guys saw it. It's a controllable video essentially. You can like give motion controls and it dreams the video close to real time of what it should look like.

• 39:54Speaker 04

  So essentially you're, you're controlling the world you're imagining a bit. Right? And you can do this in real time or you can do it, of course, off board to or offline with even larger potentially models. And so now these models are pre-trained on a large amount of video and text. And so they capture a lot of knowledge of how the real world behaves and it's a more complements the knowledge that vision language models capture from the internet corpses. And so how do these two relate? How do you mix them? Right? Which one is beneficial for which type of tasks? These are all interesting capabilities that people are doing.

• 40:38Speaker 04

  And maybe one other interesting topic is there's a lot of talk about architectures for robots that there's some combination of system two and system one architecture. So you guys may have heard it, right? Now, we know that large models are more capable when trained on more data and more compute, but in latency sensitive situations, you can't, if they're too big, you can't run them in real time. So now the question is, okay, well, what if you have a real time model that handles most cases, but then you have a slower model that does better high level reasoning, that runs at some slower hertz, that helps guide and understand additionally and provide

• 41:20Speaker 04

  this to the fast model well needed while still keeping this reflexive capability. If someone jumps in front of you, you still respond, right? Like, these are interesting questions in our domain as well. So there's many, actually, it's a really, really fascinating time, and I think we're studying a lot of these questions, just as the whole field is, and we have some very interesting findings, some of them not published yet. Generally, I would encourage people, come join us, you can, well, you know, contribute to the premier embodiment of physical AI currently out there, and you can do interesting research, right?

• 41:58Speaker 03

  Sounds like fun.

• 42:00Speaker 04

  But, yes, these are all fascinating topics. And of course, how to control hallucinations in all these models? How do you determine when these models are out of domain and potentially making clear mistakes, right? This can happen. We have research experience with VLMs like, you know, like the many of the current ones, but we have a paper called Emma, where we tried to fine tune VLM for driving task, got a bunch of learnings. It can be quite good, but it has limitations too, right? So how do you overcome these limitations? With additional system design is very interesting.

• 42:41Speaker 03

  I'm curious, as we're talking about this, and just, I'm really enjoying the conversation, and I work for another company in autonomy, but in a slightly different context, and I'm curious, one of the things that is popular in the industry I'm in right now is solving for swarming behaviors. As you're talking about many autonomous vehicles that are having to collaborate in certain ways, I'm curious from your take, that may or may not be an interesting problem for Waymo, I don't know what your thinking is on that, but I would love to know, when you look at that space, what are some of the things that you think about and are interesting to you about the notion of many autonomous vehicles collaborating together?

• 43:28Speaker 04

  That's been a very interesting area that actually there was early research that I was impressed with, where people proved that if you can control groups of vehicles, you can improve traffic flow. So to me, we're not exactly swarming yet autonomous vehicles. They're still a subset, a relatively small subset of the whole traffic. So it's mostly when I think of swarming, I imagine, say, a crowd of two hundred people on Halloween all around the car and stuff like this. That's swarming, or you go to downtown after, you know, a Giants game, and they're exiting, and that is swarming, right? They're the the human agents, so to speak, more prone these days to swarming than AVs still.

• 44:12Speaker 04

  Maybe we'll get more prominent. I think when you think of coordinating multiple AVs, in our domain already, they do send each other valuable information. For example, if one of our vehicles encounters some very complex construction, it can help pass information about it to the others. If we encounter potentially, you know, slowdowns or vehicles getting stuck, that kind of information can be passed. I think controlling jointly vehicles is, it started becoming interesting now that we're getting to some kind of scale. I think one of the interesting domains where this is interesting is when you want to charge them. So imagine you need to charge now hundreds of vehicles in a location.

• 44:55Speaker 04

  How do you control all these vehicles so that they all get to the right place and don't block each other, and it's all very efficient? That's one example of where you're fairly swarmed. It's your own warehouse, right, or a garage, where this comes up. And then down the line, potentially, there is opportunities to improve traffic flow for everyone, but that's still maybe in the future.

• 45:22Speaker 02

  Well, you took us right there, Drago, as we're kind of getting close to an end here. I'd love to talk about that future. And we were talking beforehand and I was saying, I'd love for you to share just what you're excited about. And that could be, that could be, of course, in general, related to driverless research. It could be kind of in the AI ecosystem generally, something that you're excited about as you look forward to or are thinking about a lot. Does anything stand out so that we can ask you about it? Hopefully not in five years from now, but maybe, maybe the next time you're on in less than five years, we can ask you, ask you about it.

• 46:01Speaker 04

  Sounds good. Well, I'm around, so I could come probably faster than in five years' time.

• 46:07Speaker 02

  In a way, yeah.

• 46:09Speaker 04

  Potentially, yes. I think maybe let's go in a couple areas. First, maybe as to parallel this chat we had earlier, maybe first about the product and then a bit about the AI. I think in terms of the product, I'm excited, in a way, with the safety studies we've shown, these are significant improvements over the baseline. And I think we've shown it already at scale with some fairly, starts to become fairly good confidence or some statistical significance at this point. And maybe your listeners, I'm not sure they understand, but even just on the US roads

• 46:52Speaker 04

  alone, I'm not talking world roads, US roads, 40,000 people die every year from accidents. That's a lot. And I think these gains are starting to become somewhat meaningful. So it starts becoming thinking, hey, maybe we have a mandate to expand. You're like, we should be expanding. It will save people's lives. Right, and you think about it, and then the question is, how can I contribute to expanding? I mean, ignore all the, of course, it's a, I believe, it's a great service. A lot of people love it for a lot of good reasons. We could potentially go into some reasons people found why they love it, right? But like, I think even just from the, from the mandate, okay, you know,

• 47:36Speaker 04

  it's helping in meaningful way. And I think being out there can make quite a dent against some of these numbers. And so, yes, I would love it to expand more. Now, we're doing that. I think to me then the question is, what can I do to contribute to it, right? And I think one of the most scalable solutions to tackling dozens of new cities and, you know, conditions and countries is machine learning and AI, right? And so now, for me, what I'm excited about is harness all the positive latest trends. I think for me, more directly first into the Weimar Foundation model work we're doing, where we can directly experiment and deploy them,

• 48:18Speaker 04

  and then try to push more and more of them to, to contribute similar benefits and to, to the main production systems, which is the onboard driver and the simulator, right? So that's what I think about. Now, more specifically, if you want to go into AI techniques, I think this question of, okay, how do I endow vision language models with more modalities? Right, is a fascinating one. We actually have some good results already. Like, how do you, how do you expand to new modalities? Say, right, light and radar? How do you connect it to actions, the model?

• 49:00Speaker 04

  What's an effective way to do this while preserving all the world knowledge that's present in the model that you're trying to build on top of? Is an interesting, a model and system design challenge. And then, what I'm also excited is building the simulator. I think, right, as realistic, as scalable as possible. I think the modern technologies, like the genie model that I mentioned, these world models that are still relatively few and far between, but I think it's a ton of labs are working on them today. I think taking that kind of technology and build the most generalizable possible simulator with it.

• 49:42Speaker 04

  I think is fascinating. Now, the interesting thing is you could do that, but it can still be very expensive to run. So you still need to show, it's not just enough to show that it can handle very like realistic, interesting cases. You still need to show how you can run it without breaking the bank. The amount of simulation Waymo does today to ensure that we're safe, we run like millions of virtual miles every day. That's a lot of things to simulate potentially with, you know, so many sensors on board and so on. So there needs to be, there's some very interesting question in that space. How do I get the maximum possible simulator realism? And how do we get the maximum possible scalable simulator?

• 50:26Speaker 04

  And there's a very interesting mix of technologies getting involved to do that.

• 50:30Speaker 02

  That's awesome. Well, I certainly, I'm certainly excited about that. Like I say, I encourage our listeners to check out Waymo's research page. Lots of amazing stuff to explore there.

• 50:41Speaker 04

  And folks can see our history, right? Like I think you can see the kind of work and papers people did from, I think, 2019 to now. And there's almost 100 papers there now. And maybe it's not 100, only because we may not have uploaded the most recent ones. I'll try to make sure we do soon if we're missing any. So if the readers go there, they can see the full, the full set.

• 51:05Speaker 02

  That, that sounds great. Well, thank you for joining us again, Drago. It was a real pleasure to have you on the show again. And let's, let's, uh, let's not make it five years next time. We'll, we'll try to get you on and, and hear the update sooner than that, for sure.

• 51:19Speaker 03

  Don't be a stranger.

• 51:20Speaker 04

  Thank you, guys. A pleasure to be on the show.

• 51:30Speaker 00

  All right. That's our show for this week. If you haven't checked out our website, head to practicalai.fm and be sure to connect with us on LinkedIn, X, or Blue Sky. You'll see us posting insights related to the latest AI developments, and we would love for you to join the conversation. Thanks to our partner Prediction Guard for providing operational support for the show. Check them out at predictionguard.com. Also, thanks to Brake Master Cylinder for the beats and to you for listening. That's all for now, but you'll hear from us again next week.