Getting Waymo into autonomous driving

Waymo’s mission is to make it safe and easy for people and things to get where they’re going. After describing the state of the industry, Drago Anguelov - Principal Scientist and Head of Research at Waymo - takes us on a deep dive into the world of AI-powered autonomous driving. Starting with Waymo’s approach to autonomous driving, Drago then delights Daniel and Chris with a tour of the algorithmic tools in the autonomy toolbox. Sponsors: Linode https://linode.com/changelog – Our cloud of choice and the home of Changelog.com. Deploy a fast, efficient, native SSD cloud server for only $5/month. Get 4 months free using the code changelog2019 OR changelog2020. To learn more and get started head to linode.com/changelog https://linode.com/changelog. Fastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship. Rollbar https://rollbar.com/changelog – We move fast and fix things because of Rollbar. Resolve errors in minutes. Deploy with confidence. Learn more at rollbar.com/changelog https://rollbar.com/changelog. Featuring: Drago Anguelov – LinkedIn https://www.linkedin.com/in/dragomiranguelovChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Dragomir Anguelov - Google Scholar https://scholar.google.com/citations?user=T04c3fwAAAAJDrago Anguelov – Machine Learning for Autonomous Driving at Scale https://youtu.be/BV4EXwlb3yoWaymo https://waymo.comWaymo - Twitter https://twitter.com/waymoWaymo - LinkedIn https://www.linkedin.com/company/waymoUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  So if you look at generally the stacks of modern autonomous driving companies, a large part of them use all three sensors or a subset, which is camera, radar and radar. Some also use thermal cameras, occasionally even more creative sensors, sonar and so on. But those three are the mainstay and I think each of them brings something really unique and powerful to the table.

• 0:25Speaker 01

  Being with your change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at Change Law because of Rollbar. Check them out at rollbar.com. And we're hosted on Linode cloud servers. Head to linode.com/changelog.

• 0:43Speaker 02

  Linode is our cloud server choice. Grab the Nano plan for just $5 a month, just five bucks. That gets you a gig of RAM, a blazing fast 25 gig SSD and one terabyte of transfer. Let's be honest, you can go a long ways on that five bucks. When you do need to scale up, their prices are predictable. So you can put your calculator down, you won't need it. We've been running change.com on Linode for years and we've always impressed by their award winning support team. Check them out at linode.com/changelog. Once again, that's linode.com/changelog.

• 1:24Speaker 02

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and Slack with us around various topics of the show at change.com/community. And follow us on Twitter. We're at Practical AI FM.

• 1:50Chris Benson?

  Welcome to another episode of the Practical AI Podcast. Hopefully today, we're going to have some fun stuff that we haven't covered before. My name is Chris Benson. I work at Lockheed Martin and with me is Daniel White, an Accad data scientist at S I L International. How's it going today, Daniel?

• 2:06Daniel Whitenack?

  Oh, it's going great. It was an interesting weekend. I actually got to get out of the house, which was a good thing, you know, during COVID. My pastor and his wife took us out on a, they have a little boat and it was quite an adventurous time because we went, we got to get outside. It was beautiful, but then the boat broke down and we had to like get a tow back to the dock and all that. So it was an interesting weekend. Nice, like, you know, away from the computer, out in nature, but it was an adventure. So what about yourself?

• 2:38Chris Benson?

  I think yours was much more adventurous than mine. We managed to adopt out one of our foster dogs. So we're down to only seven dogs now in the house. Only seven dogs. Went into the weekend with eight, so that's good. And so that's always a nice little family weekend in that way.

• 2:57Daniel Whitenack?

  Yeah, yeah, for sure. Now back to some fun AI stuff. I've got a few things going. Actually, I got a new project I'm working on, which hopefully I can, right now it's a, I can't share everything about it, but hopefully I'll be able to share some soon. It's, it's speech related and it's some pretty fun stuff with local languages here at S I L. So hopefully I'll be able to share that at some point. But I know today we're doing something with, you know, I was just talking before the show that after even 100 or more episodes, there's still things on the show, so many interesting topics going on with AI that we haven't had full shows devoted to certain things. And we've got one of those today, right?

• 3:38Chris Benson?

  And it's a big one as well. I'm just, you know, I was kind of shocked. How did, yeah, over 100 episodes and we missed this one? This is insane.

• 3:46Daniel Whitenack?

  Yeah.

• 3:46Chris Benson?

  And so today, wanted to welcome on to the show, if I get your name correct, it's Drago Anguelov? Did I come anywhere close there?

• 3:55Speaker 00

  Sounds good, actually. Thank you.

• 3:56Daniel Whitenack?

  Yeah, that was quite good. Yeah.

• 3:58Speaker 00

  Thanks for having me.

• 3:59Chris Benson?

  You are a principal scientist and head of research at Waymo. And we're going to talk a bit today about autonomy, I believe. Is that correct?

• 4:08Speaker 00

  Sounds like a good topic for me. I enjoy it.

• 4:11Chris Benson?

  Topic I love too. It's and one that I touch in my work life and so can't believe we haven't talked about it on the show in the depth it deserves.

• 4:19Daniel Whitenack?

  Yeah, Chris is gonna nerd out today quite a bit.

• 4:22Chris Benson?

  Oh, yeah, this, this is great.

• 4:24Daniel Whitenack?

  We've been talking about NLP too much. We need a break and.

• 4:28Speaker 00

  It's a very quickly developing area with a lot of great technology coming out. So it's not surprising.

• 4:34Daniel Whitenack?

  Yeah.

• 4:34Speaker 00

  And some good systems too.

• 4:36Chris Benson?

  Yeah, he's just schooling me on NLP on every episode. So I don't know about that.

• 4:41Speaker 00

  How many GPT three episodes have you had so far?

• 4:43Daniel Whitenack?

  Well, I think it's come up in a number of them. We had hugging face on not too long ago. So that was a good like transformer episode. Yeah, it is. Like you say, I'm a little bit biased in that way, but I'm excited. It's so cool to also see, like I mentioned at the beginning, I'm starting to work on some speech stuff. Because it works a lot with Mel spectrograms and other things, there's, there's a lot of overlap with sort of computer vision. Types of models. And so it's cool for me to kind of dip back into some of that stuff and it'll be awesome to discuss some of the computer vision things today.

• 5:19Chris Benson?

  You know, Daniel was mentioning they couldn't talk about it. I'm suspicious. Maybe he's gonna release GPT four for the world.

• 5:25Daniel Whitenack?

  Yeah, that must be it.

• 5:26Chris Benson?

  Maybe that's next week's episode, right?

• 5:29Daniel Whitenack?

  With all of my GPU clusters in my house here.

• 5:33Speaker 00

  Right, right behind you. Yes, in the picture.

• 5:35Daniel Whitenack?

  Yeah, exactly.

• 5:36Chris Benson?

  Yeah. So for listeners, since you don't have the visual, we're on the Zoom call.

• 5:39Daniel Whitenack?

  I think we've mentioned this a couple of times.

• 5:41Chris Benson?

  Yeah, we have, but Daniel always has the data center in the background behind him there. And so, you know, I think I'd be shocked to see something more domestic there, you know? I just figure you go home and go to bed in the data center, get the air conditioner going.

• 5:53Daniel Whitenack?

  Well, before we go too deep down the rabbit hole of virtual Zoom backgrounds, Drago, could you give us a little bit, maybe just about your background? And how you got into self-driving cars and that whole world? What's your background and how did you get into AI?

• 6:07Speaker 00

  I'm by nature, a machine learning researcher and engineer. That's how I think of myself.

• 6:15Speaker 00

  And when I started learning about those fields, I always was drawn to robotics. And as such, I ended up doing a PhD at Stanford in machine learning, but with a focus on perception, 3D perception at the time. I was actually Daphne Koller's first PhD student that was focusing on perception. And as such, I had a very interesting overlap with Professor Sebastian Thrun, because Daphne met him at a conference and sent me to his lab, because we had no robots or any way to collect sensor data. And Sebastian did. Daphne sent me to Pittsburgh to collect some data with his robots. And that was my first more serious exposure to robotics.

• 6:59Speaker 00

  What really happened, though, was I spent three weeks in Pittsburgh. But unfortunately, just at the time when I went there, Sebastian's whole team got to go to Hawaii. I think something that he organized.

• 7:10Chris Benson?

  So you were left alone with the robots?

• 7:12Speaker 00

  I was left alone with the robots. And since I'm primarily a software guy, I managed to short the main robot with which I was supposed to collect the data in fairly short order. So it was interesting times. Not too much data got collected. But we were lucky for Sebastian to come to Stanford. And so he became an official advisor of sorts. And I got exposed to a lot of his work. It took me a while to get into self-driving, but I had been exposed to robotics and concepts there since 2000, I would say. And slowly but surely, my path led into this field.

• 7:47Chris Benson?

  So can you tell us a little bit about going from Stanford and getting to Waymo? And what some of those steps that kind of led you there along the way?

• 7:57Speaker 00

  I initially focused more on computer vision and 3D perception. That was my bread and butter. That was a great field that clearly would be empowered by machine learning. Originally, in those times, deep learning was not really that popular. Neural nets were considered to be the second best thing to do anything, as they said in the Stanford AI lab. It was just a saying. People were just kind of jokingly saying that. And probably genetic engineering was the third best way of doing anything, right? And we did a lot of graphical models. I was lucky to have a great advisor, Daphne Koller. That was my primary advisor. And she's a very accomplished professor with a lot of seminal work in graphical models. And those were the models at the time that actually worked.

• 8:42Speaker 00

  And so I did a lot of work in that. But when I graduated Stanford, I felt that I would go into a startup. And so I spent two years in a computer vision startup, Ria, and later was known as like dot com, where we did face recognition in photo albums and we did visual search for shopping. Eventually, though, it became too much of a website and the computer vision became more of a second-class citizen and I felt that it might be a great idea to join Google. And so I spent eight years at Google, never in self-driving at the time, but in areas that are very, very closely related. So one of them was Street View.

• 9:25Speaker 00

  At Street View, I learned a lot about nonlinearly squares and feature matching and bundle adjustment, structure from motion, SLAM. I had to because I was tech lead for pose estimation and 3D vision at Street View. So that was the field I really got to exercise a lot. After doing quite a lot of very interesting work at Street View, which culminated in 2010, we did actually this, I don't know how much you guys follow this type of the literature, loop closing of the world, which is you took all the poses of all trajectories Street View cars ever drove. And we built a system with a guy called Samira Garwal, who later developed a very popular software called Seres to

• 10:09Speaker 00

  register all of these runs of all the cars using the lidar sensor data. And later we tried the cameras, compute constraints between them and bring them into joint global coordinate system. So after we accomplished this for all of Street View data, I felt, okay, it's time to go back to ML.

• 10:27Daniel Whitenack?

  You you literally built the world.

• 10:30Speaker 00

  We loop closed the world, I would say.

• 10:33Speaker 00

  Right? And after loop closes, you have all this sensor data, you can really map it well.

• 10:37Daniel Whitenack?

  Yeah.

• 10:37Speaker 00

  And afterwards, I switched to ML again, to a more experimental, researchy team, which ended up in Google Research eventually. And what we did was initially Google Goggles, which was not the glass, which everyone knows, but it used to be a little app in your phone that you can take pictures and it would tell you what the pictures are of. And at the time, it didn't work too well, but it pioneered a lot of the core concepts and even technologies. In fact, In around 2012, we were already doing quite serious deep nets at the time, only they weren't working as well because the neural net architectures were not very good. And then came AlexNet, and it was much better than anything we had that was not deep net based,

• 11:20Speaker 00

  as most people in the field also found out. And we found it very early. And so we switched to deep learning and we developed a lot of this early neural net architectures for detection and classification that became quite popular afterwards. And we started working on annotating Google Photos, so I worked a lot on that. This is large scale image recognition and detection. And a lot of these skills are also very relevant for self-driving. And around 2015, I said, okay, this perception things, we're starting to get the hang of it, at least for cameras, at least for images. It probably is finally time to go and do self-driving. And so in 2015, I got involved in self-driving, and so I'm here today.

• 12:03Daniel Whitenack?

  Gotcha. This is really the first focused episode we've had on self-driving. And it is one of those things that when people think of AI, one of the first things that come to mind is, oh, you know, self-driving is an example of applications of AI. Maybe you could just give us a sort of state of self-driving cars. In terms of like, for people looking from the outside, they might have a variety of perceptions and expectations all the way from, oh yeah, we have self-driving cars now that work great, to I'm not convinced that any car could self-drive. You know, where are we at, just sort of generally? What's the sort of state of where we're at with self-driving cars in industry?

• 12:44Speaker 00

  Overall, I feel the industry has made a lot of progress and this progress is accelerating. At Waymo, we have been at this for over 10 years now. And even in 2010, 2011, when I was not directly with Waymo, they still had a lot of awesome demos and very impressive driving and behaviors that they were doing there. Right? It's just the level and the complexity of what you need to do to launch safely, fully driverless product is unlike most products that have ever been launched. And so it takes a lot of time and iteration and understanding of all the details and corner cases. And so at Waymo, we've been working on improving the systems.

• 13:27Speaker 00

  And we've been steadily improving. So we're now giving thousands of rides a week in Phoenix metro area. And before COVID, five to 10% of them were fully autonomous. This is quite a large area and quite general driving that we support there. So that to me is something that people don't talk as much about, but to me is a tremendous accomplishment already. And we're quite good currently at taking a specific area of interest, going out there and learning how to drive it. A lot of work, though, needs to go into making this system really scale to the variety of areas, diversity of situations,

• 14:12Speaker 00

  and even conditions into types of intersections that you see, to the long tail of scenarios, to the weather, right? And you want a system that scales quite seamlessly the more you give it data. And this machine learning has a very, very big role to play. And to me, that's an exciting multiplier. And so this will help bring this technology to much more areas further.

• 14:39Chris Benson?

  You've kind of alluded to it and stuff, but as you're looking at this field, and especially as it's grown over the last few years and there are new entrants and it's maturing, what are those components that make up the field? What are all the kinds of areas of concern that you have to address when you're doing this kind of work in autonomy?

• 14:59Speaker 00

  There is actually two levels of, or two areas in which work needs to be done. And I would split it into technological area and product and policy, right? And maybe it's easier to, if we're starting from a broader and then starting to narrow, maybe the latter is a good thing to describe.

• 15:16Chris Benson?

  Sure.

• 15:18Speaker 00

  An autonomous driving product has not been created in the past, right? And the legislation for it is work in progress. And so this is an area where, you know, all the companies are focused a lot, working with governments at all levels and city councils and so on, to make sure that this product is developed. To really serve the needs of the people that are using it, that they really work for the various interests and needs in the specific communities, that it has the right features, because it's a different nature product than a car with a driver. And so this all this area around compliance, regulation, product details is something that requires more work. Also, just to build the hardware at scale

• 16:03Speaker 00

  is a non-trivial effort that needs to be set up so that you can produce these vehicles at a good price point. Automotive companies have existed for over 100 years perfection and they've been perfecting the how to build vehicles efficiently, how to put these boxes together and integrate all the components. Now we have a lot more components. There is all the sensors and the compute and the safety features that we put into the vehicles that we need to learn to integrate efficiently. So that's on the policy side, the non-technological side. On the technological side, there is a lot of interesting challenges with regards to, indeed, how do you handle the long tail? The rare events, how do you get to be robust to them?

• 16:47Speaker 00

  How do you efficiently, having said learned to drive in Phoenix, how do you easily adapt all of this stack to a new area and prove to yourself that you're safe there? Right? How do you efficiently handle rare events in new areas and weather? Because each weather and condition leads to different behaviors and performance of both of your sensors and potentially of the other actors that react to the conditions that you need to be adapting to. So all of this is very fruitful areas for research. And I think this is all of these areas in technology. There is a lot of great base and there is a lot of great robotics and probabilistic robotics of the kind that Sebastian Thrun used to do,

• 17:32Speaker 00

  modeling that has gone in it. What's really changing this area a lot is the advent of machine learning and its rapid development, which makes more and more things possible. And most of the companies, I think, we're reinventing our stack as we go as these new capabilities that become possible, a lot of them we try to create ourselves with our research. This makes your system more and more adaptable. Right? And right now, machine learning, the true successes, I mean, already for over five years, have been a lot in supervised learning.

• 18:06Speaker 00

  So you have a reasonable way to design these neural net architectures for specific tasks. They're universal function approximators, but it helps to have a good architecture. Right? And when you have this good architecture and you define a good supervised problem and you label all the data you ever wanted, which may be hundreds of millions of boxes, we're really great.

• 18:27Speaker 00

  But that doesn't scale very well, right? And so there's a lot of new paradigms over the last few years that have become very popular in how to make each step of this progress a lot more seamless and a lot more automatic. And when I got into this kind of research role, I kept dreaming like, what's the dream system that I want to build? And to me always has been, okay, I want to go to a new city. We would just deploy a bunch of vehicles there with our drivers. We will tell them how we want them to drive so that they're, you know, way more like driving. Then we collect data maybe for months and we just take all this data and we put it in our systems and they just improve and adapt to the city. And that can be in terms of perception, it can be in terms of behavior prediction,

• 19:12Speaker 00

  it can be in terms of better planning, right? By observing how everyone does everything. And hopefully you're almost all the way there. Of course, that's not yet necessarily realizable that just with a system that just observes you can go all the way. Of course, there's always in a serious domain like this, a lot of room for great engineering and modeling as well to complement it. So it's usually an amalgamation of these two approaches. But you want also the first part to be possible because that really makes your system global. And so I think a lot of our projects are motivated by this vision. How do you take a piece that maybe is, you know, has some heuristic or some too much assumptions that may not generalize to a very different environment?

• 19:56Speaker 00

  And how do we relax them with ML such that the system becomes able to handle a larger diversity of situations without people necessarily going in and coding it all?

• 20:23Speaker 02

  ChangeLog News is the best way to keep up with the fast-moving software world. We track, blog, and contextualize the coolest projects, the best practices, and the biggest stories each and every week. Make change log.com your daily destination, or hit the snooze button and subscribe to our weekly newsletter that hits inboxes on Sunday mornings. Join more than 15,000 enthusiastic readers, it'll cost you exactly $0, and you can subscribe right now at change log.com/weekly.

• 21:05Daniel Whitenack?

  So you started getting into the data side of things a second ago in terms of collecting certain types of data and you know what was needed. Maybe we could just kind of start breaking down the basic problem in terms of what data you're collecting, and then you know, we could move on to talk about the various modular components, like you're saying, that drive the system. So in a Waymo self-driving car at least, or maybe it's kind of standardized now, I'm not sure, but what are the main data inputs that you're working off of? I'm assuming cameras, but you mentioned LiDAR, I think. What's the combination of sensors that you're working with?

• 21:42Speaker 00

  So if you look at generally the stacks of modern autonomous driving companies, a large part of them use all three sensors, or a subset, which is camera, LiDAR, and radar. Some also use thermal cameras, occasionally even more creative sensors, sonar and so on, but those three are the mainstay. And I think each of them brings something really unique and powerful to the table.

• 22:10Chris Benson?

  For anyone who doesn't know what LiDAR is, because that tends to be used in this particular area, could you also just quickly say what that is for those that are not familiar with LiDAR? I think everyone knows radar, everyone knows cameras, but maybe not everyone knows LiDAR.

• 22:24Speaker 00

  So LiDAR stands for Light Detection and Ranging. The LiDAR is a laser unit that sends out a laser pulse that then gets returned from the environment and you can measure the time travel and potentially add the attributes of the signal like the intensity of the object that you measured. And so it gives a very accurate depth estimate to objects. Of course, radar also does give you depth estimate to objects. Radar has antenna that send out RF signal and then the signals come back. But the radar signal is not quite as localized and it's lower typically resolution.

• 23:05Speaker 00

  LiDAR is able to, especially the high-end LiDAR, is able to image the scene at very high resolutions with singles. So usually if the LiDAR is spinning or they're imaging LiDAR, depending currently there's different kinds, you can have up to a million returns in a spin of this or in this range image that it forms for you. And that's extremely rich description of the shape of all the objects and also their reflectance property.

• 23:32Chris Benson?

  So if you're with Waymo or any of these other companies with their stacks that all have kind of these common three sensors in, in a generalized way, what is the advantage of camera? What is the advantage of LiDAR? What is the advantage of radar? How do you choose to apply the input you're getting from each of these types of sensors to form, I don't know what you would call it, but your operating of your vehicle? How do they each contribute optimally to the overall picture there?

• 24:01Speaker 00

  So let me say first that now that I'm at Waymo, I'm just amazed by the quality of our sensors. Waymo, one of the strengths of Waymo among many, and we can talk about them. Is that Waymo has been already existing for 10 years. And it builds both its own sensor hardware and the software that then uses the sensor data. And so the sensors are top of the industry, I believe. And the setup is built such that both the hardware and software engineers can work together on the requirements that contribute to the operation domain requirements. And so that is, you know, a privilege.

• 24:47Speaker 00

  Now, one caveat is that for each different self-driving company, the balance of these sensors is a bit different because everyone uses, some of these have somewhat different characteristics, right? But overall, now let's generalize a bit. Camera is a standard sensor, very popular, it's very mature. It's the sensor that comes in the highest resolution, it has the highest amount of information. It's a passive sensor. So traditionally, we knew that all the information there in the camera is great to use, but before deep learning could not really make, take into use a lot of it for two reasons. Not only that the models themselves did not exist, but also the compute to really effectively process these large images did not really exist.

• 25:33Speaker 00

  And so now that's no longer true. So camera is an extremely powerful sensor and you can do a lot with cameras. LiDAR became potentially historically over the last 10 years, the most popular sensor for self-driving. It is an unparalleled sensor in the sense that it combines really accurate dense depth and it's an active sensor, which is important because it has additional benefit for safety. And honestly, if you start comparing the performance of these sensors, especially in areas of some reasonable proximity to the vehicle, which can be up to 100 meters, in terms of accuracy that you can obtain for your object modeling, at least at a geometric level, LiDAR dominates still, right?

• 26:17Speaker 00

  And radar is also very popular for other reasons. It's really a great sensor for weather because the RF waves can travel reasonably well, they don't degrade as much as LiDAR and potentially camera in fog and rain.

• 26:36Speaker 00

  Also radar is very complementary to cameras. Because cameras, they're really great at semantics and understanding what is there qualitatively. They're really good at estimating the relative position, the lateral position of things and the lateral motion. But they're really not so good at estimating depth or velocity in the longitudinal relative to the ray in which you observe the object. And that's exactly what radar is good at, because what radar does is, especially for metal objects, it measures super well the distance to it. And it also directly measures velocity.

• 27:14Speaker 00

  Which for camera, it's a very derivative concept, which is yet more inaccurate. Radar really decreases the uncertainty in this measurement. So it really complements the strengths of cameras. And so that is a good property. Also, radars are very cheap. But they also have their shortcomings, things like resolution. And especially vertical resolution. Often you need a lot of antennas to get vertical resolution. So things like overpasses and so on, you cannot tell if the object is potentially, especially for static objects above you or at your level, given the vertical resolution, and that's problematic, right? And this is something that radar, for example, can do well. And so each of these sensors, when put together, there is a case where one more sensor is good.

• 28:00Speaker 00

  Right? And that's how at Waymo we have arrived to this combination of the three. And of course, ideally you would say, "Oh, isn't it great to just have one sensor? What simplicity?" But if you think about safety and ability to generalize to all conditions, well, these three historically have proven very powerful together. And so Waymo's stack has evolved like this, and a lot of the main manufacturers as well of self-driving systems.

• 28:28Daniel Whitenack?

  So before we jump into the actual models that are processing this data, it strikes me as we're talking about all of this, we have these various rich sources of data. And those are being combined, and I'm guessing pre-processed in various ways. And you're also running, in my understanding, a variety of models in the unit, in the car. What sort of compute is available to you, like in the? Car itself to be able to, you know, respond in real time. What are your constraints compute wise and power wise and all of those things? I would think, you know, running in the car. Of course, it's not like a drone, maybe where you're in the air and waits like a huge.

• 29:13Daniel Whitenack?

  I mean, it's still an issue, but it may not be as huge of an issue. But what are the constraints there and what sort of hardware are you working with? Do you have to have accelerators at the edge or what? What are you working with?

• 29:23Speaker 00

  So let me just say a few things. I don't think I can really disclose the details of the Waymo compute in the car.

• 29:31Daniel Whitenack?

  And that's fine. Yeah, we're not surprised.

• 29:33Speaker 00

  All right, but I can describe roughly the considerations when you think about this.

• 29:37Daniel Whitenack?

  That's great.

• 29:38Speaker 00

  First of all, outdoors, you need to understand often objects up to quite a distant range.

• 29:46Daniel Whitenack?

  Yeah.

• 29:46Speaker 00

  Where maybe 300 meters in front of you. If you have a truck, you need to detect a cone or an obstacle that will may cause you to start slowing down. Right? And so you need to understand things over fairly long distances and also quite crowded scenes. If you're in downtown San Francisco, I live in San Francisco, you go down, there could be hundreds of different people and dozens of vehicles all around you. And you need to make sense of the scene and be able to ultimately anticipate which parts of it are relevant for you and make decisions. This results in fairly large requirement for compute compared to most systems.

• 30:28Speaker 00

  Also, as we want to run more and more deep networks, and as you guys probably know, there's this nice trade off in deep networks is more compute equals somewhat more quality. Right? You generally want a compute stack that is really good at running these machine learning models. Of course, CPU is also important that you have a good amount. You can't do everything in parallel neural network computations. So we want a lot of parallel compute for low voltage. Ideally, also certified system that is robust to shaking and the treatment it gets out there in the wild.

• 31:09Speaker 00

  And so you do want something like GPUs or TPUs ideally because that's what makes running these models tractable and possible. And of course, you can do dedicated hardware as well. So it's a bit evolving. I think the field starts with benefiting from a more flexible architectures that you can run more different models on. And then as we mature and hone down on the specific architectures and final models that work really well, it's still a work in progress, then I think the hardware will become more and more custom to those and gain more and more efficiency over time. So there's just a natural evolution, but you do want a lot of compute in parallel for low voltage.

• 31:54Speaker 00

  And this is the systems that most of our companies like ours are really looking at.

• 31:59Chris Benson?

  So I'm wondering if you could kind of give us an overview. We know that you have your modular autonomous driving system. And if you could kind of describe what that is and just from other organizations, including the one that I work for, most of them have kind of a way of approaching that and they have a lot of commonalities across it. And you don't need to give, obviously, we're not asking for anything proprietary.

• 32:23Speaker 00

  Right. I can happy to go over the architecture, typical architecture of an autonomous driving system.

• 32:29Chris Benson?

  Okay.

• 32:30Speaker 00

  That'd be great.

• 32:31Chris Benson?

  That'd be fantastic.

• 32:32Speaker 00

  And this would be the onboard version because there's also a lot of key offboard systems that come into play, which are crucial to be able to make progress on the problem. And so the onboard stack, it all starts probably with maps. It's really helpful to have a map of the environment already available to you. This can be obtained, of course, by surveying the environment ahead of time, collecting sensor data from it. And from the sensor data, you can build a representation of the environment and store it. Why is this important? Some people can say, oh, wouldn't it be nice to not have a map? I think map is a really important safety feature and in a sense, an additional sensor. It gives you a lot of prior information about the environment.

• 33:17Speaker 00

  Often, big parts of the environment end up being occluded, especially in crowded scenes. And it really helps you anticipate and understand better the cues in this data-limited regime, where you cannot easily see everything with your sensors, for example. So that's almost like a fourth sensor is your map.

• 33:37Chris Benson?

  Could you say it enhances kind of the situational understanding of the environment as a kind of as a constraint of, you know, your sensor, regardless of what the other sensors are saying, the map says if there is a road or not a road there?

• 33:49Speaker 00

  Yeah, and here's roughly these lanes, and they used to go at least to these lanes before. And afterwards, there will be something in 50 meters, you should know that there is some other road merging in. And you know all this, you don't have to just find it along the way, you know what to expect. Now, when you use maps, one core property is you can't just assume the map is true. Also, ideally, will not make the map so high fidelity and density, just model every pixel in the map, that's probably unnecessary. You just want the high-level semantic gist of the scene, right? And then you transfer this understanding into your models as an aid. But it's a knowledge that you need to be able to change as you observe the world. For example, maybe you believe there's a traffic light over here, and then someone

• 34:31Speaker 00

  moved it, right? Your system needs to be able to now understand that whatever the traffic light that in the map was telling you is no longer relevant, and you need to detect any new one and figure out what it means for you and execute, well, the allowed behaviors in the scene relative to this new object. And so that's a key requirement to be able to maintain a map is you need to track which parts of it are correct and use those to aid your models, right? But that comes from part one of the system. And again, some people make a big deal out of, oh, you may not need this high-fidelity map. You need some map, right? And it's a bit open question, and everyone has a different answer. But some information about the environment that you

• 35:16Speaker 00

  can just collect, it's a great safety feature. So it starts there. Now, to use the map, you need now a system that's called localization system. This is a system, as you drive through this environment, you need to know how you're positioned relative to your map. Once you know that, then all this knowledge stored in the map has direct transfer to your own representations from your sensors, and now it can really enrich them. So that's the prerequisite. Then you have perception. Perception, you have the map, the localized map, you have the sensor data, and you're reconstructing a model of the world. And this model of the world is actually very interesting. It can be quite rich. So you can understand and detect and track

• 36:00Speaker 00

  many different kinds of objects in the environment, all the vehicles, pedestrians, cyclists. You can detect signs and traffic lights and lanes on the road and borders of the road, the drivable surface, and so on. Flares, police cars, ambulances, gestures of people. It goes quite rich, right? And this kind of predicates having several models running in perception to obtain this data from the three available sensors. And so, once you have this representation of the world that you agree on, right? That now is fed to your behavior prediction module

• 36:42Speaker 00

  for the objects that are ultimately behaving in the environment on their own volition, things like pedestrians and vehicles, which are a lot of the things that makes our driving hard, is the ability to anticipate uncertain, potentially, behaviors of the people around you, right? Whether they're in vehicles or walking. It's not a deterministic behavior. It's not something you can project for sure. You need to be prepared for a diverse set of potential outcomes and drive an optimal policy relative to that. So you have a behavior prediction system that gives you a representation of these beliefs about what other agents are going to do. And now with the input from perception,

• 37:24Speaker 00

  which also can include things like you understanding which parts of the world occluded and not occluded. So it's not just about detecting and tracking an object. You need to understand which things you don't even know, because that's important knowledge in itself. And the predictions and all the tracked objects and all the obstacles you've detected, this now goes into the planner, which needs to plan a good, safe trajectory that gets you to the goal comfortably. Right? That's ultimately roughly the gist of what is required. You need to trade off all of these requirements in an optimal way to get a good behavior. Right? And then the planner executes or makes the decisions of how to drive. Often those then decisions get need to be translated into acceleration and steering

• 38:09Speaker 00

  wheel motions, which a controller can do. And this is the whole system working in unison with all these modules, maintaining the mapping from observed environment to active environment. So that is roughly my overview of the onboard system as a set of modules.

• 38:51Chris Benson?

  That was a fantastic explanation that you just gave us there. And I just want to kind of follow up a little bit on that. You know, you covered so many of the components that go into the process of the vehicle figuring out, you know, what's happening and where it's going down the road from behavioral prediction model to the planner, the perception, from the sensors, your world model. As you put all of that together and you are having a vehicle that is moving down the street, that's a fairly complicated situation to try to resolve.

• 39:22Speaker 00

  It's a wonderful integration problem. I mean, self-driving cars is a robot, and you need to integrate all of these systems, which also, of course, includes all the hardware and sensors and so on into one coherent whole. It's fascinating.

• 39:35Chris Benson?

  That's really where I was wanting to go with it was, you know, it is a very complicated situation. And for any one of the various things that you covered, you have to have a strategy that, you know, that the planner is going to come up with and stuff. If you're starting with, you know, maybe the mapping or the world model that, you know, from that, and you have the sensors to give you the real-time data, how does the behavioral prediction model and the planner, how do they kind of take those things in, in a generalized sense, and be able to deviate from what otherwise? You know, if you're starting with, well, I have a map of roads and therefore, but, as you said, they may have moved the traffic light. Or there may be a pedestrian or a dog that's about to dart out into the road.

• 40:19Chris Benson?

  How does that integrate in? Is there, do you kind of have a path that you're typically taking? And then those are all exception cases? Or is it more integrated than that?

• 40:29Speaker 00

  I think it's more integrated than that. Generally, there is high-level principles or understanding of how to build this thing and how it's supposed to fit together. So operationally, we'll talk operationally and then technologically. Operationally, in these areas, there's a lot of these vertical, cross-functional work groups that ultimately tailor a set of capabilities and algorithmic system designs to handle the specific types of behaviors in the environment. I think also ultimately, there is a lot of interesting high-level choices you do that are a bit more general, which is what is a good representation to power behavior prediction?

• 41:12Speaker 00

  And I think this may, different companies may have different answers. And a lot of this can be tried by trying deep learning models on different types of inputs and seeing what gains you get. This is one interesting way to understand what kind of attributes and representations are important. I will give you some examples from our recent work that we published. This year's CVPR. Traditionally, and this is some of the early Waymo work that we published, maybe early last year, 2019, early 2019, we had a paper called SchofferNet, which was based on end-to-end or mid-to-mid, as we call it, "learning to drive." That was based on the presentation of the world as an image.

• 41:57Speaker 00

  So your map, you rasterize it to an image. Then you can use your favorite computer vision models to process the image with the convolutional net and then make whatever predictions you want on top. Maybe some kind of heat maps of where you're going to be, or trajectories, or so on. That is one type of representation, but it also has a significant set of weaknesses. Typically, that's a dense representation. You're quantizing potentially continuous signal. You lose structure that you had because you just rasterized something and the structure is maybe getting lost. And conditional neural nets also are good for certain types of processing, but not others. You have these local kernels, maybe a 3x3 or 5x5 or so on. That's a very local thing.

• 42:42Speaker 00

  And often you need to reason two agents, maybe 100 meters from here will merge. That's not so obvious to model out well, at least in a more precise way with the deep net, with small kernels. And so that leads to the question of, well, maybe there's other representations possible. Also, there's a question of, and of course, that's clearly not the full answer. Do you do prediction just based on bounding boxes? So if you treat every person and every vehicle is a box, and all you do is observe how the boxes move, can you do accurate behavior prediction? Well, you can be pretty good, but probably you can do even better. Humans have a lot of different cues.

• 43:23Speaker 00

  Gestures, gaze, appearance, behavior of the body, body language, right? And so on, that also matters, especially in the more complex environments, in the dense urban downtowns. So there is a bit of this interesting, open question. There's a lot of different things you can do. One of the things we found in a paper called VectorNet that we published in CVPR this year was that if you model all the map information such as polylines.

• 43:52Speaker 00

  And you model the agent behavior, also the past history of the agent is also a polyline of sorts. And now you have a model that processes all of these, computes features from these polylines, puts a graphical neural network or transformer in this specific case on top to reason about all their relationships and produces predictions. That works better and often more efficiently than rendering everything in an image. So it's a little bit of how your models are in behavior prediction and their capabilities depend on this interface. And there's a lot of things that can be tried and potentially just rendering things as an image and putting a neural net is not it. But it's also not just a bunch of bounding

• 44:32Speaker 00

  boxes moving around. The world is rich and you need a lot of these attributes or potentially you can go even from the raw pixels in an end to end fashion, of course, but then you run into different challenges. So, so this is one of the exciting areas of engineering design, right? And each company makes a decision, what is our API? What do we put there? What is the predictive features that we're going to be reconstructing with these models?

• 44:55Daniel Whitenack?

  Yeah, it's interesting that you're talking about kind of how things are related. I also saw, so I watched your talk at CVPR is a great. I also saw you were doing other things, tracking predicting trajectories with these graph neural networks. And it kind of strikes me as a bit of a trend. I saw, I think I mentioned this on our, one of our last episodes, but I saw Sasha Rush was giving some statistics about like the number of keywords he saw being mentioned at ICLR and graph neural networks was way high up on there.

• 45:29Speaker 00

  It is definitely a trend in robotics now. And for good reason, I think, I think ultimately you want models that generalize.

• 45:37Daniel Whitenack?

  Yeah.

• 45:38Speaker 00

  And just the, well, first principle of computer vision networks from some sensor grid to controls is not a very generalized model for such a complex space. And so I need to, need to introduce structure in this model judiciously. And a lot of the structure ultimately about which we can even have modules and reasoning and humans can understand it better is things about objects and relationships and interactions and trajectories. And these are the things that things like transformers can model well. They can model all the inter-object interactions directly as opposed to using an image medium to, as a proxy, right? And that's powerful. And I think that's not to say you cannot have the

• 46:23Speaker 00

  So it's not like you can have it, you can have it both ways. You could, right? You can have some of the basic processing go in a general model and then still have all this structure tacked on. And if this is a good generalizable property, typically having more outputs in the model is nowadays likely a good idea that allows you to self-supervise them, that adds additional structure, it constraints you better to generalize to cases where you don't have as much data. So these are good trends, I think, right? And I think, I think GNNs are popular for a reason. They're quite flexible in what they can do, right? And transformers is one of the most popular versions currently of graph neural nets.

• 47:04Daniel Whitenack?

  Yeah, one of the other things I noticed in that research you were showing was the temporal piece of this that, you know, we've talked about perception and objects and relationships, but there's also, you know, trajectories. And as you're processing this data, it's all, it's all coming at you as a time series, I assume.

• 47:24Speaker 00

  Correct.

• 47:24Daniel Whitenack?

  Could you explain this sort of element of the, I think you called them temporal anchor proposals and like, how does this temporal and like projecting back in the past and projecting forward trajectories, how does that fit into the flow of controlling the car?

• 47:43Speaker 00

  So I'm not sure exactly which aspect of the work you talked about, you're talking about, because I showed several that might be interpreted in this way.

• 47:51Daniel Whitenack?

  Yeah, it was very rich. You're doing a lot. So it was hard to, it was hard to narrow down questions here. But yeah, I think the one I was thinking, you had a little guy walking and I think you had projected his historical trajectory or something like that.

• 48:08Speaker 00

  So that's your evidence, right? You have a scene, which is a static context, and you have the behavior of the agent so far. And you want to produce a distribution over the future behaviors of this agent, right? And one of the popular ways to represent that is what I call a mixture of trajectory Gaussins. We wrote a paper on this last summer in 2019, a paper called "Multipath", which was one instantiation of it. At its core, you're gonna say, hey, each agent can have some intents. It's a bit hard to say how many there should be, so, but let's assume you have some intents. And for each intent, they're roughly to do a specific behavior, a trajectory. And maybe you have some uncertainty about how exactly this trajectory will play out.

• 48:51Speaker 00

  It's a simplification, but a way to get very accurate behaviors for fairly long time horizons. Now, the complexity there is that, let's say, an agent can have maybe some number of intents. Let's say a dozen. And for each, you need a trajectory and some uncertainties. And maybe these intents change over time. So if you're somewhere almost stopped and about to start driving in the parking lot, there's some set of intents you can have. And then you come to some intersection with some constraints and you can have another. And then yet, maybe you have another.

• 49:21Daniel Whitenack?

  Yeah, and different numbers.

• 49:22Speaker 00

  Right?

• 49:22Daniel Whitenack?

  Yeah.

• 49:22Speaker 00

  How do you know which one to apply when, right? And where do you get this knowledge? Now, maybe you start hard coding based on the road graph, but that maybe makes you too dependent on the road graph. What the road graph has a mistake, right? So we ended up with a concept at the time that was called "Static Anchors", which says, hey, I'm gonna enumerate, just look at the data historically all vehicles have ever driven. And you can see all the potential things they could do. So let's just discretize this into bins. And maybe there are 128 bins now or more, but at least you know for each type of behavior, now you can classify which one you're doing. And then, of course, you adjust the specific trajectory for that behavior. It's a very simple, very straightforward model. And turns out very easy to learn.

• 50:08Speaker 00

  And so it's probably the simplest behavior prediction model for multiple trajectories. And when we did it, we just got inspired by object detection literature. I used to do object detection before. We had a paper called SSD, which was a fast single-shot detector that was way back in 2014. And so it's a bit similar to this, where you're saying, hey, I'm just going to enumerate all the things I could roughly do. There might be a few, quite a few, but at least it's fixed set. I now can train which ones get activated when and adjust them. And that works really well. So that, of course, is not the only way you can do this. But it's one of the simplest and it reduces behavior prediction much closer to object detection and it reduces it to a supervised problem. It's very friendly.

• 50:53Speaker 00

  I mean, that's what machine learning does great, right? So it's great to use it. It's considerably friendly also to production systems as opposed to sample system that just does sampling. A lot of academia work is on variational autoencoders or CVAEs and so on. And those need to sample your set of trajectories, right? And that's how they generate a distribution. Sampling is expensive and it's not that deterministic. And it's not that easy to get probabilities for it. And so while it's very nice and powerful paradigm, maybe this anchor thing is simpler. Of course, it doesn't actually stops with anchors because now there is powerful anchorless models. And we have some exciting work that we're hoping to put out on archive that does some interesting things in this space as well soon.

• 51:37Speaker 00

  So maybe you guys should stay tuned, but there's a lot of interesting things you can do with this.

• 51:40Daniel Whitenack?

  We'll be on the edge of our seat. That's a little tease there.

• 51:43Speaker 00

  But one last thing I would say is why do this thing, the anchors? Typically in object detection also in early time when we were working on it, the problem with and the reason why we introduced the concept of anchors ourselves, we did it at the time probably in parallel with Faster R-CNN, which also did it. It's a reasonable concept. Many people invented. It's logical. I think is the following, neural nets, if you train with L2 loss, right? And you don't have many different modes at output, it starts doing mode averaging. So it's hedging. If there's two different eventualities that potentially overlap and you have a single output, it will average between the two. That's the best thing you can do with L2 norm.

• 52:25Speaker 00

  And so there is this traditional trick and now by now it's perfectly standard to do this continuous regression problems is to do discrete continuous. So you discretize in slightly coarse bins and then within each bin you regress and that's dramatically better than just doing directly regression. That does not work so well. Right? And now, of course, it's 2020 and people have even more things that they can do. But this is the probably one of the simple things you can do if you do regression is to discretize parts of your output and usually it does better.

• 52:56Chris Benson?

  So I guess to finish up, if you could tell us, what kinds of things are you most excited about as you're looking here at the future of this industry, at the future of self-driving cars? With your special insight in there, what's getting you excited? What keeps you up feeling like a kid about what's going to happen next?

• 53:16Speaker 00

  First of all, I'm excited by the product and the possibility of helping society with it, right? And I think that goes in multiple different directions. And you probably have probably heard them before, but that does not make it any less valid. There is safety aspect. Right now, in the US, I think more than a million people die a year. I had heard

• 53:40Speaker 00

  kind of a comparison to this. It's as if a Boeing plane with 150 people crashes every hour all year, right? This is the amount of, it's in the world though, not just in the United States. In the world, about 1.3 million people die from automotive accidents and 50 million people get injured. And a lot of those are due to human error. I think over 90%. And so with autonomous driving systems, the hope is to be able to reduce the consequences of and the possibility of human errors.

• 54:16Chris Benson?

  As you're addressing this, and I don't want to cut you off, I want you to continue, but as you're addressing this, can you envision a world at some point in the future where humans are not driving at all anymore and it is 100% everything we do is autonomous?

• 54:29Speaker 00

  I think it may take a while to get there. People are attached to their cars and I think generally penetration will happen in a thoughtful, steady way over time.

• 54:39Speaker 00

  There is the hope and the advantages of autonomous driving that people will hopefully get attracted to. We've had very enthusiastic reception. Originally, you think, okay, we're going to roll out this self-driving product in Phoenix, and how is it going to go? A lot of people are really excited users, and especially for the fully driverless rides, which there's no one in the car and it's driving you throughout Phoenix. We got a lot of five stars. People get super excited once they experience it. It's a big thrill and they're very positive. So I think it may take some time for people to get exposed, and we work a lot of outreach and working with key groups and representations to get people to be more informed and hopefully

• 55:24Speaker 00

  give people more chances to try this. But I think, you know, it still will take some time, but I'm very optimistic about it. Some advantages of self-driving cars, in addition to safety, is economics as well. I think it can make it easier for a lot of people that currently cannot drive. There's a lot of disabled people or people otherwise limited to what they can do to more affordably do so. I think it often potentially can change how cities are designed. There might be fewer parking lots. There might be more green spaces. There might be less congestion. That's another aspect. I think fourth, I'm personally excited about it.

• 56:07Speaker 00

  Self-driving cars go really well with electric cars, whether it needs to be hybrid or electric. And I think larger adoption of self-driving technology will also contribute to larger adoption of electric vehicles over time. That's my personal opinion. I think that it's a natural progression that is likely to happen as well. And so it has all these beneficial... Consequences. There is an interesting question about with COVID, what have we learned and is it changing transportation? At least so far with the virus, it's problematic to have many people in a small space.

• 56:47Speaker 00

  And autonomous driving has the chance to remove the driver and have you be in a vehicle by yourself. And I hope this COVID problem, I mean, it's a pandemic, it's terrible and soon. But ultimately, I'm not sure what the patterns and preferences of people would be. But autonomous driving can help their potential as well in some shape. And I think it will start little by little. Maybe people will not get their second car and just use autonomous driving because it's convenient. We see this already in our pilot service Waymo One in Phoenix. There is a lot of, this is, for example, a popular way. It's much easier to just get the Waymo, especially if you need a children's child seat. It's already installed in the back and so on.

• 57:30Speaker 00

  I think it's only the beginning and ultimately the possibilities are great, but it will not happen overnight.

• 57:36Daniel Whitenack?

  Yeah, yeah. Well, thank you so much for diving into all of these topics with us. I know that there's, there's so much to discuss here. I definitely recommend people check out your CVPR talk. We'll make sure that the link to that is in our show notes so that you can explore more of the great research topics.

• 57:53Speaker 00

  I hope the people that listen to your podcast are somewhat technical because that talk was aimed at a research conference. But hopefully, hopefully it's still interesting, right? It starts general, but then it goes into a bunch of models.

• 58:07Daniel Whitenack?

  Well, I think, I think it was, yeah, that's what I enjoyed about it actually was that it started general and actually it kind of eased you into it and people can go as far as they, they want, looking at the papers or just, you know, hopping off to the Waymo website and the blog and all of those things and, and reading there. We'll make sure and include those links as well. But yeah, I know I'm fascinated by so much of what you're doing and really appreciate also your passion and, you know, how much you're pouring into this. It's really great to see. So thank you so much for, for joining us. And I hope someday we're able to meet, we're able to take the same autonomous car somewhere and have a chat. That would be wonderful.

• 58:46Speaker 00

  Well, after COVID, hopefully, let's go a little bit. I fingers crossed that it will be a possibility. So it would be good to, uh, well, be able to give you guys a test, right? Maybe that would be interesting.

• 58:59Daniel Whitenack?

  Oh, absolutely. We'll do that. We'd definitely be in for that. Yeah. So thank you so much. We really appreciate you being here.

• 59:07Speaker 00

  Thank you. And we're hoping to put more of our work out there in this year and going forward. A lot of the work you see now comes out of our lab. A lot of it was done last year because that's how long usually the publication process takes.

• 59:24Daniel Whitenack?

  Yeah.

• 59:24Speaker 00

  But we have a lot of other great things in the pipe. So I hope to share them with the community soon.

• 59:29Daniel Whitenack?

  Cool. Thank you so much.

• 59:31Speaker 00

  Thank you.

• 59:36Speaker 02

  Do you have questions, praise, or constructive criticism about the conversation you just heard? Comment on this and every episode of Practical AI on changelog.com. Just open your show notes, follow the Discuss on Changelog News link, and let your voice be heard. Practical AI is hosted by Daniel Whitenack and Chris Benson. It's produced by Jared Santo. That's me. And our music is provided by the mysterious Breakmaster Cylinder. We're brought to you by some amazing sponsors. Special thanks to Fastly, Linode, and Rollbar. And a special shout out to those listening on our ad-free Changelog++ feed. If that's you, you're awesome. If that's not you, well, you're awesome too.

• 1:00:18Speaker 02

  But you can learn all about it at changelog.com/plusplus. That's all for now. We'll talk to you again next week.