Flying high with AI drone racing at AlphaPilot

Chris and Daniel talk with Keith Lynn, AlphaPilot Program Manager at Lockheed Martin. AlphaPilot is an open innovation challenge, developing artificial intelligence for high-speed racing drones, created through a partnership between Lockheed Martin and The Drone Racing League (DRL). AlphaPilot challenged university teams from around the world to design AI capable of flying a drone without any human intervention or navigational pre-programming. Autonomous drones will race head-to-head through complex, three-dimensional tracks in DRL’s new Artificial Intelligence Robotic Racing (AIRR) Circuit. The winning team could win up to $2 million in prizes. Keith shares the incredible story of how AlphaPilot got started, just prior to its debut race in Orlando, which will be broadcast on NBC Sports. Sponsors: DigitalOcean https://do.co/changelog – DigitalOcean now offers three managed databases https://www.digitalocean.com/products/managed-databases/ — PostgreSQL, MySQL, and Redis. Get started for free with a $50 credit. Learn more at do.co/changelog https://do.co/changelog. The Brave Browser https://brave.com/cha913 – Browse the web up to 8x faster than Chrome and Safari, block ads and trackers by default, and reward your favorite creators with the built-in Basic Attention Token https://basicattentiontoken.org. Download Brave https://brave.com/cha913 for free and give tipping a try right here on changelog.com https://changelog.com. Fastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship. Featuring: Keith Lynn – LinkedIn https://www.linkedin.com/in/keith-lynn-85837011Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: AlphaPilot https://lockheedmartin.com/alphapilotAlphaPilot – Lockheed Martin AI Drone Racing Innovation Challenge https://www.herox.com/alphapilotLockheed Martin and Drone Racing League Announce 2019 AlphaPilot Teams https://www.prnewswire.com/news-releases/lockheed-martin-and-drone-racing-league-announce-2019-alphapilot-teams-300855026.htmlLook ma, no hands: UCF arena to host fast-flying, autonomous drone racing in October https://www.orlandosentinel.com/business/os-bz-drone-racing-ucf-20190912-geppy5sutvdtpceyagqpbayk4e-story.htmlLockheed Martin https://lockheedmartin.comLockheed Martin - Wikipedia https://en.wikipedia.org/wiki/Lockheed_MartinThe Drone Racing League https://thedroneracingleague.comThe Drone Racing League - Wikipedia https://en.wikipedia.org/wiki/Drone_Racing_LeagueLockheed Martin - Twitter https://twitter.com/LockheedMartinThe Drone Racing League - Twitter https://twitter.com/DroneRaceLeagueNVIDIA - Twitter https://twitter.com/nvidiaUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Being with your change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at ChangeLaw because of Rollbar. Check them out at rollbar.com. And we're hosted on Linode Cloud servers. Head to linode.com/changelog.

• 0:16Speaker 00

  This episode is brought to you by Digital Ocean, the simplest cloud platform out there. And we're excited to share they now offer dedicated virtual droplets. And unlike standard droplets, which use shared virtual CPU threads, their two performance plans, general purpose and CPU optimized. They have dedicated virtual CPU threads. This translates to higher performance and increased consistency during CPU intensive processes. So if you have build boxes, CICD, video encoding, machine learning, ad serving, game servers, databases, batch processing, data mining, application servers, or active front-end web servers that need to be full-duty CPU all day, every day, then check out Digital Ocean's dedicated virtual CPU droplets. Pricing is very competitive, starting at 40 bucks a month.

• 1:01Speaker 00

  Learn more and get started for free with a $100 credit at do.co/changelog. Again, do.co/changelog.

• 1:21Speaker 00

  Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and tag with us around various topics of the show at changelog.com/community. Follow us on Twitter, we're at Practical AI FM. And now into the show.

• 1:46Chris Benson?

  Welcome to another episode of the Practical AI Podcast. This is Chris Benson speaking. I'm Principal AI Strategist at Lockheed Martin. And with me today is my co-host Daniel, who is a data scientist with S I L International. How's it going today, Daniel?

• 2:01Daniel Whitenack?

  It's going pretty good. I'm working on not my normal amount of sleep. I've been having some extra meetings this week with a team in India I'm working with on some dialogue stuff, some conversational AI. And so it's been it's been early mornings for this week, so I'm looking forward to sleeping in in the morning. But it's been been good, good tech, just just early mornings. How about yourself?

• 2:27Chris Benson?

  I probably have more sleep. I was a little sleep deprived earlier in the week. I think I've caught up. But I'm just kind of on cruise control here. I'm about to head over to the UK for a week of work and then a couple of weeks of family vacation. So I'm looking forward to that.

• 2:42Daniel Whitenack?

  Good times.

• 2:43Chris Benson?

  Yeah.

• 2:44Daniel Whitenack?

  Looking forward to some shepherd's pie or jellied eels or something?

• 2:49Chris Benson?

  Strictly vegan everything.

• 2:51Daniel Whitenack?

  Oh, yeah, that's right. I don't even know why I asked that.

• 2:54Chris Benson?

  Strictly vegan, my friend.

• 2:55Daniel Whitenack?

  When I, when I think of England, the food scene seems like, you know, meat and a pie is what I think of.

• 3:03Chris Benson?

  And I think traditionally it has been. And since you said that, I'll just say, my wife was amazed. She's, she's always grumbling. We're both vegan together and she's, she's always grumbling that her father is like the epitome of the meat eater. And she was just delighted. She was just lethal when I came home a few days ago. And she said, she said that her dad had tried some, some like vegan beef and vegan hot dogs and stuff like that. And he liked it. And she was just, oh, it's great. So who knows?

• 3:29Daniel Whitenack?

  Well, you'll have like vegetables and pies or something.

• 3:32Chris Benson?

  There you go. So, you know, apparently even the meat eating folks of the UK may go that way. But this isn't a vegan podcast. So I'll stop, I'll stop going down that. Anyway, but when I get back, so I will kind of be back into into the world on this side of the pond, early in October, is kind of coming back into into work on the 7th. And I'm going to get to do something that's pretty fun. That's participating in a program that Lockheed Martin is has been driving with some partners. And it is called Alpha Pilot. And today with us on this podcast, we have Keith Lynn, who is the program manager at Lockheed Martin for the Alpha Pilot program. And when he tells you about this in a few

• 4:15Chris Benson?

  minutes, it's going to blow your mind. It's a super, super, super cool thing. And he's been working really hard on it. Keith, welcome to the show.

• 4:22Speaker 03

  Hey, guys. Thanks for having me.

• 4:24Chris Benson?

  We have actually been talking about having you one for quite a long time with some of the the events that we've done internally at Lockheed that you have orchestrated. And I've kind of come back and just said, Daniel, this cool thing. And, you know, at some point, you know, when it's the right time, we got to have an episode about it. And so we've both been excited.

• 4:44Daniel Whitenack?

  I've been asking Chris each week about, hey, can we talk about this thing you've been talking about now?

• 4:50Chris Benson?

  He really has. So, so before we dive fully into AlphaPilot, if you could just kind of tell us a bit about yourself, how you got to where you are, you know, and, and kind of what, what's the life path that led you up to this moment? Where you are running this amazing AlphaPilot program?

• 5:07Speaker 03

  Yeah, so I think I've been with Lockheed about 11 years now. I started out graduate in physics and mathematics at a liberal arts college here in Pennsylvania. And my first job was doing signals intelligence. And from there, I, I guess my curiosity got the best of me and Lockheed is a really good place. If you like some diversity of what you're working on. So I quickly jumped to doing some research in healthcare, then in energy management systems. Eventually made my way to our corporate office and got to do some international work. Spent a better part of a year and a half managing R and D in the United Kingdom. Some nanotechnology research down at Rice, Rice University in Texas.

• 5:52Speaker 03

  And eventually landed in the autonomous systems group at corporate. This is around the time when AI really started to take off. You know, a couple of years back, AI was, you know, if you ask people, what do you know about artificial intelligence? They'd say, you mean like robots? So the, the term in itself has, has really blossomed and it has much broader field of meaning now. And at Lockheed Martin, we started looking at Artificial intelligence and then a subset of that, which is autonomous systems. And really trying to figure out where we can apply that across our different products and services. And that's what really led me into AlphaPilot. A few years back, we, I'm sorry, a year back, it feels like a few years back already.

• 6:33Chris Benson?

  I know that when I saw you at the event that you probably mentioned a little while back, you were going, pardon the phrase, balls to the wall. You were really working hard, so I can believe it's been, it's felt like three years.

• 6:46Speaker 03

  It's, it's been one of the longest years for me on record, that's for sure. But I couldn't be happier, you know, I, I might not always say so in the heat of it, but I think I always look back and I say, man, doing this kind of thing where you're, where it's chaos and, you know, things are going haywire, but you somehow managed to land the plane. So it was just about a year ago, I was actually sitting at a hotel next to South by Southwest Conference Center where we were doing a presentation. And I think I was actually telling my boss, like, I'm getting a little bored. I'm not really sure I want to do next. And I feel like I need my next big thing, something that really pushes me to the limit. And then our, our vice president sat down and, you know, he just said, I just got this new thing, you know, it just came on my plate.

• 7:30Speaker 03

  Lockheed wants to do a global innovation challenge. And they want to do something with autonomy and, you know, maybe drones or something like that. We're not sure, but it's got to be huge. Got to really get people's attention. It's got to really show people that Lockheed cares about artificial intelligence. And now we just need some, you know, poor fool to put in front of this and figure it all out. And I mean, the timing was just perfect with what I was talking to my boss about. So I just said, hey, I'm, I'm looking for something, like, let me take a crack at it.

• 8:01Chris Benson?

  So before we even dive into AlphaPilot, you know, just to give some background on, on, before we even talk about the program, you know, it has to do with, with drone racing and stuff. And I was wondering if you could kind of talk about what that is prior to AlphaPilot? You know, what, what is drone racing and what's the interest in it? And is it, is it something that's specific to the US? Is it international? If you could talk a little bit about that and then we'll get into AlphaPilot and describe its role in that world.

• 8:31Speaker 03

  I just mentioned that AlphaPilot involves drones and drone racing specifically. That as a sport is about four years old. From what I've been told, it was first conceived of in Australia, then made its way to America and really took off. And now it's really has a global audience. I know it's very big across, all across Europe, actually. But what that is basically is humans piloting drones via a, what they call FPV, or first-person view camera. So these pilots have a really good RF signal, so they have low latency. They see what the drone sees, and they are competing with each other in a traditional first-to-the-finish-type

• 9:16Speaker 03

  race through various types of courses.

• 9:20Daniel Whitenack?

  Like real-life video games?

• 9:22Speaker 03

  Yeah, like a real-life video game.

• 9:24Daniel Whitenack?

  And how do they see?

• 9:25Speaker 03

  By wearing these goggles. And I've actually tried them on. They just look like, like a pair of lab goggles or something like that, or VR headset, kind of. And you put them on and you get this view from the drone. And a lot of times, it's kind of grainy. It's not the best. And if you are at all motion sick, or you're the type of person that gets motion sick, it's not the most pleasant experience.

• 9:48Daniel Whitenack?

  That's what I was just thinking. So I can ride regular roller coasters all day, but I remember, like, we went to Universal Studios recently, and we were in one of those, like, 3D rides where it also shakes you around and stuff. And man, I got so sick. And it's like, something about that, like, virtual motion just really does not agree with me.

• 10:06Chris Benson?

  Yeah, the older I get, the more that affects me. When I was younger, I could kind of do it. But yeah, after a while, your body works with you less and less in that way.

• 10:16Daniel Whitenack?

  So you mentioned that there's like, these people, they're wearing headsets, they're piloting these drones. I'm assuming it's like around, like, a racetrack, like, there's gates they have to go through or something? Is that how it works?

• 10:28Speaker 03

  Yeah, so it's done a little bit differently depending on who you're racing with. There's a lot of different organizations that are doing this. Some are more grassroots, some are professional. On AlphaPilot, we're partnered with the Drone Racing League, which is probably the biggest drone racing outfit out there. They're on national television. They have events worldwide. I think over 80 million viewers annually.

• 10:53Speaker 03

  Their courses are like a traditional race course where you're going through different gates and checkpoints and they try to do them in exciting locations and integrate with the local environment to the extent possible.

• 11:06Daniel Whitenack?

  So you mean like around buildings and such?

• 11:08Speaker 03

  Uh, so I was at one last year that was at the BMW headquarters. So there was literally an indoor track with all these BMW vehicles going back a hundred years and they were flying over top of the cars and between different BMW displays. So they try and make it really, really an exciting environment.

• 11:27Daniel Whitenack?

  Cool. Yeah, that, that's pretty neat. So how fast are these drones going? Like, I'm just imagining having this headset on and I'm assuming you have to have pretty quick reflexes to kind of make it around this course. What's the sort of speeds that these things are going and like the size of the course? Like, are we talking like feet, miles? Like, how big of a scale are we talking about?

• 11:51Speaker 03

  So I think even more than reaction time, it's so darn hard just to control the drone. There's so much sensitivity in the joysticks. I've tried it out with a little, you know, like training wheels drone and I could not, this thing just pinballed between the floor and the ceiling. So it's, it's incredibly hard just to keep the thing in the air. But they top out around 80 miles an hour, I believe. That's not sustained 80 miles an hour, but uh, you know, on a straightaway, they get up to about that speed. The laps themselves, they're maybe like a size of an indoor track, I would say.

• 12:26Daniel Whitenack?

  Like a track and field track sort of thing?

• 12:29Speaker 03

  Yep. And they would do multiple laps. And the, uh, the interesting thing is the, you know, they're really burning through these batteries going at that speed and with that kind of agility. So the, the flights themselves don't last very long, maybe a minute total.

• 12:42Daniel Whitenack?

  Like the full race or do they like stop for like pit stop recharge? I imagine that wouldn't be too exciting.

• 12:50Speaker 03

  Yeah, so when you're there, I mean, you got this safety net in front of you. There's a very intense safety protocol and these things just go buzzing by you. And really, sometimes all you see is just the LED lights. So it's just kind of a blur of red or blue or yellow or whatever color that particular racer is.

• 13:06Daniel Whitenack?

  It seems like almost like a horse racing experience where, you know, at least a couple of people on this call don't approve of all things horse racing. But it's like that very short period of time of like high excitement and like everything's over. And however long it takes one minute, like you said, it's a very, very intense experience.

• 13:25Speaker 03

  Yep, you're you go from looking for it to come around a corner, whiz past you. Then your eyes go back to the giant screen to see where it is now. And, you know, Drone Racing League is very good with production quality and they have the whole arena mic'd up. So if one of these things bites it, you know, anywhere, you're going to get that audio as if it's right next to you. And I mean, it's a real intense sound when these things crash going that fast. And part of the the guilty pleasure of racing always is, you know, seeing a little bit of a hardware go flying. So it's a fun time.

• 14:00Chris Benson?

  Yeah, you can describe it in a minute, but even the the thing that I was at, the event that I was at with the teams was was quite thrilling and it wasn't even production quality of what what you're describing for everyone. But I was, you know, I think I think all of us that were new to it were like, wow. Which kind of brings me to the next thing is, you know, now that you've kind of described what drone racing is with these human pilots doing their their thing really well. Could you kind of describe what Alpha Pilot is and how it fits into that context of drone racing and such?

• 14:31Speaker 03

  Sure. So Alpha Pilot is a public innovation challenge. So I would compare it to like a DARPA Grand Challenge or an X Prize where an organization, in this case Lockheed Martin, puts out a hard problem. So ours was programming a drone to fly in a race course autonomously.

• 14:54Speaker 03

  And that challenge is associated with some kind of award, an award. So ours is $2 million for the top teams and then an additional $250,000 kicker prize. If any algorithm can fly faster than a professional human pilot on the same course. So we're having a little friendly rivalry with the professional pilots here in the drone race.

• 15:19Daniel Whitenack?

  Gotcha. So there's like, it's almost like a sort of Watson Jeopardy sort of challenge thing going on where there's actually humans racing at the same time.

• 15:28Speaker 03

  It would be more like if there were four different Watsons competing with each other. And then we took the best Watson and put it against the best human Jeopardy player, just for fun.

• 15:37Daniel Whitenack?

  So there's, I guess, a series or a bracket or something too. So like people go against each other, the humans go against various drones, other autonomous, and then there's sort of like a grand showdown.

• 15:49Speaker 03

  So originally we were going to pair the autonomous racing with human racing. So we could have more of that side by side. Then we decided to move towards more standalone events because we wanted this to really be not just about the racing, but a celebration of artificial intelligence and create an opportunity for larger STEM engagement. So our events are just going to be autonomous racing, even though we'll have some human pilots there to do some commentary and some demonstrations. And then as we get later in the season, we will have one of the pilots on standby to go head to head against the top autonomous team after the racing for that day is finished.

• 16:43Speaker 04

  This episode is brought to you by Brave. The Brave team is on a mission to fix the web by building an open source, privacy focused, and performance oriented browser. Browse the web up to eight times faster than Chrome and Safari. Block ads and trackers by default and reward your favorite creators with the built in basic attention token. Yes, you heard that right, a real world use case for blockchain. Download Brave for free using the link in the show notes and give tipping a try on changelog.com.

• 17:24Chris Benson?

  So for these teams that are working on AlphaPilot, you know, we've talked about kind of the drone racers in the manual sense. There are these teams that are behind these automated AI-enabled drones that AlphaPilot's doing. Could you tell us a bit about the teams and where they're from and a little bit about them?

• 17:43Speaker 03

  Sure. So AlphaPilot has a innovation challenge launched late last year, and we had about 430 teams sign up to compete. And these teams were from all around the world. We used a simulation-based process to down-select to just nine teams to go into this first-year season of autonomous racing. And the teams that we selected are just fantastic. A lot of them are with universities. Some of them are small businesses, but we have teams from Georgia Tech, UCLA, University of Zurich,

• 18:25Speaker 03

  team from Brazil, team from Warsaw, team from Keyst in South Korea. And then we have a team that's really kind of a mix of people from three different continents, which is quite impressive considering this is all software-based and they have to run it remotely. So we're really impressed with the diversity that we received, and there are some pretty impressive folks.

• 18:47Chris Benson?

  So on these teams, I'm kind of curious. You mentioned universities. Are there kind of mostly student teams that are kind of doing this as a, let's say, a senior design project or something like that? Or is it like graduate researchers? Or what's the sort of range of experience there?

• 19:05Speaker 03

  A little bit of everything. So we didn't want to put too many restrictions on how teams formed because we wanted people to feel comfortable working with the other individuals that made the most sense and that they would enjoy being part of this with the most. And we do have some teams where it's all undergrad students. We have some where it's a mix of grad, undergrad, and we have some where the professors are even involved. So yeah, we get a little bit of everything. The only rules we put on it were no more than 10 individuals per team. So we didn't have anyone with 100 people on the roster and then we have to keep tabs on all of them and they're outsourcing tasks to, you know, the web and things like that. And then a basic age requirement, just because you're dealing with some,

• 19:51Speaker 03

  you know, flying things that are really heavy and go fast. So we want to make sure everybody's an adult. But that was it.

• 19:56Daniel Whitenack?

  And in terms of constraints like technology wise, is there, and forgive me of my ignorance of drone racing out there. People probably love drones and I'm like butchering a bunch of things or ignorant, but is there like, uh, in these leagues, is there a specific drone you have to use? Or like, what are the constraints as like hardware wise? Like, you have to have a a drone that's this model or one of these models, but you can modify it in XYZ ways. Um, how does, how does that kind of work for the teams?

• 20:29Speaker 03

  Yeah, it's a good question. And, you know, in the entirety of drone racing, the answer is all of the above. Some leagues will let, you know, it's, it's BYOD, bring your own drone. And they'll just give you some basic requirements about size, weight, power. Others are a little more structured, but still BYOD. So they might say you have to use, you know, six inch props, you have to use these motors. But the platform and everything like that, you optimize as you want. And then, uh, the Drone Racing League, our partner in, in AlphaPilot and their new league that they're creating for AlphaPilot. And then will be a standalone event moving forward is called AIR, Artificial Intelligence Robotic Racing.

• 21:11Speaker 03

  Um, they use a standardized drone and they do this in their human races as well. So each year they, uh, I believe each year, but periodically they release a new model drone. That is standard for all of their competitors. And we really like that, Alaki, because we wanted this to be about the quality of the algorithms. And, uh, if a team won, we want it to be because they were the best programmers, not because they had, I know, better motors or better batteries on the drone or something like that.

• 21:37Chris Benson?

  Yeah, so I guess to extend that a little bit, you know, we're kind of talking about some of the tech with this. Could you kind of describe the technologies that are in the drone? And, you know, in terms of the hardware, what kind of hardware is being used to run the algorithms and do inference? Just kind of take wherever you think would be appropriate. What are some of the technologies that are being used at this point by DRL's drone for these races?

• 22:03Speaker 03

  Yeah, so the main difference between drones for autonomous racing and drones for human piloted racing is that our drones have to carry a computer onboard. And to do autonomous racing, you need, you know, there's two ways you could do it, really. You could do edge computing, which is all the decisions that are made by the algorithms happen onboard the drone. Or you could offboard, which means you have a low latency comms link between the drone and some high-performance computer on the sideline somewhere. And it reads in signals, sends it to the computer, the computer makes a decision, sends it back. That's less impressive technically, and it could actually create a problem because of the latency and the speed of the drone. So we decided to do a fully onboard edge computing,

• 22:48Speaker 03

  which means you need a pretty impressive processor on this drone. And there are some drones that carry an onboard GPU, but we wanted to make sure that we were giving people enough computing horsepower. We went with the latest Nvidia model for edge computing, which is the Xavier. And that has never been put on a drone before. It's a little bit larger than previous model GPUs. So we had to design an entirely new platform altogether. DRL did a fantastic job doing heat displacement and everything for this drone. And it's definitely unique and first of its kind.

• 23:24Chris Benson?

  Yeah, I know commenting on it, and I actually have the Xavier listed here in terms of its specs. This is the same GPU computer that is used in autonomous vehicles. It's a 512-core Volta GPU with tensor cores, eight-core ARM with a 64-bit CPU, 16 gig of 256 LPDDR4X memory, 32 gigs of flash storage. It's quite a computer without going through the whole thing. It's, I know that when I saw you last at the event, you're putting a pretty serious computer on these drones, and I was pretty impressed with the performance, even so.

• 24:06Daniel Whitenack?

  Yeah, no wonder they run out of battery. I mean, I'm kind of impressed. This isn't like, you know, the humans are still wearing the headset and they're kind of like augmenting the flight somehow. It sounds like more it's like they're, and I don't know how this starts. Maybe you can describe it like there's these autonomous drones that are just kind of hovering there, ready to start and like you push the button start and like based on the, the video feed that they're processing, then they sort of accelerate, turn, switch directions, slow down, all of those things. Is that the kind of scenario we're, we're dealing with or what data are they working with to, to make these decisions?

• 24:49Speaker 03

  So the teams are primarily training for these races in a simulation environment that's built off of the DRL training simulator. So they modified that for autonomous racing. They put in the new drone physics and updated world models. And then they also created course models based on the races and we have a permanent training facility in Littleton, Colorado, where we set up a small scale course with two gates. And that's kind of the gym, so to speak. So these teams can write their algorithms and then train them like in a simulated deployment in this gym.

• 25:31Speaker 03

  And then what they can do is when they feel comfortable, they send the code over to our team. We load it up on an actual drone at this facility and deployed in real life. And they collect the actual data from the drone, the telemetry, input from the IMUs, rangefinder, the cameras, and send that back to the teams. And then the team uses that in comparison with what they're seeing in simulation to make some modifications. And then before each race, we release a model of that course so they can specifically start training for the, the race course that they're about to compete in. And then about a week out from the race, there's a final code submit. So they send us the code.

• 26:13Speaker 03

  We do all of our safety checks on it, make sure it checks out and it's going to deploy. And then on race day, they have absolutely no physical involvement with the drone whatsoever. Everything that it does when it takes off on that podium has already been been coded and pre-planned and it's just ready to go when they get there.

• 26:31Chris Benson?

  So I would like to follow up on a couple of things that you just said. And first of all, there's literally a podium that all of these drones are sitting on rather than being all the way down on the ground and they take off. So they're at a slightly elevated position, just for listeners. But you mentioned a few things, a little bit of jargon there. I want to ask you for some definitions for just for, so everybody can follow. You mentioned simulation and world models and a couple of other things, but starting with those two, can you kind of talk about what is a world model? What does that mean in simulation?

• 27:01Speaker 03

  Sure. So it's, it's really, really hard to train a drone to fly itself when you're primarily doing it in a simulation, which means, you know, you, you recreate the world that you want it to fly in, in a computer environment. And the reason that's hard is because there's always a delta between what's actual and what you're simulating, what you're modeling. So in order to do that, you need high fidelity. You need to model everything the drone's going to encounter, wind, lighting glare, drift in the sensors or the standard deviation error bar in the sensors that you're using, which is specific to the specific brand and the hardware. Gravity, you need good gravity, wind resistance model, drag.

• 27:45Speaker 03

  So all these things have, have an effect. And if you don't model them well, that means your drone isn't going to behave exactly as you think it is. It's going to behave slightly differently. And sometimes that's small enough that it doesn't make a difference. But if you're going fast and over a long enough distance, that can be the difference between going through the center of a gate or crashing into the edges of it.

• 28:08Daniel Whitenack?

  Gotcha. And you also mentioned a couple of other things, just for definition. You mentioned telemetry, which probably most people know, but I'd like to cover that for a second. And also you mentioned IMU, and I'm betting a lot of people don't know what IMU is referring to.

• 28:21Speaker 03

  Yeah, IMU is inertial measurement unit. And this is like, you can think of, maybe some people are more familiar with a gyroscope. Something that essentially can provide data on the three dimensional positioning of a drone at any given time as it's flying. So pitch, roll, and yaw are the three measures. Could be leaning forward, left or right, or side to side. Knowing your position in space is really important if you're trying to figure out how you go from where you are to where you want to be. And then also knowing your position from your last measurement is important so you can gauge your velocity and trajectory.

• 28:59Daniel Whitenack?

  And just to kind of separate, so there's like the simulated world in which you can kind of train before they ship the code off to you and they kind of go through the training at the gym or whatever. But in terms of the, so there's that simulated world, but in terms of the actual inputs that the AI models are working with in order to like make decisions, is that just like, just video, like a, like a human operator would see? Or there are other, some of these other sensors and these measurements that you're talking about, can those actually be input to the, to the models as well?

• 29:39Speaker 03

  So the simulator that our teams are using is part of what they call the integrated development environment. And it's a hardware in the loop kit. So think of it as a computer that's running a simulator. Then the simulator is connected to basically a dissected drone. So there's the cameras, and this drone in particular has four cameras. There's the inertial measurement units that I mentioned. There is a laser rangefinder which points down at the ground and that gives the drone a measure of its altitude at a pretty high sample rate, something like 200 hertz. And then there's the Xavier itself. And when they're running the simulator, everything they're getting on the screen.

• 30:24Speaker 03

  So let's say they're, they have an environment that looks like a warehouse and they're flying towards a racing game. That visual input is then run through the cameras, which then output that into the GPU, which then processes it. So it's replicating the exact process that they would be using on the drone.

• 30:44Daniel Whitenack?

  Yeah, so there's definitely various inputs to the AI models, but in this sort of simulated environment, those inputs are simulated as if they were flying around the track. Of course, like you mentioned, there's like these deltas that they have to deal with and it's not always exactly the same, but it's meant to mimic that as close as possible.

• 31:02Speaker 03

  That's right.

• 31:03Daniel Whitenack?

  Makes sense. It seems like a heavy, like with four cameras, this laser rangefinder, the GPU, all of this stuff, it's like a pretty intense drone. How big is this thing and like, how long can it run? You mentioned the races were about like, what did you say, like a minute or something?

• 31:21Speaker 03

  Yeah, the human piloted races are about a minute long. The drone itself could probably run for about five minutes before the battery started to run out. And how long it takes really is going to depend on how proficient our teams are with their algorithms, which is, you know, partially based on their ability to program, partially based on the fidelity of the simulation environment that we're providing. And, you know, we're working really hard to continuously upgrade that to make it as accurate as possible. If everything's looking great, they should be able to fly through this thing in, you know, seconds, you know, 30 seconds, 40 seconds, a minute. But if you look at the current state of the art, so to speak, you know, there's been

• 32:04Speaker 03

  other attempts to do autonomous drone racing, IROS competition comes to mind. They go pretty slow there because it takes a long time to orient and figure out what they're doing. So still, I don't think it would take more than two, three minutes tops.

• 32:18Chris Benson?

  You know, AlphaPilot is a fantastic way of showing, you know, how current AI technologies are driving autonomy, kind of in a small, a small scale and stuff. But just in a larger sense, in general, how do you see AI as an enabler for autonomy? What has AlphaPilot brought you to in terms of how you think about this, since you're part of this world on a day-to-day basis?

• 32:41Speaker 03

  Well, the work I've been doing with AlphaPilot and with artificial intelligence in general has been pretty enlightening for me. It's something I haven't specifically studied, but the range of applications is just so broad. And it's not always in the most obvious ways or the way that people would think, you know, self-driving cars always come to mind, or robotic systems come to mind. But basic optimization of business processes or human functions like scheduling, calendars, and email and things like that, you know, AI is already making a huge difference. And I think that it can't be overlooked because that's really how it touches everybody's lives. That really gets it out there. Everybody

• 33:23Speaker 03

  gets some level of exposure to it. Things like home assistants work in the language space, others work in the written space, you know. And I think the more we all become familiar with AI and the more we have opportunities to see it doing cutting-edge things like flying a drone all by itself in a very friendly environment, in a competitive environment that's fun and encouraging, the more everybody is going to get inspired by it and we're going to come up with even greater ideas.

• 33:51Chris Benson?

  So partly, I'm assuming that

• 33:54Daniel Whitenack?

  Lockheed is interested in this sort of innovation challenge because of the algorithm and the automation part of it that could be applied even outside of flight. But inside of flight, like when we're thinking about maybe like commercial airline flight in the future, or even recreational planes or military planes or all of that, like what do you see as kind of the the near term applications of of AI within flight? I know that like the, for example, the car industry, they're kind of going through all, you know, figuring out all of this stuff. And there's talk about, oh, well, it's first going to happen in, you know, long range trucking. And there's always going to be like a human operator there.

• 34:38Daniel Whitenack?

  And like, what, what does that look like in terms of of flight? What do you, what do you kind of see as the sort of near term applications of AI in the in flight, whether that's commercial or or military or whatever?

• 34:51Speaker 03

  Yeah, so it's always fun to talk about what AI could do. But I think it's equally important to talk about what it can't do, what it should do, or what it shouldn't do. And when you're talking about military aircraft or space exploration, or anywhere where you're putting a human in a dangerous or difficult and extreme environment, there are certain things that AI should be doing or could be doing that really improve the safety or overall quality of that mission. And those are generally lumped into things like cognitive assistance, where somebody under stress has a high cognitive load and they make mistakes, or they're more prone

• 35:34Speaker 03

  to make mistakes, or where their reaction time is is limited or it suffers to some extent. That's where AI can really help save lives. And and we are using it in that capacity already. Another area that's really interesting is teaming environments where you have a human piloting an aircraft and you may have that human commanding a fleet of autonomous vehicles.

• 36:01Speaker 03

  So it's really the interchange between human pilot of vehicles and unmanned or autonomous vehicles working together. And then they really become an extension of the human. They provide additional mission capacities or they help them with certain functions and basically extend the reach.

• 36:20Daniel Whitenack?

  So in terms of like, if I'm thinking of commercial airline flight, for example, it seems like if I compare the sort of drone racing you're doing, which already like within this confined space with like a small device, we're talking about like four cameras, a laser rangefinder, all of these inputs that it's receiving. And it's a challenge. Would I be right in saying that at least in the in the near term, like AI might be processing inputs like within a flight to, you know, help augment pilots, but it may be unlikely that we're going to be seeing like autonomous 747s everywhere really soon.

• 37:01Speaker 03

  Yeah, I think we can certainly get to that point, you know, someday. But in the near term, you know, whenever you have, you know, precious cargo or human lives onboard, you know, this is what I was talking about where you need to think about the limits of AI or what AI shouldn't do. And right now there is a certain need for a human to have a level of control in that decision making loop. And if you're talking about passenger aircraft, I don't think you're going to see them fully automated anytime soon. There's a certain level of comfort and dynamics that a human can bring to decision making that I think you need a pilot for. But you might be able to say, so let's say there's a pilot flying a 747. Mount David or 747 is so old now.

• 37:45Daniel Whitenack?

  Yeah, I was trying to think of a Don't make it a Max, okay? Just just being fully transparent here, being on a call with two people from Lockheed Martin and trying to come up with a valid aircraft name. 747 was the only thing I was really confident in. So that's what I went with, but I realized that's probably extremely out of date. I wouldn't, wouldn't want to go for like the Boeing Max, whatever.

• 38:07Speaker 03

  So I think it would be totally feasible in the near future to see a pilot land their 747. And the onboard AI is flying in parallel. And then that pilot can compare their performance to the AI's performance. And either the AI learns from the pilot or the pilot learns from the AI. So those types of things are already in motion. And they have tremendous benefits attached to them.

• 38:33Chris Benson?

  I'm curious, outside of, you know, we've been talking about light, you know, within, within air, within the atmosphere. I'm just curious what your thoughts are in general about other environments and, you know, you know, whether it be space, whether it be underwater. And, you know, where, when we might see inroads there, do you have any insight into any of that?

• 38:54Speaker 03

  Yeah, so, you know, we're at Lockheed Martin, we're looking now at autonomous vehicles for use in humanitarian aid and disaster relief. And there's a lot of different scenarios, especially in that area where you have difficult to reach environments. Or you have destruction of local infrastructure. And it's really hard to put a human in there and it's very time sensitive. So that's a great near term application where autonomous vehicles can do a lot of good. You're already seeing things in the news about them being used to deliver, you know, blood and medical supplies to hard to reach places. That's another immediate application.

• 39:32Speaker 03

  And then as we really start to grow into things like space exploration, going back to the moon or putting a human on Mars. You get in these situations where you have communications latency given to the huge distance. And you need to make decisions local. So you need an autonomous vehicle that can process information and make its own decisions. So I think you'll see as we get into the new space age, you'll see AI playing a huge role there. And then I think you mentioned undersea as well. And this is an area what's actually already in practice with some of the autonomous vehicles. We're developing at Lockheed Martin. We've already used them for things like undersea oil rig inspection after some of the hurricanes in the Gulf.

• 40:15Speaker 03

  And just yet another domain where it's hard for a human to go safely. And it's also hard for a human to analyze the situation very well because it's not their traditional element. So these, these types of things are just right for AI.

• 40:30Daniel Whitenack?

  So if I'm, and I've probably shown my ignorance of drones and flight throughout this interview, but if I'm out there, maybe our listeners are wanting to, you know, get involved with drones and maybe do their own experiments, are there accessible ways that they can kind of experiment with maybe it's computer vision or other things with drones without kind of being part of this elite group of teams competing at this level? I think if I remember right, there's like, there's at least some drones that have AI inference chips on, am I right about that? Maybe, maybe you know of some good ways to get involved?

• 41:08Speaker 03

  So there are a couple of commercial drones that are starting to have onboard computers where you can, you know, you could do some amazing stuff with them and they're a lot of fun, things they have like an onboard Raspberry Pi or a Nvidia Nano. And that's enough compute for you to basically learn the basics of how to code so that a drone can read input from its cameras and use that to make guidance, navigation and control decisions and fly autonomously. So there's a lot of great beginner kits out there. And then if you want to experience what some of the pilots are dealing with for Alpha Pilot and for the air season, we actually have the simulation environment we used in

• 41:52Speaker 03

  our qualifier, which is different from the one the teams are using now, but it's, you know, high caliber simulation environment uses the robotic operating system. That has been made available, open source through MIT and it is called Flight Goggles. It was developed by a consultant on our program, Dr. Surtash Karaman at MIT, who's been a huge help in setting this whole thing up. And it's a great, fantastic tool. I encourage anyone to just go check it out, even if you're just flying manually, it's a lot of fun.

• 42:22Daniel Whitenack?

  Sounds great. So I guess as we finish up, if you could possibly maybe tell us a little bit about the first couple of races that are coming up, where and how people can, can access them, but also as you, as you do that, as people engage on this and, you know, families and kids are watching, what would you like to see them take away? What, what kind of impact would you like to have on them as they, as they engage on these, these initial few races?

• 42:49Speaker 03

  Yeah, I'll start with that one. This is something that I I find really fascinating. I I'm so thrilled to be doing racing with AI because racing has a history of being a catalyst for getting people inspired with new technology. And I mean, I I gave a talk at MIT a few weeks ago where I was showing photos of London in the eighteen nineties. And I showed an article in in the London Times about how they were predicting in several years the city would be buried in hu- horse manure because the current solution of horses and buggies was just not sustainable.

• 43:31Speaker 03

  And people were really freaked out about this at the time and they thought, you know, we're at the edge, you know, this this is as far as society can go, right? And, um, you know, the automobile was invented and people just kind of wave it off and they're like, that that thing's a monstrosity, you know, I don't know how to use it. You know, everybody, every horses are so integral to society and you just want to get rid of them and you want to use these things now. I don't know how to use it and I don't know how to work with it. And, um, you know, it just floundered for quite a few years and because nobody was demanding it, the scale of production was low, the prices were high. And I I think that, you know, when people started racing cars and made a spectacle

• 44:14Speaker 03

  of it is when really people started to come around to the idea of these things being a technology of the future. And it took kind of the the mysticism out of it. If you can watch, if you can, you know, pay a dollar and go watch these things race, then they're not something that's reserved for a select few. They're part of your life now. And that's what we want to do with Alpha Pilot. We want to make Artificial intelligence and autonomy, part of everybody's lives. Everybody can watch these races on NBC Sports or Twitter and feel some ownership in it. Or they can come to our race events, which are free to the public, and feel a little bit of ownership in that AI and feel like, maybe this isn't something that is closed off to them.

• 44:58Speaker 03

  Maybe it's a lot easier to get involved than they initially thought. And if they're inspired to go home and do a little more research, or maybe try and see what other great things are going on in AI, then I feel like we've done our job to help inspire that next generation.

• 45:11Chris Benson?

  Fantastic. And when are those first couple of races and where are they? So people can show up for them or watch them online?

• 45:17Speaker 03

  So the first race is going to be October 8th at the University of Central Florida. And it's going to be an all-day event. We're going to have some great Lockheed Martin products on display. We're going to have the UCF Drone Racing team there showcasing some of their capabilities. Some of the DRL professional pilots will be there doing the same. And we'll have some great commentary about what people are seeing in the races and about AI in general. So tickets will be made available on Ticketmaster. And you can find that through the DRL website. It's the DroneRacingLeague.io, DRL.io. And yeah, the races will also be included in the Drone Racing League's content

• 45:59Speaker 03

  on NBC Sports and on Twitter. And we are filming all of this, all the behind the scenes, what the teams are going through, what we're going through, what happens at the races, all this great human drama associated with the AI is going to be put together throughout the season and then hopefully released as a documentary on one of the great on-demand Blackboard.

• 46:23Chris Benson?

  That sounds fantastic. I'm personally very excited. My small role on the team is to do some AI commentary. And so I can't wait to be in Orlando on the 8th of October. And looking forward to seeing you there, Keith. Thank you so much for coming on Practical AI and kind of taking us into a deep dive into AlphaPilot. And thanks so much.

• 46:46Speaker 03

  Of course. This is a lot of fun. Thanks, guys.

• 46:50Speaker 00

  All right. Thank you for tuning into this episode of Practical AI. If you enjoyed the show, do us a favor, go on iTunes, give us a rating, go in your podcast app and favorite it. If you are on Twitter or social network, share a link with a friend, whatever you gotta do, share the show with a friend if you enjoyed it. And bandwidth for change log is provided by Fastly. Learn more at fastly.com. And we catch our errors before users do hear change law because of Rollbar. Check them out at rollbar.com/changelog. And we're hosted on Linode cloud servers. Head to linode.com/changelog. Check them out. Support this show. This episode is hosted by Daniel Whitenack and Chris Benson. The music is by Breakmaster Cylinder. And you can find more shows just like this at changelog.com. When you go there, pop in your email address, get our weekly email, keeping you up to date with the news

• 47:35Speaker 00

  and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.