Mapping the intersection of AI and GIS

Daniel Wilson and Rob Fletcher of ESRI hang with Chris and Daniel to chat about how AI powered modern geographic information systems (GIS) and location intelligence. They illuminate the various models used for GIS, spatial analysis, remote sensing, real-time visualization, and 3D analytics. You don’t want to miss the part about their work for the DoD’s Joint AI Center in humanitarian assistance / disaster relief. Sponsors: DigitalOcean https://do.co/changelog – DigitalOcean’s developer cloud makes it simple to launch in the cloud and scale up as you grow. They have an intuitive control panel, predictable pricing, team accounts, worldwide availability with a 99.99% uptime SLA, and 24/7/365 world-class support to back that up. Get your $100 credit at do.co/changelog https://do.co/changelog. AI Classroom https://datadan.io – An immersive, 3 day virtual training in AI with Practical AI co-host Daniel Whitenack Fastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship. Rollbar https://rollbar.com/changelog – We move fast and fix things because of Rollbar. Resolve errors in minutes. Deploy with confidence. Learn more at rollbar.com/changelog https://rollbar.com/changelog. Featuring: Daniel Wilson – LinkedIn https://www.linkedin.com/in/daniel-wilson-a274b218Rob Fletcher – LinkedIn https://www.linkedin.com/in/rob-roy-fletcherChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: ESRI https://www.esri.comESRI - GeoAI Blog https://medium.com/geoaiESRI - YouTube: ArcGIS & Artificial Intelligence (DoD JAIC) https://www.youtube.com/watch?v=laDc9TKhpdIPractical AI #72: How the U. S. military thinks about AI (DoD JAIC) https://changelog.com/practicalai/72Books “Pattern Recognition and Machine Learning” by Christopher M. Bishop https://www.amazon.com/dp/1493938436Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Being with your change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at Change Law because of Rollbar. Check them out at rollbar.com. And we're hosted on Leno cloud servers. Head to leno.com/Changelog.

• 0:17Speaker 00

  This episode is brought to you by Digital Ocean. Digital Ocean's developer cloud makes it simple to launch in the cloud and scale up as you grow. They have an intuitive control panel, predictable pricing, TM accounts, worldwide availability with a 99.99 uptime SLA and 24/7 365 world-class support to back that up. Digital Ocean makes it easy to deploy, scale, store, secure, and monitor your cloud environments. Head to do.co/Changelog to get started with a $100 credit. Again, do.co/Changelog.

• 0:59Speaker 01

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and Slack with us around various topics of the show at changelog.com/community and follow us on Twitter. We are at Practical AI FM. Okay, here's Daniel and Chris.

• 1:24Chris Benson?

  Welcome to another episode of the Practical AI Podcast. My name's Chris Benson. I'm a Principal AI Strategist at Lockheed Martin. And with me as always is Daniel Whitenack, a data scientist with SIL International. How's it going, Daniel?

• 1:38Daniel Whitenack?

  It's going great. Got a few more guests in the house these days because my brother-in-laws are back from college due to college being canceled. Or at least virtual for now. So it's a fun household right now. And yeah, that makes working from home, I get a little bit more interaction with humans.

• 1:56Chris Benson?

  Yep. Coronavirus is making life interesting. We have my daughter home. Our school system is closed indefinitely. So, yep, we're doing homeschooling and working and, uh, and no travel. So like I said, life is interesting right now. Uh, hopefully not in too bad a way. We're, we're in early days. Hopefully things will, uh, will not get terrible. I'm crossing my fingers to see.

• 2:18Daniel Whitenack?

  Yeah, yeah. And regardless, people across the world really are, I'm guessing, looking at maps a lot, um, and geography and all sorts of related things, which is very relevant to our conversation today. Who do we have on the show today, Chris?

• 2:34Chris Benson?

  We have today two guys from ESRI, which is the Environmental Systems Research Institute, whose names are Daniel Wilson, who is the AI Lead for Professional Services, and Rob Fletcher, Senior Data Scientist at ESRI. And, uh, welcome to the show, gentlemen.

• 2:50Speaker 04

  Thank you.

• 2:51Speaker 05

  Thanks.

• 2:52Chris Benson?

  So I guess, uh, as we get started here, if, uh, maybe first Daniel and then Rob, if you could each give us a kind of a, an introduction to yourself, kind of, uh, what you do and kind of how you got to this point in your career, before we kind of dive into learning a little bit more about ESRI.

• 3:08Speaker 04

  Sure. So I have basically spent my entire career doing data science, um, but I didn't start that way. Uh, I went to school originally for, uh, engineering physics and applied mathematics. Thought that's where I wanted to go.

• 3:21Daniel Whitenack?

  Uh, I went to school for engineering physics. It's, uh, I think there's a first engineering physics other than me that's been on the podcast. So that's great.

• 3:30Speaker 04

  Cool. Colorado School of Mines.

• 3:32Daniel Whitenack?

  No way. Me too.

• 3:34Speaker 04

  Yeah.

• 3:35Daniel Whitenack?

  Oh, wow. Yeah, this is so weird right now.

• 3:39Chris Benson?

  It's a coincidence.

• 3:40Speaker 04

  That's funny.

• 3:40Chris Benson?

  Yeah. You're probably in the same class at the same time and just don't know it yet there.

• 3:45Speaker 04

  That's, that's very possible.

• 3:47Daniel Whitenack?

  Yeah, I graduated in, in, uh, 2004. Yeah, I'll have to talk to you afterwards to catch up on that.

• 3:54Speaker 04

  Yeah, it sounds good. And I'm sure there was a lot of overlap. Yeah, so really I got started there because I was interested in in math and physics and I I kinda just wanted to get into studying the universe. Found it really interesting, especially on the applied mathematics side. Really wanted to get into theoretical physics. So that's kind of the path that I started down was studying things like general relativity. And I was really fascinated with string theory at the time. And then life happened as, you know, life tends to. And I ended up getting a job at a software company, mostly in the federal government space. Started there and I I basically told my my manager, hey, I want to do some math. I don't really know what math means in this context, but I love math and I want to figure out how to do it.

• 4:37Speaker 04

  So I kinda got put on a a data science team, got a trial run, um, working with the team. It wasn't really called data science at the time. It was back in two thousand nine. Data science wasn't really commonly commonly said, at least most of the circles that I was in. But that's kind of what we were doing. We were working with a lot of different sensor data types and spatial data, primarily spatial data. Um, and really what we did was apply machine learning, statistics, or other algorithms to geospatial data. So my career kind of grew out of that. I'm pretty much self taught. Most of my machine learning that I learned initially was from Christopher Bishop's Pattern Recognition Machine Learning book. And then I had some really great mentors there.

• 5:21Speaker 04

  So that's kind of where I learned it, but I got hungrier and hungrier for more machine learning, artificial intelligence, statistics, any field of applied mathematics, really. And just kept learning more and more and more. And as things started to grow in the the industry as well, I I tried to keep pace and follow it. I've been with Esri for about two years and since then, things have been really exciting, especially in the the geospatial realm, which is where we're seeing a lot of a lot of changes happening right now.

• 5:48Chris Benson?

  Very cool. Rob, how about you?

• 5:50Speaker 05

  Yeah, so I started off very similar to Daniel, actually. I was very interested in engineering to begin with, but I ended up switching over to when I did my bachelor's degree, I studied physics and mathematics. I was really obsessed with this stuff at the time, and so I decided to go straight through to grad school. And so I ended up going to the University of Pennsylvania to do my PhD in particle physics. And while I was there, the main experiment I worked on was Large Hadron Collider, Geneva, Switzerland, on the ATLAS experiment. So I did quite a bit of work there, kind of all over the place. I did a lot of coding. You know, we don't really hire software engineers in physics experiments because they're expensive and grad students are very cheap. So we kind of had to do everything and just get our hands in everywhere.

• 6:35Speaker 05

  And so I, to start off with, I was doing a lot of software, kind of writing a lot of packages that sort of helped do analysis, things like that. As I got further into my PhD, I started working on more actual analysis-type topics. One of the first kind of major projects I worked on was doing an electron identification-type task. So we get all these signals in the detector and we needed to make a decision whether they were an electron or, you know, something else that was the originating particle. I guess I didn't realize it at the time, but this was, you know, a pretty traditional kind of classification problem in machine learning. So I was, you know, doing data science, you know, even quite a while ago and just didn't really understand that that's what I was doing. I was just sort of solving problems.

• 7:18Speaker 05

  Later on in my PhD, I went on to work on some physics that has to do with extending the Higgs boson model, something called the two Higgs doublet model. So we were specifically looking for these kind of additional Higgs bosons that might be lurking somewhere in physics and then produce two photons. So I wrote a lot of code, did a lot of analysis in that specific area right there. And then I think really kind of tipped me over the edge and got me thinking about data science. I noticed that as we were doing these things, you know, this detector was collecting just so much data all the time that... A lot of these older kind of statistical methods that were used to assess uncertainties and really kind of backbones a lot of these analyses, these things were just kind of crumbling

• 8:02Speaker 05

  under the weight of all of this data that we had. It was just so much data that these things were falling apart. And so I really kind of started thinking out of the box and trying to see, well, like, there's got to be something better. I mean, a lot of these methods were, you know, old methods that were developed in, you know, the 60s and 70s that some of the original particle physics experiments. And so I really wanted to start looking around and seeing what other kind of advances were made. And that sort of got me into some of the machine learning data science stuff that I started finding. And I ended up for a portion of my thesis using Gaussian process regression to estimate backgrounds in this model that I was working on. And, you know, again, it was sort of, you know, just the kind of tip of the iceberg for me. I didn't really completely understand that

• 8:46Speaker 05

  I was doing data science at the time. It was just sort of, hey, here's some new fancy technique, which is actually not even all that new, but I can start using to make a real difference in these analyses. And so from there, I decided I really wanted to get out of academia. I just kind of didn't really want to pursue that. I wanted to go into industry and was trying to decide what to do. I started applying for data science jobs because it was kind of the hot thing at the time and got a couple of offers and just really wasn't crazy about any of them. None of the jobs really sounded that interesting. It was, you know, a lot of them were kind of, you know, how many, you know, more clicks can you get on a button that generates ad revenue, you know, things like that. I just didn't really, it wasn't interesting to me at all. And then I actually came across

• 9:30Speaker 05

  Esri because my wife has worked for Esri for quite a while. And she let me know that, hey, we're actually looking for a data scientist. Maybe you should apply. So, you know, it's something that never crossed my mind, but that's why I applied for that. And I was one of the first ones. I was actually the second data scientist hired at Esri next to Daniel. And so very new team and really kind of had this very uncertain sort of direction that it was going. And that really excited me. I thought it was really interesting that there was this whole new realm of topics that this company and to some extent, entire space of GIS just really hadn't tackled yet. That just sounded like a ton of fun to me and I couldn't pass that up. So I ended up accepting that job. And so I've been at Esri now for almost two years, a little less than two years now.

• 10:12Daniel Whitenack?

  Super interesting backgrounds and I'm excited because I didn't know we'd be talking about particle physics or I'd be talking to an ore digger today, but excited, excited about those things. But let's turn a little bit to Esri. For those that aren't familiar, what does Esri do? What businesses are they involved with and what are their primary products or services?

• 10:36Speaker 04

  I guess I'll take that one. So Esri is a GIS company, GIS software company. GIS is geographic information systems. So basically Esri deals with all aspects of geospatial data from, from collection, storage to analysis. A large amount of our users are in the state and local government, federal government spaces where they use our, our software as a system of record for city management, land data, road networks, parcels, really you name it. If it's a place on the earth, it's probably in a geospatial database and Esri maintains a lot of different geospatial databases.

• 11:17Daniel Whitenack?

  So with that data, are you providing some of that data or are you serving mostly as a system to, to manage that sort of data or a little bit of both?

• 11:27Speaker 04

  A little bit of both. So we actually have what's called the, the living atlas and it's basically a collection of geospatial data, authoritative geospatial data that ArcGIS subscribers can use or Esri subscribers. I kind of glossed over this, but ArcGIS is really the, that's the name for our, our primary software platform.

• 11:48Daniel Whitenack?

  And could you tell us a little bit about what ArcGIS is and how is that, I guess that's a, a particular implementation or product, if you will, of GIS? Is that fair?

• 11:57Speaker 04

  Yeah. Well, so GIS, kind of geographic information systems, the software behind it started several decades ago and

• 12:06Speaker 04

  I think it's fair to say as we, as we really started that side of things, ArcGIS specifically has evolved into a collection of desktop and server products for geospatial analysis and data manipulation. Some of that's for like production mapping where people take imagery or other content or even paper maps and turn that into products that people can use.

• 12:33Daniel Whitenack?

  So just as a side note here, I actually got exposed to ArcGIS a little bit more recently. So SIL, some of our listeners will know that we do language related work around the world. And one of the things we do is we have a mapping team that does language survey work and they produce maps of, you know, what languages are used where. And so there's all these sorts of polygons around the world where certain languages are used and populations exist and, and all of those sorts of things. And so just to give an example, that, that was one example that I was exposed to recently. So I'm assuming that, I, I know they're using ArcGIS for this. So I'm assuming that ArcGIS is kind of managing those, those polygons and allowing them to

• 13:18Daniel Whitenack?

  do analysis over them, combine them and that sort of thing. Would that be accurate?

• 13:23Speaker 04

  Yep. Basically all functions of that from geographic coordinate systems to computational geometry, to normal database, relational database utilities, kind of larger scale spatial databases as well.

• 13:38Chris Benson?

  Yeah. Just, just looking through your website here while we're talking, it, it talks about spatial analysis and remote sensing and real-time visualization and analytics, both in 2D and 3D. Are all these applications centered around ArcGIS or other product? Rob, would you take that one?

• 13:57Speaker 05

  Yeah. So most of them are centered around ArcGIS. We do have a lot of kind of extensions and other products that kind of add on to all of these things. Like you said, there's the desktop products, which are the ArcGIS ones. And then we also have some server products, you know, things that are cloud-based to do kind of like large data consumption or analysis. And there's also a bunch of other extensions we have that can do things like even producing the data, like for instance, drone to map. So these are things that, you know, you can take drone footage and as long as you have the correct metadata, it can do all the sort of math under the hood to actually, you know, put those, they're taken by drone onto a map. So you can get, you know, produce your own kind of imagery products and things like that.

• 14:34Daniel Whitenack?

  Interesting, and you mentioned that, I forget who mentioned that, that one of the sort of big clients of Esri are local and national governments. I guess, who's primarily like using ArcGIS for like the main use cases? What are the main use cases, I guess I should say? Why is it important to have this sort of specialized ability to deal with spatial data?

• 14:59Speaker 05

  So a lot of the state and local government usage, we see some of it on like the county level where they might track, say like parcels of land. This allows them to keep a really good system of record and inventory of where all these parcels are, any historical information about them. So all of these different geometries that you see in there are actually enriched with data. So you can kind of think of it as, you know, being like a relational database, but a relational database that has spatial properties to it. And so it's not just, you know, a row in a table, it's, you know, a polygon that sits on a map that also has all that relational information associated with it. And so doing this allows state and local governments to, you know, keep track of, like I said, things like parcel data, tax information. We get a lot of utilities where they can look at where, you know, water lines are. They can

• 15:44Speaker 05

  use this to actually do analysis to try to find out, you know, what kind of precautions need to be taken if, let's say, somebody needs to go in and dig something. They can basically give an area, we say, we need to dig right here. And based on this whole system of record with all this geometry, you can come up with things like, oh, are there any gas lines or water lines or power lines? Or is there anything nearby? Is, are you in some zone that needs to be low noise? All of this analysis can be kind of done inside of the platform. And so the state and local governments use this to do a lot of planning, to keep track of all of their things, but also to do analysis and kind of forward looking as well. Like, for example, if you want to, you know, do some, like a site selection type problem, you know, we need to put something new somewhere. Where do we put it? Where's

• 16:28Speaker 05

  the best place to put it? Where are we gonna not infringe on, you know, other people's property or, you know, keep it near where, you know, some other utility that it needs to have access to, things like that.

• 16:40Chris Benson?

  So, quick question, just as a quick follow-up. You mentioned the word polygon in reference to this, and I was wondering if you could take a second and just tell us what that is in this context?

• 16:49Speaker 05

  Sure. A polygon in this context is really just a series of points that exist on, you know, in some map space, their x, y coordinates. And in general, what makes it a polygon in our realm is that the first point is equal to the last point. And that means that it's just, you know, it's a closed thing, and so you have some defined shape that's just a series of x, y coordinates on a map, basically.

• 17:27Daniel Whitenack?

  Hi there. This is Daniel Whitenack, one of the co-hosts of Practical AI. And when I'm not working on Practical AI, I'm developing my own AI applications, or I'm training teams at other companies. I've been doing this for over 10 years now, and I've trained more than 1,000 people. Now, I'd like to invite you to my new, live, online training event called AI Classroom. In AI Classroom, I'm going to teach the practical skills I've learned over the years using the latest open-source AI technology. You will learn both AI theory, along with practical, hands-on implementations in both PyTorch and TensorFlow. After attending AI Classroom, you'll be able to understand the latest models, implement

• 18:10Daniel Whitenack?

  your own models in code, train computer vision and NLP models, create model inference servers, and experiment with state-of-the-art methods like reinforcement learning. AI Classroom is taking place this May. It'll be taking place live and completely online in a high-quality virtual classroom, so no travel is required. There will be two cohorts with convenient time zones for Eastern and Western hemispheres, so don't miss out. Tickets and more information is available at datadan.io. That's datadan.io. And early bird pricing lasts until April 3rd. See you online in AI Classroom.

• 19:01Chris Benson?

  So Daniel, I've got a question for you. We've been kind of getting this crash course in these last few minutes about GIS and specifically ArcGIS and kind of some of the use cases and applications in a general sense. I'd like to start pulling artificial intelligence, machine learning ideas into it and kind of understand how is Esri using AI and ML in the context of GIS and specifically ArcGIS and your other products and services.

• 19:29Speaker 04

  Yeah, it sounds good. So that's a really big question and there's definitely a lot of sides of that that I'd like to address.

• 19:35Chris Benson?

  Sure, absolutely.

• 19:36Speaker 04

  I'd like to first start with what I think most people tend to think of when they think about applying artificial intelligence to spatial data or geospatial data, geographic data. And that's things like aerial drone and satellite imagery and extracting features from that. So that could be something as simple as, you know, we have a satellite image of an area and we want to extract all the buildings in there. And historically, that's been a manual process. You know, a human has gone through satellite imagery and they've drawn boxes around the building footprints of a house or drawn borders around things or drawn lines where there are roads or anything like that. And we're finding that we can automate a lot of that using mostly modern day deep learning

• 20:20Speaker 04

  techniques, convolutional neural networks and the like. And that's where a large amount of our use case and a lot of our focus on our AI strategy has revolved around. But I also want to say that that's just a piece of everything. Really what we're trying to do is find these areas where geospatial information in geographic context adds value to an analysis problem. There are countless areas where we found applications of AI. Love to talk about some of those, definitely.

• 20:48Daniel Whitenack?

  Yeah, so before we get into the specific applications, I guess I just have a sort of general question about AI and spatial data. So you talked about like polygons and relational information to geographic entities. When we're thinking about like AI models that use geospatial data, are there specific like types of types of models or types of encoders that work with this sort of data out of the box? Or most of the time, are you kind of thinking of using like pulling features out of, let's say, ArcGIS

• 21:27Daniel Whitenack?

  that are related to geography and using those as kind of more, I guess, quote unquote, standard features in an AI model?

• 21:35Speaker 04

  So I think the answer is all of the above. I think the most common applications that we're seeing are things like using geospatial information. So when we say data in a relational database has a geometry like a polygon, more generally, we're talking about an arbitrary shape for an object. So that could be a polygon, it could be a series of points in a line that we call a polygon line, a point, or things like that. Sometimes you can extract features purely off of that geometry and then tie that back to the attributes that are in the attribute table. So an example is if we're trying to make predictions about something on, say, a road network, an application that I've put a lot of work into is car accident prediction.

• 22:19Speaker 04

  There's a lot of features that you can apply to a point on the road that could be extracted from both its geographic context, the attributes of the road itself, or the geometry, such as like how curvy that is. And when we apply this in machine learning models, there's a lot of different applications. So this could be a normal kind of standard out of the box machine learning algorithm, something like a random forest or support vector machine. It could be a graph neural network if you're talking about something on a road graph. But really, it's not limited to any one set of models or one type of data. There's a lot of different applications.

• 22:55Speaker 05

  Yeah, and I'd also like to just say that one of the things that really attracted me to this problem set in general is that using geospatial information in especially deep learning models, it's not always clear how to do that. You mentioned kind of extracting these into kind of the standard sort of inputs that you would normally get. And that's definitely something that we do sometimes. But I think also we try to put a lot of thought into, well, how can we use, you know, the objects themselves as input? And this is just, I think, in general, a very difficult question because, you know, you take something like a polygon, for example. You know, try putting a polygon into a neural network and it can be hard because if you have some arbitrary number of points that make up vertices of the polygon, you know, it's really difficult to fit that into something that's

• 23:38Speaker 05

  like a, you know, a confined or a predefined neural network that has a certain number of inputs. So I think in general, this, it's actually a pretty difficult problem to kind of tackle in the general sense. And that's one of the things I think is really exciting about this field.

• 23:51Daniel Whitenack?

  And so on that point, and I know Daniel mentioned graph neural networks. And so are you actively exploring these kind of unique ways? Because I know this is something that you face with like text, for example, too, which could be, you know, you could have three words or you could have 17 words. But most of the time you sort of encode this information into a fixed size vector somewhere in your in your model. Is it a similar strategy that you're pursuing? Or maybe what are some of the methodologies that might be worth noting that people are exploring in this area?

• 24:26Speaker 04

  One area that I found really interesting and been following very closely over the last couple of years. And it's very similar to the NLP side of the world is attention. So attention normally has been used within natural language processing context to look at the relationship between words in a sentence or characters in a word.

• 24:46Daniel Whitenack?

  As you go, can you define attention also for anyone out there that doesn't know it?

• 24:50Speaker 04

  Sure. So that's basically training a neural network. What parts of an input are relevant to its computation?

• 24:58Speaker 04

  So it's a learned way of kind of approximating. You know, if you look at a scene, you're not, you're not looking at every last part of a scene. What you're doing is you're parsing it into the most important parts first and paying the most attention to those. So in the context of a neural network, attentions are similar to that. And mathematically, it's a pretty simple operation. But applying attention outside of a natural language processing context, we've been trying it across spatial entities. So let's say you have a bunch of points on the ground. We've actually been working on a project to apply reinforcement learning to police patrol optimization. And within that, that particular application, the units that we're applying attention over

• 25:41Speaker 04

  are things like locations of crime, or police patrols, or, you know, basically spatial entities that have an X, Y coordinate, and then maybe other attributes. In this case, what we're doing is we're looking at where we're applying attention to that is how do all of those relate to each other and using that as the inputs to the model? So it's not about bringing in new features, but how the features are handled within the model.

• 26:06Daniel Whitenack?

  That's interesting. It makes me wonder about like with that application, it seems like there's these really interesting intersections with geospatial data and time as well. Cause I'm, I'm assuming when you're optimizing those sorts of police patrols, obviously crime in certain areas is going to be very time dependent as well, or possibly even weather dependent. So there, it seems like there could be multiple layers of geospatial data that's also changing per time. It seems like a quite a rich data set.

• 26:38Speaker 04

  Yeah, absolutely. And we're finding that a lot, a lot of the times that we're applying machine learning to geospatial problems, it's not just one data set. So a lot of machine learning models are, you know, built on top of relational data. So bringing multiple tables together and then creating one big set of input features and running the model. And that's still kind of true for geographic information, but it's about bringing together a lot more disparate sources of information together into a model that I think makes it an especially unique challenge.

• 27:10Speaker 05

  And I think that understanding the time characteristics and some of the problems, especially with something like crime prediction and modeling how crime is distributed, not only over space but over time, is pretty difficult and it's something that we've spent a lot of time on. But also you mentioned weather in there and that's something that obviously comes up all the time. Weather is a big factor whenever you're talking about, especially larger areas, where even weather can be different across an entire area where you're trying to make predictions. So sourcing very good weather data has been a big task of ours that we're always on the lookout for better weather data, more predictions in shorter quantities of time and things like that.

• 27:54Chris Benson?

  So, one of the things that I was noticing as I was looking through Esri's material here that really caught my eye and Daniel's eye as well, actually related to some stuff that we had already talked about in past episodes. So you have done some work with the DoDs, the US Department of Defense's Joint AI Center, which shorthand is called the JAKE. And before I lead into the question, I'll note to listeners that we actually had an episode with Greg Allen, Chief of Strategy and Communications at the JAKE, which was episode 72, it was entitled "How the US Military Thinks About AI." So if you and anybody that's interested in that can dive into the JAKE's perspective on that episode. But going into that, Lockheed Martin, we apparently have done some similar work to you guys in terms

• 28:38Chris Benson?

  of the context of humanitarian assistance and disaster relief. And you had a YouTube video talking about some of the work that Esri's done with the JAKE. And I was wondering if you would just kind of tell us about that as a use case. And then you have a bunch of other interesting use cases that you note online as well. And later on, I wanted to ask you about those as well. But if you would tell us about what you're doing with the JAKE, that would be fascinating.

• 29:01Speaker 05

  Yeah, sure. So we are specifically involved with a portion of the JAKE that we call Hatter, which is the humanitarian aid and disaster response. The main focus of this particular mission is this rapid response to any kind of natural disaster. The kind of rapid response we're talking about is, you know, picture kind of zero to two weeks after something like a hurricane. So this is when, you know, allocation of resources are very, very important, and it can be very difficult to gather information at the same time. And so one of the big things that we are supporting with the JAKE is using satellite imagery and any other data that we can access to. How can we enable them to make better, smarter decisions in a shorter frame of time

• 29:46Speaker 05

  that they can help the most people possible that are affected by these types of disasters?

• 29:51Daniel Whitenack?

  So I appreciate you leading us into this topic of humanitarian assistance, disaster relief. So you talked about kind of gathering whatever imagery and data might be relevant to responding quickly in the case of a disaster. Could you give us kind of a scenario of how that might play out and how the system might help in terms of what you're trying to achieve?

• 30:12Speaker 05

  Yeah, definitely. So how we kind of envision this being used is immediately after a disaster, as soon as this imagery becomes available, which, you know, kind of varies depending upon what kind of imagery it is. But there's lots of satellite imagery you can get, and also NOAA captures aircraftborne imagery pretty soon after these events happen. And so one of the main use cases that we're working on has to do with road detection and specifically road debris detection as well. So what we kind of want to do in the end is to be able to give them a system that allows them to understand how the state of the road network where in the affected area. So let's say, you know, you could find, because of, you know, blockages in the road,

• 30:54Speaker 05

  you could find that there are some community that is completely cut off from any sort of aid because, you know, all of the roads are completely blocked in there. So this gets to this idea that we call a service area, where we can say, yeah, we know that this place is blocked off, and we need to maybe, you know, contract special vehicles come in. And actually deliver resources to them. It can also help with things like routing. So if an agency like the Jake comes in and they have some sort of base setup and they need to be able to distribute resources or send out emergency crews. They need to know where they can get to as well. So this kind of real time routing is something else that we're looking at where we can say, you know, I need to get from our base of operations or whatever point we have here out to some other point.

• 31:36Speaker 05

  How do I get there so that I'm not going to get blocked by trees or by flooding or, you know, things like that? So we, we really want to be able to give them the information that they need to be able to move around in this area without being hindered by, you know, running into a dead end and turning around and to try to find another way around.

• 31:54Daniel Whitenack?

  So I'm curious on that front. One of the things I'm thinking about as I'm thinking about these use cases is like the, the work that's going to have to go into, to the data set behind this. And I'm guessing like, you know, if, let's say the satellite imagery exists of previous disasters and you kind of have that data set. It seems like there's like, I guess, two major challenges from, from my side. I guess one is the data labeling bit and the, the overhead with that. Um, and then second is the sort of variability you could get with disasters, right? So like, let's say a tornado or something in one part of the country is going to look very like the, the ground cover is going to look very different from a tornado in a, in a separate part of the country.

• 32:39Daniel Whitenack?

  So are you thinking about those problems at all in terms of, have you been able to deal with those? Is it, is it a matter of brute force and, and data labeling or there, there are tricks to, to kind of dealing with this robustness issue?

• 32:52Speaker 04

  So on the data labeling side, we've, uh, we've actually partnered up with a company called Figure8. Um, that does large scale image annotation for deep learning models and they've been doing it for a very long time. The, there are those challenges that you mentioned though. There's a large amount of disaster imagery collected by, uh, Digital Globe and NOAA that's publicly available. So Digital Globe has a, open data program that you can get disaster imagery going back, I don't remember how many, but, but several years. So we have quite a few earthquakes and tornadoes and hurricanes, um, from disparate areas. And then.

• 33:33Speaker 04

  Kind of bringing all that data together, we can label that at scale so that we have a large data set to to build our model upon. That's a good start, but we also have a large collection of high resolution pre-disaster imagery. Some of that is from Digital Globe, from their pre-disaster imagery in the Open Data Catalog, and some of that is internal imagery that we have at ESRI from a variety of sources that we can use for this. Sarah, was that a question?

• 34:02Chris Benson?

  No, that was great. I was just gonna actually add a little bit to that is it's a pretty amazing time. I love the work that you guys are describing that you've done there. I know at my employer, we've done some similar stuff. We're basically creating data sets that are multi-sensor, multi-platform, and spatiotemporally synchronized and stuff. So similar stuff, but it's really, I just wanted to comment that it's really an amazing time, I think, that we're going to come into for the hater use case, for the humanitarian disaster relief use case, because, you know, we're going to be able to finally apply AI technologies and data science in general to make a really meaningful impact through the DOD. And I love the way the DOD, through the JAKE, has really engaged a lot of different

• 34:47Chris Benson?

  organizations, each contributing what they can, just as ESRI is, into this. You know, it's a pretty inspirational thing that we're all trying to accomplish. So I think, I was pretty thrilled to see you guys doing that work there and understand that going forward, I think that we're going to have people all over the world as they look at these humanitarian assistance disaster relief scenarios, being able to have tools that they've never had available before to save lives. And I just wanted to really kind of draw out that that social good on it.

• 35:15Speaker 05

  Yeah, I think that, you know, we're really happy working on this. This is one of those projects that makes you feel really good, you know, when you can, when you get results and really deliver things to people, because you know it's making a difference. And also to kind of touch on some of those other things, you know, there's a lot of other people in the community contributing to this as well. And this is actually one of the goals of the JAKE, is also to kind of establish a platform that allows everyone to come in and contribute to this. It's not just ESRI building models and You know, running them, you know, for, for these different use cases. What we're also trying to do is we're trying to use, you know, this huge GIS platform that Esri has been building over a very, very long time as this sort of basis that other people can come in and start contributing to.

• 35:59Speaker 05

  So I think one of the big things for me that really makes these AI platforms really an attractive offering for something like this is that, you know, there's a lot of companies out there that can produce AI and do a great job of it. They do really amazing work, but in the end, you know, they give you sort of a, like a model file, maybe they give you weights, maybe they even give you some results of, you know, the output of their AI model is, but then what, you know, what, what do you really do with that? And I think that that's one of the things that Esri really brings to the table is that we have this entire platform with these hundreds of geo processing tools that have been built over a very long time by some very smart people that can now take that data and run further analysis on it and really make use of it. So in this case, you know, AI can be thought of as just sort of producing the data that

• 36:42Speaker 05

  you can then do even more analysis on and, and, you know, you can kind of unleash this entire platform on it.

• 36:48Daniel Whitenack?

  I really like what you're saying about the platform and we've discussed like a lot about satellite imagery. And I'm just kind of scrolling, as Chris said, you've got a lot of great use cases on, on your blog as well. And I see a lot of those are talking about 3D data as well, which we haven't really talked about much. So there's like the satellite imagery part and we haven't really talked a lot about 3D data and AI on this podcast. So before we get too much further, I kind of wanted to see what your thoughts were in terms of the current state of, of AI utilizing 3D data and challenges around that and what sort of use cases that fits into.

• 37:26Speaker 04

  Yeah, 3D data has actually been a pretty big focus for, I'd say the last year for my team and some others within Esri. There's a lot more LiDAR being collected. That's where a lot of the 3D data that we're talking about is coming from is from vehicle mounted LiDAR. So sometimes it's a A Department of Transportation for, you know, a state or or city that is is driving around to collect roadside assets such as street signs or, you know, anything that they need to know exactly where it is. And in the end, a human has to go through and extract those manually. So what we're doing with 3D deep learning, it's it's all been deep learning to this point. We is trying to extract objects like that or run segmentation models to find

• 38:11Speaker 04

  buildings or vegetation, or there's a lot of different applications that we've been seeing.

• 38:16Daniel Whitenack?

  So with 3D data specifically, I'm just thinking like, oh, 3D data, like, it must be so big and hard to work with. Is that the case? Or are these, you know, these 3D data sets, are they in general like very large and and hard to deal with computationally? Or or is that just kind of a misconception on on my part?

• 38:38Speaker 04

  Well, so it depends. That's usually the answer to most questions is it depends. But there's actually been a lot of really good advances in the last couple of years that make that problem less simple. A lot of times when I I've worked with 3D data in the past, it has been with a what's called a voxel model, where, you know, with an image, you can have a 2D array of pixels and, you know, work in that space, or you can stretch that into three dimensions and then your pixels become voxels. Now, data explodes when you start getting into voxel space, because now you're talking about, you know, the cube of the the number of points, but you can actually run deep learning models directly on groups of points themselves.

• 39:21Speaker 04

  So as long as the number of points in the, you know, the type of objects that you're detecting isn't absolutely massive, you can run these on regular hardware such as GPUs. To talk about a couple of specific models, we've been looking at things like PointNet and PointCNN. So PointNet and then NutsPoints, convolutional neural network. Those actually take in the individual points as features and they use a set of shared weights, basically, and then aggregations further down in the network. So you can have a network that takes an arbitrary number of points as an input and it basically embeds that collection of points into some lower dimensional feature

• 40:03Speaker 04

  space that is then used by just, you know, pretty much a normal neural network to do classification and segmentation and object detection.

• 40:12Chris Benson?

  And I noticed, you know, as you talk about that specifically, I noticed I'm looking at your blog as you're talking and you, because you guys have so much interesting stuff here. There's the PointCNN article. Could you take us through a use case? I want to ask about actually several of the use cases that you had here. Would you take us through that PointCNN one where you talk about replacing 50,000 man hours with AI?

• 40:33Speaker 04

  Sure. I'll do my best on that one since I did not directly work on that project, but

• 40:39Chris Benson?

  No worries.

• 40:40Speaker 04

  Dmitry, the guy who wrote the article, would be the perfect one to ask if you were here. But, so that was using that algorithm I mentioned before called PointCNN. It's close, maybe not quite the state of the art at the moment, but it's still a very, very good algorithm. And in that case, what we did was we had a large training data set that was created manually. So some humans went through and they labeled utility poles and wires within a large LightR data set. I believe that was an airborne LightR data set, but I'm not entirely sure on that one. And it was a TensorFlow-based implementation

• 41:18Speaker 04

  of PointCNN and it was trained on a single Nvidia GV100 GPU. So it has 32GB of RAM. So to answer your question about size, you know, to get a high performance, you still want to have a pretty large GPU or a set of GPUs. But in this case, we didn't even need multiple GPUs to train that model.

• 41:38Daniel Whitenack?

  Yeah, I just had one other quick follow-up. A moment ago, you were talking about voxels. And I'm wondering if you would go ahead and kind of tell us, not everyone, I've worked with it, obviously, but not everybody has, if you would describe what a voxel is and talk about how that works into a typical process. So when you're working with three-dimensional space, what is a voxel? How does it apply to a modeling process? And if you have a favorite use case that, you know, where you've used it, that would be great. Just to, I'm trying to make it tangible in this sense.

• 42:10Speaker 04

  Sure. So 3D data can be represented in a lot of ways. So one way is a point cloud, which is just each point has X, Y, Z and some attributes. You can represent 3D objects as meshes. So when you're looking at like 3D models, such as in, in video games or in three-dimensional, 3D animation movies, like Pixar Animation and stuff like that, that's when you deal with mesh surfaces. And there's some other categories too, that the whole field of computer graphics talks about that. Voxels is what happens when you aggregate information into a fixed grid. So just like an image is a two-dimensional grid, a voxel, you can represent a 3D object as a 3D image in our voxels instead of pixels. So just instead of a X width and a Y width,

• 42:56Speaker 04

  you'd also have a Z width as well.

• 42:58Daniel Whitenack?

  Awesome. Thanks for sharing. And I guess as we kind of get to the end here, there's obviously a lot of things that we don't have time to cover. And I encourage people to check out the Esri blog and some of their videos online, which we'll, we'll link in the show notes to explore further. But I was curious from both of your perspectives, what are you most excited about in terms of geospatial data and AI in the near future? What are you most excited about either working on personally or, or things that you've seen people working on out there that really excites you? Rob, you want to start out?

• 43:33Speaker 05

  Yeah, sure. One of the big things that I've been kind of tackling recently and been doing a lot of reading on is, I guess, not, you know, particular to spatial, but I think could play a big role in it, is unsupervised learning. We kind of consistently see the case that, you know, when we do engage with a client and a client asks us to do a problem, especially if it's something like an imagery type problem, they tend to not have enough imagery to really kind of make a data scientist comfortable in, you know, in results you might get from that. And so one of the big things that I've been thinking about a lot lately is some way of kind of doing unsupervised pre-training on, you know, some of these networks so that, because satellite imagery all looks very similar. I would say, you know, for

• 44:17Speaker 05

  the most part, satellite imagery from different parts of the world looks a little bit more similar than, you know, you just your kind of random like ImageNet dataset or something like that, you know, just sort of cellphone pictures or oriented imagery. And I think that that could be a big advantage in that if you can extract a lot of information, you know, I kind of think of something like the image equivalent of BERT, for example, which, for those who don't know, is a language model that gets pre-trained on a massive corpus of text in order to sort of learn, not necessarily specific tasks, but learn kind of about language in general. I've been really interested in this idea for sort of imagery where, you know, instead of learning to do some specific tasks, say, find a car or find a tree, you

• 44:57Speaker 05

  can teach some network to kind of understand what an object is. You know, if I see a line kind of going, you know, that's perfectly straight over here, you know, I can assume that, you know, that line is probably going to continue over here and that these all kind of constitute maybe one object or one thing that is like a discrete unit. And I think if you can get to that, that state, I think that the amount of data that you'll actually need to make really good progress, some of these imagery related tasks go way down. And this really opens up AI to a lot of companies that maybe don't have the capital or the time to invest in huge data sets and to label them as well.

• 45:34Chris Benson?

  Sounds good. Daniel, do you have, uh, what are you excited about?

• 45:38Speaker 04

  Well, uh, I'm excited about most of it, but let me just kind of mention that, um, a lot of the work that I've been doing for my career has been on the computer vision side. And I want to say I kind of lost interest in it as soon as deep learning started getting good. So really what I'm interested in is everything that's not imagery. Imagery is huge. It's, there's a lot of amazing opportunities and, and ways that we can apply deep learning to it and extract a lot of value. So it's, don't get me wrong. It's a very amazing place to be. But the problems that I've had the most fun working on, and I think are really kind of the future of a lot of this geospatial data are either multimodal or non-imagery

• 46:20Speaker 04

  data sets. So that's bringing things like imagery and text and what we call vector data, or you know, basically tabular data that also has, you know, geometric information, all of those together. So just to motivate that a little bit, the problem that I've been working on most recently is a car accident prediction. I actually have an article on that, uh, that Medium blog about the way we used to do it. And I hopefully will have a one about the way we're doing it now coming out fairly soon. But that's, that's a situation where if you think about it, you know, imagine yourself on the road and there is, you know, the, the terrain affects how far you can see, affects the slope of the road. The road itself could be made in a curvy way. The speed limit changes things, the,

• 47:04Speaker 04

  the number of lanes changes things, the position of the sun in the sky changes things. The weather, so cloud cover or precipitation, snow, the conditions of the roads themselves. And all of that can be captured in a lot of different data sets. Some of that could be imagery. Some of that could be in the, the actual geometry described on the road. Some of that could be in the table. It's all in different places and it becomes really difficult to assemble all of that information together into a model that can actually make a real world impact. So that's really what excites me is within the geospatial context is all of this data can be brought together in really interesting and novel ways that I think people are just starting to scratch the surface of.

• 47:48Speaker 04

  My whole career, I've been, you know, working on spatial data science and it's been really hard to find, you know, oh, someone's done it before. I mean, chances are someone's tried it before, but it's such a hard and interesting problem that now that I think with the advent of so much large scale spatial data, a lot of people are starting to take notice of this field and there's a lot of great work happening that we can capitalize on.

• 48:13Chris Benson?

  I'll tell you what, that's a great place to close it out. Well said, both of you. And so I guess we'll wind up. Daniel Wilson and Rob Fletcher, thank you so much for coming on the show with Daniel Whitenack and myself to tell us all about what Esri is working on. It's been really fascinating and talk to you soon.

• 48:32Speaker 04

  Yeah, thank you for having us.

• 48:34Speaker 05

  Thank you for having us.

• 48:39Speaker 01

  Thank you for listening to Practical AI. If you're not following Practical AI FM on Twitter, you're missing out on clips and highlights from past episodes, links and repos from around the AI and data science community and more. Follow us, Practical AI FM. You won't regret it. Practical AI is hosted by Daniel Whitenack and Chris Benson. It's produced by me, Jared Santo, and our music is brought to you by The Beat Freak, Breakmaster Cylinder. We have awesome sponsors. Support them. They support the show. Special thanks to Fastly, Linode, and Rollbar for helping us do what we do. If you aren't receiving Changelog Weekly every Sunday, you are missing out. It's our take on this week in the world of software, what's interesting and why. Head to changelog.com/weekly to subscribe.

• 49:22Speaker 01

  Get it for the price of a free cheeseburger. Thanks again for listening. We'll talk to you next week.