How Microsoft is using AI to help the Earth

Chris caught up with Jennifer Marsman, Principal Engineer on the AI for Earth team at Microsoft, right before her speech at Applied Machine Learning Days 2019 in Lausanne, Switzerland. She relayed how the team came into being, what they do, and some of the good deeds they have done for Mother Earth. They are giving away $50 million (US) in grants over five years! It was another excellent example of AI for good! Sponsors: Fastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship. Rollbar https://rollbar.com/changelog – We move fast and fix things because of Rollbar. Resolve errors in minutes. Deploy with confidence. Learn more at rollbar.com/changelog https://rollbar.com/changelog. Linode https://linode.com/changelog – Our cloud server of choice. Deploy a fast, efficient, native SSD cloud server for only $5/month. Get 4 months free using the code changelog2018. Start your server - head to linode.com/changelog https://linode.com/changelogAlgolia https://www.algolia.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our search partner. Algolia’s full suite search APIs enable teams to develop unique search and discovery experiences across all platforms and devices. We’re using Algolia to power our site search here at Changelog.com. Get started for free and learn more at algolia.com https://www.algolia.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship. Featuring: Jennifer Marsman – GitHub https://github.com/jennifermarsman, X https://x.com/jennifermarsmanChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonShow Notes: Jennifer Marsman’s blog https://blogs.msdn.microsoft.com/jenniferAI for Earth https://www.microsoft.com/en-us/ai-for-earthProject Premonition (mosquitos/genomics) https://www.microsoft.com/en-us/research/project/project-premonitionFarmBeats (agriculture) https://blogs.msdn.microsoft.com/jennifer/2017/05/01/precision-agriculture-with-iot-machine-learning-drones-and-networking-researchWildbook (recognizing individual animals) #1 https://www.wildme.orgWildbook (recognizing individual animals) #2 https://www.wildbook.orgRepository of data sets http://lila.scienceMicrosoft Green Twitter https://twitter.com/microsoft_greenUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Baiting for change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at change law because of roll bar. Check them out at rollbar.com and we're hosted on Lino servers. Head to linode.com slash change log.

• 0:20Speaker 00

  Welcome to Practical AI, a weekly podcast about making artificial intelligence, practical, productive and accessible to everyone. This is where conversations around AI, machine learning and data science happen. Join the community and slack with us around various topics of the show at change.com slash community. Follow us on Twitter. We're at Practical AI FM. And now into the show.

• 0:46Chris Benson?

  Welcome to the Practical AI Podcast. This is Chris Benson, your co-host, as well as the Chief AI Strategist at Lockheed Martin RMS APA Innovations. This week, you're going to hear one of a series of episodes recorded in late January 2019 at the Applied Machine Learning Days Conference in Lausanne, Switzerland. My co-host, Daniel Whitenack, was going to join me, but had to cancel for personal reasons shortly before the conference. Please forgive the noise of the conference in the background. I recorded right in the midst of the flurry of conference activities. Separately from the podcast, Daniel successfully managed the AI for good track at Applied Machine Learning Days from America, and I was one of his speakers. Now, without further delay, I hope you enjoy the interview.

• 1:29Chris Benson?

  I have Jennifer Marsman, who is Principal Engineer on the AI for Earth team at Microsoft with me today. Welcome to the show, Jennifer.

• 1:36Speaker 02

  Thank you for having me, Chris.

• 1:38Chris Benson?

  So I was fascinated to learn that Microsoft had an AI for Earth team. I think that's super cool. And I would like to know more about it. Could you tell me a little bit about yourself? And you know, kind of how you got on the team and a bit about the team itself? Give us kind of a broad intro to it.

• 1:54Speaker 02

  Absolutely. So I'm very excited about the AI for Earth team as well. I actually heard about it before I joined the team as one of the kind of new initiatives Microsoft was going forward with. And I wrote an essay to the hiring manager on why they needed to hire me because I was so excited. Excited to be a part of it. I love that. Oh, yeah. Oh, yeah. I went hardcore for this one. Um, but here's the idea behind it. AI for Earth started by the guy who is now my manager, a dude by the name of Lucas Japa. And Lucas is actually Microsoft's first Chief Environmental Officer. Okay. Which I did not even know that we had prior to taking this job, but apparently that is a thing. And what's really cool is Lucas worked for this guy Eric Horvitz for a very long time.

• 2:38Speaker 02

  And Eric Horvitz is, of course, a big name in the machine learning world. He's been at Microsoft Research for a long time. And Lucas's background was actually in biodiversity and conservation. And that's where he got his PhD. And so coming from that kind of environmentalist background and then working with like for Eric for a time really got him thinking all the time about the intersection between AI and these hard environmental issues. And when you think about the progress we've made with artificial intelligence, just in the last 15 years. It, I mean, it's astronomical. It's so much like things that were, so I'll give away my age here a little bit.

• 3:18Speaker 02

  But when I finished grad school, I did my grad school in, actually, my, my degree is in artificial intelligence, which gives you an idea of how long ago I did my thing, that that was still a degree then. Now that's way too broad a field to have as a degree. Now it's NLP or computer vision or whatever. But back in the day, this was maybe 16 years ago, at least when I got my degree in, in AI. The, um, we've made such progress since then, like with deep learning, just in the last, what, 10 years or so. We've been able to solve really hard problems like automatic machine translation and speech recognition and a lot of these other things. And so the idea was since machine learning is having this exponential effect,

• 4:03Speaker 02

  could we apply it on hard things like climate change? Or like being able to conserve animals that are going extinct at a rate that's like thousands of times the natural rate currently? Or how to build or how to grow more food on land in a sustainable Way so you're not, you know, burning out the land by doing, you know, you can get the the short term benefits, but then it's not good for the land long term. But to be able to increase the yield while still doing it in a sustainable way. And then provide clean water to people. So all of these really hard challenges. Like what would happen if we applied machine learning to them? And that's what was keeping Lucas up at night. And so he wrote a white paper and that white paper became the basis of essentially the AI for Earth program.

• 4:47Speaker 02

  So at its heart, or at the beginning, it is a grant program. So Microsoft publicly committed $50 million over a five year timeframe. And that is money that is available as grant money to anyone who is doing machine learning or data science work in the areas of agriculture, water, climate change and biodiversity. So any one of those for is fine. And it can be a startup, it can be a nonprofit, large corporations, independents, anyone at all is welcome to submit a grant and we accept grants on a quarterly basis. I think the next deadline is in April 2019. And so folks are welcome to apply for a grant.

• 5:30Speaker 02

  And so that's kind of the one, the start of it. But our kind of aspirations are even grander than that. As you said in the intro, I am an engineer. So like I'm, I want to be building models. I want to be doing the fun data science work myself. And so we're actually doing a great combination of that. So there's, we partner very closely with Microsoft Research. So there's several Microsoft Research projects that we're funding and working very closely with. We're actually, our engineering team is kind of building some of our own APIs where we take models that may be useful to a number of people who are doing sustainability efforts and then exposing those models as APIs so that people could call them and utilize them. And then the third bucket of engineering work is in the area of our AI for Earth grant recipients.

• 6:15Speaker 02

  So being able to support them wherever they're at in their machine learning journey. So some folks that we work with are these brilliant environmentalists, but maybe they don't have as much of a machine learning background. And so they have these fantastic data sets and so we help. Get started with how they can apply machine learning techniques to that data. And then some people are, you know, PhDs in machine learning, academics and such. And those people, you know, may be just, you know, absolutely brilliant at machine learning, but maybe they would want to know how to use Azure, you know, to be able to scale out their work or, you know, get results faster, any of those things. Yeah. So my job, like I try to, I actually own the three Es for AI for Earth. Engineering, evangelism and education. So I do.

• 7:00Chris Benson?

  That sounds like a fun job right there.

• 7:01Speaker 02

  It is. Oh, it is. So I'm just kind of trying to work across those. It's a lot of, it's a lot of work, a lot of stuff, but I'm really, really exciting to be able to be a part of it.

• 7:10Chris Benson?

  So that sounds super cool with it. So it sounds like you both, you know, fund these other projects that are out there through the grants that you talked about. But you also mentioned that you have your own engineering team. So what is the, what kinds of things would you focus on helping other people from a grant perspective? What kinds of things would your own engineering team and how do those interact?

• 7:31Speaker 02

  Yeah. Great question. So essentially the grant proposals are whatever people submit. So those are not something that I have control over. It's whatever people want to submit. We accept things that seem feasible, that seem like they're doable. We want to make sure they have a data set that will, will actually work and something that seems like it's possible. But as long as it's a feasible, a data set that would support what they're trying to do. And it seems feasible and they seem like they have the knowledge and background that they could do it. It's probably worth at least a small investment of Azure. Give them at least some Azure credits to let them try it, that sort of thing. And then our own engineering team, what we're trying to focus on is are there things that we could build?

• 8:13Speaker 02

  That are things that would work for a large variety of people, right? So almost trying to do like the way that I think about it personally. And this, this is kind of just Jennifer Marsman speaking, not an official AI for Earth thing.

• 8:25Chris Benson?

  No worries.

• 8:25Speaker 02

  For me, I try to think of what are things that would be useful to a variety of people, right? Because a lot of times the way a grant recipient should work.

• 8:37Speaker 02

  So let me take a quick tangent. Sure. So for a long time, there was a guy who actually does a lot of speaking now about AI in China, Caifu Lee, who has worked at Microsoft and Google and such. And so he's also a big venture capitalist and funds a lot of startups. And one of his words of wisdom about startups is that startups should never build a platform. They should build a product, right? They should solve one specific problem and solve it well and then expand out to a platform. Right? Okay. And so I think about the work that these nonprofits and academics and such are doing the exact same way. I think that they, a lot of times, they are trying to solve such a specific problem, like I need to be able to predict how well flooding for this particular river in Africa, in this one place in Africa.

• 9:22Speaker 02

  So it's a very specific problem. And they shouldn't focus on the high back. They should solve the problem that they need to solve because if it's a nonprofit trying to solve a thing, like they need to focus on their mission. Right? And so what I think our team should be focused on is taking that step back and building real APIs, like, okay, is there something, is there a way that we can then build things that would work for a wide variety of people, kind of in the spirit of that, you know, rising water lifts all boats. Sure. So that we can enable the entire sustainability community to do great work. So we've started with two specific APIs that our engineering team has produced. One of them is around land cover mapping.

• 10:06Speaker 02

  So we did, and essentially, if you're not familiar with land cover mapping, what that means is that given some aerial imagery as input, so typically, like what we're using is four band imagery. So it's the NAPE imagery, if you're familiar with that. But it's essentially RGB, so natural color, as well as a near infrared channel. Okay. So those are the four bands. And then take that as input and then be able to say, okay, what? What pixel by pixel level, and this is, we're actually doing it at one pixel resolution, is this water or tree or barren? So essentially, it's a road or a, you know, house, something you can't build on. And we have this enumeration of different classes. And so pixel by pixel level, we can enumerate those things. And so that's great for use if

• 10:51Speaker 02

  you are, you know, disaster relief kind of work, trying to look at, or flood levels, like realizing, okay, where are the water lines now?

• 10:58Chris Benson?

  Absolutely.

• 10:58Speaker 02

  You can use that. It's great for urban planning. It's great for a lot of different scenarios. So that's one great general purpose thing that we're trying, that we have an API for that's available already. If people would like to email, I think it's AI for Earth APIs at Microsoft.com. I can give you a key.

• 11:15Chris Benson?

  Okay. And we can add that to the show notes as well later on. So we can check that and make sure that's good.

• 11:19Speaker 02

  That would be perfect. Thank you, Chris.

• 11:20Chris Benson?

  Sure.

• 11:21Speaker 02

  And then another one we're working on is iNaturalist, if you're familiar with them, has released a large public data set. They do really amazing work. And we've partnered with them as, and have given them some funding as well. And what they have, if you haven't downloaded it, oh my gosh, you should, because it's really good. They have a really great mobile app that's available for iPhone and Android. And essentially, what you can do is they have a classifier that allows people to, you can take their app, and then wherever you are, any animal, any plant, or any fungi, you can actually take a picture and then record an observation of where that is in nature. So it's great. Then biologists and

• 12:02Speaker 02

  such can use that data to get a good sense of the, you know, the ecological makeup of any given area. And it's just citizen scientists, everyday people like you and me, who like, might care about the environment, but are not necessarily a biologist, can take pictures. And then that way scientists know that, okay, this particular species of plant is located here, and this animal was seen here.

• 12:22Chris Benson?

  That's fantastic.

• 12:22Speaker 02

  It's so cool. And so what's really nice is that they released this data set then and make it available for folks. So we actually trained a classifier that will distinguish between

• 12:33Speaker 02

  And like it's remarkably accurate for plants and animals, giving you the species. So given an image of something, here's the species, it will return the exact species name. And so you can do this across animals, you can do it with plants. And I believe here at this conference, Grant Van Horn is actually speaking on iNaturalist. He did a lot of the machine learning work.

• 12:54Chris Benson?

  Okay, I'd like to hear him speak then.

• 12:55Speaker 02

  Yeah, it's really cool. It's really amazing stuff.

• 12:57Chris Benson?

  So.

• 12:58Speaker 02

  And so they're one of our great recipients as well.

• 13:00Chris Benson?

  Awesome. So a little follow up question to that. Just so how is it when you normally, when most people are thinking about Microsoft, they're thinking about technology, they're thinking about cloud, obviously thinking about Windows and Office and other, other things that Microsoft is doing. How did they get into this in particular? What was the motivation for the company to back this?

• 13:17Speaker 02

  Yeah, great question. So as Microsoft's Chief Environmental Officer, Lucas owns not only the AI for Earth program, but he also owns our environmental sustainability work. And in terms of environmental sustainability, like Microsoft has won numerous awards for that. We've been carbon neutral since 2012. We actually have different departments inside Microsoft pay a carbon tax to offset usage and flights and stuff like that. We pay a carbon tax and that's something that was not very popular when first introduced to the company, but has been great. But the way we look at it is even if Microsoft was absolutely perfect. So let's say our data centers were all completely like underwater data centers and everything was just fabulous. That's only so much.

• 14:01Speaker 02

  There's only so much impact Microsoft as a company is having, you know, just in our own operations.

• 14:06Chris Benson?

  Sure.

• 14:06Speaker 02

  So how could we scale out even more? And so AI for Earth was really our answer to that question by dedicating this $50 million over five years to people. That enables everyone to be able to partake. And, and, and what's, what I, what I love about that is that we are asking oftentimes the people with the least resources to solve the world's hardest problems, right?

• 14:32Chris Benson?

  Sure.

• 14:33Speaker 02

  It's nonprofits, it is these, these sustainability groups, academics who are, were asking to solve these hard problems of like climate change and, you know, reducing, you know, the rate that these species are going to extinct and a lot of other really, really hard challenges. And so. This is a way that Microsoft can kind of dedicate what we're good at, namely, we can provide machine learning knowledge, and we do that through the education arm of the AI for Earth program, as well as our data centers. So here's Azure, here's like all this infinite compute power with using the cloud. Great, you know, use this. And then that gives people both the knowledge and the access to cloud that they need to help them be successful

• 15:16Speaker 02

  in their areas of expertise. So these folks who are grantees are tend to be good at the stuff that I'm not good at, which is the environmental end of it, and that strong domain knowledge in the areas of agriculture and water and such. So put those things together, and you can do something really powerful.

• 15:31Chris Benson?

  That sounds great. So do me a favor, and in your mind, think of maybe one of your favorites that have come through, and take us through what it looks like from the point where they decide to apply, and how you help them, what are the different, both from the what you might call, I don't know, the business side of it, as well as the machine learning side, and just give us a picture, because there may be people out there in nonprofits listening to this right now that suddenly jump at this opportunity, and tell them what this project will look like when they engage you.

• 16:03Speaker 02

  Absolutely, absolutely. Okay, so Chris, I'm gonna have a really hard time choosing a favorite, because I love, there's so many good stories. Like, our grant recipients are doing such amazing things. So I'll try, let me tell you a few stories, and then we'll kind of go back to the, you know, if you're a startup, how do we get started here, or if you're a nonprofit, how would we get started? All right, so let me do one in particular that is a, something that's kind of inside of Microsoft Research, and then one grant recipient. So let me do a Microsoft Research one first.

• 16:35Chris Benson?

  Sounds good.

• 16:35Speaker 02

  Okay, so there is an amazing gentleman by the name of Ranveer Chandra, who is part of Microsoft Research, fabulous guy, very, very smart. He got his start doing networking and battery technology, which is kind of cool. And he, essentially, there is a very, very hard problem where by the year 2050, it is predicted that we are the The pace of the amount of food that we can grow is is just not going to be sufficient in in no way.

• 17:05Chris Benson?

  I've actually heard that separately outside this.

• 17:07Speaker 02

  Yep, this is, I think it was spoken at the meeting of the United Nations in 2009. Someone put forward something that it was, we would need to essentially double our current rate of food production if we're going to be able to feed everyone with the world's growing population rate. And so like, he started out like really trying to think about that problem, like, how could we solve that? Like, what what kind of things could we do? And so he hit on the idea of precision agriculture. And I wasn't familiar with this prior to joining the team, so let me explain it for the folks who who aren't familiar. So the idea behind precision agriculture is instead of, for instance, like homogeneously watering a field where every part of the field gets the exact same amount of water, like we traditionally do with the large sprinklers and such. Precision agriculture or precision irrigation, for example, is let's only water the parts that actually need water.

• 17:52Speaker 02

  Because if there's little dips in the field or things like that, water runs down, some of the field may get more water than others. And so obviously, that seems pretty easy to do, right? Because, you know, there's, if anyone's been out and does any IOT work and has ever played with these things, you can go to atefruit.com or whatever and get little moisture sensors that you can put in the ground. So pretty, pretty easy to do. But, um, so I was talking with Randvier and I'm like, okay, this seems doable. And then he told me it actually worked out to about a thousand dollars a sensor to do this. And I was like, you must be kidding me. Here, look at this website, it's much cheaper than.

• 18:24Chris Benson?

  So what was driving that cost up?

• 18:25Speaker 02

  Great question. So the actual sensor, of course, is only like tens of dollars. My next guess was maybe power, right? But it turns out we use solar panels to drive in in power and those little chips don't take that much anyway. A little, um, little board doesn't take that much power anyway. And a solar panel is like 50 to 100 dollars US. So still nowhere in the neighborhood of 1,000. Turns out where that 1,000 dollars is coming from is actually connectivity, getting data from the actual farm into the cloud. And so there's actually a couple of different ways that we can handle that. A lot of times when you think about it in this farm, if you've ever driven through a more rural area and your cell phone coverage kind of goes out, right? There's, the cell phone coverage tends to be not be quite as good because there's not the population to support it.

• 19:09Speaker 02

  And then there's also, when you think about third world countries and stuff like that, where the wireless access may not be as good, or Wi-Fi is good, but the problem with Wi-Fi is that it only stretches so far. Like, I know when I'm going to get my kids from the bus stop across the street, I'm literally right across the street from my house and can't reach the home via Wi-Fi. You lose it, right?

• 19:26Chris Benson?

  Totally get.

• 19:27Speaker 02

  So if you're dealing with like hundreds of acres of farmland, like Wi-Fi is just not, you have, need a million repeater. So it's just not feasible. So there's actually two different ways we've solved this problem. One of them uses machine learning and one of them uses cutting edge networking technology. So the first one using machine learning is you can actually reduce the number of sensors that you need by putting sensors in fewer places and then augmenting that with either drones flying overhead or other ways of collecting aerial imagery. And then you can feed a machine learning algorithm, both those aerial images as well as the data from the sensors that you do have, and from that be able to extrapolate

• 20:08Speaker 02

  the values of the entire field. So it uses essentially, conceptually, what the machine learning model is using is visual smoothness as well as spatial smoothness. So the idea that two things that are close together are likely going to have similar values and then things that have similar colors. So think of if we, if we have a, like a patch of farmland that's darker because it's been freshly watered and another patch is darker, those, you know, two, both dark patches might indicate that they have similar amounts of water applied to them. So that sort of thing.

• 20:39Chris Benson?

  It makes me think of topographical maps, you know, with the contours on them, maybe different colors.

• 20:44Speaker 02

  Yep, yep. That also, so, so having that information, feeding both of those in, we have a machine learning model that's based on Gaussian processes that can extrapolate and then give you essentially, here's how much water in the field. So that's, that's one example. The other part of it that's really cool, not as machine learning related, is that he also brought in his, his old networking background to do the concept of television white spaces. So this is really cool, Chris. So he's actually using-

• 21:10Chris Benson?

  And for the audience, she's leaning in and she's so excited about this.

• 21:13Speaker 02

  So this is the idea of using unused television channels to be able to- Send data packets over there like your own internet. Okay.

• 21:22Chris Benson?

  Sure.

• 21:22Speaker 02

  So think about it. So a television station can broadcast very, very far because it's lower spectrum.

• 21:28Chris Benson?

  Right. So those frequencies are going to stretch further.

• 21:30Speaker 02

  And so you can actually use one television right spaces router and it can, you know, stretch miles and miles and miles, right? Tens of miles, right? So that's a really, really cool thing. And a lot of times when you get to these either third world countries or rural areas where we're setting these things up. There's a lot of unused television stations, so a lot of empty bandwidth there. So you do have to work with the local government. You can't just start broadcasting things on television stations. Right. It is very regulated. But if you work with the government, you can actually send data over these television channels. And so that's the other part of what they're doing. And that's also driving down the cost as well.

• 22:04Chris Benson?

  That sounds great.

• 22:04Speaker 02

  It's such cool work. So it's amazing stuff. So that is called Farm Beats is the name of the project. And I can provide a link to that in the show notes as well. Okay. And then let's talk about one of our grant recipients, our fabulous AI for Earth grant recipients. So I love all of them and they're all doing such cool work. But one in particular that would be fun to talk about is a company called Wildme. And their platform is something called Wildbook. And so the problem they're trying to solve is to be able to recognize individual animals. So not just, you know, zebra, not a zebra, but rather this is Ziggy the zebra versus Zoe the zebra, like zebra 5715, you know, kind of thing. So, and this can be really, really helpful. Because when you think about it, there was actually an amazing article in, well, not an amazing, a very sad article

• 22:50Speaker 02

  in National Geographic a while back where some folks were actually trying to tag this rare whale in the Pacific Northwest. And they actually botched the tagging job and ended up killing the animal, which was the exact opposite of what they wanted to accomplish by tagging it. That it was such a rare animal, they wanted to track it and all that sort of thing. And so the idea is can we use computer vision to do that instead? And that can lead to a whole, when you understand like the animal population, that can drive so many benefits. You can estimate population densities. You can track migration patterns, all kinds of cool things. And so.

• 23:25Chris Benson?

  So how, and let me, as you answer this, how are they going to use computer? Vision to accomplish this particular task.

• 23:31Speaker 02

  Great question. So they are taking images and training essentially a wild book per animal. So when you think about it, I don't know if you've ever heard this, but you know, humans have very individual fingerprints where each of us have a unique fingerprint. Well, with zebras, their stripe patterns are

• 23:50Chris Benson?

  Heard that as well.

• 23:51Speaker 02

  And then same thing with giraffes. Actually, the spot patterns on their neck, their long necks have uniqueness.

• 23:56Chris Benson?

  Really? I didn't know that.

• 23:57Speaker 02

  Yeah, that and then like the shape of an elephant's ear, the spot patterns on cats. So they've actually created a wild book for a lot of different large mammals and they train it with this kind of data. And then, and then they're able to recognize individual animals. So it's a very, very cool process. But here's where the story goes from good to great.

• 24:22Chris Benson?

  I'm waiting.

• 24:22Speaker 02

  They actually are then augmenting their data using social media. So the idea is, let's say, so one of their, one of their wild books is whale shark.org.

• 24:33Chris Benson?

  Okay.

• 24:33Speaker 02

  That is the wild book for whale sharks. And let's imagine that some, you know, random person goes on a whale watching trip and they see whale sharks and they're taking videos of them and then they're posting them to YouTube. Well, Wildbook has an intelligent agent that wakes up every night at 10:00 PM and it searches the internet and it looks on these social media sites. And then it will find instances of people using just like natural language processing, it can find or just regular search, you can find people who are posting about whale sharks.

• 25:02Chris Benson?

  Sure.

• 25:02Speaker 02

  And then they're extracting frames from those videos and then running their object detector to find the whale shark in the object. And then classifying those things and recognizing that individual animal. So they're finding both new whale sharks that way. Their researchers haven't been exposed to before. Isn't that great? Isn't that so much better than using Instagram to show your food?

• 25:21Chris Benson?

  Yeah.

• 25:22Speaker 02

  Right? Like what an innovative use of social media.

• 25:24Chris Benson?

  I could be on a vacation with my family. We're on a boat. We see a whale shark. We take a picture. We, we tweet it.

• 25:29Speaker 02

  Yeah.

• 25:30Chris Benson?

  And then later on that evening, it comes in, not only is it able to detect it, but it's also able to know exactly which individual in the population we're looking at. Is that correct?

• 25:38Speaker 02

  That is correct. That is correct. And they actually give you a little comment, which is kind of cool. The agent will, it's looking not only for the animal, but it also wants to know the, the when and the where. So when did they see that and where did they see it? And so they're actually using some NLP actually on the, on the, like the YouTube page itself. For example, if it's posted, it said, last month, we saw this and you get the post date there and you can do some, their, their NLP is good enough to like do the math and figure out, okay, it's approximately this time. Um, and then where, and they'll again, try to extract that from the video. But if they can't, it will, um, this is all automated. It'll post a comment and say, oh, where did you see that? Or they'll take whatever misinformation is missing, post the information and then get that data back. And then, um, uh, and then they actually post a little comment saying, hey, you know,

• 26:22Speaker 02

  your picture was able to help science and help us help this conservation effort. And then they'll point to the whale, the, the page of that particular whale, um, that they've contributed images to. Um, which is really, really nice. Uh, so YouTube is a great one because that's, that all is public domain. By default, any public things are, are shared based on their licensing. So anything like that where it's a public image, they can track those kinds of things and then be able to utilize that, uh, for data. But it's really cool because then you can like go click on it and be like, oh, there's my whale shark that I saw. And, um, they, they have some neat stuff where they allow people to nickname them. So you can like give your whale shark a name and stuff like that. And then you can see all images, other images that people have taken of them. There's like an adopt a whale feature type thing. And then it actually shows, oh, and this is why they call it Wild Book.

• 27:05Speaker 02

  They call it like the Facebook for animals because it actually shows, um, co-occurrence charts. So the social graph of this whale has been seen with these other whales.

• 27:13Chris Benson?

  Yes.

• 27:13Speaker 02

  So you can see exactly kind of the, um, some of the, um, you know, parent child type relationships.

• 27:18Chris Benson?

  So you get the whole graph of the relationships that their whales have with each other.

• 27:21Speaker 02

  Exactly. Exactly.

• 27:22Chris Benson?

  That's pretty amazing.

• 27:23Speaker 02

  So isn't that neat? And then you can actually see, um, using the when or the where, sorry, information. You can see, um, like migration patterns of individual whales. It's phenomenal stuff. It just, oh, it gets me so excited.

• 27:35Chris Benson?

  So all this sounds incredibly fascinating. I'm an outdoors person, so I'm particularly interested in covering this. And frankly, this may be the first practical AI episode that my six-year-old daughter wants to listen to because she's really into animals and stuff. So I suddenly have a hook for the family on this. So like, if you are out there and you are interested in engaging you guys, I guess there's a whole lot. One of the big challenges, obviously, is data sets. Do you have any recommendations on where people can find data sets that will help him? Do y'all have a repository? Are there good go-to sources that you recommend?

• 28:12Speaker 02

  That is an awesome question. 10 points to Gryffindor. So we are actually in the middle of that process right now. So currently, we are collaborating with several other organizations and we established something. If you go to http:lila.scienc.

• 28:29Chris Benson?

  Yes.

• 28:30Speaker 02

  That is a repository of camera trap data, which one of my esteemed colleagues, Dan Morris, is very passionate about camera trap imagery. And so he started putting together this data set on lila.scienc. And the Lila is actually a reference to the Library of Alexandria, which is kind of cool. So that is available today. So you can go and there are several data sets that contain lots and lots of different animals and imagery there. So that's one great start that we have right now.

• 29:04Chris Benson?

  Okay.

• 29:05Speaker 02

  And then we're trying to do even more work. I would love to be able to, we're currently in the process of trying to host more and bigger and better data sets. So that is work in progress. So hopefully, this time next year, I will have even more great data sets to share with you. But in the meantime, Kaggle is also a wonderful resource.

• 29:25Chris Benson?

  It always is.

• 29:26Speaker 02

  Oh, my God, I love Kaggle. Isn't it the best?

• 29:28Chris Benson?

  It is.

• 29:29Speaker 02

  So Kaggle is another great resource for, they always have AI for Good type challenges on there. I mean, currently, right now, there's some great earthquake predicting data that's there. There's an Identify Whale type data, which is very similar to like the Wildme work is available right now. There's a data set on that. So there's so many cool sources for data sets, but we're trying to compile even more of that as part of our AI for Earth mission as well. So we have a start, but we want to go even further with that.

• 29:54Chris Benson?

  Gotcha. So what types of deep learning algorithms do you find that are the most common that you use with people? Are there areas in particular that are on the rise or that you're most interested in? You know, whether they be CNNs or GANs, or even if we move over into reinforcement learning and such. What are some of the things that you're seeing out there that are news?

• 30:15Speaker 02

  Great question. So in particular, on the AI for Earth space, a lot of the problems revolve around computer vision. It's the idea, a lot of things just seem to reflect, okay, given like this data, maybe there's cameras in a farm or something, and we want to recognize pests on leaves, like that sort of thing is one example. Or we're trying to recognize like in camera trap, I should mention, I haven't defined camera trap. So what that means, for those who aren't familiar, is mounted cameras that are usually enclosed in waterproof casing, that may be mounted in trees or other remote places. We've worked with the snow leopards, for example, and those guys are so elusive, they're so quiet, they're so hard to spot.

• 31:00Speaker 02

  There's one researcher that devoted his life to snow leopard research, and he was up in the mountains, like living there in the area, only saw a snow leopard twice in 11 years.

• 31:12Chris Benson?

  Really?

• 31:12Speaker 02

  It's insane. So you really need camera traps to be able to do that.

• 31:15Chris Benson?

  Sure. So they move in front?

• 31:16Speaker 02

  Exactly. They move, exactly. Only the problem with camera trap, the thing that's historically challenging about camera trap data, is that camera or animals don't line up nicely for their selfies, right? They don't, they're not in the middle. So you get somewhere, it's like an animal sniffing, and you'll get like a close up of a nose that's kind of checking out the mounted camera itself. Or you'll get them very far away, or if it's at night, which is when a lot of animals are most active, you'll see like two eyes squinting out of the darkness, but it's hard to tell what they are. So there's all kinds of fun challenges with camera trap data. And so. That in particular is is one one thing, but that's very much a computer vision problem. So of course, computer vision, CNNs are a big, you know, convolutional neural nets are a good way to go there.

• 31:57Speaker 02

  So there's a lot of work in in object detection and computer vision, also with climate. So given satellite imagery, for example, spot the little swirl that means like a hurricane is going to form or something like that. Or in the, you know, in the water space, there's like monitoring plastic flow into the oceans, which another group, Ocean Cleanup, is doing that we've worked with. And they have like mounted cameras on that on bridges. And then as water flows underneath, they're tracking plastic flow that going into the ocean. So and being able to tell the difference between a water bottle and a fish can actually be, or pieces of plastic is actually harder than you think because they're both kind of clear or silverish in the water and that sort of thing. So it's just an interesting problem.

• 32:41Chris Benson?

  As a side thing, before you go in, do you actually have a project associated with that this time?

• 32:45Speaker 02

  Yes, there is. So the Ocean Cleanup has done a lot of great work with them. And they actually came to Microsoft for one of our hackathons and have partnered with some of the folks here as well. But they are doing amazing work in terms of tracking plastic flow into the ocean.

• 33:00Chris Benson?

  Yeah, I mean, everyone is hearing in the news these days about the that the giant amount of plastics, particularly in the Pacific where that is and, you know, what are what's being done to help that. So it's really interesting to hear about something.

• 33:11Speaker 02

  Ocean Cleanup is a great organization doing exactly that.

• 33:13Chris Benson?

  I love hearing about about how these technologies that we love so much are being used to solve these real world problems that we actually hear about in the news or watch TV on.

• 33:24Speaker 02

  Right.

• 33:25Chris Benson?

  So I guess, other than than CNNs, is there much room for things like natural language processing or capsule nets or GANs or anything?

• 33:34Speaker 02

  Great question. So I've seen, I'm actually would love a good RNN problem. So if anyone has any, please let us know. We have a few of them. There's one where people, one of our AI for Earth grant recipients is doing some work using, like working through text to be able to find what makes a difference to people the most. So for example, Like, let's pretend there's all these flyers or communications that go out to people. What actually inspires them to act to prevent climate change, that sort of thing?

• 34:05Chris Benson?

  Sure.

• 34:06Speaker 02

  And so they're doing some NLP work around that, trying to find the most motivating things for people. So I know that's one particular project. And there's a couple like chatbot type scenarios, but there's not as many RNN things as you would think. And then GANs, I am just in love with, as I think everyone is.

• 34:23Chris Benson?

  We all are.

• 34:23Speaker 02

  Yeah, they're the hotness right now.

• 34:25Chris Benson?

  It keeps expanding. There's so many more use cases that even month by month right now.

• 34:29Speaker 02

  Oh, my gosh. Yes. So if anyone has a good GAN project, let me add it. Like, I'd love to see a few more GAN projects here, but I don't think... There's a few. There's a few GAN projects. But again, it's primarily dominated by convolutional neural networks and computer vision problems. I will say on reinforcement learning, I think that is another really big area for AI, for Earth in particular. Because if you can do things like modeling climate change and then tweaking different variables to see, okay, running, you know, using reinforcement learning to run simulations and see, you know, as I toggle these different things and I try these different actions, what is going to make the most impact and help us fight climate change the quickest? And so that's a great area for reinforcement learning as well.

• 35:11Chris Benson?

  That's fantastic. You know, the thing that I love about all these stories you're telling me is that, you know, historically, we've kind of, we've gotten into data science. And then we might, we might do environmental things, you know, on the weekend or, you know, as a hobby or donating our time to charity. But it's, it's for most people, kind of a separate activity.

• 35:31Speaker 02

  Right.

• 35:31Chris Benson?

  I love the way your team has brought this together to where you can both have a career in this field and also do tremendous good for the world. And I think that is, as people realize that Microsoft is there supporting organizations and people that want to do this work, I think it's a fantastic, whole new way of using artificial intelligence and machine learning technologies to do this.

• 35:55Speaker 02

  Absolutely, absolutely. You had asked earlier about how people could get started. So if you go to

• 36:00Chris Benson?

  Go for it.

• 36:00Speaker 02

  Yes, if you go to the AI for Earth webpage, please come join us, come help us save the world because my background is in machine learning and AI, but and so I'm there's some folks on the team who are stronger at the earth side, some people are stronger at the AI side. I'm definitely one of the AI people and I'm just in awe of what people are doing. These these amazing environmentalists and conservationists who are working so hard on these these huge problems that face our earth. And so if there's anything you know, Microsoft can do to help, like I would love to support folks. So people are welcome to apply for grants. Again, we evaluate all the applications every quarter. So I think it's January, April,

• 36:41Speaker 02

  was it October? And I'm forgetting, but every every four months, we evaluate proposals and then give things out or apply or award grants. We have a couple of different grant types. One of them is just standard Azure compute hours. So just getting some cloud computing time. We actually have a data labeling grant as well now where if you have a great data set, but you need to pay someone to get it labeled and sometimes it may be a specialist. It's not, you know, something that a mechanical Turk type thing can

• 37:08Chris Benson?

  Yeah.

• 37:08Speaker 02

  can solve because you need very specialized knowledge of climate, for example, to be able to read these satellite things to label it properly. And so we actually are providing funding for for data labeling as well.

• 37:19Chris Benson?

  So that before you even go on, that is fantastic because I know in real life, at the companies that I have worked for, that has been one of the biggest problems about getting a project off the ground. It's not only getting the data set, but then after you have it, getting it labeled in a useful way so you can use it. And and at times that can slow down a project by weeks or months to get that done.

• 37:38Speaker 02

  Completely agree, Chris. I completely are with you, you know, for for those machine learning folks. I mean, we everyone listening to this can probably relate like data is like, that's the hard part, right? The algorithms are the easy part. It's getting the data is and getting it in right form. All that stuff is usually 85% of the work. So I completely agree. We're on the same page there. It's how can we help people if they're if you kind of think of it as like a a pipeline of, okay, we start with some data and then we need to get to the stage in the stage. We're trying to find things to help at each stage of that pipeline. And so the data labeling grant is essentially, how do I get from just having some data to actually having data that is useful that a machine learning algorithm can consume or supervised machine learning algorithm, I should say. Um, yes. And then we provide education as well. Uh, we have some online resources that

• 38:23Speaker 02

  we've compiled for our grant recipients. Um, I hold office hours monthly so that people can just show up and ask questions. And then we also twice a year have an AI for Earth Summit where we bring some of our grant recipients, um, to Redmond, Washington, where Microsoft's main campuses. And we give them, um, training. So it's like a day of training. There's networking opportunities, um, which is great because if someone is working on hard agricultural problems in India and someone else is doing it in North Carolina, great. Like, we see people coming together and collaborating and sometimes sharing data sets and stuff for getting even better results. So it's really amazing to see the, the, you know, the power of what we can do collectively.

• 39:05Chris Benson?

  I would love to be able to see that and actually talk to these people with this passion firsthand.

• 39:09Speaker 02

  Oh, you should maybe come do a podcast there.

• 39:11Chris Benson?

  I may come do that. Be careful what you offer there. Um, so I guess, uh, you know, as had been mentioned before, we are at Applied Machine Learning Days in Switzerland at this conference. And you, in just a few minutes, are about to go up and give your keynote. Uh, could you tell us quickly a little bit, by the time this airs, it'll, it'll be past that, but could you tell us a little bit about what you're gonna talk about?

• 39:31Speaker 02

  Yes. So, uh, what I'm going to do is I'm going to try to cover, um, kind of the basics of the AI for Earth program. Basically, not spend too much time on that, but just let people know that it's a resource that's out there. Cause the worst thing is when there's this free pot of money that people don't know about, right?

• 39:45Chris Benson?

  Sure.

• 39:45Speaker 02

  So just spread awareness of the, the grant program that it exists. And then I'm just gonna tell, um, a couple of stories of AI for Earth grant recipients and what they've done. So we'll talk about WildMe, we'll talk about FarmBeats, um, which we've already spoken. And then the final thing I'm gonna talk about is, um, another project, Project Premonition, which also started with Microsoft Research. This one is very cool as well. It's focused on, how can we predict outbreaks of disease before they happen? So think about like Zika virus and West Nile and mad cow disease and some of these things that we've seen in recent years. So the idea here is, is there a way we could get out in front of that and be able to predict these things before they happen? And, um, the way, this is so cool. So the principal researcher on this one is a guy by the name of Ethan

• 40:30Speaker 02

  Jackson.

• 40:31Chris Benson?

  Okay.

• 40:31Speaker 02

  And the idea behind this one is, What if we take advantage of little data collectors who are out in the environment already collecting random blood samples? And those are, of course, mosquitoes.

• 40:43Chris Benson?

  Oh, yes.

• 40:43Speaker 02

  If you can collect data, basically mosquitoes, use them as data collectors, they're collecting all these random blood samples and they bite humans, they bite animals, they, you know, they're collecting essentially-

• 40:53Chris Benson?

  They're indiscriminate.

• 40:54Speaker 02

  Exactly. And so they're getting a great random sampling of the environment and feeding off of these various hosts. And so what we do is we actually have, there's two kind of big contributions of this work. Number one is a smart mosquito trap that can selectively just trap mosquitoes. Other insects can go in and go out of the trap, like they'll head in, check out the lure and then fly away. And then there's little trap doors and they only close if it's a mosquito. And we can actually differentiate between species of mosquitoes as well, because I did not know this prior to this work, but it's actually like the 80s egypti species of mosquito is the one that's responsible for Zika. And then there's a different one, the culix is responsible for West Nile and all these other things. It's fascinating. And so we can actually differentiate between those two species of mosquito if we need

• 41:39Speaker 02

  to focus on one disease or the other, which is very cool. And 75% of these large diseases actually originate with animals and then come to us. And I know a lot of them have these cutesy names of like mad cow disease or avian flu or some of these other things, but a lot of them outside of the ones with the cutesy names do actually start with animals and then spread to us. So if you can catch them while they're still at the animal stage, that's great. So the first thing is the smart mosquito trap, which uses machine learning to be able to differentiate mosquitoes from other insects. The second big contribution of this work is our metagenomics pipeline. And so what we've done there is now that we have those mosquitoes, we can actually take the blood meal they've consumed, reverse engineer it and be able to tell what host animal it came from and what diseases they carry.

• 42:24Speaker 02

  And then from that, you can actually get a sense of what diseases are there and then recognize them before the outbreak happens.

• 42:29Chris Benson?

  That's amazing.

• 42:30Speaker 02

  So cool. Such amazing work.

• 42:32Chris Benson?

  I am looking forward. I'm going to come watch your talk. I can't wait to see it. And I wanted to finish up by saying, how can people listening reach out to you or to the organization and start that conversation with you?

• 42:46Speaker 02

  Great question. I am on Twitter, so you can find me at Jennifer Marsman on Twitter. I have a blog as well, which is blogs dot MSDN dot Microsoft dot com slash Jennifer. And then the AI for Earth team has a has a great website and there's information about the grant program and such on there. So and some of our amazing grantees is all available on the AI for Earth website, which is Microsoft dot com slash AI for Earth. And then you can also contact us. I think there's another Twitter account, Microsoft underscore green, which deals not just with the AI for Earth, but other environmental initiatives around Microsoft.

• 43:29Chris Benson?

  All right. Well, Jennifer, this has been a fantastic conversation. Thank you very much for taking some time out of your day. I know you're rushing around doing a lot of stuff, but it was a fantastic conversation. Thank you so much.

• 43:39Speaker 02

  Thank you.

• 43:42Speaker 00

  All right. Thank you for tuning into this episode of Practical AI. If you enjoyed the show, do us a favor, go on iTunes and give us a rating, go in your podcast app and favorite it. If you are on Twitter or a social network, share a link with a friend, whatever you gotta do, share the show with a friend if you enjoyed it. And bandwidth for changelog is provided by Fastly. Learn more at Fastly dot com. And we catch our airs before our users do here at changelog because of Rollbar. Check them out at Rollbar dot com slash changelog. And we're hosted on Linode cloud servers. Head to Linode dot com slash changelog. Check them out. Support this show. This episode is hosted by Daniel Whiteknack and Chris Benson. Editing is done by Tim Smith. The music is by Breakmaster Cylinder. And you can find more shows just like this at changelog dot com. When you

• 44:25Speaker 00

  go there, pop in your email address, get our weekly email, keeping you up to date with the news and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.