Insights from the AI Index 2019 Annual Report

Daniel and Chris do a deep dive into The AI Index 2019 Annual Report https://hai.stanford.edu/sites/g/files/sbiybj10986/f/ai_index_2019_report.pdf, which provides unbiased rigorously-vetted data that one can use “to develop intuitions about the complex field of AI”. Analyzing everything from R&D and technical advancements to education, the economy, and societal considerations, Chris and Daniel lay out this comprehensive report’s key insights about artificial intelligence. Sponsors: DigitalOcean https://do.co/changelog – DigitalOcean’s developer cloud makes it simple to launch in the cloud and scale up as you grow. They have an intuitive control panel, predictable pricing, team accounts, worldwide availability with a 99.99% uptime SLA, and 24/7/365 world-class support to back that up. Get your $100 credit at do.co/changelog https://do.co/changelog. The Brave Browser https://brave.com/cha913 – Browse the web up to 8x faster than Chrome and Safari, block ads and trackers by default, and reward your favorite creators with the built-in Basic Attention Token https://basicattentiontoken.org. Download Brave https://brave.com/cha913 for free and give tipping a try right here on changelog.com https://changelog.com. Featuring: Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: The AI Index 2019 Annual Report https://hai.stanford.edu/sites/g/files/sbiybj10986/f/ai_index_2019_report.pdfStanford Institute for Human-Centered Artificial Intelligence https://hai.stanford.eduUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Being with your change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at Change Law because of Rollbar. Check them out at rollbar.com. And we're hosted on Leno cloud servers. Head to leno.com slash change log.

• 0:17Speaker 00

  This episode is brought to you by Digital Ocean. Digital Ocean's developer cloud makes it simple to launch in the cloud and scale up as you grow. They have an intuitive control panel, predictable pricing, TM accounts, worldwide availability with a 99.99 uptime SLA and 24/7 365 world-class support to back that up. Digital Ocean makes it easy to deploy, scale, store, secure, and monitor your cloud environments. Head to do.co slash change log to get started with a $100 credit. Again, do.co slash change log.

• 0:58Speaker 00

  Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and slag with us around various topics of the show at changelog.com slash community. Follow us on Twitter. We're at Practical AI FM. And now into the show.

• 1:23Chris Benson?

  Welcome to another fully connected episode of the Practical AI Podcast where Daniel and I keep you fully connected with everything that's happening in the AI community. We'll take some time to discuss the latest AI news and we'll dig into learning resources to help you level up on your machine learning game. My name is Chris Benson. I'm Principal AI Strategist at Lockheed Martin and with me as always is Daniel Whitenack, a data scientist with S I L International. How's it going today, Daniel?

• 1:49Daniel Whitenack?

  It's going great. It was, you know, a cold and rainy slash snowy weekend, but on the good side, as some of our listeners will know, we had a season in our household of flu and sickness and that's kind of ending. So very, very happy about that. You know, no coronavirus yet.

• 2:06Chris Benson?

  Oh, yeah. Well, I was about to make a joke. Welcome back to the land of the living, but you know, yeah, with that in the news right now as we're recording this, that's been a big thing the last few days.

• 2:17Daniel Whitenack?

  Yeah, and I know you pointed a link to me earlier, which was pretty interesting for our listeners. You want to tell people about that?

• 2:23Chris Benson?

  I was just scanning across news articles and it was actually on Wired and it noted a couple of days ago, it says, the title is, "An AI Epidemiologist Sent the First Warnings of the Wuhan Virus." And so, which is the virus that started in China and is spreading to some parts around the globe as a type of coronavirus. And I think the, the short part of the article is that there's a company called Blue Dot who has algorithms that take a lot of data sources, you know, from health and airline ticketing data and such to predict the spread of disease. And in this case, they really kind of got there first, actually on December 31st, before New Year's, they actually sent out their first note that this outbreak was expected.

• 3:07Chris Benson?

  And it was another, really another week before, here in the US, the CDC, which is the US Centers for Disease Control and Prevention, got the word out. And so it was a case of, of I don't know the detail of their algorithm, but they refer to it as an AI-driven algorithm and they, they got those first reports out. So we like to talk about AI for good. That, that certainly seems like a good thing to get early warning of a major outbreak like this.

• 3:31Daniel Whitenack?

  Yeah, definitely. It's, it's a super interesting thing. And in some ways, it seems very much science fiction-y to me.

• 3:38Chris Benson?

  Yep.

• 3:39Daniel Whitenack?

  Like they're kind of, you know, like movie-like, oh, we're detecting all of these signals around the world and correlating them to say there's going to be this pandemic or whatever. Their article says that they scour foreign language news reports, animal and plant disease networks. I'm not quite sure what that network is. Official proclamations and other things like that. So it's, it's definitely pretty interesting and they're, they're really doing this. It's, it's pretty cool. If anyone knows anyone at Blue Dot out there, we'd love to have them on the podcast to discuss that. Maybe we can make that happen sometime soon.

• 4:12Chris Benson?

  That sounds fantastic. Yep. Timely news about AI having an effect on the world and possibly able to save lives. Especially as they gain notoriety and others start really watching them, it might make a big difference. So, yeah, I mean, I think we're really, over the next few years, you know, the way AI is really revolutionizing, you know, medicine at large. You know, this is one of many of those cases.

• 4:34Daniel Whitenack?

  Yep. So, speaking of how AI is revolutionizing things, you also found an interesting thing that we're going to talk about today on our Fully Connected episode. The goal of these episodes, again, just being to keep in the news, the AI news, and keep ourselves updated, but also our listeners, and kind of dive into the topics that people are currently looking at. And Stanford's Human-Centered AI Institute, in partnership with a variety of others, came out with an AI Index Report 2019, which kind of provided a lot of, I mean, we did a sort of top five things of 2019, I think, but that wasn't really based on rigorous research and data collecting and that sort of thing.

• 5:19Daniel Whitenack?

  This was more of actually, you know, who published articles, what's going on in the AI world as we move into 2020? And there were some interesting things in there, I think, that would be great for us to dive into. So that's what we're going to do today.

• 5:33Chris Benson?

  Yeah, it's quite a lengthy document. I haven't counted, but I think, you know, somewhere, what, 150-ish?

• 5:40Daniel Whitenack?

  291 pages.

• 5:42Chris Benson?

  I was far short, okay.

• 5:43Daniel Whitenack?

  Yeah.

• 5:44Chris Benson?

  It was quite long. I didn't look at the page line, but they had, yeah, they broke things out into a number of sections. And it was a really interesting kind of overall view of kind of the world of artificial intelligence in general, and gave a lot of statistics and facts. The breakdowns were pretty interesting, actually.

• 6:02Daniel Whitenack?

  Yeah, and they have, I'm just looking at the steering committee here from Stanford, also McKinsey Global Institute, Partnership on AI, Harvard, OpenAI, MIT, SRI International. So it wasn't just Stanford that put this together. They also mentioned partners of Google, PWC, and others. So, yeah, it'll be interesting to see what they think were the noteworthy things that are happening now in AI, some of which are maybe different than the things that we talked about in that, you know, first episode of the new year, so. Yeah, let's go ahead and dive in. It looks like the first section of what they talk about is research and development, which I know both of us aren't professors or anything like that,

• 6:47Daniel Whitenack?

  or involved in academic research in that context, but we both interact with research people. What if anything kind of surprised you by what they talked about, Chris?

• 6:58Chris Benson?

  Well, you know, they started off kind of really focusing on how much growth there had been in the space. And I don't think that was surprising in itself, but we've really seen the rise of China in terms of just raw numbers of publications coming out. And as part of that, they finally passed Europe. They had previously passed the US already. And so, you know, we're still seeing that, I guess, despite the fact that you're seeing those raw numbers, the fully weighted citation impact, though, of US publications is still about 50% higher than China's. And that really rang true. I was actually in an internal artificial intelligence meeting last week with my employer, and we actually were discussing that specifically, just the fact that you're seeing more citations.

• 7:41Chris Benson?

  The discussion was in terms of the quality, and obviously, we were speculating on why that is and when that might change going forward.

• 7:49Daniel Whitenack?

  Yeah, and I think actually on our next episode, we're going to be talking to someone from the Symantec Scholar team at Alan AI. And of course, that project is really concerned with discoverability of research and kind of weeding through the noise and all of that sort of thing. So that'll be interesting to talk to them about how this, you know, this giant surge in research and archive papers and all of those things has increased and in some ways made it hard to find, you know, the real, you know, notable things that are happening in some cases. I thought, you know, another thing that was interesting was that they noted several small

• 8:29Daniel Whitenack?

  countries that were having a relatively high increase in deep learning papers in terms of per capita. So these are countries like Singapore, Switzerland, Australia, Israel, Netherlands, Luxembourg. They're talking about these countries who, despite their small size, are really kind of I guess making AI research or encouraging AI research and believing in that as a future driver of economic prosperity and innovation and all of those things.

• 8:59Chris Benson?

  Yeah, it's interesting. And we've talked about this before. You know, I know we've talked specifically about Singapore. It was very obvious to me about a year ago when I was in Switzerland for a conference. These countries have really kind of committed to some degree, maybe their national identity a little bit to to saying, hey, this is something that we're going to do, you know, technology in general and specifically AI in a lot of these cases.

• 9:22Daniel Whitenack?

  Do you think, I mean, obviously, we're both pro AI, I think. I wonder, just kind of off the top of my head, if that's kind of making a dent in other research areas in terms of taking away funding or putting the focus less on other still very important areas. But now that everybody's kind of all in for AI, what effect that's having on sort of more traditional biology and medical research and those sorts of things?

• 9:49Chris Benson?

  That's an interesting idea. I'm not sure. I would say that obviously, there's a finite amount of, you know, research and development dollars and time.

• 9:57Daniel Whitenack?

  Or whatever your currency is.

• 9:59Chris Benson?

  Yeah, available and stuff. And I suspect as we look at different strategies over time, you know, AI is one of the great enablers of our time. And so I think they may be, you know, selecting certain specialties and then kind of going after AI as an enabler in this.

• 10:14Daniel Whitenack?

  I mean, I know that that's why, at least partly why I'm in industry versus in academia because I originally wanted to go the sort of professor route and that sort of thing. But I was in physics in grad school and I mean, physics as a discipline, I think it's exciting. There's a lot of people that think it's exciting. But in terms of like a paradigm shifting things that have happened in recent years, there haven't been a lot in terms of the type of paradigm shifting things in physics of, let's say, the 20s and the 30s and that sort of time period. And so, you know, physics has become a sort of, I think it's plateaued in terms of its excitement to some degree. And that's made a lot of

• 10:57Daniel Whitenack?

  the jobs in physics research very competitive, right? Because there's not maybe not as many universities that are really. Filling their bench with physics people, maybe they're now filling their bench with computer science, AI type of people. I think it's definitely will be interesting to see how that that plays out on those fronts.

• 11:19Chris Benson?

  Interesting. You know, another interesting note there was the fact that we're seeing, especially in Western European countries, but but not limited to that, in countries such as Netherlands, Denmark, Argentina, Canada, and even Iran, a relatively high presence of women that are involved in AI research.

• 11:36Daniel Whitenack?

  Yeah. Where's the US on that list?

• 11:37Chris Benson?

  Yeah, I would ask the same question.

• 11:41Daniel Whitenack?

  What's up with that?

• 11:42Chris Benson?

  Yeah, but it's it's nice to see this field. I know that that is, since the very beginning of this podcast, that's been a big goal of ours is to see this field be a truly equal field in all respects. So that was a stat that that caught my eye that I was really thrilled to see.

• 11:56Daniel Whitenack?

  Yeah, I think we need to take note of some of these countries and see what they're doing to promote that and try to increase that that more. I know there's been another thing in the next section of the index report is about conferences. And one of the things maybe that has driven that change is they talk about the Women in Machine Learning workshop that happens throughout. I'm not sure how long it's happened. They talk about 2014, but they say that has 20 times more alumni than it had in 2015. And so I think that this is one of the contributing factors. I think it's a pretty big contributing factor that there's been intentional effort to have these sorts of workshops and kind of local

• 12:41Daniel Whitenack?

  chapters and all of these things focused on women in machine learning and AI.

• 12:46Chris Benson?

  Yep, I agree. It's a good sign. It's a good sign that we're moving in the right direction. And and, you know, just in general, we're seeing conferences, their popularity and the number of people attending in AI related conferences just exploding.

• 12:59Daniel Whitenack?

  That's insane.

• 13:00Chris Benson?

  Yeah, I mean, NURIPS was, I think it upwards of well over 10,000 attendees, I believe, at this point.

• 13:07Daniel Whitenack?

  Yeah, I think it was like almost 14,000, 13,000, something, which is just insane to me.

• 13:12Chris Benson?

  It is.

• 13:13Daniel Whitenack?

  Yeah, I mean, 13,000 people like, I don't know what we're gonna have to start doing like run out like football stadiums or something for AI conferences. Like, what is the future? I don't know. I would say, I mean, one personal note on that is I would love to see, and there was, I think, some of this last year where there's a lot more effort to live stream things. And so I watched several conferences live streamed at least some of the content. I really appreciated that. And also, from an environmental standpoint, if we have 13,000 people taking plane rides to go to a conference, I would love to see that many people involved. But,

• 13:54Daniel Whitenack?

  you know, having live streaming resources and better live streaming and remote conference events would be something I would love to see.

• 14:02Chris Benson?

  I would too. I mean, we're seeing this explosion and being able to participate as we see so many people wanting to get to conferences and, you know,

• 14:10Daniel Whitenack?

  Getting visas rejected.

• 14:12Chris Benson?

  Getting visas rejected, in a lot of cases, just not able to get in. I mean, NERIPs is famous for its lottery. And, you know, so many people that would like to go cannot go. And that's despite the massive number of identities it already has. So yeah, live streaming would be, would be a fantastic way of, of being a little bit more inclusive for those who either can't travel or are wanting to be responsible and avoid the environmental impact by getting on a plane.

• 14:38Daniel Whitenack?

  Yep. The next thing, which is, is kind of obvious that they would go into this, but the technical performance of AI models has shifted in several ways. Generally, they talk about image classification as an example task. And they talk about how it's, you know, the time required to train these sorts of models has drastically decreased, and the cost to train them has drastically decreased. So there seems to be this sort of more general availability of architecture or of compute resources in the cloud that allow you to train these systems. So the availability of that, but also efforts to, to speed this up, maybe architecture or framework or language wise as well.

• 15:21Chris Benson?

  Yeah, you know, we're seeing, you know, I know that one of the topics you and I like to talk about a lot are transformers, for instance. And I've noticed just a few weeks ago, I was talking to some folks and you'll see these large transformer models come out and then these follow-ups that are huge performance enhancements. And they may, you know, they may reduce the size of the model, but you end up getting a dramatically faster training, you know, based on these optimizations. And I think when you counter that with the fact that we're seeing GPU, TPU and other hardware architectures really, really accelerating, you've got cloud options, you have options for being able to have one, you know, maybe have a GPU right on your desktop that you're working, whatever. And I think, you know, the combination of that has made a huge difference in accessibility for people to

• 16:06Chris Benson?

  be able to actually do these.

• 16:17Speaker 03

  You like this show, so I bet you'd love listening to GoTime. Not working with Go? Don't fast forward quite yet. GoTime covers a wide range of topics, including cloud infrastructure, distributed systems, microservices, Kubernetes, and Docker. Here's a ridiculous clip from a recent episode about the defer keyword.

• 16:34Speaker 04

  I think, I really think that Matt missed his calling as a stand-up comedian. Totally. Yeah, funny.

• 16:41Speaker 05

  And he can still be one. He just has to choose his audience very wisely. It's gotta be a tech audience.

• 16:46Speaker 04

  Well, he has GoTime FM. I think the funniest low-key podcast out there.

• 16:54Speaker 06

  Thing is, no one in tech, no one likes a stand-up comedian. You just want them to get on, tell us what you did yesterday, tell us what you're doing today, and if you've got any blockers, and get off. You know what I mean?

• 17:04Speaker 04

  Aww.

• 17:05Speaker 06

  No one wants the, uh, yeah, there you go. See? That's why I didn't. I'll stick to programming.

• 17:10Speaker 05

  I mean, there's only three people here. You might have a whole audience that's live listeners that's laughing right now.

• 17:16Speaker 06

  Oh yeah, let's assume that.

• 17:22Speaker 06

  I'm pretty sure this could be edited to make me not sound like an idiot.

• 17:27Speaker 03

  You heard Carmen. GoTime is low-key hilarious. Check it out at changelog.com/goTime, or just search for GoTime in Apple Podcasts, Spotify, or your favorite podcast directory, you'll find it. Once again, that's changelog.com/goTime.

• 17:50Daniel Whitenack?

  So Chris, I found one really interesting thing in the index report that is related to technical performance. And if people are following along while they're listening, this is on page 68 of the report. So a good ways in there. But they go through and talk about the milestones that have been achieved in terms of human level performance and AI reaching or beating human level performance in certain tasks. And I hadn't seen something like this. Probably are things like this compiled in other places, but I thought the compilation and timeline that they compiled here was really interesting. So this starts

• 18:28Daniel Whitenack?

  with Othello back in 1980, and it goes all the way to 2019, detecting diabetic retinopathy with specialist level accuracy. This is really cool. I don't know how many of these things were familiar to you, Chris?

• 18:44Chris Benson?

  A few of them are that, you know, they talk about AlphaGo. There's a number of them, especially given back when I was doing gaming when I was younger. A lot of the older ones, ironically, are more familiar with me than some of the new ones. The ones that have been the news a lot in terms of AI in the last few years, I'm familiar with. But they had some that I had not noticed before. The prostate cancer grading I had not seen.

• 19:05Daniel Whitenack?

  Yeah. So pre 2011, there's three milestones that they list, which are Othello, Checkers, and Chas. And then post, you know, 1997, then we skip all the way to 2011, and you can kind of see this, you know, rapid advance. So there's like between 2011 and 2019, I'm not going to be able to count these, but there's probably at least like 15 or something?

• 19:32Chris Benson?

  Yeah, something like that.

• 19:34Daniel Whitenack?

  15 to 20 after that, including things like, you know, Dota and video games, machine translation, all of those things.

• 19:41Chris Benson?

  And you know, something I noticed just as a side comment, since it leaps from 97 to 2011, those are the years of the most recent AI winter. I almost said nuclear winter. It's not, not a nuclear winter. But yeah, the most recent AI winter was right there. And you saw zero progress made as everyone turned away from neural networks for those years.

• 20:02Daniel Whitenack?

  Yeah, also, kind of going back to the conference thing, they have this graph in the in the report where they track AI conference attendance and you can see, so they track back to 1985.

• 20:15Chris Benson?

  Mm hmm.

• 20:15Daniel Whitenack?

  And I see, so this is one of those, aside for a minute, this is one of those plots that, being slightly colorblind, I have no chance of reading because all of the lines are like colors that blend together for me.

• 20:29Chris Benson?

  Yep.

• 20:30Daniel Whitenack?

  So just FYI, there's, there's color palettes out there to help with that.

• 20:33Chris Benson?

  Okay.

• 20:34Daniel Whitenack?

  But maybe you can help me know which one goes back to 1985.

• 20:37Chris Benson?

  Uh oh.

• 20:38Daniel Whitenack?

  There's a couple of conferences back in 1985 that have at or over 5,000 attendees. And then you can kind of see as, you know, 1990, 1995, 2000, 2005, it actually decreases all the way to about 2010, some of them start to increase again and then, you know, 2015 and on, it's just like skyrocketing attendance in these conferences. So it is just historically interesting to see, you know, back then, there were these very high profile, you know, 5,000 attendees at a conference is no joke.

• 21:15Chris Benson?

  Yeah, that's quite a conference. And for, for those of you who haven't been to one that size, I mean, it's just, you get lost in them in terms of trying to find your content and everything. So I actually prefer smaller conferences. They're a lot more intimate, a lot more fun from my standpoint.

• 21:29Daniel Whitenack?

  Yep. Before we leave the technical performance side of things, one of the trends that I saw them point out, which was interesting to me, and they, they drew it out in terms of NLP, but I think it's true of computer vision in some ways as well, is that in terms of benchmarks, like they had benchmarks for certain tasks in NLP or computer vision, like object recognition, or like machine translation, or entity recognition, reading comprehension, co-reference, all of these different benchmarks. And as they've reached, you know, human level performance on these, they've actually had to go, like they, I'm meaning the research community, or those in the research community, many have had to go back and say, how can we make this more challenging?

• 22:12Daniel Whitenack?

  Because we're reaching, you know, we're reaching human level performance in so many of these tasks. So for example, in the AI world, there's this benchmark called glue, which I'm gonna mess this up off the top of my head. I think it's general language understanding here.

• 22:31Chris Benson?

  Come up with that.

• 22:32Daniel Whitenack?

  Something like that. Let's see, there's glue and then there's super glue. So there's a general language understanding evaluation. There we go. And then they, they had that and then that wasn't enough or that wasn't challenging enough. So they reached, you know, human level. And that this graph is in the index report as well. So you can see kind of models surpassing human level performance in glue. This sort of task that combines a bunch of NLP tasks to make it harder itself. And so human level performance was beat there. And so then they developed this other one, which is super glue, which kind of ups it from there and others like Alan NLP or Alan AI Institute. And, and others are producing other benchmarks to kind of further challenge things.

• 23:14Daniel Whitenack?

  So as we've reached a lot of those milestones, now we're kind of in the season of, you know, what's next? How do we make this harder for computers? And a lot of those things are like common sense, understanding and, and reasoning that are really hard for computers to do. We, we still have a lot of room for growth there. And of course, in, you know, languages other than English and, and, you know, multimodal settings where we're combining video and imagery and text and all of those things. It'll be interesting to see what benchmarks come about there.

• 23:45Chris Benson?

  I agree, you know, just as a side thought there, we're having a couple of interesting conversations, meaning the community at large right now. We're seeing these things that we're calling out here in terms of how far we're coming and having to adjust for benchmarks. And then you see people in the kind of artificial generalized intelligence community saying, oh, we, we need to have completely new models and stuff. And I think people tend to get caught up in one or the other. It's interesting that in terms of kind of the deep learning basis where we're at right now, we're really still making pretty immense progress in terms of applicability and, and performance improvements. And while that may not be AGI, I think some people. tend to get caught up in one or the other conversation.

• 24:24Chris Benson?

  I think it's pretty remarkable that we're in an industry where you can have both of those conversations with the specifics of whether we are going fast or whether we are not making much progress at all in the in the larger scheme of things. But I really think it it shows how vastly artificial intelligence has has moved into culture and society and all, you know, and industry at large. That astounds me repeatedly, and I think this report on the applicability of these technologies is really amazing. It just as you just called out, the fact that NLP is making such fast progress.

• 24:58Daniel Whitenack?

  Yeah, sometimes I think we overcomplicate the sort of how much progress are we making? I mean, one indication of that is economic investment in in AI and application within industry. That's another point that they call out in the report. And they throw out, you know, really huge numbers like, oh, AI investment was over 70 billion with AI related startup investments over 37 billion. Honestly, I mean, I don't know about you, Chris. Those numbers don't, it's hard for me to grasp those numbers because I've never seen a billion dollars.

• 25:34Chris Benson?

  Yeah.

• 25:34Daniel Whitenack?

  But it's a lot of money, right? And there's definitely, they also cite different percentage increases in jobs and AI investment. And I'm guessing, you know, some of that may be hype, right? But there is actual proof that AI applications within industry are driving a lot of change and people are responding with investment.

• 25:57Chris Benson?

  There was a particular stat, it's a US centric stat that I noticed, and that is the total number of jobs in the economy relative and taking that total number and looking at the share of those that are kind of AI related jobs, or at least, at least in terms of, you know, maybe titles or tangentially. It's approaching nearly 1% in terms of AI jobs to total jobs.

• 26:18Daniel Whitenack?

  And that's AI jobs, meaning like humans doing AI.

• 26:22Chris Benson?

  Correct.

• 26:22Daniel Whitenack?

  Not AI doing jobs.

• 26:25Chris Benson?

  I'm glad you called that out just for clear, for clarity on that. But yeah, the the number of jobs that we as humans are. Engaged in that are AI related compared to the total economy. It's approaching 1% and that is remarkable because we're still at such an early stage in this industry. And so, you know, you can see, I believe back around 2015, it was just, you know, just a fraction of 1%. I think it might've been 0.3%. So we are growing so fast in terms of how AI is impacting the economy, represented by the number of jobs being created to do just that.

• 27:00Daniel Whitenack?

  And job wise, they did a bunch of analysis of LinkedIn, which I found this interesting because they have a lot of, so they have asterisks next to like India, for example, because I think they said like 40% of workers in India are on LinkedIn. And so that numbers are likely not accurate in that sense. If, if anything, they're kind of diminished, I would say. And yet, India was kind of at the top of a lot of the job statistics in terms of how many people are involved in AI and also the fastest growth in AI hiring. Again, these other countries that have invested heavily as part of their national strategy in AI, like Singapore, Brazil, Australia, Canada,

• 27:45Daniel Whitenack?

  were right at the top of those AI hiring stats. And it's interesting to, I mean, it, it makes sense with what I've heard. So I've, I've got a few people that I work with in Singapore. And from what I understand, you know, AI people in Singapore are pretty much snatched up instantly. So if you're trying to hire someone in AI in Singapore, there's just so much hiring going on. And there's not enough AI people to go around, which is one of the reasons why they establish some of these things like AI Singapore, which is trying to feed AI expertise into industry. But it's, there's just so much hiring going on. The demand is, is so high that, that AI people, you know, could, could get hired pretty much right, right away.

• 28:30Daniel Whitenack?

  So if you're interested in getting an AI job, you know, consider Singapore, it's a beautiful place.

• 28:34Chris Benson?

  Although I don't think it's, they may be the most extreme case, but I don't think it's a problem just for Singapore. We're seeing that really everywhere, you know, all. All employers that are invested in AI, which is obviously an increasing number steadily, are contending with that same issue in terms of finding qualified people who can be productive quickly. And, you know, so the university system, you know, they're snatching them straight out of universities that are oriented on AI. You know, it is just an explosive growth area. You mentioned all the billions of dollars a few minutes ago in terms of kind of global private AI investment. And, you know, along with that, I was really astounded to see that year over year, the annual growth rate being around 50% in terms of investment

• 29:16Chris Benson?

  in startups is continuing to go. And that's despite, you know, various economic worries and things that people are concerned about, about life in general. So it's still quite staggering, the explosiveness of the field in general.

• 29:47Speaker 07

  This episode is brought to you by Brave. We deserve a better internet. That's why the team behind Brave reimagined what a browser could be. Brave is like Chrome, the good parts. Even your extensions will just work. It has built-in add-in tracker blocking, easy anonymization with the Tor network, earn tokens while you browse and use them to tip your favorite creators. And did I mention it's lightning fast? Turns out the web is super fast when you remove all the cruff. Download Brave today using the link in the show notes and give tipping a try on chainsaw.com.

• 30:29Daniel Whitenack?

  All right. So you kind of started into the conversation about education and me being in a university town, I also am kind of monitoring this and thinking about it. But more people are going to school to learn specifically AI-related things than than ever before. And, you know, one of the things they talk about is international PhD students pursuing AI specialization in computer science. So that's up.

• 30:59Daniel Whitenack?

  But also, there's this sort of interesting trend of PhDs that are graduating in AI are not going the sort of academic route in general, in terms of getting professorships and that sort of thing. But there's kind of this drain of AI talent where a lot of these people are going into industry to work at awesome, cool places, whether that be like Google Brain or OpenAI or whatever it is. So much so that, you know, AI faculty are leaving academia for industry and that's continuing to accelerate. And so that's, you know, exciting in some ways that there's kind of this infusion of expertise into industry, but it's also concerning in

• 31:40Daniel Whitenack?

  some ways because, at least from my perspective, it seems like, you know, when I'm thinking about academia, this gap between how in touch academia is with industry is kind of separating more and more in certain ways. And so, yeah, I would love to see industry and academia be closer and people coming out of university programs, you know, really ready practically to do jobs. But I'm not sure that that's, that's really happening because the people, the AI professors in academia that are interested in industry stuff are just leaving to go to industry. And

• 32:22Daniel Whitenack?

  then what's left is kind of just, you know, pure academic research, which is interesting, but maybe in some ways less connected to industry problems.

• 32:32Chris Benson?

  Yeah, it's really changed both academia and industry, this trend. And it's changed the relationships we have. And one of the things that I'm involved with at my own employer, at Lockheed Martin, is I engage a lot of universities in terms of artificial intelligence with that. There's a number of us that do this. And the nature of those collaborations are changing, whereas once upon a time, you know, you might think of kind of the brain trust at these universities and that industry would access them to help them. But we're seeing so many people that, as you pointed out, might otherwise have been on an academic career path. Are going to industry because of the opportunities and because of the compensation that that has changed.

• 33:14Chris Benson?

  And so it's interesting to see the partnerships that we're having between industry and academia, where both sides are doing cutting edge research in different topics and collaborating across. And you're seeing, you're seeing, you know, a substantial amount of that in industry, whereas it used to be mostly in academic and then industry would kind of take that and apply it to what they're doing. But you're, this field is moving so fast and the brain drain is happening from academia into industry for those reasons, that it's kind of rebalanced that. And I think both sides are trying to figure their way through that at this point. It's also driven compensation rates through the roof for AI specialties, obviously. And so that's, and that becomes another thing where different companies are competing for the talent.

• 33:56Daniel Whitenack?

  Yep, it will be interesting. I know one of the other things we've talked about before on the podcast is sort of trend to more formalization around data science and AI programs within universities. So to where before a lot of universities took the strategy of, you know, AI as a graduate discipline within computer science, which it definitely is and should continue to be. There's efforts to kind of embed data science and AI across all organizations, with some universities even taking steps to establish, you know, the Center for Data Science or Center for AI or whatever.

• 34:39Daniel Whitenack?

  And there's kind of cross-discipline collaborations that happen within those centers. So that's interesting. I think there's success stories within that and there's not so great success stories with trying to apply that. So yeah, I don't know what all of the solution is to this sort of balance between industry and academia. And maybe it kind of just flows back and forth. But yeah, it'll be interesting to follow for sure.

• 35:04Chris Benson?

  Absolutely.

• 35:05Daniel Whitenack?

  So other things that was emphasized in the economic point with the index, but also actually called out as a separate whole chapter in the index report were autonomous systems.

• 35:18Daniel Whitenack?

  Typically autonomous vehicles. So autonomous vehicles receive the largest global share of, or the largest share of global investment over the last year, followed by things like drug, cancer, therapy, facial recognition, video content, fraud detection, other things. But autonomous vehicles were at the top. And I know this is something that you've been involved with personally. And of course Lockheed Martin is interested in. But it kind of took me off guard a little bit because you hear a lot about self-driving cars and that sort of thing, but it doesn't seem to me to have penetrated markets as much as something, let's say like facial recognition or computer vision. And yet it's at the top of investment.

• 36:02Chris Benson?

  I think you will though. You know, one of the things that was notable is that the state of California, you know, licensed testing for over 50 companies with an enormous number of autonomous vehicles. And they noted that they had already driven over 2 million miles. And when people hear autonomous vehicles, they're often thinking about cars on the road. But what we're really seeing here is a transformation. And they kind of call it out in this report that you're seeing, I think we're right on the cusp. You know, now that California has done that, you're going to see other states and other countries as well engaging in the same thing as people recognize that these vehicles can be safely integrated into society,

• 36:44Chris Benson?

  whether on the road or in the air or on the water, wherever we happen to be. And we're going to see that more and more. And I think we're kind of right. It, I think this, this action by California demonstrates that we're kind of right at that tipping point as we're recording this, where we're going to see it everywhere. I know in the company that I work at, autonomy is a big part of it as it is in many different industries. And so I think you're going to see autonomy becoming fairly common over the next few years, whether it be on our, on our nation's roads or those of other nations, or from the bottom of the ocean to the surface, all the way to outer space. I think that it's going to become.

• 37:21Daniel Whitenack?

  In the Space Force.

• 37:22Chris Benson?

  Yeah, it's, we have you. That's exactly right. That's another thing worth calling out is in the last few months. The US Space Force has been created out of what had been the Space Command and the United States Air Force. So we're now at a point where it made sense to separate those out as their own concerns. And so I think you're going to see autonomy in every facet of transportation sooner than most people might expect.

• 37:47Daniel Whitenack?

  So I'm full of asides today, but just as an aside, did you see how close the kind of the Space Force logo and the Star Trek thing?

• 37:55Chris Benson?

  Yeah.

• 37:56Daniel Whitenack?

  So I saw people have them like side by side on Twitter and, you know, it's kind of, yeah, I don't know what to think about that, but it's kind of interesting.

• 38:06Chris Benson?

  We'll leave it to people to call out on Twitter and Slack to us. We welcome your comments. I've seen some pretty funny ones so far.

• 38:13Daniel Whitenack?

  What's your opinion about Star Trek and the Space Force? Anyway, I guess that really good. So, see, I made a really good transition with that aside to talk about public perception and societal considerations, which were some of the last things that were talked about in the index along with national strategies. So they talked about public perception of AI, societal considerations around things like fairness and interpretability. One of the things that I thought was good in the report is they did specifically call out that there's these 17 United Nations Sustainable Development Goals, which cover a lot of

• 38:55Daniel Whitenack?

  things around education and climate and other things. Oh, yeah, there's so there's the 17 and then there's 169 targets and they talk about how AI can contribute to each of these. And if you remember, we had another guest on the show with the AI for Good Foundation, who is directly working with the United Nations to apply AI to the Sustainable Development Goals in really interesting and amazing ways. And so I definitely, if you're interested in that side of things, take a listen to that other episode. But I think that's really worth calling out is now more than ever, because we have reached so many milestones

• 39:35Daniel Whitenack?

  in terms of AI, we're at a point where we can really apply AI to all of these different problems that really matter and make a difference for the quality of life for people to give them a better life. So if you're interested, you know, that's really a great. Effort to be a part of. And in terms of if you're looking for side projects or just to learn about AI, why not, you know, take on, you know, some side projects related to the Sustainable Development Goals, related to AI for Good. Yeah, I think it's a really great time to be part of that sort of thing.

• 40:12Chris Benson?

  I agree completely. And, you know, it wouldn't, as we talk about AI for Good and societal impact, I think is a, maybe finishing up with, with one last point that they note in this document, is they really point out kind of the rise within the context of AI fairness, interpretability, explainability, what we tend to call ethics. And they identified that really those topics in terms of references to AI ethical principles have become an enormous, enormous conversation that we're having globally at this point. And we're recognizing that we have these powerful tools. And before unintended consequences could arise that we need to be thoughtful. I love the fact that people are engaging on this and, and

• 40:57Chris Benson?

  trying to say, how can we think about fairness before we have brought in? We've, we've had some bumps in the road over the last few years, obviously. But I'm very optimistic as we kind of go into the 2020s here about people at least engaging on these topics, on these kind of ethical AI principles on the front end of the decade as we, as we search forward. So I just wanted to end on that note of optimism and ask people to continue to do that. Don't just do the engineering side and the data science side of AI, but think about the world that you want. And AI for Good, as you mentioned, is, is a great place to be thinking, whether it's in your primary job, or whether it's what you're doing for a side project when you go home at night.

• 41:37Daniel Whitenack?

  Yep, definitely. And we always like to share learning resources as part of these fully connected episodes. And one related to what Chris was just talking about, which we could share, which I've poked around a bit with is the AI Fairness 360 Toolkit from IBM. I think we've mentioned it maybe once on the show. But if you just go to ai360.mybluemix.net

• 42:03Daniel Whitenack?

  There's a toolkit there where you can experiment with their tools for fairness and analyzing data sets and modifying models and all of those sorts of things. They have a web demo, but also as a resource, they have links to read more about bias mitigation concepts, terminology. They have a Slack channel where you can ask questions related to that. They also have tutorials that show kind of examples of code that checks bias and in different industries and different applications. And I think I'm scrolling down, it seems like there's even more here than what I remember the last time I checked it. So they're talking about all sorts

• 42:48Daniel Whitenack?

  of things. So disparate impact, Manhattan distance, average odds difference, equal opportunity difference, all sorts of different methods. Then also talking about adversarial, deep biasing, reweighting, really cool stuff. So I would suggest to check it out. And of course, they have notebooks where you can try things. And it's easy these days to spin up a notebook on Colab or other resources to try out a toolkit like this. Absolutely. Sounds good. Yeah, awesome. Well, great to go through this with you, Chris. Interested to see what the index looks like next year, but it was great to talk through it with you and looking forward to a great year of AI. As am I. Sounds good. Talk to you later, Daniel. Thanks.

• 43:33Speaker 00

  All right. Thank you for tuning into this episode of Practical AI. If you enjoyed the show, do us a favor, go on iTunes, give us a rating, go in your podcast app and favorite it. If you are on Twitter or social network, share a link with a friend, whatever you gotta do. Share the show with a friend if you enjoyed it. And bandwidth for change law is provided by Fastly. Learn more at fastly.com. And we catch our airs before users do hear change law because of Robar. Check them out at robar.com/changelaw. And we're hosted on Linode cloud servers. Head to linode.com/changelaw. Check them out. Support this show. This episode is hosted by Daniel Whitenack and Chris Benson. The music is by Breakmaster Cylinder. And you can find more shows just like this at changelog.com. When you go there, pop in

• 44:14Speaker 00

  your email address. Get our weekly email, keeping you up to date with the news and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.