Analyzing the 2021 AI Index Report

Each year we discuss the latest insights from the Stanford Institute for Human-Centered Artificial Intelligence (HAI), and this year is no different. Daniel and Chris delve into key findings and discuss in this Fully-Connected episode. They also check out a study called ‘Delphi: Towards Machine Ethics and Norms’, about how to integrate ethics and morals into AI models. Sponsors: iMerit ML DataOps Summit https://imerit.net/dataops – A FREE virtual event on December 2nd, 2021! iMerit is excited to gather 1000+ senior leaders at the forefront of deploying data solutions that power machine learning and artificial intelligence. Learn more and register to attend at imerit.net/dataops https://imerit.net/dataopsThe Brave Browser https://brave.com/cha913 – Browse the web up to 8x faster than Chrome and Safari, block ads and trackers by default, and reward your favorite creators with the built-in Basic Attention Token https://basicattentiontoken.org. Download Brave https://brave.com/cha913 for free and give tipping a try right here on changelog.com https://changelog.com. Fastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorshipFeaturing: Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: The AI Index Report: Measuring Trends in Artificial Intelligence https://aiindex.stanford.edu/reportIEEE Spectrum: 15 Graphs You Need to See to Understand AI in 2021 https://spectrum.ieee.org/the-state-of-ai-in-15-graphs/number-of-ai-journalsDelphi: Towards Machine Ethics and Norms https://arxiv.org/abs/2110.07574IEEE Spectrum: Machines Learn Good From Commonsense Norm Bank https://spectrum.ieee.org/ai-ethics-machines-learn-goodUpcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Chris Benson?

  I would say the generative stuff worries me the most in the short term because it can be taken advantage of so easily and it's so accessible to such a wide audience and the output is of such high quality that it can make you question what is real. And when you are shaping the data or the information that gets to people that they are choosing to process and you can channel that and you can target people, it can make the whole system fall apart if you let it go far enough. And so that worries me in the short term a whole lot more than AI becoming sentient and taking over the world.

• 0:40Speaker 01

  Big thanks to our partners, Leno, Fastly and Launch Darkly. We love Leno. They keep it fast and simple. Check them out at leno.com/changelog. Our bandwidth is provided by Fastly. Learn more at fastly.com and get your feature flags powered by Launch Darkly. Get a demo at launchdarkly.com.

• 1:01Speaker 01

  This episode is brought to you by iMerit and their upcoming ML Data Ops Summit in partnership with TechCrunch. It's a virtual event happening December 2nd, 2021. Check out the speakers and register at imerit.net/dataops. The event is gathering more than 700 attendees from top AI and ML companies and feature major speakers including Facebook AI, Cruise, Zooks, GE Healthcare and more. And I'm here with Ivan Lee, the founder and CEO of DataSor, who's also speaking at the event. Ivan, I know you'll be speaking at the conference on this subject, but can you share a teaser of what's happening right now in the NLP space?

• 1:38Speaker 02

  If we look at the advances in NLP over the last few years, there have been some really exciting developments. Perhaps most notably OpenAI's GPT-3 and their ability to just really start mimicking humans in generating snippets of English language. What we've noticed is that perhaps of all the branches of AI, NLP is one of the most mature. And there were some obvious use cases when we were starting out. There's things like the ability to handle customer support, improve upon chatbots. These were very clear.

• 2:10Speaker 02

  That we wanted to go after. But as we learned more, it turns out there's applications in the legal industry, in healthcare, in financial. There were a number of nonprofit organizations using us to label COVID-19 research and be able to just make sense of all the abundance of research that was coming out. We were kind of astounded by the creativity and the ways in which NLP could be produced.

• 2:34Speaker 01

  All right, learn more and register to attend for this free virtual event at imerit.net/ dataops. Again, you'll hear from top AI and ML speakers who've successfully deployed machine learning data operations in their organizations. Again, this event is free and is virtual. Learn more and richer at imerit.net/ dataops.

• 3:08Speaker 03

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and Slack with us around various topics of the show at change.org/community and follow us on Twitter. We are at Practical AI.

• 3:34Daniel Whitenack?

  Welcome to another fully connected episode of Practical AI. This is where Chris and I keep you fully connected with everything that's happening in the AI community. We'll take some time to discuss the latest AI news and we'll dig into learning resources to help you level up your machine learning game. I'm Daniel Whiteknecht. I'm a data scientist with SIL International and I'm joined as always by my co-host, Chris Benson, who is a strategist at Lockheed Martin. How are you doing, Chris?

• 4:02Chris Benson?

  I'm doing very well, Daniel. How's it going today?

• 4:05Daniel Whitenack?

  It's going good. I have some paper deadlines or paper submissions coming up very soon. So here in like a week and a half. So a lot of my life right now is finishing up graphs and writing things and revising things and making sure references are put together and all of that fun stuff.

• 4:30Chris Benson?

  Understood. You know, I work for a big company these days and we have all the usual PowerPoint and we're documents to navigate ourselves.

• 4:39Daniel Whitenack?

  Yeah, yeah. Well, sometimes that's more different. I feel like my Google Drive, it includes so much, but it's really hard sometimes to navigate and find what I want, which I think actually we talked about on a recent episode, or you talked about with a guest on a recent episode.

• 4:56Chris Benson?

  Yep.

• 4:57Daniel Whitenack?

  Yeah, I feel the pain there. But yeah, mostly my, my life right now is thinking about, should this line be colored or should I do?

• 5:07Chris Benson?

  Those are the worst.

• 5:08Daniel Whitenack?

  dashed lines or what font size should this be? All of those.

• 5:13Chris Benson?

  Yeah, I sympathize with you. I imagine most of the people in our audience sympathize with you as well, because we know, talking about technology and data science rather than actually doing it.

• 5:26Daniel Whitenack?

  Yeah, it's definitely useful though in terms of communicating what you've been working on helps you yourself also formulate like, oh, this was sort of like a nice story arc that we did, or maybe there's like gaps in our thinking. Usually it's the latter where you sort of try to formulate and they're like, oh, I really should have like done that in addition to the other things to sort of fill in the gaps of our thinking, or at least that's normally how it happens.

• 6:00Chris Benson?

  Yeah, and I gotta say, you have a great attitude about it. I just, I just hopefully solitarily when I'm doing it, but I just, I just whine. I just complain. I just go, oh god.

• 6:13Daniel Whitenack?

  Well, I'm not saying I never do that.

• 6:14Chris Benson?

  God, I gotta do, I gotta do a bunch of PowerPoint slides before 2 o'clock today or whatever. So yeah. You're a great attitude, man. You're like, oh, if I do this, I can learn from it. I'm just like grumbling. Okay?

• 6:25Daniel Whitenack?

  Yeah, yeah. Well, we'll see how I feel tonight after after another day of of that. Yeah, well, today we've got a we've got a fully connected episode. There were a few things that came across our path over the last couple of weeks that I think we were we were texting to one another and saying, hey, it'd be really good to to chat through some of these things. One, one was the updated AI index report from Stanford University. Another one, though, so we'll move on to that. Actually, we went to that AI index and discussed it last year. But in this year, there's an updated version, which we want to talk through.

• 7:09Daniel Whitenack?

  But before we do that, maybe there's a quicker one that I that I had, which is this article that I saw originally in the IEEE Spectrum called Machines Learn Good from Common Sense Norm Bank. A New Moral Reference Guide for AI Draws from Advice Columns and Ethics Message Boards. Very, very interesting. So that caught my attention. I don't know if that seems interesting to you.

• 7:37Chris Benson?

  It did.

• 7:37Daniel Whitenack?

  I thought we might talk about it for a second.

• 7:39Chris Benson?

  I read the article when you brought it up, and it is interesting. And and and it's a it's funny, there's so many different approaches to to AI ethics out there because there's no you know, there's no standard way. We're so early days still. And I like the fact that they were literally like, well, let's just go to the ethics folks and and see what they have to say and see if we can train it that way, which is very practical. You know, for practical AI, I thought that hit the mark. So interesting article.

• 8:05Daniel Whitenack?

  It was interesting for sure. In some sense, it seems a little bit like, I don't know, a little meta for me. That like, we're training models. So maybe I should describe the premise of of the thing. So the premise of the thing is like, OK, we're all talking about like how we need sort of like ethical and moral considerations on how we build AI models and the decisions that they make and all of that stuff. So a lot of people are talking about this subject. So in this work. Alan AI or some researchers at the Alan Institute for AI used a bank of sort of common sense moral judgment data to train a model to make moral judgment calls.

• 8:50Daniel Whitenack?

  That's why I mean it's sort of maybe a little bit meta in that like we see that there's a problem that we need to consider ethics around AI. So let's train an AI to determine certain ethical things. I don't know. Maybe, maybe that's like, that's one thing that just sort of struck me.

• 9:09Chris Benson?

  It is, but, but I would argue, how can they go wrong? I mean, they're actually using Dear Abby, you know, the advice column as, as one of their basis. So, you know, one of the data inputs that they're using there. I mean, and which raises a little tiny side point I just need to raise. I'm in my early 50s and Dear Abby has been around all my life. I'm just saying Abby must be really, really old at this point.

• 9:31Daniel Whitenack?

  Yeah, yeah, that's probably true. Maybe Abby is an AI.

• 9:34Chris Benson?

  Oh, wow. Maybe she always has been. Maybe we're living in a simulation. I won't go down that rabbit hole. Okay.

• 9:40Daniel Whitenack?

  Yeah, okay. On that note, I'm just reading through the data here that they're using and what it talks about is to, to train the model, they use this common sense norm bank, which is a collection of 1.7 million examples of people's ethical judgments on a broad spectrum of everyday situations. Very interesting. I'm just looking at one of the things that's highlighted here, which I, I think is, or I don't know if it's exactly represented like this in the data set, but the, the sort of highlighted example here they have is killing a bear to please your child is bad.

• 10:25Daniel Whitenack?

  Killing a bear to save your child is okay. Exploding a nuclear bomb to save your child is wrong. So I guess there, when they say this is ethical judgments on a broad spectrum of everyday situations, I guess everyday situations include nuclear bombs.

• 10:44Chris Benson?

  I guess so. I mean, and, and we're all, we all have access to exploding, apparently. It's, it's a curious example.

• 10:51Daniel Whitenack?

  So I don't know. Yeah. So this is very interesting that there's different sort of. Aspects of the same kind of entities or things represented in this data set. My understanding, and if anyone from the Allen Institute is listening, they can come on the show and go into more detail. But my understanding is they trained this model on this data set. And then they actually employed some crowd workers.

• 11:20Chris Benson?

  Yep.

• 11:21Daniel Whitenack?

  Using Mechanical Turk to evaluate a thousand examples of example moral judgments from the model. And so doing this, they found that the model, which they called Delphi, achieved 92.1% accuracy compared with lower accuracies for other sort of existing models like GPT-3, which we've talked about before on the show. So, yeah, I mean, I guess nine out of 10 moral judgments from the model, at least on an average from crowd workers, seem to be justified or right, or, you know, however you want to put it.

• 12:02Chris Benson?

  There is one little side thing I want to hit here, since they mentioned, I think you're the right person for me to ask of, and that is GPT-3. With its much lower performance overall, what do you think that says? I mean, GPT-3 was, you know, from a massive, you know, amount of the internet being sucked in as a source and processed. Would that imply that the internet is, you know, that the context?

• 12:29Daniel Whitenack?

  Less moral.

• 12:30Chris Benson?

  I don't know. Or at least is the context of ethics largely absent from the internet in terms of how different material out there is presented, just, you know, on any given website that they happen to pull from? It's interesting that they really, they had to focus on something to give it the context, to give it the specificity of that.

• 12:53Daniel Whitenack?

  Well, I would have to read a little bit more into how they use GPT-3. I mean, certainly, I would agree that large portions of the internet are immoral, but that's like separate from my own, any of my views of, of anything related to AI. But, you know, just based on my, my own thinking. But when you use GPT-3, typically you're looking at sort of a few-shot scenario, where you have this pre-existing language model and you sort of start feeding it examples of the type of thing that you want it to do. And then it starts doing this action. Now, I don't know exactly how they sort of, how much of the, maybe the common, uh,

• 13:35Daniel Whitenack?

  common sense norm bank they used in terms of fine-tuning GPT-3, or if it was truly, you know, few-shot or, or, or something like that. Um, so I'd have to read a little bit more there to, to understand it. But I think that GPT-3 is sort of this general purpose model. And the Delphi model was very specifically trained to do this task. So it's not, not incredibly surprising to me that it performed better. And I think, you know, that's good that people are thinking about this, this side of things. Um, it's definitely, there's a lot of sort of common sense, complicated, subtle things that happen with language.

• 14:18Daniel Whitenack?

  And I think the Allen Institute for AI has done a lot of thinking with regard to common sense and, you know, pitfalls that many language models fall in. So I'm glad, this seems to be a continuation of some of their thinking with regard to that subject. And that's, um, you know, that's good from, from my perspective, I think.

• 14:39Chris Benson?

  I'm looking as we're, as you're saying that, at one of the other highlights uh, that they have just below. Um, and they, they note that the system stumbled on time of day judgments, is running a blender rude at 3:00 PM or 3:00 AM? And unfamiliar topics as well, such as sports and law, uh, regarding, you know, what they were doing there. Do you have any, any thoughts on why, why some things might be easier or harder than others, uh, in NLP models in terms of picking up on that?

• 15:07Daniel Whitenack?

  Uh, yeah, I mean, it's still an open topic of of research, probably, because a lot of what happens in these large models is not incredibly interpretable. So I think that's an open, open area of of research. But I know that there have been a lot of, well, I don't know about what would people consider a lot, but there's been some work with regard to sort of adversarial NLP examples. And we've talked about those, I think, very briefly in certain times on the show where, so like for sentiment analysis, if you're to say, I really love the United States, you know, that might be judged by a model as positive.

• 15:52Daniel Whitenack?

  And then you could say, I really love Turkey. But then all of a sudden, by just changing the United States to Turkey, then it's viewed as a negative because most of the examples in the data set regarding Turkey are probably some sort of like negative examples. And, you know, there's an underlying behavior of the model that isn't really probed until you do these sort of adversarial examples, which is probably true in this case as well, where there's just, you know, topics or scenarios that aren't well represented in in the data or very scarcely represented. And so that might play into into the behavior for sure.

• 16:34Daniel Whitenack?

  But yeah, I mean, it's interesting to see if if others will try similar things to this. And maybe, you know, AI models can start writing their own ethics principles for AI models.

• 16:46Chris Benson?

  That would be interesting. And to that, kind of to that point, they also note in the article, when you're talking about it writing its own, that this particular model tends to reflect the status quo, the cultural norms of today's society. And yet, we know that with, you know, to tie this into another topic that we've hit a bunch of times, with this type of automation becoming more and more pervasive in every aspect of our lives, we know that our society, our culture, has to change to accommodate, you know, work, you know, humans at work or not at work and and others such as that. It would be interesting if we could use a model, maybe a next generation of this, that can talk about what might be ethical in a different society

• 17:31Chris Benson?

  that's reachable going forward to help us get there and use AI to help us solve the AI at work problem.

• 17:37Daniel Whitenack?

  I did, just reading a little bit further along, they do make us, you know, some statement about the explainable and transparent part saying they'd like to grow the data set since at the current stage, it's hard to know why exactly it said what it did. So that that kind of gets to your gets to your previous point and sounds like they'll have more for us in the future.

• 17:59Chris Benson?

  Absolutely. And we can put in the show notes, the paper that this is based on is called Delphi Towards Machine Ethics and Norms, and we can include that for people to go read as well as the article.

• 18:18Speaker 03

  We deserve a better internet and the brave team has the recipe for bringing it to us. Start with Google Chrome, keep the extensions, the dev tools and the rendering engine that make Chrome great. Rip out the Google bits, we don't need them. Mix in ad and tracker blocking by default, quick access to the Tor network for true private browsing, and an opt-in reward system so you can get paid to view privacy respecting ads. Then turn around and use those rewards to support your favorite web creators like us. Download Brave today using the link in the show notes and give tipping a try on changelog.com.

• 19:02Daniel Whitenack?

  Okay, Chris, well, last year, I don't know if we'd consider this a follow-up fully connected episode, but last year we did talk about the AI Index Report, which came out from the Stanford Institute for Human-Centered Artificial Intelligence. And we have an updated version of that for this year. There's sort of a bit of a summary article, which I think you ran across also in IEEE Spectrum, but you can also look at directly at the AI Index Annual Report site, and they have some sort of major takeaways as well. So yeah, this is, as always, very interesting. I encourage any practitioners

• 19:43Daniel Whitenack?

  who are working in the space to take a look at this and to see what some of the trends are from from the past year. I'll start off by just... Mentioning that the number one takeaway that the Stanford site mentions is that AI investment in drug design and discovery increased significantly. So there's more than thirteen point eight billion dollars of investment in in this area, apparently, four and a half times higher than in 2019. I don't know what possibly could have could have spurred on that that shift.

• 20:20Chris Benson?

  Yeah. What could it be? Hmm. Pandemics.

• 20:23Daniel Whitenack?

  What sort of health related thing happened in the past?

• 20:26Chris Benson?

  And yet, and yet they don't mention that in their little in their lineup. But but it's interesting. It is notable in that it's a very specific use case. And the ones around it that we'll hit in a moment are a little bit more general than that. And that was the very first thing I noticed when we when we start when we started looking through the article before the show was was just how I was like, wow, OK, that's number one that they're calling the attention out there. So it'll be interesting to see over the next few years where this goes with with drug development.

• 20:56Daniel Whitenack?

  Yeah, I know that there've been there've been efforts for some time and sort of protein folding and genomics bioinformatics as related to AI techniques. And it sounds like that that's increasing a lot. But now sort of in the commercial sector where people are maybe where I heard about it a little bit more was on the academic and research side. So it sounds like this is shifting a little bit into industry. It's a difficult problem with a lot of data that is also very large and complicated data like genomics data or

• 21:36Daniel Whitenack?

  bioinformatics data. And AI is is particularly good at those sorts of problems. So yeah, it'll be be interesting to see how people apply this. And also how one of the things I'm always thinking about is how are the experts going to be involved in the loop with the model? So are they giving some hints to the model to as the sort of probes the landscape of drug design? They I'm guessing that not all of the

• 22:07Daniel Whitenack?

  People working in drug design won't be a part of the process moving forward. And I'm, I'm not in that field, so I don't know exactly what sort of input they might give. But I hope that, you know, that's being thought about as well.

• 22:20Chris Benson?

  Indeed. The next one they mentioned is the industry shift continues. And what they're talking about there is they note that 65% of graduating PhDs in North America in the AI space went into industry instead of staying in academia. And that's up fairly substantially from 44.4% a decade ago. And when I think about that, I think about you, Daniel. You represent that trend. And when I think about our guests on the show over the last three years, we've had many, many people with PhDs in either this or related field, or they've migrated into this field. And the majority of them have been in industry as opposed to academia. We've done both. I think the show represents that trend very directly.

• 23:05Daniel Whitenack?

  Yeah, and I'm a little bit curious as to, and I'll have to do a little bit more reading if, you know, what the category of PhDs in AI quote is. I know that, you know, part of my background, like you said, is, you know, from physics. And in physics, you know, at the time, and I think, I think it continues to today, although I haven't followed it as much is in physics, there hasn't been like paradigm sort of shifting things happening for the most part. If you look on the whole, you know, since the sort of twenties and thirties, 1920s and thirties, and you know, the huge revolution that happened. And certainly there have been advances, but because of that,

• 23:49Daniel Whitenack?

  the sort of investment and the amount of academic positions in physics weren't that many and is very highly competitive, right? So you might end up doing like. Three postdoctoral positions, you know, before you end up getting a getting a tenure track position and then and then, you know, you're still not quite to like a stability yet because you've got to go through the, you know, tenure type stuff. So yeah, it's very difficult. Um, I'm sure that that impacts other fields as well. But when you can, you know, come out of your PhD and still do sort of cutting edge research, but at a cool place, like whether

• 24:30Daniel Whitenack?

  it's Google Brain or the Allen Institute, or, you know, a cool startup like Hugging Face or something like that, and you're still writing papers, but you're into it and you can have a little bit more of your own research agenda and participate in a significant way, sort of out of the gate, and also not teach. A lot of people don't like to teach. Um, it's it's pretty appealing, not to mention that the salaries are higher. So it will be interesting to see how much is drained out of academia and if the the competition for academic positions changes as a function of this.

• 25:09Chris Benson?

  True. The next thing they mention is generative everything. They talk about the fact that text, audio, images are so high standard at this point that it's pretty hard to tell the difference. And that's true. I've noticed that on a number of AI-driven services with, you know, generative output. You would think a human robot.

• 25:32Daniel Whitenack?

  Yeah, and it's kind of multimodal at this point. I mean, used to, they had sort of images or images of people or something like that. But I know that we've had people on the show thinking when we talked about machine learning for music and generative music or text in the case of GPT-3 or something like that. Um, is that like in your, as you think about AI and the, you know, what you worry about in terms of applications of AI, does that factor into like something that weighs on your mind that, you know, generative things are out there, it's hard to tell the difference, there could be a contribution to fake information?

• 26:14Chris Benson?

  Oh, sure.

• 26:15Daniel Whitenack?

  Does that that factor up there in in your mind? I know you do a lot of thinking on Sort of ethics, principles and strategy?

• 26:22Chris Benson?

  Yeah, I mean, at that level, it's a huge concern. Governments are concerned about it. Corporations are concerned about it. I think people should be concerned about it. It's one of those things we've let the genie out of the bottle and the genie is really convincing. And yet it is quite difficult to tell the difference between the genie and the human. So if you're putting tooling out there with a specific agenda around it, that maybe the affected parties might consider nefarious, then there's a lot of things that we have to sort through. I think that's one thing we've learned. We started with the ethical conversation at the beginning of this, but there are so many areas in AI that we have not figured our way through

• 27:06Chris Benson?

  in terms of how to have a safe, productive, good world with these tools. It's not that the tools are bad, but they fall into the hands of people who were out to affect an agenda. So it definitely affects all of us and it's definitely something I worry about.

• 27:24Daniel Whitenack?

  So in terms of your own just day-to-day thinking, when you're talking to your families or family or others about AI, what should be prioritized as maybe the biggest problem that we're facing with AI? Is it the sort of talent and diversity among that talent within the AI community? Is it like generative things and misinformation? Is it bias in data sets? Is it AI becoming sentient and, you know, taking over the world? What's on your mind as you're talking to people?

• 28:03Chris Benson?

  I would say the generative stuff worries me the most in the short term because of those issues, because it can be taken advantage of so easily and it's so accessible to such a wide audience. And the output is of such high quality that it allows us, I mean, we've really seen this over the last few years here in the United States in particular, in our political system. Is that it can make you question what is real? And I have really, really good friends here in the American Southeast that see the world in a completely different perspective. You know, everything about the world is different to them. It's almost like we don't live on the same planet in the same country.

• 28:47Chris Benson?

  And when you are shaping the, the data or the information that gets to people that they are choosing to process and you can channel that and you can target people, it can make the whole system fall apart if you let it go far enough. And so that worries me in the short term, a whole lot more than AI becoming sentient and taking over the world and, and eating us all as, you know, Matrix Neos or whatever, you know, or non-neos maybe. That may someday become a problem, I don't know. But it's not the problem that we have today or in the near future, but the problem of, of understanding what our reality is. I know that's real because I can talk to some of my very closest friends.

• 29:31Chris Benson?

  It's, it's completely different stuff.

• 29:33Daniel Whitenack?

  Well, one of the interesting links that you forwarded along to me, Chris, was, so there's the AI index itself, but then I triple A spectrum did a sort of take on the index, which is titled 15 graphs you need to see to understand AI in 2021. I would recommend people, if, if they don't want to read the whole index, this is a good, well, I don't know, this is pretty much always what I do when I read a paper, right? You read out the paper and you sort of like flip through and look at the graphs and see what catches you. Maybe, maybe other people aren't as vain as me and just looking at pictures. But that's normally what I do.

• 30:13Chris Benson?

  That's what they're there for.

• 30:13Daniel Whitenack?

  Yeah, exactly. So this is an interesting sort of take on everything. One of those graphs that stood out to me, which I think is a really interesting idea with a whole variety of.

• 30:28Daniel Whitenack?

  Implications and aspects is that they say number three faster training equals better AI. So in other words, training has become faster based on certain, you know, AI training of of models has become faster. They talk about the standard ImageNet dataset and training, training a state of the art model on that dataset. In 2018, it took 6.2 minutes to train the best system. And in 2020, it took 47 seconds. And of course, this is due to a lot of the advancement and sort of accelerator chips and distributed training and,

• 31:09Daniel Whitenack?

  you know, specialized hardware that's designed specifically for AI and machine learning. And the implication is that, you know, if you're able to run your experiment in like 30 seconds rather than waiting like three hours, then you can run it a lot of different ways with different parameters and all of that stuff. And eventually get to a better model than you would have if you were, if your training was, was slower. To me and to some degree, training is getting faster. And so that makes me encouraged because we're also having a significant sort of like sustainability problem in in AI

• 31:49Daniel Whitenack?

  where like these large language models and other ones, it takes so long to train and it takes so much power. It's great if you can run things faster, but then if, if you can run things faster, then maybe you just end up running more things rather than like not needing to run as, as many things for as long. Which, yeah, is kind of the other side of this, I guess.

• 32:13Chris Benson?

  Put simply, if you have models that can be trained that much faster, you have a lot more options. Cause it's still humans on a human time scale, on a human day at work that are doing. That are putting these together. We're still filling those hours. And if you can train many more times in the given, a given span of time, then you get better stuff. I thought a lot of the things that were in this made a lot of common sense. That being one of them is you have more options and I don't see that slowing down anytime soon. So.

• 32:44Daniel Whitenack?

  There's also a couple of graphs about citations and some sort of like implications of that. The one that they've labeled is we're living in an AI summer.

• 32:55Chris Benson?

  Yep.

• 32:55Daniel Whitenack?

  Where if you sort of look at the graph and this is what's hard about being on a podcast with only audio, but if you look at the graph of sort of citations per year, number of AI citations and journal articles per year, that percentage or number, and you look at that graph over the years, it's sort of from 2000, it kind of is rising up, rising up. And then it kind of maybe plateaus or even goes down a bit to where I remember people were talking about, hey, well now we're just like the peak has happened and now we're just sort of going back into some type of winner or normality or something like that. But then it comes sharply back up again in 2019 and 2020.

• 33:38Chris Benson?

  This chart I found really, really interesting because we've had that conversation ourselves and I've been one of those people that was a little bit disappointed. But if you look at the dates on it with it, it rising steadily up to 2013, 14, and then kind of taking a dip over the next few years. And then once again, starting that uptrend in 2019, it made me realize the perception of what's happening AI trails the publications of the research papers a bit because if you look at the over three years, you and I have been doing the show, a good bit of those early episodes, we're kind of catching up and us educating ourselves and educating folks in our audience about all the research that had already happened. And then we kind of went through that. And I was, you know, going

• 34:23Chris Benson?

  back to me at the beginning of the show, talking about whining, I'll whine a little for a second. And I was whining that, wow, we haven't, it hasn't felt like we had big breakthrough things like we used to feel like were happening a lot. But I think we have, it's just taking us a little while for those to get out in the space and people to take advantage of it. At least that's how I'm interpreting the data I'm looking at, given the little bit of time lag. So I would say that bodes very, very well for for us continuing to have these, these great conversations.

• 34:52Daniel Whitenack?

  Yeah, yeah, for sure. The other trend within the citations that they note is that China takes the top citation honors. And one of the things that they highlight about sort of AI research in different geographic locations or countries is that China sort of has a stated policy of getting journal publications, that they really push for that. Whereas in the US, maybe a good portion of AI R and D happens in corporations. And so if you're in a corporation and you're doing AI research, generally, you know, this isn't true across the board, but you might have

• 35:33Daniel Whitenack?

  less of an incentive to publish journal articles, especially if the company is wanting to keep trade secrets, or if maybe you're just trying to get the product finished, or advance something,

• 35:47Chris Benson?

  or whatever. Or if there are national security implications, which is something I'm obviously familiar with in my industry.

• 35:54Daniel Whitenack?

  Yeah. So this wouldn't maybe be saying that the US isn't doing an increasing amount of AI research, but in terms of the publications, China is definitely now dominating. And you know, they've, they've put a lot of investment in that area. And so it's not, not surprising. And they keep, their numbers keep rising and they keep doing more and more AI research. So yeah, it's definitely maybe changing the landscape a little bit.

• 36:24Chris Benson?

  It is. And we've been watching that trend, you know, you know, we've gotten the crossover. We've been watching that for a while. It was expected. The big question mark is with their focus on publishing, and yet we know that there's a lot of unpublished research. What's the delta between what's published and what's not on the non-China side? I can't help but wonder.

• 36:46Daniel Whitenack?

  Yeah, yeah. Another interesting fact that they talk about in this report is the global AI job market. So I know a lot of our listeners, you want an AI job. Talked to many of you about it. And so this I know is an interesting, interesting aspect of what people are thinking about and interested in. And the AI index talks about Brazil, India, Canada, Singapore, and South Africa's having the highest growth in AI jobs. And so we're thinking of strong representation in Asia and Latin America in AI jobs, which is really interesting. I remember

• 37:28Daniel Whitenack?

  a few years ago, I visited AI Singapore, which was a really innovative and great effort that the government and universities in Singapore were behind in terms of becoming a leader in that area. And so, yeah, there's, you know, countries are promoting this and trying to establish AI talent in these places, not just sort of have AI practitioners from those places, go to other places to do their AI work.

• 37:59Chris Benson?

  Yeah, I agree. There are a couple of points they make, and it's actually in both the IEEE Spectrum, they highlight some of this, and it's also in the in the core article. They note that AI continues to have a diversity challenge in terms of its practitioners. And then, you know, relative to diversity, you just now were talking about those different countries where it's really on the rise. But they also note that the majority of US AI PhD grads are from abroad and that they're staying in the US. It's an interesting mixture of how that all shapes together and looking at it, you know, but I can't help but ask along the way, why do you think that most of our PhD grads are from abroad? Any thoughts

• 38:42Chris Benson?

  or opinions or anything on that versus, you know, why are we not attracting here in the United States, more students that want PhDs in AI and get out there in industry? What do you think's happening with that?

• 38:54Daniel Whitenack?

  Yeah, I don't know. I mean, there's a lot of, um, a lot of things factoring into the makeup of grad students and PhD students. And I'm probably not an expert in that, in that area to, to comment, but I think there is a trend sort of for, and I, I don't know if you picked up on this, but it seems a little bit like from the US side of things, if you think about and look at some sort of like data science, either data science curriculum or bootcamps or content that's out there on the internet. Used to, I think it was fairly common for a data scientist to be a PhD.

• 39:38Daniel Whitenack?

  But now I think people are promoting more the path of like data science as a sort of post undergrad career, where you can sort of come out of a computer science background and into data science, not necessarily as a researcher, but as a practice, data science practitioner, a non PhD data science practitioner. My guess would be that that would have grown, but while maybe the PhDs are staying more static.

• 40:07Chris Benson?

  I, I mean, to some degree, that's me, you know, I, earlier I said something reminded me of you, but I'm not a PhD, uh, and I come from a software development background and my perspective was to pick up data science as, as yet another skillset to add value into the types of activities I was trying to, to accomplish. Speaking empirically, I, I see that in the people around me a lot. And so, yeah, rather than going all the way through on the PhD track to just go ahead and get out there and start doing it, but it's one of many things that they're doing. Which actually leads me to one other thing that I don't see on either of the, the article or called out in the report, it may be in the report, because it's 222 pages and I have not read that whole report.

• 40:50Chris Benson?

  I focused on the highlights, but it doesn't call out anything about, um, job concerns and stuff, which is a huge thing on people's mind at this point. I have people ask me about that, just knowing me from the show and stuff, all the time. It's one of the most common subjects that people approach me on, but I don't see anything in there. Do you think they've they've missed things? Or maybe that's another, maybe that would have been the the 16th takeaway in the article or the 10th takeaway in the in the the source.

• 41:18Daniel Whitenack?

  And by concerns, do you mean like the availability or competition for positions?

• 41:24Chris Benson?

  Or? Yeah, I mean, you have so much automation happening out there.

• 41:29Daniel Whitenack?

  Oh, I see the impact of AI on other.

• 41:32Chris Benson?

  On jobs, and I'm seeing that a lot and in different communities. And a lot of times it's pretty subtle now that you now that deep learning deployment has become cheap. If you especially if you have a very specific focus and you're seeing really common mundane tasks being replaced by automation that has this benefit of of deep learning to help drive what its task is. I have people ask me about that a lot, but maybe that's, you know, maybe it's not a key takeaway. Maybe we're not quite there yet. What do you think?

• 42:04Daniel Whitenack?

  Yeah, it could be that it's a lot of companies that are trying to introduce that automation still need a similar workforce, but maybe just doing slightly different things. So there's maybe not as many areas where automation is completely taking away positions, but they're more morphing that position into into something else. That that might be a guess, but or maybe it's just not a massive change in the past year. With respect to that, it's sort of still happening as it was in previous years or something like that. And not as much of an acceleration and in sort of the automation of jobs away factor

• 42:47Daniel Whitenack?

  seems to be like something we can make an acronym for. Yeah, so I encourage encourage our listeners, we'll post the links to the report and IEEE. Article which goes through the different graphs on that. I have one thing that I wanted to highlight in in these episodes, the fully connected episodes. We always like to do a learning resource or two for listeners. And something came across, uh, came across my Twitter feed. And I was looking into a little bit more because I'm also helping connect some of our practitioners with some professional development opportunities here at S I L.

• 43:27Daniel Whitenack?

  But Hugging Face now has a Hugging Face course for transformer models. So the course includes things like an introduction to natural language processing and transformers. What can they do? How do transformers work? Encoder models, decoder models, sequence to sequence models, talking about bias and limitations, using transformers, fine tuning a pre-trained model and sharing models and tokenizers. So this is really, it seems like a great time for this to exist as Hugging Face now has integrations with even sort of multimodal things like speech and image related models.

• 44:10Daniel Whitenack?

  And so learning about transformers, learning about how to share models and work with data sets using an open source framework like Hugging Face data sets. All of that seems super relevant and timely. So if you're interested, I highly recommend that you check out the Hugging Face course and learn a little bit about that. So we'll include a link in our notes.

• 44:32Chris Benson?

  I think I'm gonna go do that one myself.

• 44:35Daniel Whitenack?

  Yeah, and it includes some sort of video components, text and images. And yeah, check it out. Well, Chris, it's been really good to, uh, really good to chat through some of these things and revisit the AI index this year and talk through some of it. Thanks for pointing me to it and, uh, and for talking through some of the points. Well, I'll look forward to talking, talking to you about it again next year.

• 45:01Chris Benson?

  It was a good conversation this week and good, interesting material like always this year, and I will talk to you next week.

• 45:08Daniel Whitenack?

  Sounds good.

• 45:12Speaker 03

  That's our show. Thanks for listening. For more like this, check out our master feed. It is all Changelog podcasts in one easy to consume place. Let your podcast app snack everything we produce and then pick and choose which ones to listen to. Subscribe today at changelog.com/master or just search for Changelog Master in your podcast app of choice, you'll find it. Special thanks to Brake Master Cylinder for providing our music and to our longtime sponsors, Facely, Launch Darkly, and Linode. That's all for this week. We'll talk to you again next time.