AI in the majority world and model distillation

Chris and Daniel take some time to cover recent trends in AI and some noteworthy publications. In particular, they discuss the increasing AI momentum in the majority world (Africa, Asia, South and Central America and the Caribbean), and they dig into Hugging Face’s recent model distillation results. Sponsors: DigitalOcean https://do.co/changelog – DigitalOcean now offers three managed databases https://www.digitalocean.com/products/managed-databases/ — PostgreSQL, MySQL, and Redis. Get started for free with a $50 credit. Learn more at do.co/changelog https://do.co/changelog. All Things Open https://allthingsopen.org/?utm_source=changelog – Exploring open source, open tech, and the open web in the enterprise. Raleigh, NC — October 13-15, 2019 The Brave Browser https://brave.com/cha913 – Browse the web up to 8x faster than Chrome and Safari, block ads and trackers by default, and reward your favorite creators with the built-in Basic Attention Token https://basicattentiontoken.org. Download Brave https://brave.com/cha913 for free and give tipping a try right here on changelog.com https://changelog.com. Featuring: Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: AI in the majority world: NeurIPS publications by country https://medium.com/@chuvpilo/whos-ahead-in-ai-research-insights-from-nips-most-prestigious-ai-conference-df2c361236f6“Best” universities according to the Times Higher Education https://twitter.com/theworldindex/status/1173797410352578560ICLR in Ethiopia https://iclr.cc/Google AI offices: Bangalore https://blog.google/around-the-globe/google-asia/google-research-india-ai-lab-bangalore/Ghana https://www.blog.google/around-the-globe/google-africa/google-ai-ghana/Beijing https://ai.google/research/join-us/beijing/Deep Learning Indaba http://www.deeplearningindaba.com/Southeast Asia Machine Learning School https://www.sea-mls.com/Facebook’s AI for India summit https://facebookaiforindia.splashthat.com/AI Singapore https://www.aisingapore.org/Indonesia AI research center https://en.tempo.co/read/922922/joint-effort-drives-indonesias-first-ai-research-centerArtificial intelligence probes dark matter in the universe https://phys.org/news/2019-09-artificial-intelligence-probes-dark-universe.html DistilBERT from Hugging Face https://medium.com/huggingface/distilbert-8cf3380435b5 Three People-Centered Design Principles for Deep Learning https://sloanreview.mit.edu/article/three-people-centered-design-principles-for-deep-learning Papers with code https://paperswithcode.com/ Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Being with your change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at Change Law because of Rollbar. Check them out at rollbar.com. And we're hosted on Leno cloud servers. Head to leno.com/Changelog.

• 0:16Speaker 00

  This episode is brought to you by Digital Ocean, the simplest cloud platform out there. And we're excited to share they now offer dedicated virtual droplets. And unlike standard droplets, which use shared virtual CPU threads, their two performance plans, general purpose and CPU optimized. They have dedicated virtual CPU threads. This translates to higher performance and increased consistency during CPU intensive processes. So if you have build boxes, C I C D, video encoding, machine learning, ad serving, game servers, databases, batch processing, data mining, application servers, or active front end web servers that need to be full duty CPU all day, every day. Then check out Digital Ocean's dedicated virtual CPU droplets. Pricing is very competitive, starting at 40 bucks a month.

• 1:01Speaker 00

  Learn more and get started for free with a $100 credit at do.co/Changelog. Again, do.co/Changelog.

• 1:21Speaker 00

  Welcome to Practical AI, a weekly podcast about making artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and slag with us around various topics of the show at changelog.com/community. Follow us on Twitter, we're at Practical AI FM. And now into the show.

• 1:47Daniel Whitenack?

  Welcome to another fully connected episode of Practical AI. In these episodes, Chris and I keep you fully connected with everything that's happening in the AI community. We'll take some time to discuss some of the latest AI news and trends. And we'll dig a little bit into learning resources to help you level up your machine learning game. I'm Daniel Whiteneck. I'm a data scientist with S I L International and my co-host is here, Chris Benson, who's a principal AI strategist at Lockheed Martin. How you doing, Chris?

• 2:21Chris Benson?

  I'm doing great. How's it going today, Daniel?

• 2:23Daniel Whitenack?

  It's going pretty good. Yeah, it's been reasonably normal week in the sense that I've just mostly been at my desk here, working on things. But yeah, a lot of exciting NLP and, you know, language-related stuff coming up this fall and into the spring for me. So I'm excited about that. But your week's been a little bit more exciting, I hear.

• 2:46Chris Benson?

  Well, I'm in London right now as we're recording this and started off doing some stuff with the Royal Academy of Engineering, representing Lockheed Martin and

• 2:55Daniel Whitenack?

  Yeah, were you knighted?

• 2:57Chris Benson?

  I was not knighted and I was so disappointed that I wasn't.

• 3:00Daniel Whitenack?

  Wait, is it only, I guess it's only possible for English citizens or UK citizens to be knighted? Is that true?

• 3:09Chris Benson?

  Yeah, I'm not sure, but you know, I'm married to a Brit and my daughter is half Brit.

• 3:15Daniel Whitenack?

  That should count.

• 3:16Chris Benson?

  I was hoping that they'd make an exception, but they didn't. I'm going to register a complaint with Buckingham Palace.

• 3:21Daniel Whitenack?

  But they didn't kick you out. It was a good experience.

• 3:24Chris Benson?

  They didn't kick me out, did a talk earlier in the week and did a panel and was privileged to meet a lot of really, really, really smart people. And that was really good. I still had to work remotely after that was done because we're about to start a family vacation over here. So I was just kind of waiting for family to arrive and I dashed off to North Wales and climbed Snowdon and then still put in a full workday every day. And so it's been busy, but I'm back in London now and tomorrow is vacation. So all is good, my friend.

• 3:55Daniel Whitenack?

  Yeah, you're almost there. We gotta, we'll get through this podcast recording and then it's all vacation from there, at least until I talk to you next.

• 4:06Chris Benson?

  I'm already drinking a Welch beer right now, so I'm getting started. Okay.

• 4:11Daniel Whitenack?

  Nice. Cool. Well, today on the Fully Connected episode, a few of these episodes in the past, we've kind of dug deep into individual topics. And the last one, we talked about high-performance computing, which was great. But today, we're just going to take some time and discuss a little bit about some of the trends that we're seeing in the AI community, a few news stories that caught our attention. And we've done this in the past, but every once in a while, we like to kind of update things in this way. So to get things started, I just kind of wanted to bring up a trend that I'm definitely seeing, which I'm encouraged by, and I think is a really good

• 4:52Daniel Whitenack?

  sign. And that's what's seeming to be some good moment in terms of AI contributions, AI education, AI research activity, all of these things in the majority world. So outside of the US and Europe, in places, countries in Africa, or maybe Indaba, Southeast Asia, and where most of the people in the world live. So it's great to see a lot of AI activity starting to happen in these areas. There's definitely still a lot of, you know, room for growth. There's an article that we'll reference in the show notes, and we'll reference all the articles in the show notes that we talk about. But they kind of look at the publications for the NERIPs

• 5:36Daniel Whitenack?

  conference. And of course, those are still highly dominated by the US mainly, but also European contributors. But you can kind of start to see some other regions of the world contributing. So there's a lot of room for growth. But I've definitely been seeing a lot more attention being placed on education, AI research, AI innovation, in other places around the world. I don't know if you've seen this as well, Chris.

• 6:00Chris Benson?

  I have. And it's really good to see, you know, we talk so much about the democratization and commoditization of AI in terms of accessibility. And so seeing it, seeing it really blossom all over the world, and not just in kind of, you know, US and Europe, and kind of key nations around the world. Really good. I, you know, we talk a lot about, I like seeing it every time, you know, it's in Africa and I know we've talked about in the past and I think we're going to talk about that a bit today. So I'm pretty happy and I've also noticed it's going into a lot of colleges and universities around the world that are not like top tier strictly. And so instead of everything strictly being, you know, like in the US, you know, Stanford, MIT, that kind of thing.

• 6:44Chris Benson?

  There's a lot of second tier universities that are trying to do their thing in it and I'm very encouraged by that. It's really becoming available to everyone.

• 6:53Daniel Whitenack?

  Yeah, definitely. And I mean, there's a lot of problems that are relevant to AI that are probably people from those regions have more domain expertise and more empathy for these sorts of problems around things like translation or things like certain agriculture related applications or other applications that, you know, I'm sure people are working on, but these sorts of problems. The major application of those seems like they would be in places where there is a lot of language diversity or where there is a lot of agriculture, right? You know, some of the things that I've seen recently are the ICLR, ICLR conference is happening in Ethiopia this year, which it's a, it's a big AI research

• 7:38Daniel Whitenack?

  conference, which it's really great to, to see that happening there. I wish we could be there. Maybe, uh, you know, if there's any organizers listening to this and you want our podcast there, definitely, uh, let us know. I'd be up for going to Ethiopia.

• 7:52Chris Benson?

  Yeah, absolutely.

• 7:53Daniel Whitenack?

  Would you?

• 7:53Chris Benson?

  Yeah. Totally. And I agree. And I say this, uh, I'll say it quietly. I'm sitting in London right now and I don't want to see all AI stuff in, you know, New York City and London and, and San Francisco and places like that, you know, it's fantastic to see Ethiopia. I think, I think we should start tracking that a little bit in terms of, uh, places you wouldn't expect to see it that is popping up and, and give a shout out to people like that.

• 8:18Daniel Whitenack?

  Yeah, definitely. So I know Google AI, for example, has started opening, I should say, opening offices in other places around the world. They just announced, I forget when this announcement was, it was just in the last week that I saw it at least, that they're opening up an office in Bangalore, in Bangalore, India, but also they have offices in Ghana and Beijing. And then also, we're mentioning Africa. I know there's a series of conferences and events and workshops, you know, called the Deep Learning Endaba that's happening there. I've seen other things that are being sponsored by companies like Facebook, Google, Microsoft, and other sort of large tech entities that are happening there. There was a Southeast Asia Machine Learning

• 9:02Daniel Whitenack?

  School. There was an AI for India Summit that Facebook put on in India. I think that was actually also in Bangalore. It seems like a lot of these tech companies are really interested in developing AI talent and AI expertise in these, you know, majority world countries. And I have my own thoughts about, you know, why that might be taking place. But do you have any thoughts on sort of why tech company like Google or Microsoft would be interested in that?

• 9:34Chris Benson?

  Well, I mean, I think part of it is it's really hard to find great talent. And so I think having a diversity of your, in terms of your search for the people that are interested and capable of doing this, it makes sense to go to third world countries and take a bunch of interested people and, you know, bring them into the fold. I love that. I'm looking at the Endaba website from a few weeks ago when they had the conference. And, you know, to your point, the sponsor list is a who's who. It's, you know, DeepMind, Microsoft, Google, Facebook, IBM, Apple. I mean, it's the same tech companies that you're going to see sponsoring stuff in San Francisco and New York and London and such. And so I love seeing that attention. And I would

• 10:15Chris Benson?

  love to see, in this case, since we're talking about Africa, Africa being able to really get a great AI community behind it. And while they're at it, they should listen to our podcast.

• 10:26Daniel Whitenack?

  Yeah, and we should get involved as we can, for sure. I mean, I definitely agree with you. I think that the talent is one side of things. I also think that, you know, hopefully, these companies are starting to realize, like I mentioned, that some of the problems that they're really putting a lot of focus into, like Facebook is putting a ton of focus into machine translation and language tech. And really, the, you know, the sort of real world experience and domain knowledge that people from Africa or Southeast Asia could bring to those sorts of efforts. I think you're just gonna end up with better results by using expertise that's rooted and has experience in those areas.

• 11:09Daniel Whitenack?

  And of course, you know, AI, I think, is going to be kind of ubiquitous in the software stack as we move forward. And so as we try to build up, you know, software engineering in general in these different areas and education in general, I think it definitely needs to be a part of it. I wanted to share too, recently, in July, I was in Singapore and had the chance to stop by the AI Singapore offices, which you might have heard of programs like this. So it's kind of like the Prime Minister's Office in Singapore says Singapore needs to be a leader in AI. How are we going to do that? How are we going to develop local talent in AI?

• 11:52Daniel Whitenack?

  How are we going to contribute to research, et cetera, et cetera? And they established this AI Singapore organization, which is associated with the Prime Minister's Office, and their job is to basically figure that out. I have to say, I was, I was super impressed with the program that they've put together there. It seems to be run like a well-oiled machine. So companies can come to AI Singapore and they basically say, okay, if you have a problem that's related to AI, let's partner together to solve that. What we're gonna do is we're going to form a team in AI Singapore, which includes, you know, really top-notch mentors in AI and researchers in AI paired with AI.

• 12:36Daniel Whitenack?

  Apprentices that they're training up and your engineers from your company. So basically, everybody's leveling up in AI at the same time. And their focus is really not just to kind of learn interesting things, but they're working on, you know, real world problems that can be solved. And they really want to take on problems that will be pushed into production in commercial entities. So they have this whole program around it. And I have to say, I was, I was super impressed with the talent that was there and the program that they have going on. So it's just cool to see that sort of thing happen. And I shouldn't have been shocked, but I was kind of a little bit shocked that, you know, this was going on in Singapore and at such a great level and I had no idea about it.

• 13:18Chris Benson?

  So is it accurate to say that it's sort of running like an AI incubator? The way you see incubators in the US, you know, where they're nonprofits and, you know, university affiliated organizations that are, you know, taking the talent in. They're just doing it instead of being around a university setting specifically. They're doing it out of the prime minister's office. Is that a fair way of assessing?

• 13:38Daniel Whitenack?

  Yeah, it's not unrelated to that sort of incubator idea, but it's really more maybe an accelerator or something like that paired with a educational piece is there. Because what happens is every year they have, you know, engineers and whoever from Singapore apply to become AI apprentices in the program. And I think they said they were like, you know, it's really competitive. I'm probably going to get the numbers wrong, but there were like 800 people applied and 20 get in, right? So it's really competitive. And so you've got these top notch applicants and they're trading them up over this sort of nine month period to be AI engineers. And so they go through a little bit of training, but then they're also kind of their capstone

• 14:23Daniel Whitenack?

  or the project that they work on is an actual problem within a company, tech company that they're working with. And so they sort of form this collaborative team with mentors and the apprentices and the industry company to, uh, to actually solve a problem that will go into production. So it's kind of existing companies leveling up their AI expertise, while at the same time, you know, developing AI talent within Singapore. It's pretty cool.

• 14:50Chris Benson?

  Do you think you're going to see more of these popping up associated with, with various nation states around the world? Do you think, do you think this is going to be a kind of a common blueprint that we'll see?

• 14:59Daniel Whitenack?

  Yeah, I certainly hope so. They kind of seem to have this down. And I'm kind of actually shocked because most of the times when I think about like internship or sort of accelerator programs or project focused program partnerships between academia and industry, a lot of times those seem like they're run really poorly. And I think what I was shocked by was this was run really well. So I think if other countries are really serious about this, I think it is a model that we might see more of. I think it's hard to get right for sure because, you know, how many times have we seen, oh, you know, this company is going to work with this university or there's going to be like this center of excellence at a university that we're going

• 15:42Daniel Whitenack?

  to work with certain companies from industry and then, you know, some stuff happens, but it's not really that impactful. So I hope that these sorts of things are a little bit more impactful.

• 16:01Speaker 03

  Greetings, AI practitioners. Jared here, wanting to let you know that Changelog will be at All Things Open on October 14th and 15th. We're hosting a live JS party onstage, and as a special thanks from the organizers, we're giving away five free passes to the conference. All you have to do is tweet, "I want a free pass to All Things Open because", state your reason, and mention @changelog or @practicalaifm so we see it and we will DM you if you win. Okay, that's all for me. Let's get back into it.

• 16:40Daniel Whitenack?

  It was definitely great to talk about some of the things that have been going on in parts of the world that we're not currently in, but there's certainly a lot of AI news coming from the rest of the world. As well, and I think that you had something you wanted to highlight that you saw in, was it from MIT or where was it from?

• 17:01Chris Benson?

  Yeah, I saw it on fizz.org and so it's an article that I ran across and called "Artificial Intelligence Probes Dark Matter in the Universe."

• 17:10Daniel Whitenack?

  Ah, yeah, from ETH Zurich.

• 17:12Chris Benson?

  Yep.

• 17:13Daniel Whitenack?

  Sorry, got that wrong.

• 17:14Chris Benson?

  Correct. And so I know that you're a physicist by background and just as an amateur, I love physics. And so it certainly caught my attention and it was interesting that they were trying to explain it by drawing an analogy with facial recognition in terms of how they're using models to scan, you know, scan the universe and try to understand what both dark matter and dark energy are. And for listeners who may not be familiar, you know, the dark matter exerts gravity on the universe and you can measure that, but we can't see it. It's kind of like gravity coming from an invisible source. And Daniel, if I mess any of this up, you should correct me.

• 17:54Daniel Whitenack?

  No, you're doing great. Yeah.

• 17:56Chris Benson?

  And also, the universe is expanding, constantly accelerating in this expansion. And so that's dark energy. And once again, we cannot see that source. And so those are two of the great mysteries in physics at this point is trying to identify and understand what we're observing. And so seeing that they're using models to try to recognize those features and find the patterns that maybe otherwise we're not seeing, that was pretty cool.

• 18:24Daniel Whitenack?

  Yeah, definitely. I think that there's a sort of general trend in science where these sorts of AI techniques are being applied that are making an impact. And I've mentioned a couple of times on the podcast where machine learning started to make an impact on the field that I was studying when I was in grad school. But I think the pattern was similar there. And that's the sense that, you know, we have these experimental observations, which are kind of, you know, they're rooted in reality. And we have a little bit of knowledge about how things work and, you know, constants and other things and certain laws that shouldn't be violated. So constraints essentially. But the sort of relationships between input

• 19:06Daniel Whitenack?

  and output could be incredibly complicated to write down in terms of equations or maybe computationally too expensive. So the problems that we are working on were You know, we know that this atom or molecule has this many particles in it, right? Electrons and neutrons, but to kind of write down the equations and actually make the computations about how all these things work together, it's actually computationally feasible to do that, you know, and just by the equations that we write down. And in this case with the dark matter, you know, we know some of these constraints, we know the experimental observations, but we're not able to sort of maybe write down well

• 19:50Daniel Whitenack?

  the statistics like that they're talking about that govern these things. And so kind of plugging in a neural network into that gap and having it try to learn some of the features that are important to some input-output, whether that's input, in this case, input, it sounds like these experimental observations, and output cosmological constants and other things, or in the case of atoms and molecules, inputting like geometries or numbers of particles and outputting energies. Putting a neural network into that gap where things are really hard to model can make a lot of sense, I think.

• 20:28Chris Benson?

  Yeah, I mentioned that they draw that analogy, which they talk about how Facebook uses its algorithms to find eyes, mouth, and ears and images, and that they're looking for these telltale signs of dark matter and dark energy, that they're basically looking for light bending, you know, and so as light is bent by the gravitational influence, I'm assuming that they're using convolutional neural networks, although they don't explicitly say that, to try to notice the subtleties in terms of identifying the relationships. But it's just really interesting to see it in being used in this way. I'm always fascinated to see all the different use cases across industry as we see this becoming more and more pervasive over time.

• 21:07Daniel Whitenack?

  Yeah, data science for science, I think that's kind of in vogue right now, I think.

• 21:13Chris Benson?

  It's a delightful redundancy of the word "science," isn't it?

• 21:17Daniel Whitenack?

  Yeah, very, very meta. Yeah. So, speaking of language, you know, listeners always know I'm keeping up with language-related things. So maybe you're out there, you're listening, you're like, oh, Daniel's gonna share another language-related thing. I'm tired of that.

• 21:31Chris Benson?

  Don't worry, we're gonna do an episode where you're gonna share all of that with us soon. So you're not getting out of it that easy.

• 21:37Daniel Whitenack?

  Cool. Well, I think that this kind of has a more general angle on it, so not just language. So hopefully our listeners are okay with that. But I was really intrigued by this recent thing that Hugging Face released. So if you, if you don't remember, the CEO of Hugging Face was on the podcast here talking about social AI and conversational AI, and that was a great episode. We'll link it in our, in our show notes. But they've kind of really been working on these, you know, large-scale language models. You might have heard of BERT or Elmo or GPT-2 from OpenAI.

• 22:17Daniel Whitenack?

  And so with this most recent release, or one of their recent releases and publications, they kind of took a different angle on it, and it's called Distill BERT. And so what they said was, okay, the trend in these language models is that they keep getting larger and larger, and they're trained on more and more data. And in fact, one of the references they give is a latest model from Facebook AI that was trained on 160 gigabytes of text, which that might not seem like a lot for those that do images or videos, but if you think about 160 gigabytes of text.

• 22:54Chris Benson?

  That's a lot of text.

• 22:55Daniel Whitenack?

  Raw text data, that's an enormous amount of text data. So they were motivated by the fact of looking at those things and then saying, okay, well, if we actually want to use those types of models in production, how do we do that under the sort of low latency constraints where we might want to make a lot of predictions in a very short period of time, like in a microservice or something like that? Also, How would we run those types of models on a smartphone where there's obviously constraints around energy efficiency, you know, maybe memory constraints? And also, just generally, the environmental cost of running large models like this is huge in terms of the computing requirements for them.

• 23:38Daniel Whitenack?

  And so they took this and said, what would it take to create a smaller, faster, cheaper, lighter version of BERT, which is one of these large-scale language models? And they ended up doing that. And this is what they're calling Distill BERT. And it actually has very small penalties in terms of evaluation, but it's smaller, faster, cheaper, lighter. It's in comparison, they have a little graph of how many millions of parameters various models have. And the latest, so there's an Nvidia model, language model, that has like 8,300 million

• 24:19Daniel Whitenack?

  parameters, and Distill BERT has 66 million, so a significant reduction in size. But only, you know, Distill BERT still maintains 95% of BERT's performance on language-understanding benchmarks like GLU, which I think for such a reduction in size with only a very small, in some cases negligible, penalty in performance, it's just really encouraging to see.

• 24:47Chris Benson?

  So since we're fortunate that you're kind of an expert in this topic to some degree, I've seen lately a lot of different articles about different types of compression and different types of architecture construction in an effort to make it more performant. Do you have any insight into how they might be approaching that? Or even if not, how would you tend to do that? And what kind of benefits can you get out? And what kind of applications might it enable?

• 25:13Daniel Whitenack?

  Yeah, definitely. I'm not an expert in all areas of this field. There's a lot of different techniques used here. Of course, there's a lot of, a lot of people working in this area, you know, Intel and Google and others are working to kind of shrink down models to put them on smartphones or even microcontrollers and other things. And so there, there's a lot of work going on there. Some of the methods kind of fall under, I think what's called pruning, which is kind of like cutting out parts of your network that might not be having an impact. Hopefully I'm saying that somewhat right as I'm not an expert on that.

• 25:49Chris Benson?

  I had a conversation with a professor at Georgia State who was, who was doing a version of that for compression. And, and so, yeah, keep going. I wasn't trying to cut you off.

• 25:58Daniel Whitenack?

  Yeah, no. So that's like, you could kind of think about that technique as a like, you train your model and then afterwards you go through this sort of optimization or compilation, which kind of prunes things out or makes the model smaller. So it's kind of like a post-processing thing, if that makes sense.

• 26:14Chris Benson?

  Do you think that'll be common in terms of as a technique in this area, you know, with you doing NLP all the time? And, you know, is this going to be a standard part of NLP deployment going forward?

• 26:24Daniel Whitenack?

  Yeah, I think it'll probably, at least based on my understanding, it'll depend on the, the type of model, the type of task, and also the type of target architecture in terms of like, actually how small do you need to get? I think that some of the goals of the Hugging Face team were to get the model small enough to where they could run it efficiently in production and maybe on smartphones, right? Which still are actually pretty computationally powerful if we at least compare them to like microcontrollers.

• 26:55Chris Benson?

  Sure.

• 26:56Daniel Whitenack?

  So they use this technique called knowledge distillation, and that's why the model is called Distilbert or Distilbert.

• 27:05Chris Benson?

  Which has to be a knockoff Dilbert, right?

• 27:09Daniel Whitenack?

  They should have come up with a logo as such. Yeah, so this model, you might have heard of sort of teacher-student training models. And the idea is, I think, again, you know, please, our listeners, correct me if I'm wrong. But I think the basic idea is that you have a sort of teacher model is a larger scale model, maybe like full BERT, let's say. And then you have a smaller model that is supervised during training by the larger model. So you try to get as close as you can to the larger model's performance and output distribution using this kind of teacher-student supervision. And that's kind of interesting because you kind of do a full-scale training on a larger model, so that that still happens.

• 27:54Daniel Whitenack?

  But maybe it doesn't happen over and over and over. You kind of are able to train these these smaller models to still get most of the performance out of the tasks that you're concerned with. And so I think this was the type of methodology that the Hugging Face employed. Hugging Face is a very open source focused company. And so in their blog post even, they show some of the PyTorch code and illustrate how to do this in PyTorch. So if you're interested in knowledge distillation and teacher-student models and want to actually get your hands dirty trying out some of these things, maybe trying your own distillation, then that would be a good place to start, I think, because you could look at some hands-on examples.

• 28:53Speaker 03

  This episode is brought to you by Brave. The Brave team is on a mission to fix the web by building an open source, privacy-focused, and performance-oriented browser. Browse the web up to eight times faster than Chrome and Safari. Block ads and trackers by default. And reward your favorite creators with the built-in basic attention token. Yes, you heard that right, a real-world use case for blockchain. Download Brave for free using the link in the show notes and give tipping a try on changelog.com.

• 29:33Chris Benson?

  Okay, so I found an article that was interesting to me from the title, but as I started reading through it, I realized that I actually know one of the authors. It's an article called "Three People-Centered Design Principles for Deep Learning", and it's by Dr. David Bray and R. Ray Wang. And Dr. David Bray and I both used to work at the same company together called Intellinet many years ago, and we've kind of loosely kept in touch.

• 29:59Daniel Whitenack?

  Yeah, so that's super cool. And I'm glad that you, like, when you said it out loud, it made so much more sense to me because when I first read the title, I was thinking like, "Three people-centered design"? Like, how do you center design around three people? I guess sometimes you're designing things for three people, you know, interacting kind of thing. So my mind was totally in the wrong direction on that, but thanks for clearing that up.

• 30:23Chris Benson?

  No problem. Actually, when I, just a minute ago, when I started saying it, I started saying it that way in error as well, and corrected myself if you noticed. So, yeah, a little bit of an awkward title there, but very interesting article. And a lot of that, from my standpoint, has to do with the fact that I'm a big advocate of keeping people and users at the center of technology. And that's not an AI-specific principle. You know, it's used across many different types of development processes and technologies. And I liked seeing this AI-centered piece. And when I go out and do keynotes and things like that around AI, it's one of the points I'm often making, so I was kind of delighted to see it. And he kind of starts off talking about in deep learning and how you think about the

• 31:06Chris Benson?

  outcome first with the intention of kind of avoiding bias in your process, which is very easy to allow to happen, as we all know. It's probably the most common problem in deep learning that we all face with our data. But how easy it is to get to potentially a bad outcome. And so by kind of keeping your people-centered outcome at the center of your process, he kind of says you're more likely to get an outcome and better training than you would otherwise get. And so... The three kind of keys that he's talking about here is transparency and he's talking about the fact that you really need to understand what you're trying to get to and be very

• 31:50Chris Benson?

  clear with what your intentions are in the training and make sure that your, your data is, is oriented on the outcome that you want and, and making sure that the process of doing the, the model training is obvious in terms of what your inputs are to get your output as possible so that no mistakes are made. And the second point is explainability. And this is obviously a huge area inside AI research in terms of being able to understand how the inferences that a model is making, how those inferences are being reached. And so that's certainly in the industry I'm in where we have a lot of autonomy in terms of autonomous vehicles.

• 32:29Chris Benson?

  You know, being able to put people's lives in that and being able to explain how your model's getting that is, is a, is kind of a key to people having confidence in that. And then the last thing is, is reversibility and understanding how you reverse out of a model about what it knows. It's kind of tied to explainability. So you're really looking between transparency, explainability and reversibility. You're really looking at something where your outcome is a little bit less black box. It's a little bit less mysterious and you have kind of a map on how you're working your way through the network. Obviously there are limitations on what we can do today and in each of those areas because obviously there's some fairly significant research going on, but it's a good focus to try to

• 33:12Chris Benson?

  kind of know that that's the direction you're going. And so I'm, I'm looking forward to seeing where we go. Have you, have you come across people centered design principles in the past, Daniel?

• 33:20Daniel Whitenack?

  Yeah, I definitely have. I mean, I wouldn't say I haven't had any formal introduction to the topic, but it's definitely come up in a lot of teams that I've worked on and different organizations. And I think that it's, you know, it's, it's an important piece. I just remember being in a meeting just the other day when one of my supervisors, we were talking about, you know, the topic just came about, you know, in general, what is the purpose of a commercial entity? And, you know, the idea that, oh, it's, it's to make profit came up. At least that's part of it. And, you know, he was saying, well, some people might think about it that way, right? But I think that

• 33:59Daniel Whitenack?

  Whether you're a kind of social good company or organization, nonprofit, or a commercial entity, ultimately, you're wanting to satisfy customers, right? And that should be at the very forefront of your design, of your product, of how you go about your business, that you're wanting to satisfy your customer or your user. In this article, I was kind of wrestling and trying to figure out this idea of reversibility. Because initially, when I read that, I immediately thought of reproducibility, which is something we've talked a lot about on the podcast. Then had an episode where we talked to one of the founders of Pachyderm about data versioning and such, but reversibility, it seems, it seems very different.

• 34:41Daniel Whitenack?

  So I was kind of interested to, to read a little bit more about that.

• 34:46Chris Benson?

  Yeah, that caught my attention as well. And I also went to reproducibility initially in my head and went, nope, that's, that's not what he's saying. So it was interesting to see, I'm, I'm kind of curious to see, I think of the three concepts that he talks about. That's the one that I probably need to ramp up on a little bit more and understand how it would be utilized here. But at the end of the day, I think the intention is good. It's funny, I have a personal tie a little bit to this interest. And that is, you recently interviewed me on one of our episodes about high-performance computing as it relates to AI. And I can say that in the effort that we did at Lockheed Martin and are still doing, that aspect of really centering on the person that is building the model and on those outcomes that you're trying to achieve,

• 35:30Chris Benson?

  it was really one of the core design principles that we built into our effort. To the point of it was every bit as important to do that for the ability of getting to the outcomes that we needed and, and need as quickly as possible with very high fidelity. And so anyway, when I saw this article, I was, I was pretty, pretty interested. And maybe at some point, maybe Dr. David Bray might come on to the podcast and tell us a little bit more about it. And I think that would be an interesting episode.

• 36:00Daniel Whitenack?

  Yeah, I definitely would be interested in hearing more about some of the details that he goes into. So he also talks about creating data advocates, which I thought was an interesting idea. And also a huge piece of this, which I think is important and was mentioned by one of our guests, Lindsay Zulaga from HireVue, is really putting a lot of effort into mindful monitoring systems to test data sets for biases. So if I remember right, I think Lindsay was talking, and this probably intersects there, where she was talking about, well, it's good if you put in some effort into thinking about your users and thinking about bias when you're training AI models. But a lot of things drift over time and the performance of things drift over time.

• 36:45Daniel Whitenack?

  Your user base might change in different ways. And so really putting in a monitoring system that is actually monitoring your online models to really judge whether you're actually dealing with any sort of bias in the input or if your model is all of a sudden generating predictions that are biased between two different groups or something like that. So I think that's a really important piece of the puzzle. And I'd love to hear more from the author here about the types of monitoring systems that he has in mind. He does even have like a chart for the mindful monitoring system for AI,

• 37:29Daniel Whitenack?

  which I think is kind of interesting. I'd like to hear more about that.

• 37:34Chris Benson?

  Yeah, I would too. Do you, I'm just curious, just from your own personal experience, how do you, when you're thinking about bias in the data that you're working with and however you may choose to monitor it or try to address it ahead of time, how do you approach that? It's a little bit of a tangent, but if there is one thing that every data scientist that works in AI is going to contend with, it's that. I'd love to know how you approach bias and what are some of the processes or exercises you do to try to eliminate bias that might produce a bad outcome for you?

• 38:08Daniel Whitenack?

  Yeah, I think, I mean, right now I'm working on a lot of language and voice tech. And I think people, when they think of bias, they always think of some, some sort of like discrimination against certain groups, maybe like based on race or gender or whatever it is. So I don't think it always has to be that side of things, but of course, we should be aware of that side of things. I mean, it could just be as simple as like, I'm trying to translate this piece of text into Hindi. But in my training data, all I had was news data. I only had politics data. I didn't have any sports data, right? And so it could just be a simple bias in that data set in terms of what it's been exposed to.

• 38:48Daniel Whitenack?

  But also, you know, there's, there's certainly cases where I think marrying that sort of mindset with a mindset geared towards your users can be really powerful. So especially with voice, you want your voice system to work equally well for men and women, right? You want it to work equally well for those that are highly educated and, and not highly educated, maybe. Or those with certain accents and, you know, not certain accents or certain regions, certain dialects or not. And so the only way that you're going to be able to, you know, be able to do well in that scenario

• 39:27Daniel Whitenack?

  is if you, for one, try to have diversity in your training data when you initially put that system together. But also in terms of monitoring, I mean, one thing is you may never be able to anticipate all the types of people that are going to interact with your system, right? So, you know, anticipating that in advance and putting a monitoring system in place where you could actually tell, oh, in these scenarios, our system isn't doing well based on, you know, our feedback we're getting or based on some metric that we're measuring. And if you look further into that, you could identify, you know, certain groups that are using your system that you just didn't expect before. And so now you should circle back and integrate that into your training data.

• 40:09Daniel Whitenack?

  So I think it's, it's a cycle and you have to think about it both in production, in training, and kind of this feedback in between the two.

• 40:18Chris Benson?

  Gotcha. Okay. Thanks for going there.

• 40:20Daniel Whitenack?

  Yeah, definitely. Well, I appreciate you coming up with some, some good articles this week, Chris. Before we head out, like in all of our fully connected episodes, we like to end with just a couple, at least one learning resource that you can use as you're trying to learn more about the most recent trends in AI and kind of level up your skills. So the one that we wanted to highlight this week, which I've just really enjoyed over the past year is called, and we may have even highlighted this before, I'm not sure, but it's worth highlighting again. It's called Papers With Code. So if you just go to paperswithcode.com,

• 41:02Daniel Whitenack?

  it's a site that is essentially what it is named. It's Papers With Code. So it's AI research papers with the links to the code of the implementation on GitHub. And in some cases, a sort of ranking on various tasks. So on the, on the front page, if you go to this site, it'll show trending research. At the very top of the trending research right now is PyTorch Transformers, which is not a, not a huge surprise there since hugging faces is killing it. But you can click on PyTorch Transformers on that paper. It's going to take you right to the GitHub repo of the implementation. But you could also click on the paper, right? And it's going to take you to the, to the paper for that.

• 41:44Daniel Whitenack?

  And you can click on, some of the papers are tagged with little things like, oh, this model or this paper is state of the art in common sense reasoning or something with a certain data set. And so they've also got these pages of what are the most state of the art papers with code for X task? Like, you know, reading comprehension or question answering or, you know, sentiment analysis or these different things. So just an overall really well put together site. And something where, for example, the other day I wanted to know, what are people doing in terms of sign language recognition? And who's doing the best stuff? And I was able to get just a few leads on this site that led me in the right direction.

• 42:29Daniel Whitenack?

  So I find it really useful.

• 42:31Chris Benson?

  Yeah, we have, I do remember that we have highlighted it before, but I also love it just like you do. It is definitely worth highlighting again. And actually, as I'm looking at this right now, the third one is one that I'm going to dive into as soon as we're done recording this podcast, which is "Deep Privacy: A Generative Adversarial Network for Face Anonymization", you know, because we had an episode recently on deep fakes. And so I'm looking forward to diving into that one and see what they have. But thank you for highlighting this site again. It's a fantastic one. And after a little while, we might even need to do it one more time.

• 43:08Daniel Whitenack?

  Yeah, for sure. I think it's a good reminder. And a lot of things have been added over time on the site that have made it really useful. Well, I appreciate you taking time before your vacation to talk through a little bit of AI stuff, Chris.

• 43:21Chris Benson?

  I wouldn't miss it. This is fun stuff to me. As far as I'm concerned, vacation's already started. This is part of it. So yeah, I mean, this is great. I always love our conversations. We always love to get the feedback that our listeners give us. We talk to people on Slack because we have our Slack community. We talk to people on LinkedIn. We talk on Twitter. And a lot of what this show is about is coming from engaged listeners who ask a question or say, "Hey, I would really love to hear more about this." And so I really hope that everyone keeps engaging us on this and let us know what you're interested in. So that it's part of what makes this fun.

• 44:03Daniel Whitenack?

  Yep, definitely do. And have a great vacation, Chris. We'll talk to you soon.

• 44:08Chris Benson?

  Thanks a lot. Take care, Daniel.

• 44:11Speaker 00

  All right. Thank you for tuning into this episode of Practical AI. If you enjoyed the show, do us a favor, go on iTunes, give us a rating. Go in your podcast app and favorite it. If you are on Twitter or a social network, share a link with a friend, whatever you gotta do. Share the show with a friend if you enjoyed it. And bandwidth for change log is provided by Fastly. Learn more at fastly.com. And we catch our airs before users do hear Change Law because of Rollbar. Check them out at rollbar.com/changelog. And we're hosted on Linode Cloud servers. Head to linode.com/changelog. Check them out. Support this show. This episode is hosted by Daniel Whitenack and Chris Benson. The music is by Breakmaster Cylinder. And you can find more shows just like this at changelog.com. When you go there, pop in your email address.

• 44:54Speaker 00

  Get our weekly email keeping you up to date with the news and podcasts for developers in your inbox every single week. Thanks for tuning in. We'll see you next week.

• 45:09Chris Benson?

  You