Democratizing ML for speech

You might know about MLPerf, a benchmark from MLCommons that measures how fast systems can train models to a target quality metric. However, MLCommons is working on so much more! David Kanter joins us in this episode to discuss two new speech datasets that are democratizing machine learning for speech via data scale and language/speaker diversity. Sponsors: Changelog++ https://changelog.com/++ – You love our content and you want to take it to the next level by showing your support. We’ll take you closer to the metal with no ads, extended episodes, outtakes, bonus content, a deep discount in our merch store (soon), and more to come. Let’s do this https://changelog.com/++! The Brave Browser https://brave.com/cha913 – Browse the web up to 8x faster than Chrome and Safari, block ads and trackers by default, and reward your favorite creators with the built-in Basic Attention Token https://basicattentiontoken.org. Download Brave https://brave.com/cha913 for free and give tipping a try right here on changelog.com https://changelog.com. Fastly https://fastly.com/ – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorshipFeaturing: David Kanter – GitHub https://github.com/thekanter, X https://x.com/thekanterChris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Press Release about MLCommons datasets: MLCommons™ Association Unveils Open Datasets and Tools to Drive Democratization of Machine Learning https://mlcommons.org/en/news/neurips21/NeurIPS Papers: People’s Speech Dataset https://openreview.net/forum?id=R8CwidgJ0yTMultilingual Spoken Words Corpus (MSWC) https://openreview.net/forum?id=c20jiJ5K2HGradient article: New Datasets to Democratize Speech Recognition Technology https://thegradient.pub/new-datasets-to-democratize-speech-recognition-technology-2/Blog posts for more insight: People’s Speech https://mlcommons.org/en/news/peoples-speech-blog/Multilingual Spoken Words Corpus (MSWC) https://mlcommons.org/en/news/spoken-words-blog/Downloads: People’s Speech https://mlcommons.org/en/peoples-speech/Multilingual Spoken Words Corpus https://mlcommons.org/en/multilingual-spoken-words/Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  I think of data sets as sort of, we want them to be living data sets. Right. It's sort of like a garden, you know, you gotta prune it. You gotta water it. And so one of the things that I think ML Commons is uniquely positioned to do is to be that organization that does sort of maintain it for the community. We've got engineers who are, their full time work is to help maintain and improve this. So, you know, we are looking at new data sets that again, we think will push the needle forward.

• 0:32Speaker 01

  Big thanks to our partners, Leno, Fastly and Launch Darkly. We love Leno. They keep it fast and simple. Check them out at leno.com/changelog. Our bandwidth is provided by Fastly. Learn more at fastly.com and get your feature flags powered by Launch Darkly. Get a demo at launchdarkly.com.

• 0:49Speaker 02

  Change Log++ is the best way for you to directly support Practical AI. Join today and unlock access to a private feed that makes the ads disappear, gets you closer to the metal, and helps sustain our production of Practical AI into the future. Simply follow the Change Log++ link in your show notes or point your favorite web browser to changelog.com/plus. Once again, that's changelog.com/plus.

• 1:21Speaker 03

  Change Log++ is better.

• 1:32Speaker 02

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and Slack with us around various topics of the show at changelog.com/community and follow us on Twitter. We are at PracticalAIFM.

• 1:58Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist with SIL International, and I'm joined, as always, by my co-host, Chris Benson, who is a tech strategist at Lockheed Martin. How you doing, Chris?

• 2:12Chris Benson?

  I'm doing very well. Happy New Year, as we are in the early winter of 2022 now, recording this.

• 2:19Daniel Whitenack?

  Yes, the very cold, early winter of 2022. Yeah, I think our listeners will probably have heard another episode this year, but this is the first that we're recording in the new year. So, Happy New Year to everyone again. But it's always good, you know, at the new year to revisit certain things or look forward to things that you thought about last year. And it's cool that we have David Cantor with us, who is the Executive Director at ML comments, because we had a great conversation with David last year. It'll be good to catch up on all the amazing stuff that ML comments is doing. So, welcome, David.

• 2:58Speaker 00

  Thank you. It's great to be back. Yeah, Happy New Year to everyone. I also have to point out, it is only cold if you aren't in California.

• 3:07Daniel Whitenack?

  Yes. Well, I certainly am not, at least judging by my walk into the coworking space today. Definitely not California. Although our listeners can't see, but you are wearing like a hoodie, zip-up type of thing, which-

• 3:23Speaker 00

  Oh, no, no, no, not a hoodie. It's a fleece, but it's also-

• 3:26Daniel Whitenack?

  It's a fleece.

• 3:27Speaker 00

  I gotta have my own logo.

• 3:28Daniel Whitenack?

  Right, it's swag, yeah.

• 3:30Chris Benson?

  You're looking good, man.

• 3:31Daniel Whitenack?

  Sure, sure.

• 3:32Speaker 00

  Yeah. I'm in San Francisco, for those who don't know, and like, we should be clear, San Francisco weather is not LA weather.

• 3:39Daniel Whitenack?

  Yeah, that is true. Yeah, I've been there on occasion when that is apparent. So, yeah. Welcome back. I know we had some introduction to ML comments in our last episode, which we'll link in our show notes, but for those maybe that haven't heard that, or maybe want a sort of refresher, could you just give us a little bit of an idea about what ML comments is, why it exists, and what are some of the things that you're doing?

• 4:06Speaker 00

  Yeah, absolutely. So, the mission of ML comments is really making machine learning better for everyone. So, I think of our goal as being, you know, sort of how do we stimulate innovation in ML in a way that, you know, really benefits society and the whole world? So we were started, you know, it was a very informal collaboration starting in 2018. But we actually formed a nonprofit in 2020 that I am, you know, the founder and executive director of. We've got an amazing team and it's an industry consortium. So we're bringing together a lot of heavyweights in the ML world, from the system side, from the software side, from the cloud side, you know, who are all focused on this.

• 4:49Speaker 00

  And we, you know, that mission of making ML better for the whole world, we have sort of three real pillars to that. And one is benchmarks and metrics. And so the first thing that a lot of people know us for is ML perf, which are the industry standard benchmarks for the speed of training neural networks and doing inference. And, you know, actually, one of the things that's been really cool since we've talked is, you know, I think when we first talked, we might've only had two of the benchmarks in our suite up and running. And we're now, what I like to say is covering from micro watts to megawatts. So the smallest systems we've measured performance on are like, you know, maybe tens of micro watts,

• 5:29Speaker 00

  you know, really deeply embedded IoT devices up to the world's number one supercomputer, which is 20 megawatts. And sort of everything in between. And so, you know, you can see what's really cool. And I have some slides that I use in keynote saying like, hey, if you look at the two and a half years we've been around, you would expect that just through Moore's law alone, ML solutions would be about two and a half X faster. And that's great, right? You know, we want to make your life easier, Daniel, when you're doing the work. But if you look at the ML perf data that we have, actually, it's more like 16 to 30 X faster. And so it's really cool to see, you know, when you get benchmarks and when you get metrics and everyone starts really rowing in the right direction, the kind of momentum you can get in the industry.

• 6:14Speaker 00

  So that's one example of ways we can make ML better for everyone is just, you know, being able to train bigger models and do it faster and really bring those capabilities out. Sort of the second pillar is the one that brings me here is our data sets. And you know, I like to think of data sets as really being the raw ingredient for ML. If the industrial revolution was powered by iron and coal, not the most environmentally stuff. You know, we're really powered by data, right? And it's, you know, I think at SIL, right? You guys understand that just as well as everyone else. You know, if you wanna start talking in new languages or working with new texts,

• 6:55Speaker 00

  you can't get from English to Urdu without some new data, right? And bringing large-scale, open, nicely curated data is a huge boon for the industry. Because even at some of the biggest shops in the world, you know, places like Google or Amazon, their researchers wanna use public data because the whole point is that you're sharing techniques that everyone can use and driving the industry forward.

• 7:22Daniel Whitenack?

  Yeah, that's really interesting. Sometimes I get the question when I'm working on various data sets, like what the benefit? And like you said, we deal with this at SIL too. Like, we have a bunch of data in our archive or wherever that linguists have gathered, like over a very long time. And sometimes the question comes up, well, like, first off, why would anyone in the industry be interested in sort of like this strange data? And we can answer that. But then secondly, like, why would we want to make this open? Like, how would that benefit like our organization to work on open data sets versus like closed and proprietary stuff? When you're having those conversations, how do you express that to people?

• 8:05Daniel Whitenack?

  Like, the, maybe the business sort of case for contributing to open data sets? Because you do engage with large organizations on the data sets you work with. So how is that phrased to their sort of leadership?

• 8:19Speaker 00

  Yeah, so I think it depends on the companies, right? So there's some companies where their data really is a critical differentiator, and they probably don't ever wanna open it. But even in the case where that is true, so like I'll give you an example. So there's a company, Criteo, in Europe that actually was really fantastically cooperative and opened up one of their data sets for our use. It was an older data set that they weren't super concerned about a competitor having access to, right? And they said like, look, if you use this data set for your benchmarks, systems are going to get faster for what we want to do.

• 9:00Speaker 00

  So this is a way that you can kind of, in some ways, punch above your weight in getting the attention of the industry. Right? And I'm sure you've talked to folks about how, you know, when you start training on a data set, the models begin to specialize a bit for that. And so, you know, if you want your use case to be popular, making it open is a really good one. And then a lot of our member companies, you know, if you look at someone like Intel or Nvidia, more open data means more ML, means more people doing cool things with computers means more sales. So, you know, in a lot of cases, I think you hit the nail on the head, which is, as a nonprofit and industry consortium, I can't force people to do what they don't want to do.

• 9:42Speaker 00

  I can provide persuasion and I can try to help them understand where our interests are aligned. And I think there are a lot of ways where that's great. Or, you know, Chris, you work in the defense and intelligence community. So I had a really cool meeting late last year with some folks there. And one of the things they were saying that they love about open data is certain parts of that community have real difficulty spending money through normal channels.

• 10:08Chris Benson?

  Indeed.

• 10:09Speaker 00

  And one of the interesting things, specifically about speech, is by and large, the intelligence community tends to care about languages that are not very commercially viable for like regular products, right? There's not as much interest. Let's rewind to 10 years ago. If you asked like, you know, the CIA or whoever, like, what language would you love to have? Like an auto magical translator? They'd be like, oh, Arabic for sure, right? And they wouldn't have said English or Mandarin, right? Which are, you know, probably the two most popular speech data set flavors out there, right?

• 10:42Chris Benson?

  Sure.

• 10:42Speaker 00

  And so, yeah, I think just open data is hugely powerful in enabling researchers and everyone to sort of work together on these problems.

• 10:53Chris Benson?

  Do you think there's a cultural aspect to it in terms of people getting used to this idea? Because when you position it like that, you can clearly see the business case for it. I know in my experience, it seems like some years back, we went through this with open source software and people struggled to see that business case and understand. And now it has swept almost all aspects of business. Do you envision that use case benefit in terms of being able to influence how things go and participate that in a larger scale that that is going to be more widely adopted over the years ahead?

• 11:25Speaker 00

  Yeah, I think so. And I mean, I think one of the other things that I'd say is, you know, when you look at open data, in a lot of cases, even where you might be differentiating via your own data, like in a lot of cases, it's going to be additive, right? So let's say you want to do a medical transcription thing. You might not want to donate to the wide world, like all of your work on translating and decoding specialized medical terms like tachycardia, right? But if you were going to do, train like a natural language model, you know, some version of say BERT, right? You might start with BERT on a large public data set like Wikipedia or a crawl of the web.

• 12:08Chris Benson?

  Yep.

• 12:09Speaker 00

  And then you'd say, well, you know, I'm going to add on my special sauce later to fine tune or supplement or augment that. And so one of the ways I think about this is that getting from zero to product is a really big push. And you need the data to get there, but it's not the only thing you need, right? There's also the whole process of how do I productize it? How do I test it? How do I make sure it's not going to do, you know, wild and crazy things that I don't expect? And if I, as an organization, can provide some open data and get the whole world a few steps down that path and sort of accelerate things, that's just good for everyone.

• 12:46Chris Benson?

  I'm curious, I got a follow up for you on that. And that is, I know that there are people listening to this because it's in my head as well that are thinking my organization needs to do better on that. And as we talk about the decision point about how do I make the business case? And, and what should go open? What should we keep as our secret sauce and all that? Any guidelines on how to make such an evaluation? I realize that use cases vary a lot, but I have had conversations with people where they either want to just throw everything open or they want to do everything as their special sauce, but there doesn't seem to be a, a method to the madness on what, what constitutes the two. How would an organization do that? Recognize that by going open, they selfishly, they get the benefit of being able to steer

• 13:31Chris Benson?

  things, but they still get to keep their secret sauce? Any thoughts about how to make such an evaluation?

• 13:37Speaker 00

  Yeah. So the other thing I was actually going to mention is, you know, one of the things that our organization is very dedicated to is the maintenance and upkeep of these. And, you know, so if you think of ImageNet, right? That really started this round of innovation in machine learning, right? And then it was ImageNet and then the realization that, hey, AlexNet plus GPUs plus ImageNet beats humans at image recognition. Oh my gosh, right? And, and now ImageNet is fantastic. It's amazing, but you know, now it's old. It hasn't been updated. There's some legal issues associated with the licensing and, you know, so imagine we're doing speech to text and we did that in 2019. That model, that data set would not have anything about COVID or coronavirus or any

• 14:22Speaker 00

  of these other things. So, you know, you want to keep these things up to date and that, that does cost some amount of money. And so by offloading that to the community or, you know, ML Commons, we're curating our stuff. It's part of where our budget goes. That can be very helpful in terms of combining community resources. And so that's another aspect of the business case. But to answer your question, I think it is very easy to slide into these binaries, right? It's just, it's a very natural way of thinking. And where we've had a lot of success, I think is

• 14:54Speaker 00

  Usually where there is some sort of research, folks involved or, you know, some sort of specialists who can think about this and reason about it. These kind of endeavors inevitably involve your legal department.

• 15:07Chris Benson?

  That's a great point.

• 15:08Speaker 00

  And we don't want to get folks in trouble. And so there's all sorts of issues about how was it collected? Do you have the right permissions, GDPR? And a lawyer is probably not going to be able to give you the insights as to what is commercially valuable, what is commercially threatening? Could it be monetized in other ways? Or no, is this just the thing that makes sense? So you kind of do need some element of the business and potentially technical community. And so we've found that that is oftentimes a very helpful driver. One of the things that is beautiful about the machine learning community is I think there's a openness is in our DNA. And so I think for a lot of researchers, that is sort of a default mode of thinking, and that is helpful.

• 16:11Speaker 02

  We deserve a better internet and the brave team has the recipe for bringing it to us. Start with Google Chrome, keep the extensions, the dev tools and the rendering engine that make Chrome great. Rip out the Google bits, we don't need them. Mix in ad and tracker blocking by default, quick access to the Tor network for true private browsing, and an opt-in reward system so you can get paid to view privacy respecting ads. Then turn around and use those rewards to support your favorite web creators like us. Download Brave today using the link in the show notes and give tipping a try on changelog.com.

• 16:58Daniel Whitenack?

  So David, as ML comments was thinking about this sort of pillar of data sets. And obviously, there's a lot of different data sets that could be valuable to the ML community and all sorts of data, whether that be one type of data or multimodal data or like all sorts of things. How did ML comments sort of settle on an initial maybe focus on speech data? Because I know we'll be kind of transitioning to talk about a couple of the very impressive data sets that were recently released, but they're both speech data sets or speech-related data sets. So how did that come about?

• 17:39Speaker 00

  Yeah, so when I think about the data set problems that I want my organization to or the organization I serve to tackle, I sort of think about it as two sides of the equation, return and investment. And by by return, I really mean what's the impact, right? And so I want to be working on problems that'll have an impact where we're going to provide some very clearly differentiated, very valuable things that the community doesn't have. If all you do is produce something that already exists with a slightly friendlier license, that can have a big impact. But you know, it's not as great as, you know, I'm now one of our data sets, right? Is the first in many, many languages, right? That's like a qualitative change in the landscape that's like really exciting.

• 18:24Speaker 00

  And then the other side of that is, do we have the right people to do it? And what is the investment that's involved there? And can we bring that skill set together? And are we uniquely positioned to do that? And for us, speech was very much ground zero for a couple of reasons. So one is, we actually want to build up the infrastructure around building these data sets, because part of the theory behind our organization is that actually a lot of the plumbing behind the scenes, there's some commonality to it. And sharing that can be helpful. I mean, obviously, there's greater alignment between two speech data sets than, you know, speech and vision, but still, there's a lot of just general wrangling you got to do.

• 19:04Speaker 00

  And also, some of the leads on our project had prior experience building speech data sets. And there was a very real desire to have, and all of our data sets are permissively licensed for both research and commercial use. And there's a real desire to build something big enough to train an end-to-end speech model on, and then begin to tackle diversity. And so those things all kind of lined up for us in that there was sort of a gap in what was out there, and we had the right expertise, and we thought that the impact would be big.

• 19:36Daniel Whitenack?

  Yeah, so maybe you could just describe, generally, there's two large data sets that you've released recently. You know, we'll link, of course, to the specific information about those, and people can obtain them and download them and start working with them, but maybe just give us a picture of each of those and what the goal was with each one.

• 19:58Speaker 00

  Yeah, so the first one is called The People's Speech. And I think I talked with you guys about this last year. And that is 30,000 hours of labeled speech data, and it's sort of intended for speech recognition purposes, but you can use it for a lot of other things. We've had people who downloaded it to do like denoising filters for acoustical applications. And it's, like I said, permissively licensed conversational speech, so it's not just read audiobooks, as was sort of common before. And it's very big. It's 30,000 hours, it's about, I think, 2 to 3 terabytes.

• 20:39Speaker 00

  And part of the point here is we've got, you know, when I think about speech, I sort of think about three dimensions to the data sets. One is size, one is the languages, and then the other is sort of the context. And like, do you have noisy speech, or is it, you know, something really clearly recorded? Are there kids playing in the background and other things like that? And so this is putting the stake in the ground on the size, and, you know, we can work on the other things. And like I said, it's big enough to train an end-to-end model, so kind of over the 10,000 to 15,000 hour tipping point. So that is sort of going deep and big in the size dimension. And then the second one...

• 21:19Speaker 00

  Which is nicely complementary is the multilingual spoken words corpus. And this is exciting because this is really pushing the boundaries on the diversity angle from the number of languages. And so that it's not for speech recognition, but for keyword spotting. Right, so recognizing keywords, a modest number of keywords. And it is 23 million clips that are each about one second long, covering 340,000 keywords with 115,000 different source speakers in 50 languages. And the thing to me that's really cool about that is those 50 languages cover 5 billion human speakers.

• 22:02Speaker 00

  Right, so this is the majority of the world's population. And most of these languages, this is the first existing data set for keyword spotting in those languages, and certainly the first under such a nice permissive license. And, you know, we're talking about languages that are not supported by these home assistants. Right, if you, you know, I don't want to pick on any one company, but if you look at a lot of the home assistants or things like that, they might support a dozen languages. But there's a lot of languages where there's not a lot of support. And so, you know, bringing that capability to these new communities is really exciting. Like, it was very cool to see some of the people tweeting about, hey, my language is in here, or getting emails about, how can I add my language?

• 22:45Speaker 00

  So that's pushing forward in a different dimension. But again, also speech.

• 22:49Chris Benson?

  So I'm kind of curious, you know, I've worked with data sets from whatever, you know, the task that I'm focused on, but I've never done anything at the scale of having to put the data together. What's involved in that? You know, if you're looking at, you have the keyword spotting and you have the speech recognition use cases, and you have massive amounts of content that you're doing, I would imagine that there is a level of organization and effort required way beyond what most of us that are just doing typical day-to-day machine learning models are having to deal with. How do you even start that? How do you approach such an engagement?

• 23:25Speaker 00

  Hmm, that's a really good question. So our team is pretty small in terms of the number of engineers. So I'm not sure that, you know, resource wise, it's not like, you know, we're coming in with 10 people or whatever, full-time or anything like that. You know, I would say, I think the thing that we really wanted to focus on was because we knew, as you said, this is like a big problem. I think we had to focus on things that scaled initially. And, you know, a really good example of that is we did a back of the envelope calculation of like, "Okay, what would it take to label the data manually?" And, you know, I think we calculated it would be on the order of about $10 million.

• 24:07Daniel Whitenack?

  That's a bit of a budget right there.

• 24:09Speaker 00

  Right, yes, that's several years of budget for my nonprofit. And we've got to keep the lights on, we've got to pay the employees, you know. So that was out of the question. And so we actually, I think, one of the things we really wanted to focus on was building the tools and leaning into compute rather than manual labor. And so, you know, we were able to label the data and generate the labels using computer systems for under about $20,000. So I think it's sort of attention to that and building up the tools. And all of the tools are open source, by the way. You know, not only can you go and get the data set, but you can get the tools we use to create it.

• 24:51Speaker 00

  And, you know, we'd love to see people updating them, fixing bugs, whatever it is, and see community adoption of all of those pieces. But, you know, it's also, again, yeah, I think it does come down to the right tools, right? If you're building one house, you want one set of tools. If you are building a 50-story apartment building or a dozen houses, you're gonna, you know, have slightly different infrastructure.

• 25:14Daniel Whitenack?

  Yeah, I would be kind of curious to maybe dig a little bit into that because when you're creating a labeled data set, people might think, "Oh, I want gold standard data," right, which necessarily in their mind means human labeled, right? Without a sort of machine in the loop of creating the gold standard data. So maybe describe a little bit more like why this sort of machine process can create something that's useful to train a machine on, I guess, would be the question.

• 25:46Speaker 00

  Right, or put another way, okay, so the snake is eating its own tail. Does that actually work?

• 25:51Daniel Whitenack?

  Right.

• 25:52Speaker 00

  Right, or are we left and we're like, "There's no snake anymore." So first of all, I think the technical team had some great instincts here. And, you know, one of the big hypotheses is, as you say, like, a lot of people will focus on extremely high-quality data. But it's also pretty common that if you... Have some bad data in there, it may not really be a huge problem. And so I think part of this is accepting that a large amount of modest quality data. And actually, we did, we can talk about some of the details because we did, in generating the labels, we actually did figure out or try to estimate how many of them were good. And like a big chunk of it was done, you know, sort of perfectly.

• 26:35Speaker 00

  Some of it was done to sort of human quality, some of it was worse. And, you know, I think there was a bit of a gamble there that that would ultimately prove to be useful. But we do know that machine learning is very good at handling kind of rough inputs. And so it seemed like a pretty good hypothesis there. You know, it's also the, the ways that a machine in the loop for labeling go wrong are going to be probably a lot different than a human.

• 27:06Chris Benson?

  I'm, I'm just kind of curious. I'm sure Daniel already knows this, but as you, as you're putting these data sets together, is it kind of one shot and you're done and it's there? Or is there a management of that data set over time? Do you go back and, and relabel with new tools or anything? Or is that, or do you kind of move on to a whole new data set? How do you think about producing things?

• 27:27Speaker 00

  I guarantee you, like any engineer, we tried a few things and the first things did not work. I'll give you an example. I'm, I know we evaluated several different forced aligners. So we started for people's speech by scraping data where there was a transcript and the audio, but they weren't necessarily temporally aligned, which you need to train a speech to text model. So we did our own transcription using a CALDE with an N-gram language model. And so we came up with our estimated transcription. And then we reinforced alignment on that to get the timestamps right.

• 28:13Speaker 00

  And all of this audio came with subtitles. But just as an example, sometimes the subtitles are a little wonky, right? Like, you might get a subtitle, maybe the language spoken is French and the subtitle is in English. Or maybe it's a description of a picture or something like that. So you can get all sorts of really interesting problems. And so we tried different aligners. And we did find one that we thought, you know, solved some of these problems, didn't solve all of them. But we definitely tried out a bunch of different things. And to us, one of the proof in the pudding sort of things was actually training a speech to text model on our dataset and seeing how comparable it was to using something like Libra Speech, which is much smaller, but is sort of the gold standard today.

• 28:56Speaker 00

  And, you know, we got in the ballpark of Libra Speech, which we're like, that's good enough. But I will tell you, we did not start out in the ballpark of Libra Speech. There's, you know, all sorts of problems, you know, you have like mismatched transcripts where things are just, you know. But you find the problems and you hammer them down, one by one. That's my experience. I don't know. Daniel, what about you?

• 29:15Daniel Whitenack?

  Yeah, I mean, this is a sort of never, never ending process, right? And I think, like you were talking about ImageNet and others, you know, this is only, like, things get stale and need to be updated. And language especially is like always changing very rapidly. And like, like you say, if something was released like before COVID and you need like COVID health words and stuff after COVID, it needs to be updated. As part of your release, do you have a mechanism for people to like make sort of contributions or feedback or sort of a human evaluation type of feedback or anything like that?

• 29:58Speaker 00

  Yeah, so, you know, again, we're big on community. We have a, an open Discord channel that you can drop in and, and tell us what's great, tell us what doesn't work. When we released it, as you might expect, the initial sets of comments were, "This is really amazing, but we, you know, we found this hitch." So we're nailing those down and we've got, you know, a Google group and mailing list for this. And so, and again, all the code is open source. So, you know, if folks want to file bugs and, I mean, the thing to me that is most exciting is now that it's released, like a year from now, what are people doing with it? Like, to me, that's really the sort of thing that's very cool. One of the other things I would actually mention, just briefly, you said, "An example of a scale thing that we had to work on is when we were initially doing the forced alignment,

• 30:43Speaker 00

  one of the things we found is that, you know, just using out of the box software was not sufficiently fast. And so we actually had to go through and optimize our forced aligner, both the acoustic model and the language model, and get them running on an accelerator, on a GPU. And at that point, we were able to get 250x real time. And so that's an example of paying attention to scaling and sort of systems where it's like, yeah, 30,000 hours is a lot. You don't want to be renting an Amazon instance for 30,000 hours because A, we got to release it. But okay, you cut that down by a factor of 250. It's like, okay, we can rent a few of those. Right? So there's all sorts of things that go into this.

• 31:41Speaker 02

  Changelog News is the best way to keep up with the fast-moving software world. We track, blog, and contextualize the coolest projects, the best practices, and the biggest stories each and every week. Make changelog.com your daily destination, or hit the snooze button and subscribe to our weekly newsletter that hits inboxes on Sunday mornings. Join more than 15,000 enthusiastic readers. It'll cost you exactly $0.00 and you can subscribe right now at changelog.com/weekly.

• 32:36Daniel Whitenack?

  So being with S I L, I think I would probably get fired if I didn't ask a little bit about the particularly the data set with the spoken words and your sort of attempt to, like you say, bring in a bit of diversity into speech data sets that were out there. Kind of as you look back on that, how does this data set stack up against maybe like, what are other examples of data sets out there for spoken words? And how does this change the picture in terms of the diversity of languages?

• 33:11Speaker 00

  No, that's a great question and that hits on sort of the things that I'm most proud of in some ways. So, you know, I think the gold standard is Google Speech Commands. And I am not as much of an expert on the keyword spotting area as the speech to text. But my understanding is that's a data set with about 105,000 one-second utterances covering 35 words in English. And it's great, you know, again, fantastic community resource. You know, and if you go back to what I said, we've got 340,000 keywords in 50 languages with 23 million clips. So there's 50 different languages we're covering and you know, some

• 33:52Speaker 00

  of them have very good coverage. We sort of bucketed it into three categories, sort of low resource languages, which is under 10 hours of data. So that'd be something like Georgian, Tamil, Vietnamese, Arabic. And that's about half of the languages. And then we have medium resource languages with between 10 and 100 hours. And that's 12 languages. And so some examples of that are like Czech, Ukrainian, which is actually, that's the first Ukrainian data set. Of its kind for sure. Turkish, Portuguese, Indonesian. And then we've got high resource languages, which is over a hundred hours. And that includes like, actually some reasonably obscure languages like Basque, Catalan, Persian,

• 34:37Speaker 00

  and then of course, some standard ones like English, Welsh, et cetera. And so I, you know, to me, the really exciting thing is that first of all, it's a lot of languages. And also, some of the work in this paper was about, okay, when we say low resource, does that mean you can't use it? Or, you know, what does that really mean? And actually, what we found is that you can use a lot of the low resource languages, I think, for like, few shot training and fine tuning examples, right? And so that is really powerful in terms of bringing out new capabilities. And, uh, you know, I think the blog post I wrote was sort of cheekily like giving a voice to five billion people, right?

• 35:20Daniel Whitenack?

  I think especially with speech, you know, text is a slightly different scenario, but especially with speech, you actually, like, we always talk at SIL about the sort of long tail of languages, which is you've sort of got the top 100 and then you drop off sort of very quickly in terms of resources that are out there, all the way out to 7,000 something languages. So you actually don't have to go very far. As, as soon as you get past like the first 10 or so top languages in the world with speech data, it drops off like very rapidly. And so, yeah, it's really cool to see this effort that pushes that out there. And that's really encouraging and my brain's already going with ideas of, of how to use this.

• 36:05Daniel Whitenack?

  I think it's also interesting, you know, looking at it, how you've, I mean, it's one thing to release data, but also it's very useful, at least in my view, to provide additional metadata and annotations along with those words. Like I see you have parts of speech and semantic categorization, which is all really interesting and, and something that I think will be one of those things that drives maybe surprising uses of the data set. So, yeah, I'm not sure when that came up in your planning, but I, I'm glad you went for that extra kind of metadata information.

• 36:41Speaker 00

  Yeah. This is part of Sort of the curation process and how can you improve things over time and make it more useful? And part of it is, you know, we talked to the member organizations along the way as we were doing this to get some feedback on what would be useful. And the other thing I should mention is actually the multilingual spoken words corpus is actually a data set generator. So you can sort of dial in your own keywords, right? And I think because of that capability, it's useful to know like what parts of speech are there and what are sort of the content types and the topics and domains. So to me, it's part of making it low friction and usable. And also just understanding really what's

• 37:23Speaker 00

  going on. And I think that kind of analysis is super important for large scale data sets. Like to jump around a little bit and talk about the people's speech. One of the things we did is we sort of randomly sampled about 5,000 hours to find out what kind of background noise we had. Because, you know, if you're going to train a model, you want to know. And, you know, we found lots of music in the background, conversation, basketball bounces, all sorts of things. And ultimately, as a data scientist, just getting this is really good. And I should mention for the multilingual spoken words corpus, it's built on top of Common Voice. So what we have is this really cool pipeline that can pull

• 38:03Speaker 00

  in a data set. And what's cool about Common Voice is it's ordinary people. So anything that goes into Common Voice will eventually get incorporated in into ours. And there's going to be good background noise and different kind of recording circumstances. But I think it really helps to the extent that we can to try to characterize that.

• 38:24Chris Benson?

  Kind of in the spirit of if you build it, they will come. What kind of member organizations are you kind of pulling in now that you have this, these new data sets out there? And as you're attracting new participants, how are you organizing them? And how are you able to put them all together in such a way that it kind of continues the development forward in a cohesive way? How do you manage that process?

• 38:46Speaker 00

  Oh, that's a good one. Well, so we released these data sets in the middle of December. So it hasn't been that much time. So I don't know if we've gotten any new members to join yet. But I, you know, it's certainly part of my goals for 2022 is getting new members. And getting folks who are excited about contributing and using and look, there's a lot of ways that folks can contribute. If you're using the data set and you give us feedback, that's super helpful. I mean, obviously, we have to get funding from somewhere, and so we do want new members. I would say that as an organization, a lot of our folks are very benchmark centric and focused on MLPerf. But you know, I'm looking at, as we evolve

• 39:25Speaker 00

  the organization, how can we allow for participation from folks who are much more data centric? And the motivation there is a little bit different, right? It may be that it's not, oh, you know, pay us to join this consortium. It may be, do you think this is a great effort? Like, cut us a check, right? Or working with, you know, there are a lot of governments and other organizations, Daniel, you're at one, that are very invested in speech diversity. And they might, you know, maybe consider doing grants or something like that. So this is something we're just starting to explore. I mean, yeah.

• 39:59Daniel Whitenack?

  Yeah, I'm, uh, my mind is already like thinking of follow-up conversations that I, that I need to have with you, David, but uh...

• 40:08Speaker 00

  That's great.

• 40:10Daniel Whitenack?

  Yeah, I hope others and our listeners will be as well. So as you kind of move forward this new year, you know, since this is sort of a first recording for the new year, what is in the kind of future roadmap for ML Commons, both in terms of the data sets, but maybe other things too, that you're excited about diving into?

• 40:31Speaker 00

  So the first thing is specifically on these data sets, I think of data sets as sort of, we want them to be living data sets, right? It's sort of like a garden, you know, you gotta prune it, you gotta water it. And so, one of the things that I think ML Commons is uniquely positioned to do is to be that organization that does sort of maintain it for the community. And so, we've got engineers who are, their full-time work is to help maintain and improve this. So, you know, that's sort of incremental, get better, you know, maybe add a little bit more data. But, you know, we are looking at new data sets that, again, we think will push the needle forward. There's some, a vision data set that we're sort of looking at that should have, you know, some really

• 41:15Speaker 00

  new, nice, sort of diverse aspects to it, compared to what's currently out there. I think that on the data set side is a big thing for us. And then we've got some other projects that actually, we sort of announced at the same time. You know, there's sort of the data centric AI movement, right? Which is sort of at a high level saying like, look, we have a lot of competitions where people are showing off their models, right? You know, I saw on Twitter, Yann Le Cun was talking about sort of, do we want to use transformers or convolution based networks for vision, right? And with convolutions having been sort of the gold standard for a while, but there's a lot of really interesting research focused on using transformers in that area. That's very model centric. And one of the things that we want to do is focus on data centric AI because we think

• 42:00Speaker 00

  that is really powerful. And so we're, we have an initiative called DataPerf. And sort of the idea is to say, hey, let's run a competition where instead of showing off the coolest model, you show off things like data augmentations or different splits that can help get better accuracy, faster time to train, things like that. And so I'd also love to see folks doing cool things with multilingual spoken words or people's speech and stuff that we've never even thought about. So I think that's some of the stuff ahead that really excites me. And we may also, um, you know, I should mention, one of the motivators around people's speech, it's not the size of a data set that someone like Google or Amazon's going to train on,

• 42:45Speaker 00

  but it's definitely moving in the right direction. And so having that for our benchmarking at MLPerf is actually very nicely synergistic. So, you know, maybe we'll start building some bigger benchmarks with that, especially because one of the things we wanted to focus on is making these data sets very legally easy to use, right? They're all CC BY. Tell the world we're awesome because we built the data set and do whatever you want, right?

• 43:08Chris Benson?

  Well, David, thank you very much for coming back onto the show a second time. These are a little bit special for me because I get to sit here not only with you, but with Daniel also and the expertise that he has in language. I kind of feel like a kid in a candy shop every time I talk to you on this and I learn a lot. So, um, really, really, really cool work that you guys are doing after talking in that the previous episode mostly about MLPerf and now getting to, uh, to learn more about data sets and everything. So thank you very much for your time and for coming on the show.

• 43:38Daniel Whitenack?

  Looking forward to next year's conversation.

• 43:40Chris Benson?

  Absolutely.

• 43:41Speaker 00

  Absolutely. No, that'd be great.

• 43:43Daniel Whitenack?

  Yeah. All right. We'll see you, David.

• 43:48Speaker 02

  All right, that's Practical AI for this week. Thanks for listening. If this is your first time with us, subscribe now at practicalai.fm or simply search for Practical AI in your favorite podcast app. We're in there. And if you're a longtime listener, do us a solid by recommending the show to a friend. Word of mouth is still the number one way people find new podcasts they love. Special thanks to our partners for supporting our work, Fastly, Launch Darkly, and Linode. We appreciate it. And to the mysterious Breakmaster Cylinder for cranking out new beats for us all the time. That's all for now. We'll talk to you again next week.