Speech tech and Common Voice at Mozilla

Many people are excited about creating usable speech technology. However, most of the audio data used by large companies isn’t available to the majority of people, and that data is often biased in terms of language, accent, and gender. Jenny, Josh, and Remy from Mozilla join us to discuss how Mozilla is building an open-source voice database that anyone can use to make innovative apps for devices and the web (Common Voice). They also discuss efforts through Mozilla fellowship program to develop speech tech for African languages and understand bias in data sets. Sponsors: Linode https://linode.com/changelog – Our cloud of choice and the home of Changelog.com. Deploy a fast, efficient, native SSD cloud server for only $5/month. Get 4 months free using the code changelog2019 OR changelog2020. To learn more and get started head to linode.com/changelog https://linode.com/changelog. Pace.dev https://pace.dev – Minimalist web based management tool for your teams. Async by default communication and simplistic task management gives you everything you need to build your next thing. Brought to you by Go Time panelist Mat Ryer. Try it out today! https://pace.devFastly https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship – Our bandwidth partner. Fastly powers fast, secure, and scalable digital experiences. Move beyond your content delivery network to their powerful edge cloud platform. Learn more at fastly.com https://www.fastly.com/?utm_source=changelog&utm_medium=podcast&utm_campaign=changelog-sponsorship. Rollbar https://rollbar.com/changelog – We move fast and fix things because of Rollbar. Resolve errors in minutes. Deploy with confidence. Learn more at rollbar.com/changelog https://rollbar.com/changelog. Featuring: Jenny Zhang – Website https://www.phirephoenix.com, X https://x.com/phirephoenixRemy Muhire – GitHub https://github.com/kenessajr, X https://x.com/kenessajrJosh Meyer – GitHub https://github.com/jrmeyer, X https://x.com/_josh_meyer_Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Mozilla Common Voice https://commonvoice.mozilla.org/enAnnouncement of Josh and Remy’s fellowship work on speech tech for African languages https://foundation.mozilla.org/en/blog/mozilla-welcomes-two-new-fellows-voice-technology/Artie Bias Corpus https://www.artie.com/blog/the-artie-bias-corpusReadings on Demographic Bias in ASR: Voice recognition still has significant race and gender biases https://hbr.org/2019/05/voice-recognition-still-has-significant-race-and-gender-biasesGender and Dialect Bias in YouTube’s Automatic Captions http://www.ethicsinnlp.org/workshop/pdf/EthNLP06.pdfRacial disparities in automated speech recognition https://www.pnas.org/content/117/14/7684Common Voice LREC Paper https://arxiv.org/pdf/1912.06670.pdfCommon Voice + DeepSpeech collaborators for Low-resource languages: Digital Umuganda https://digitalumuganda.com/AI Lab, Makerere University http://www.air.ug/Language Technologies Unit, Bangor University https://www.bangor.ac.uk/canolfanbedwyr/technolegau_iaith.php.enLinguistics Department, Indiana University Bloomington https://www.chinlanguages.org/community-resources“under-sampled majority” is a quote from Joy Boulamwini (see this article https://www.bostonmagazine.com/news/2018/02/23/artificial-intelligence-race-dark-skin-bias/)Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  The definition that we've come up with is if you have a speech to text model that performs statistically worse for a certain demographic group, then that model is said to be biased. So using this corpus, the R D biased corpus, you can take any off the shelf speech recognition model and look at what is its accuracy or word error rate, as we call it, for a certain demographic group. Thanks to Jenny and everybody at Common Voice, we have information from people who opted in to self identify their gender, their accent, and their age. So along any of those lines, you can look at how well your model performs on every group

• 0:45Speaker 00

  versus every other group.

• 0:48Speaker 01

  Being with your change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at Change Law because of Rollbar. Check them out at rollbar.com and we're hosted on Linode cloud servers. Head to linode.com/changelog.

• 1:06Speaker 02

  Linode is our cloud server of choice. Grab the Nano plan for just $5 a month, just five bucks. That gets you a gig of RAM, a blazing fast 25 gig SSD, and one terabyte of transfer. Let's be honest, you can go a long ways on that five bucks. When you do need to scale up, their prices are predictable, so you can put your calculator down, you won't need it. We've been running change.com on Linode for years and we've always impressed by their award winning support team. Check them out at linode.com/changelog. Once again, that's linode.com/changelog.

• 1:47Speaker 02

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and slacked with us around various topics of the show at change.com/community and follow us on Twitter, we're at Practical AI FM.

• 2:13Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I am a data scientist with SIL International. And I've got Chris Benson with me, who is a Principal Emerging Technology Strategist at Lockheed Martin. How are you doing, Chris?

• 2:30Chris Benson?

  I'm doing very well today, Daniel. How's it going?

• 2:32Daniel Whitenack?

  It's going really good. This last week was a fun week. I've been writing a lot of inferencing API code, which has been fun. It's been one of those weeks that I've got to be, you know, heads down in code a lot. And I always enjoy that bit of it. It's like getting the various pieces plumb together and working, and then you have your models together and you actually see it work in a practical way. That's always fun for me. So what about yourself?

• 3:00Chris Benson?

  Last week was pretty interesting. I'll relay really quickly, kind of what for me was a work-related thing, but not in a direct way. So DARPA, the Defense Advanced Research Projects Agency, had a competition, and they had a bunch of autonomy-focused companies, including my, just as a, we were there too, Lockheed Martin, that were competing in what was called Alpha Dogfight. And so they had gotten, at John Hopkins University, they had gotten an F-16 flight simulator. And they had all the teams over the last few months getting their models of their F-16s ready to go into dogfighting mode. And so they had the competition last week, and they livestreamed the entire thing on

• 3:42Chris Benson?

  YouTube. So it was pretty cool to watch. And so in the end, a company called Heron Systems won out overall. Our own team came in second out of eight teams. But it was interesting is that Heron actually went up against a, if it was the Navy, it would be a Top Gun instructor. It was the Air Force. The call sign was "Banger." And within a set of constraints, it was interesting, the AI consistently beat the human pilot instructor.

• 4:11Daniel Whitenack?

  Interesting.

• 4:12Chris Benson?

  Yeah, it was, nobody was expecting that. So the AI models did very, very well, much better than anyone realized.

• 4:19Daniel Whitenack?

  When you first said that, I was, I was thinking of like, Battle bots, Boston Dynamics, like dogs going against you.

• 4:26Chris Benson?

  Yeah.

• 4:26Daniel Whitenack?

  When you said dog fighting, I guess I'm not around, you know, aeronautics very much, and that seems way more violent.

• 4:33Chris Benson?

  Yeah, no, airplanes in the air is still quite violent in theory, but yes.

• 4:37Daniel Whitenack?

  I guess that's true.

• 4:38Chris Benson?

  Yeah, but not adorable furry little creatures, definitely. So, but anyway, it was pretty fascinating to watch. It was a moment where people went, hmm, things are definitely changing right now. So that was my week.

• 4:50Daniel Whitenack?

  Interesting. That brings up all sorts of interesting questions and ethical things and all of that that I'm sure we'll get into as the podcast goes on as well.

• 5:00Chris Benson?

  I'm sure we will.

• 5:01Daniel Whitenack?

  But yeah, I'm, I'm pretty excited about today's episode. As some of our listeners will know, I think I've mentioned a couple of times that I've been getting into speech technology a little bit more for my own work in recent months. And one of the things, like if you're going to train a speech recognition model and you're looking at what, what speech data is out there. One of the things that pops up fairly, you know, prominently in that is Mozilla's Common Voice project. And Mozilla is doing some really cool things also through their fellowship program with speech technology. And so today we've got Remy Muhire, Jenny Zhang, and Josh Meyer with us, who are going to talk a little bit about Common Voice

• 5:43Daniel Whitenack?

  and other things going on through the Mozilla Fellowship program. So welcome, everyone. It's really great to have you here.

• 5:50Speaker 05

  Thanks, Dan.

• 5:50Speaker 00

  Thanks. Nice to be here.

• 5:52Daniel Whitenack?

  Yeah. So before we jump into Common Voice and speech technology and all the wonderful things that you're doing, maybe we could start out by just having a brief discussion about your respective backgrounds. Remy, do you want to start us out and tell us a little bit about your background and how you got involved in speech technology?

• 6:12Speaker 06

  Thank you, Daniel. Yeah, so I think my background started as a software engineer. So I still actually write codes, but not often. And also contribute on some of open source library in Africa as well with a couple of friends. Yeah, I would say I spend actually. Six years, this is by years actually writing codes. And and the journey actually started in two thousand thirteen, learning about C, Python, and JavaScript. And been working in actually a couple of startups uh in Kigali, especially startup in the tech technologies, yeah.

• 6:51Speaker 06

  And before that, I actually get into voice technology was a project tried to actually do with a friend, so which were about actually uh voice recognition. So how like enough would actually help people around uh speaking actually detective language, actually uh seeking for information, yeah. But unfortunately, we couldn't actually move the project since uh there weren't actually any uh Kinyaranda data set. And Kinyaranda is uh one of uh the local languages spoken actually in Randa in Kigali, yeah. Currently working with Mozilla as a community fellow on voice tech, yeah.

• 7:33Daniel Whitenack?

  Awesome. Yeah, that's great. And I'm really excited to dive into the details of some of that in a bit. But before we do, uh Jenny, uh would you uh let us know uh the same, a little bit about your background?

• 7:45Speaker 05

  Yeah, thank you. Uh I'm a software engineer primarily, and I have a strong focus in my work on ethical tech, data governance, and privacy rights. And so I kind of found my way through common voice and open voice tech via those avenues. So currently, I lead the engineering work on the Common Voice project, both in terms of the web platform and the data infrastructure around it, and kind of helps set the technical strategy.

• 8:07Daniel Whitenack?

  Awesome, yeah. And then Josh, what's your background?

• 8:11Speaker 00

  Yeah, I got into speech technology from linguistics, actually. I uh was in academia doing more kind of cognitive science and phonetics, phonology, perception, kind of the acoustic side of things. And I started to learn more about computational linguistics and NLP and speech technology, and and that really sucked me in because that's something where you can see users actually interacting with what you're doing, and that was really rewarding. So recently, just last year, I finished my PhD on speech recognition for low-resourced languages. And during that, towards the end of the dissertation, I was working with the machine

• 8:55Speaker 00

  learning group at Mozilla, doing some Things with Deep Speech, trying to make it kind of, trying to find an approach that could scale to new languages easily for speech recognition.

• 9:07Chris Benson?

  Yeah, and Deep Speech, just for those that aren't familiar, that's a project coming out of Mozilla as well?

• 9:13Speaker 00

  Yes, it's called Deep Speech, Baidu originally named it, but Deep Speech is an open source, speech to text, software stack, includes code for training, code for inference, and also pre-trained models. And so today, I am a fellow at the Mozilla Foundation, working on this project for African languages and Deep Speech and Common Voice. And also, my normal day job is lead scientist for speech technology at a startup in Los Angeles called RD, Inc.

• 9:49Chris Benson?

  Very cool. Um, I guess, to get us started, Jenny, I'm, I'm wondering, when Common Voice started, what was the state of speech data in general at the time? And, you know, what were you able to get your hands on? You know, what was the practical aspect of getting started at that point?

• 10:05Speaker 05

  So, Common Voice started in the summer of 2017. I don't know exactly what the state of spoken speech data was at the time, but I do know that it was not the norm to have data sets. So for context, you know, Siri came out in 2011, Alexa came out in 2014. So by mid-2017, there really hadn't been a ton of work on speech recognition and voice technology stacks in general that was available to the general public. The big companies, obviously, are not particularly interested either in releasing the training data that they have been working on, or making those stacks open source. And so there had been a few data sets that have been doing a tremendous amount of work

• 10:49Speaker 05

  trying to find these open data sets, and trying to basically look at large corpuses of voice data like audio, like open source audiobooks and those sorts of things, in order to turn a data set that was initially intended for something else for voice recognition. But the, as is often the case with open source projects and as is often the case with sort of internet volunteer labor, it was heavily focused on English. It was not diverse and oftentimes there was not really a good sense of a roadmap to get from, you know, the initial 200, 500 hours of English to a place where the speech dataset can actually represent the user base of the internet more broadly. And so that was kind of the initial impetus for Common Voice.

• 11:34Speaker 05

  We wanted to experiment with having something that was crowdsourced and decentralized in the best way and look at how we can really democratize tech and bring people into the fold who were traditionally left out.

• 11:48Daniel Whitenack?

  Yeah, and Josh, maybe this is a question for you since you're working in your PhD, but in terms of state of the art right now and what people are doing, what is the sort of, for something like speech recognition, how much data or how many hours of speech data do you need sort of on general? I know that's difficult because it's, there's accents and gender and it's different for different languages and all of those things, but what sort of scale are we talking about in terms of getting something that's functional for speech recognition? What is the sort of scale in terms of numbers of hours of audio that's transcribed that we need?

• 12:28Speaker 00

  Yeah, this is a very good question and it's, it's kind of the million dollar question. Everybody wants to know, how much data do you need to make an application that's useful, that's based on speech to text? This is, it's really dependent on the application. You can actually get by in some cases with zero data. I know it sounds kind of crazy, but if you have a really, really low resource language, you can actually take an existing model from a high resource language, you can take an English model, let's say, and basically just...

• 13:07Speaker 00

  transliterate the words that you care about for the target language. And so you can kind of hack an English model to recognize, let's say, Welsh. I know people have done this for Welsh, for example. But that usually works only when you have kind of limited domain tasks, when you have a small vocabulary. So if you're, let's say, you're trying to recognize numbers, the single-digit numbers from zero to nine, you might be able to do that for a new language just by kind of transliterating what the English model thinks it heard. If you want something that's going to be more open-ended, like a really robust speech recognition model

• 13:51Speaker 00

  that can recognize any phrase, any word in some target language. This is hard to put a number of hours. People will usually say 2,000 hours, but that number comes from some standard English corpora, like LibriSpeech and Fisher. But that's a number that's, I think, a pretty safe one to say. If you want something that's really open domain for some target language and you're not assuming that you're going to do some kind of transfer learning from a source language model. I think 2,000 hours is something people would say. But it really depends on what your target application is.

• 14:31Daniel Whitenack?

  Yeah, and I know that, you know, Jenny, you were saying that the kind of state of speech data, at least how it developed. Of course, there was a lot of data available for English and maybe other major languages starting out. And I know that it was hard to put a number on that, Josh, but at least gives us a sort of scale. I'm assuming, maybe Remy, you could speak to this, but I'm assuming that for many, for example, African languages, we might have kind of zero data available, at least in open data. In some cases, we might have some more. In terms of the languages that you're working with, Remy, what is the sort of state of the data?

• 15:11Speaker 06

  Okay, so yeah, I think like some of the language we're working on, regardless of written on, Kinyaranda, which is actually a very well-documented language. And so Kinyaranda actually has like over like 12 million speakers, like in Rwanda. And to give you just like a background.

• 15:34Speaker 06

  Most of the administration, administration actually, uh, administration government in Gigali. So use actually Kenya Rwanda as a language in their daily operations. So uh, giving that's like, so a plenty of resources, uh, like consulting as well, like the Bible, which is the with the Bible, which is actually very well documented in Kenya Rwanda as well. Uh, so a lot of newspaper around, uh, yeah. So I think for Kenya Rwanda, we have actually a very useful actually, uh, this corporate to use.

• 16:09Chris Benson?

  So Jenny, I was kind of wondering, like, at this point, as we're looking at Common Voice today, kind of what, what is the state of Common Voice currently? And what kind of functionality is available in the platform?

• 16:22Speaker 05

  Yeah, Common Voice, the platform has the core platform, which you see when you go to commonvoice.mazala.org really focuses on the data collection aspect. So on, on one hand, you go to Common Voice and you click on speak and you will be given any number of randomly selected sentences from our available text corpus for you to read out loud. And if you would like to validate, you will be given recordings that other contributors have recorded. And you can say this does or does not match the text. And generally speaking, we have a criteria for when we consider a recording to have been sufficiently validated by the community. And so the Common Voice platform is really the first step in the pipeline to getting to our data set. And that's just a web app.

• 17:07Speaker 05

  It's available on all platforms. There's a little bit of trickiness with mobile Safari, you know, due to just the vagaries of media recorder API implementation, but we're doing our best to make sure it's as widely accessible as possible. And then the Common Voice data set, the output of all of this volunteer effort, the last data set we released in June had 7,200 recorded hours in 54 languages, of which about 5,600 have been validated by volunteers. And so when, you know, this is a constantly growing number, so that data set had 54 languages, but the current Common Voice corpus overall has 56 languages and you know, it's that

• 17:52Speaker 05

  number is getting bigger every time. So right now we're looking at approximately. Just a little bit under 400,000 unique contributors who have either done recording or validation and, you know, over 130,000 unique voices in the dataset.

• 18:07Chris Benson?

  And quick follow-up to that, and speaking as the probably the non-NLP, you know, person on this podcast, when you say validation by volunteers, what exactly does that mean? Just for those of us who aren't familiar with it.

• 18:21Speaker 05

  Yeah, so the common voice approach to validation, and Josh can correct me if this is wrong, is a little bit unorthodox when it comes to how we normally think about NLP annotation. So basically, for the purposes of training, you want to make sure that a piece of audio matches the transcribed text as much as possible, you know, without the ums and ahs of the usual speech patterns, without any punctuation, or, you know, any of those sorts of things. And normally how that happens is you're given an audio and some, the person who was doing the annotation would write down what they thought the audio was, or vice

• 19:02Speaker 05

  versa. Because all of our contributions are community-led and the scale is such that we can't internally as a team do QA on it, we're really relying on our community's members to listen to the recordings that have come through and say, do I think that this recording exactly matches the sentence that it's supposed to match? So, you know, if the sentence is "The quick brown fox jumped over the lazy dog," and somebody reads "A quick brown fox jumped over the lazy dog," that would be the kind of thing where we would say there is a mismatch between the sentence and the recording that the training model would be confused by. And so this is the kind of thing that we would want to filter out as not

• 19:45Speaker 05

  matching validation exactly. Does that about cover it, Josh?

• 19:49Speaker 00

  Yeah, I would say actually that it's not completely unorthodox. I think that

• 19:57Speaker 00

  In reality, I think with speech technology and NLP, what's orthodox really just comes down to what the application is and what the constraints are of the project. So there are some folks out there who would say, you know, the best corpus of of speech for training speech to text models is conversational speech from humans that has then been transcribed. Common Voice is is a a red corpus. In the past, there have been red corpora that people have issues with, rightly so, because they're not seen as applicable to most projects' needs for at inference time. So, for example, concrete, I'm talking about LibriSpeech.

• 20:39Speaker 00

  LibriSpeech is a corpus of mostly American English that was taken from, was formatted from open books that were recorded from Project Gutenberg. So people who like to read books out loud for kind of the general public, a lot of the consumers of these books might be people who are visually impaired. There's this giant project, people read books out loud. And then some researchers, I think mostly from Johns Hopkins, they took this set of audio and they're not actually transcripts, it's it's the book. And they did some alignment and they used that. And that kind of became a benchmark for doing speech recognition, especially for English. However, the way that people read books out loud,

• 21:24Speaker 00

  they try to read it in a very quiet room, they try to make sure that there's no echo, they try to make sure there's no dogs barking in the background. That kind of audio is not what most developers are expecting for their applications that use speech to text. It's better to have actually kind of messy audio. It's better to have cars honking in the background and and wind blowing and dogs barking, because that's a lot of times what you actually get at runtime at inference. So Common Voice is red speech. However, it's very valuable in the diversity of voices. It's just an enormous diversity of voices. And also the diversity of background noises, the diversity of acoustic environments.

• 22:07Speaker 00

  You have people who record Common Voice just for fun on their morning commute when they're on the train. That is really, really valuable data because it's so noisy. So I think that People who might knock on common voice because it's red speech as opposed to conversational are also kind of, I mean, it's valid. People do speak differently when you're reading something out loud. That's linguistically proven. That's not really debatable. But the kind of noise you get in the background is just so valuable that common voice, I think, is going to be, if it's not already becoming, the benchmark for speech recognition for all sorts of languages.

• 22:53Speaker 07

  Hi, I'm Matt and I'd love to tell you about pace.dev. Pace.dev is a minimalist task management and async by default communication tool. Our screen recording feature is actually very popular. Wherever you can leave a comment, just like how easy it is to upload a file, you can record your window or the entire screen and upload it as a video to the team. Sometimes a screen recording is the perfect way to explain something, you know, whether it's a bug that only happens for you, or maybe more optimistically, a new feature that you can't wait to show off. And the showcase feature takes that a step further and lets you highlight progress, which is a much more positive experience than trying to make up estimations out of thin air. So please, learn more and start your free trial

• 23:38Speaker 07

  at pace.dev.

• 23:55Daniel Whitenack?

  So Jenny, I'm curious, you know, while we're still on the topic of common voice, I was curious in terms of the playbook as you move into the future for expanding the set of languages that are available and maybe involving, you know, actively involving local language communities. What's the sort of playbook looking to the future for common voice in terms of expanding that and making sure you have, you know, more and more language communities getting involved?

• 24:24Speaker 05

  Yeah, thanks for the question. It's something that the Common Voice team spends a lot of time thinking about because, you know, as many languages as we share between us, we can't cover all of the 7,000 languages in the world. And so what we really want to do is make Common Voice as self-serve as possible in terms of a platform for, and really any language community that wants to make use of it. And so the pipeline for getting a new language on Common Voice first starts with being localized. You know, if the Common Voice website can't be used by speakers of a language because all of the website copy is in English and that's not particularly useful. And at the same time that this localization is happening, the thing that we ask language

• 25:05Speaker 05

  communities to do is really find a corpus of sentences to be read. So our threshold for that is 5,000 sentences. That's kind of what we found sets a language community up for success, both in terms of having enough sentences to generate a sufficiently large corpus to start doing training work with, but also so that people who are coming to the site are not all just reading the same sentences over and over again. And these sentences can be from any source that the community thinks is appropriate as long as the licensing is open source, so as long as the licensing is CC zero. And then once the site has been localized and translated and once we have the sentences,

• 25:44Speaker 05

  5,000 sentences available, then the language gets activated on the Common Voice site and people can just go and go start contributing. And so to that end, we've built several satellite tools for making that happen. There is a tool called Sentence Collector, which is primarily currently maintained by a community contributor, Michael Kohler. That is where anybody can go and just, if they want to, start typing in sentences to submit for a certain language, which will then also go through a community validation process to make sure the sentences are correct and appropriate and, you know, all of these things.

• 26:22Speaker 05

  We also have a tool that scrapes large open source text sources such as Wikipedia in order to, you know, automatically put together a corpus so that you don't have to come up with 5,000 sentences to read. A lot of our largest languages have been compiled by scraping Wikipedia and scraping open source encyclopedia type. Sources, I keep saying sources, but this could also include things like transcripts from the European Parliament, or, you know, we're really very open to anything that the community deems will help them succeed is something that we're really interested in supporting.

• 27:02Chris Benson?

  I appreciate that. Hey, Remy, I have a question for you. What can technologies like speech recognition enable, for lack of a better word, in local language communities in Africa and elsewhere? And what are the potential applications? And are they different than those that we're finding in places in the West, in the US, and in Europe and such?

• 27:22Speaker 06

  Yeah, I think voice, like voice technology can actually enable like a unique opportunity to reach out to people who have like been excluded to the traditional, like to the traditional from digital services. And yeah, I think some of the kind of application would be especially a fintech application and regarding healthcare. Just to give you an example, so two years ago, I was working at this startup and we were actually building a product for people actually to contribute money into savings groups. So those groups were formed like with 20 to 30 people.

• 28:06Speaker 06

  So most of those contributors were actually farmers. And looking at the literacy level, like in population in Africa, so you actually see that 30% of agents in South South Africa are actually illiterate. And so it was very like difficult for those people actually to use like this application to contribute. And so my thought was like, if you would actually have like a Kenya Randa language model, which is actually a local language in Randa spoken with 12, over 12 million people. So this would have actually is the way those farmers contribute on their savings group.

• 28:50Speaker 06

  Yeah, so there also a lot of innovation happening, especially in Randa. Most of the government's offices have been digitalized.

• 29:01Speaker 06

  And so it's not very accessible actually uh by people, especially in rural areas. Uh because someone would actually live thirty like like ten to thirty kilometers from actually uh a cyber a coffee, a cyber cafe, uh internet cafe. So where actually people go and seek actually for like services. And they always go look for uh agents, agents who actually help them actually seek those government services. So I think, yeah, uh voice technology can actually enable a lot of solutions. And helping actually farmers to actually access uh digital financial products, yeah.

• 29:40Daniel Whitenack?

  Yeah, and uh that that's super interesting. I wonder um as a follow up to that, if so, the language Kenya Wanda, if people in maybe that language community and others, do you think that they would find more value in a sort of voice application versus a text application in the local language? What do you think provides the most value there?

• 30:04Speaker 06

  Uh so like, yeah, just giving you a context. So a lot of people, especially in urban areas, communicates like with voice over WhatsApp, like using WhatsApp, so they they tend to send more voice voice notes to actually people that are chatting uh in the local language. And yeah, and people will actually prefer actually communicating through voice and and as well like seeking for information through voice. But just a quick example, so people will tend to call, especially call centers, uh like telecommunications call centers, actually to seek information. So the recently uh with actually the crisis, uh there's this institution in Rwanda called RBC. It's actually the Rwanda Biomedical Center.

• 30:49Speaker 06

  So they've been actually receiving over one thousand call like respected in in getting information through uh Covid, for the coronavirus. Yeah, and I think people will be seeking more value like with voice than actually texting, yeah.

• 31:02Daniel Whitenack?

  Yeah, definitely. It seems like there's a lot of great potential there. And I know that, of course, one of the things, like you were talking about some of the data that's available in the language you're working with. And I know, Josh, you've worked a little bit with sort of bias and data sets. And it seems like, of course, as data scarcity might be a problem with some of these, you know, lower resource languages, there's probably a heavy bias in those speech data sets for the lower resource languages towards, you know, whatever it is, maybe it's only males in the corpus, or only a very small representation of the potential accents in the data.

• 31:43Daniel Whitenack?

  You've done some work in this area. I remember reading a recent blog post about a data set that you're working with to help researchers diagnose bias in speech data sets. Could you give us a little bit of an introduction to that work and what you're doing there?

• 31:58Speaker 00

  Yeah, I'd be happy to. I think that this question's a really important one, and it's one that anybody working in language technology needs to think about, needs to face, because whatever language technology you're working with, this bias problem is going to be present. So the work that you're referring to is the RD Bias Corpus. It's a corpus of English right now. It's a subset of the Mozilla Common Voice corpus that has been kind of filtered so that we have demographic data for every audio clip in the speech corpus.

• 32:36Speaker 00

  And also, we've done extra steps of validation to really ensure that the transcripts are 100% accurate. So this data set is, it's about 1,700 audio clips. So it's not enough for training, but it's enough for diagnosing, as we call it, diagnosing bias in your speech to text model. So this is, it's a hard problem to kind of get at. When we talk about bias, to make it really concrete. So the definition that we've basically come up with is if you have a speech to text

• 33:16Speaker 00

  model that performs statistically worse for a certain demographic group, then that model is said to be biased. So using this corpus, the RDBIAS corpus, you can take any off-the-shelf speech recognition model and look at what is its accuracy, or word error rate, as we call it, for a certain demographic group. And we have information, thanks to Jenny and everybody at Common Voice, we have information from people who opted in to self-identify their gender, their accent, and their age. So yeah, along any of those lines, you can look at how well your model performs on every

• 34:01Speaker 00

  group versus every other group. And it does, the kind of beginning of your question was, this problem with more underserved language communities, this problem becomes more exaggerated. If you have a small sample from any distribution, that the sample is more likely to be biased, especially if you're collecting it in a kind of biased way. So a lot of the language communities we have, they know about these problems and they're working very hard. Like Remy and the folks on the ground working at Digital Umuganda, they have done a lot of work to make sure that the Kenya-Uwanda dataset is as balanced as possible.

• 34:42Speaker 00

  And they've done some really great things with that, especially for gender diversity. It's a hard problem because you want to get data that's balanced, but also you want to, you need to be mindful of people's privacy, so you don't always have people reporting.

• 35:02Speaker 00

  Their demographic status because it's an opt in project. But even given that, Digital Umuganda has been able to do some really great things through community recruiting to make sure that the corpus is as balanced as possible.

• 35:15Daniel Whitenack?

  Yeah, Remy, could you talk about that a little bit more in terms of, you know, your strategy around data collection and making sure you have these kind of balanced groups and involving the right people in the community?

• 35:30Speaker 06

  Yeah, so basically, Digital Umuganda did actually an incredible job in actually collecting voice data sets. And so the strategy was more like hosting like offline events and bringing the awareness to actually contributors. Why is actually voice technology very important? And especially for underserved languages. Uh, yeah, they really did a really good job. And so far, we have on the platform over a thousand hours of voice collected. This has been done by setting up actually a community of communists. So communists are actually members who actively like organize local events in their areas,

• 36:15Speaker 06

  in their universities, and in order to get people to contribute.

• 36:20Daniel Whitenack?

  I was just wondering if you could say this Digital Umuganda, is that like a, is it a company? Is it a for profit? Is it a community thing? What, could you describe a little bit of what it is?

• 36:30Speaker 06

  So, uh, yeah, Digital Umuganda is actually a company which actually worked on AI products. And especially what they're trying to actually build is actually an AI voice, AI chatbots, and in Kenya Randa. And the AI chatbot would be actually in charge of actually giving information related to COVID and also in like to other topics. So, and then facing the challenge of not having actually a Kenya Randa actually data set. And they actually started by actually collecting voice data on the Common Voice Platform. And the next phase for them would be actually like working the data sets.

• 37:15Speaker 06

  And Josh had been like working on the Kenya Randa model, which they can actually now use to integrate to their chatbots.

• 37:24Speaker 06

  I think that's actually the answer to your question.

• 37:26Chris Benson?

  Yeah, it does. And I'm kind of curious, kind of going back to what both you and Josh were saying before. Really wondering, as you're looking at what impacts performance in terms of getting these recordings from different people, different groups, is it gender, you know, the sound of gender, accent, noise? What is it that really affects, you know, what the end product is? Josh, could you address that one?

• 37:51Speaker 00

  Yeah, this is, this is a great question. What kind of demographic factors are going to have the biggest impact on performance for speech recognition systems? So I think accent and gender have been just hands down proven to have a huge effect. Accent, depending on what we classify as accents versus dialects versus languages. I'm sure Daniel has thoughts on this too. Language is a continuum, right?

• 38:22Daniel Whitenack?

  Everybody has a different definition.

• 38:26Speaker 00

  Yeah, okay. A concrete example, try using Siri or Google Home with a Glaswegian accent or any kind of Scottish accent. It might be better now. I'm assuming they put more work into it, but pretty famously a few years ago, they, it was just abysmal. And so accent has a really big effect because the, I think, the kind of continuum is so broad. And then after accent, gender has been shown to have a, a big effect, like a very reproducible bias. There's lots of different ways to think about why there's bias in a model.

• 39:07Speaker 00

  A lot of times, the, the typical answer is, well, the, the training data set was not, was not sampled correctly, that there are undersampled majorities, as, as it's been called, in the data set. And that, I think, for, for gender might be one of the big problems. And also, on a kind of technical level, speech technologies were developed, basically, in a way that makes men's voices, because they're lower pitched, they have a different kind of frequency range, to make it easier to, to work with those kinds of voices. So there's, there's a sampling problem, and there also are problems that could be inherent to the technology itself.

• 39:51Speaker 00

  But at the end of the day, the demographic factors like accent and gender are usually much more pronounced than than age. Age can also have have an effect, but it's I've seen that less often.

• 40:21Speaker 02

  ChangeLog News is the best way to keep up with the fast-moving software world. We track, blog, and contextualize the coolest projects, the best practices, and the biggest stories each and every week. Make change log.com your daily destination, or hit the snooze button and subscribe to our weekly newsletter that hits inboxes on Sunday mornings. Join more than 15,000 enthusiastic readers. It'll cost you exactly $0 and you can subscribe right now at change log.com/weekly.

• 41:03Daniel Whitenack?

  All right, so as we've got into talking a bit about the impacts of these demographic factors in in terms of the performance of certain speech technology like speech recognition, I want to kind of circle back and and see, you know, to Common Voice and Jenny and and ask like, if if this work that that Josh has been doing in terms of the demographic factors, if that's sort of influencing thought on the Common Voice team in terms of if I I may be mistaken about this, but I think like you have some processing that actually, when you download a data set, actually, there's already a segmented out training test dev set.

• 41:46Daniel Whitenack?

  Is the work that Josh is doing or maybe other similar efforts, is that influencing your thought process around how you're forming those training data sets for people and segmenting out that data? Or maybe it's, you know, making some some changes to the collection interface to promote. Certain diversity, what's the sort of feedback from this this kind of work?

• 42:11Speaker 05

  Yeah, we work quite closely with Josh and rely a lot on his research and expertise in these matters to guide sort of our technical roadmap. So I think from, I would say from the inception of Common Voice until this year, we've really focused a lot on language diversity with the idea that that was the thing that we could bring the most value to, especially around being able to mobilize Mozilla's international contributors and, you know, kind of counting on our reps and our local communities to do the kind of organizing that Remy was mentioning as well. And so now that we feel like we have a pretty good momentum going on the languages side, we're really starting to look more at the demographics

• 42:55Speaker 05

  inside of a language. And gender is something that we are very cognizant of as something that we also need to correct. The kind of interesting thing about being pretty much the only voice dataset out there that releases our demographic stats also means that we are very aware of all of the ways in which we could be doing better. And so the the ways we've thought about remedying that do definitely include the segmentation that you had mentioned. Right now, the segmentation optimizes for diversity of speakers, but there's certainly post-processing we can do there to, you know, try and optimize for gender as well. But some of the other things we are thinking about on the platform side

• 43:38Speaker 05

  include things like, we know, open source in general tends to feel less accessible to underrepresented groups and looking at what campaigns we can do, what, you know, we could run challenges and events specifically reaching out to underrepresented groups to make it very clear to them that we welcome and actively seek out their contributions. But also, one of the other features that we haven't been building on the platform is the ability to collect data for a specific segment. So the thing that Josh had mentioned earlier on, you know, collecting a dataset of just reading out the digits, uh, one to 10, and, you know, the words yes and no, that's a very specific target segment that

• 44:20Speaker 05

  we can then, you know, go out into the world and say, hey, this is a thing that we want to collect for the specific purpose. Can you help us? And we're looking at how we can use that segmented segmented campaign ability to then drive other diversity as well.

• 44:36Daniel Whitenack?

  I realize there's some good expertise with our guests on the ethical side and data governance side of things. And I wonder, like you had mentioned, that only certain of the people that contribute to Common Voice, they self-identify. Of course, it seems like you know, you could, if you wanted to, create some data augmentation method that would detect the accent or even the gender based on those who did self-identify to then label those who didn't. Has this come across your ethical discussions in terms of what you do with people's speech that comes to you and what you don't and how you form those principles?

• 45:19Speaker 05

  So, the Common Voice as a team, and I think Mozilla more broadly, really takes a data minimalist approach wherever possible. I personally am a strong believer that the only data you can guarantee to be secure is data you never collect in the first place. And so, we, as much as possible, want to minimize what we are tracking on our contributors. We respect you not track. We ensure that if people don't want to be identified, that they are not identified. This is also why we allow anonymous contributions on Common Voice. So that's kind of the first principles we're operating on. In terms of using augmentation methods to gather more information on those demographic pieces from people who are not

• 46:02Speaker 05

  self-identifying, I would say that any augmentation method that is available to us is in self also biased. And so, I would be very concerned about introducing additional SKU if we're using identification methods that are trained on uneven data sets to then further segment our own data set.

• 46:20Daniel Whitenack?

  Right. You might just kind of be propagating a problem that is already there.

• 46:25Speaker 05

  Yeah, totally.

• 46:26Speaker 00

  There's something I'd like to tack on there. I definitely echo everything that Jenny said. And a little bit, I guess, further, let's say that we do have some kind of gender identification, or age identification, or accent identification that is 100% accurate.

• 46:44Speaker 00

  Even in that case, I would not want to be using that at all. Because the fact that this is an opt-in choice means that if the person didn't give you that data, they don't want you to have it. It's not that they just don't want to give it to you, it's that they don't want you to have it. So before I got into speech technology, I spent a lot of time working in kind of psychology labs. And if I had folks coming in and doing some kind of cognitive tests, and they have a sheet of paper and they're filling out demographic information and they choose not to fill out their gender. And then after they leave, I think, oh, well, I saw that she was a woman, so I'm just going to fill in woman. That's something that would set off a lot of alarms.

• 47:29Speaker 00

  And I think that this kind of approach where you're filling in kind of holes in the data after the fact is the same thing. If, when we're thinking about kind of the ethics of using machine learning models, I think it really helps to make it concrete if you say, okay, if it weren't a machine doing that, if it were a human doing it, would that be okay? And I think in this scenario, it's something that I really do not feel comfortable with because the person doesn't want you to have the data.

• 48:00Daniel Whitenack?

  Right, yeah, and I'm sure this is a problem. I think we're going to have a discussion pretty soon as well with one of the major image dataset groups that's out there. And yeah, I mean, you could, you could identify a lot of things in images or you could probably, you know, attempt to identify even people through speech. I, I think I, I remember when I downloaded some of the data from Common Voice, I had to agree as a user of that data to not try to identify the, the speakers. So I don't know if that's always been there, but I definitely resonate with what you both are saying for sure.

• 48:36Chris Benson?

  So a question for Josh and Remy, as you were looking at the at the future of where your Mozilla fellowship can take you at this point, kind of what are you thinking? Where do you expect things to go? What would you like to get accomplished under that fellowship?

• 48:51Speaker 06

  So I think what I really want to accomplish is actually having, making sure that we have actually a strong ecosystem around that model, which the local startup ecosystem can actually use and could actually relevant application for people who have been actually left out from the digital world. And yeah, I think that would be actually something really good for me to accomplish. And as well as making sure like we have actually enough of this case where we can actually use voice technology.

• 49:26Daniel Whitenack?

  Yeah, that's awesome. I really like how you put that because one of the things that really been privileged to sort of learn a little bit more and grow in as I've been doing my own work is just growing in my own desire to have sort of two-way communication in the in the digital world. So not so much that, you know, like we want to have, for example, machine translation models that translate everything that's in English to every other language. You know, it's really great to, you know, on the other hand, think about, well, there's so much value and amazing things happening in these language communities around the

• 50:07Daniel Whitenack?

  world, local language communities that we as high resource language speakers are missing out on. So that's really great and exciting stuff. What about you, Josh? What are your thoughts for the future of the fellowship?

• 50:20Speaker 00

  Yeah, so right now, for the, at least the duration of the fellowship, there's a couple of projects that Remy and I are working on. There's the, the one that we've been talking about, the digital Umuganda kind of base project that is focused on Kenya Rwanda and collecting Kenya Rwanda data, engagement, community kind of mobilization, and also model training for deployment and making a product that people actually can use. And people care about. That's kind of one side of it. We're also working with some researchers out of Makera University in Uganda. It's also related to coronavirus. It's kind of a project

• 50:59Speaker 00

  that's keyword spotting for broadcast radio so that folks in the Ministry of Health can know kind of more details on what's going on in the country with regards to coronavirus because the broadcast radio is a really well-used kind of medium to get up-to-date information. So these two projects, they're both about data and models and community engagement. And right now, these are what's really driving and seeing these to some kind of useful product is the near future. But moving forward, what I'm really interested in is leaving a set of kind of best practices,

• 51:42Speaker 00

  guides, kind of playbooks for how to do all the things we're doing right now, but without me and Remy. If a new community wants to start using Common Voice, using Mozilla's Deep Speech, they want to start along this path that ends in a product, how do they do that without needing some kind of expert? Like, my hope is that we can make documentation and resources that are useful for developers everywhere so that they can do this at home. Common Voice is all about this democratization of data. Deep Speech is about democratization of tech. And I think that

• 52:23Speaker 00

  the work that we're doing right now is about this kind of democratization of expertise. Because for every project, you can't get somebody to help you out with voice tech. There's not a ton of us around. There's 7,000 languages in the world, and within each of those languages, there's lots of projects. And so to make this really open, we need to get the knowledge out there, too.

• 52:46Daniel Whitenack?

  So were you always wanting to have products as the end goal here? Or developer tools that you could use? Or you seem to talk a lot about enabling other developers to build things through maybe a speech technology that you put out there. Was that always the goal? Or did you ever think about building the end product? Like the fintech application or the healthcare-related application directly? What's the balance there in your mind?

• 53:14Speaker 00

  So in my mind, these two projects we're working on, there's the Rwandan project and the Ugandan project for the fellowship. These are very, each, they're very product-focused or end-goal-focused. But the overarching context, at least for me, is these are two examples of taking an idea and speech and going through the entire pipeline with data collection, data validation, community engagement, making something that can be deployed. That whole pipeline for both of these projects is something that's reproducible. So it's the answer to your question, I think, is both.

• 53:56Speaker 00

  These are both, in my mind first, important because each of these projects is helping us hone the pipeline. And then, once we have that honed pipeline, we can communicate that to a broader community.

• 54:10Chris Benson?

  So Jenny, I'm curious. You're right at the center of speech, data gathering, and governance. And I guess, are you hopeful? And what are your thoughts toward the practicality of being able to expand speech tech to more and larger communities around the world over the next few years? And what are you excited about in terms of Common Voice? Or speech tech in general, actually?

• 54:32Speaker 05

  Hmm, that's a very interesting question. For Common Voice, the thing that I saw, I've only been with Common Voice for about a year. And in that time, the thing that I've been repeatedly blown away by is the enthusiasm and the dedication of our various communities. That says to me that there's really a need for something like this in the world, that Common Voice is filling a gap. In both the marketplace and also just sort of the broader language community, you know, we've seen applications of common voice that we hadn't originally intended for. We were looking for it, using it to collect voice data for speech text models and those sorts of things. But we're also seeing that common voice is being used for language preservation.

• 55:17Speaker 05

  And I think there's potential applications for common voice for doing language acquisition even. The thing that I'm the most excited on is just how can we broaden the reach and the diversity of common voice so that people who have ideas that would have never occurred to us internally are empowered to use these tools as much as they can. And I think at the same time, because common voice is or, you know, our collection methodology, as Josh said earlier, is very consent minded, very privacy minded. And I think there's a lot of opportunity to do leadership in the data governance space around how we think about, you know, data licensing for something like this.

• 55:58Speaker 05

  There's, I think, to some extent, voice data is inherently biometric, you know. If somebody downloads a common voice dataset and goes, hey, I recognize my friend, there's no amount of obscuring or anonymization that we can do for that dataset. That would prevent that from happening. And so I think there are going to need to be some very difficult conversations in the next few years on how we handle that in such a way that, you know, it doesn't get abused. And it doesn't get used to create a deep fake or, you know, any number of nefarious and problematic applications. And I think being at the forefront of the governance question before things get to the point where they are now with ClearView AI, just to use an example,

• 56:43Speaker 05

  is a really powerful way that we can show up in the world.

• 56:47Daniel Whitenack?

  Awesome. Yeah, I think that's a great perspective to end with. And I just want to thank you all for taking time out of your. Really, really important work to talk to us about speech tech and data and bias and speech data and your really exciting work for African languages. We're just really thrilled to have this conversation. Of course, in our show notes, we'll link to all of the things that we've been talking about, about our Common Voice and Digital Umuganda and the fellowship program and all sorts of things. But thank you all for joining us. Really appreciate the conversation and hope to chat again soon. Thanks for having us.

• 57:28Speaker 05

  Thank you so much.

• 57:32Speaker 02

  Do you have questions, praise, or constructive criticism about the conversation you just heard? Comment on this and every episode of Practical AI on changelog.com. Just open your show notes, follow the Discuss on Changelog News link, and let your voice be heard.

• 57:50Speaker 02

  Practical AI is hosted by Daniel Whitenack and Chris Benson. It's produced by Jared Santo, that's me, and our music is provided by the mysterious Breakmaster Cylinder. We're brought to you by some amazing sponsors. Special thanks to Fastly, Linode, and Rollbar. And a special shout out to those listening on our ad-free Changelog++ feed. If that's you, you're awesome. If that's not you, well, you're awesome too. But you can learn all about it at changelog.com/plusplus. That's all for now. We'll talk to you again next week.