Cloning voices with Coqui

Coqui is a speech technology startup that making huge waves in terms of their contributions to open source speech technology, open access models and data, and compelling voice cloning functionality. Josh Meyer from Coqui joins us in this episode to discuss cloning voices that have emotion, fostering open source, and how creators are using AI tech. Featuring: Josh Meyer – GitHub https://github.com/jrmeyer, X https://x.com/_josh_meyer_Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: Coqui https://coqui.ai/BibleTTS - Data and models for African langauges https://coqui.ai/blog/tts/african-languages-dataset-collaborationCoqui TTS Hugging Face Space https://huggingface.co/spaces/coqui/CoquiTTSCoqui on Github https://github.com/coqui-aiJoshua’s previous Practical AI episode about Common Voice https://changelog.com/practicalai/104Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  I think a lot of the discussions around machine learning and also voice tech and the kind of the place in humanity and where it's going to be in the next few years. There's a lot of this kind of dichotomy of machines are going to do everything here and they're going to completely replace humans there. I don't think that that makes a lot of sense, especially when it comes to even voice technology. I think that there's a lot of, you know, call center technology, speech to text, speech recognition. I don't see any time in the near future as replacing people who work at call centers. I see it as being very useful to people who work at call centers. They are the customer. And I think that this kind of using machines to augment what humans normally do and replace some parts that are

• 0:44Speaker 00

  maybe tedious or annoying or whatever, like parallel parking. I think that makes sense.

• 1:00Speaker 01

  Welcome to Practical AI, a weekly podcast making artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join us at practicalai.fm/community and follow the show on Twitter. We're at Practical AI FM. Thank you to our partners at Fastly for shipping our pods super fast all around the world. Check them out at fastly.com.

• 1:34Daniel Whitenack?

  Welcome to another episode of Practical AI. This is Daniel Whitenack. I'm a data scientist with SIL International and I'm joined as always by my co-host Chris Benson, who is a tech strategist at Lockheed Martin. How are you doing, Chris?

• 1:46Chris Benson?

  I'm doing well, Daniel. You sound a little funny today, actually. So do I. What's going on today, Daniel?

• 1:53Daniel Whitenack?

  Nice, Chris. We have clones. And we might have frogs in our throat. Would that be? That's a good one. You know, for listeners who might have been with us a while and know our voice well, those were our voice, but not quite our voice. So today, we're going to be talking a lot about synthesized voices. And we've got Josh Meyer with us, who is co-founder of Koki. Welcome, Josh.

• 2:21Speaker 00

  Yeah, thanks for having me, Daniel. Good to be back.

• 2:23Daniel Whitenack?

  Yeah, so just because we have to, we have to say what that was first, and then we can, then we can launch into other things. So you're co-founder of Koki. A few, I don't know, it was a few weeks ago, or whenever it was, you, you came out with sort of a live, I guess, demo, a prototype of some of the functionality that your, your company supports, which is text to speech. But it's a sort of voice cloning thing where what Chris and I did to create those was just upload. I think mine ended up being like 11 seconds. It was like 11 seconds of my voice, and then I was able to synthesize that, that intro bit. And Chris, Chris did the same. So,

• 3:05Daniel Whitenack?

  so that was, that was pretty cool. Maybe before we launch into like all the AI stuff, and what you're doing, and the company, the projects, like, what's the general reception been to this, this kind of voice cloning thing that you're doing?

• 3:21Speaker 00

  Yeah, it's, it's been, honestly, a very, I'd say, very positive, very interesting reception. Most people that end up showing it to, they're like, wow, that was fast, and that sounds like me. And it's, we've been working on this tech for a while, right? We, we published in ICML, the kind of the, the core technique that we're using for this. And it's been there for a while. The tech's been there for a while. But you really needed to be a coder. You needed to know your way around the command line. You needed to know, you know, how to navigate through GitHub and download the models and all this stuff.

• 4:01Speaker 00

  And what's different now, the reason that we're getting this kind of like wow reaction is you don't need any of that. You can, you know, send the link to for the website, you know, koki.ai. You can send it to anybody you want. They just use the inter, you know, the web, uh, web app, the, uh, whatever browser they're used to. And you, you don't even need to actually upload audio technically in terms of, you know, you don't need to go find a file on your computer. And or you don't need to pre-record, you just hit the microphone button and, uh, say a few sentences, seconds. I think right now we have it capped at thirty seconds just to kind of be reasonable with, uh, server fees.

• 4:45Speaker 00

  Because more audio takes a little more server fees. But in general, we found that five seconds is good enough. You know, five seconds is, uh, is enough to get a nice voice print or however you want to call it. So it's, it's been fun and I, I think the reception, especially from non-technical people, has been the most, the most fun. And I, I really like working on this because I get to interact with people who don't know AI, who don't know machine learning or deep learning. They think it's cool. They maybe read some articles in the New York Times about it, but they're not, they're not in the weeds like all of us are. And so getting it in a place that we can show it to them and they're like, wow, this is cool.

• 5:29Speaker 00

  It's fun. It's, it's, it's refreshing, if that makes sense.

• 5:32Daniel Whitenack?

  Yeah, I could, I could definitely see that, like, this is something, like, I would not hesitate to kind of bring my wife in, who definitely does not. Her perception of my work is like, I always have a screen up and it's a really dark screen with a lot of text on it. And that's like my life, but when I pull this up, it's very like welcoming and like easy, just like record your voice button. Yeah, it's cool. I think it's a similar, we, we recently, um, interviewed Abu Bakar from, uh, a gradio, uh, at Hugging Face. It's similar, like, as soon as you put a demo in front of people, it's like a lightbulb moment. Like, like this is, this is how this thing like works or this is what I could expect as output, right?

• 6:16Chris Benson?

  You know, it kind of feels a little bit like we've been waiting for this moment for a while because we've been talking through the show, you know, up until now, mostly to kind of technical tooling and technical use cases where people are putting together amazing things. But now it's, you know, as you point out, you can bring people in who have no technical capability and get something really, really interesting out of it. And we've been kind of waiting for AI to take this turn. So it kind of feels like maybe this conversation is kind of the beginning of starting to turn toward really broad usage by people that would not otherwise have access.

• 6:52Speaker 00

  And I think that there's kind of generally speaking outside of speech, outside of language technology, even there's this, uh, in the last few months in particular, maybe just last month. I mean, with all the image generation stuff coming out with Dolly and Party and Crayon and all this stuff.

• 7:11Chris Benson?

  Imagine?

• 7:12Speaker 00

  Yeah, yeah, yeah. You're seeing what creative people can do, like creators or creative types. I don't know how to, but people outside of the coding community, uh, you give them the tools, you give them more tools in their toolbox to do creative things and they're coming up with awesome stuff. I mean, there was, I remember there was that one week where it was just like Twitter was full of, you know, koalas riding unicycles.

• 7:42Chris Benson?

  It was a bit overwhelming. Yeah, it was awesome. Yeah.

• 7:46Speaker 00

  What I'm really optimistic for is, is, is that kind of, uh, creative use for these, these voice tools because there's tons of applications I have in my mind of where I see these tools being useful and, and helping the creative process. But also I know that, you know, the people who came up with all these image generation stuff didn't think people were going to be doing everything that they're doing now with. And it's even more, more interesting than what they, the creators could have come up with, right? And so that's, that's something that I'm really excited to see in the coming weeks, months is to see what cool stuff people are doing with the voice tech.

• 8:28Daniel Whitenack?

  Maybe that brings me to something that's been on my mind in this discussion, which is, so a while back, we had you on the show as part of a discussion about Mozilla Common Voice, and we were talking about speech tech and also like open data and that sort of thing, open speech data. I think it's episode 104, if you're looking for that, so take a look there. But I've always kind of, even at that point, I think, had in my mind this perception of like speech tech and Alexa or Siri or whatever it is, almost like a kind of novelty type of thing from my perspective, like coming to a computer, where my first computer, like, I interacted with it with keyboard and mouse, that's like

• 9:13Daniel Whitenack?

  my standard interface, and like, that's what I've grown used to. But there's tons of people all around the world where like, maybe the first interface that they're interacting with a computing device with is their voice using Siri or different things. So I'm wondering, like, how do you perceive as someone that's like very close in the sort of speech technology space, how do you see the trends shifting in terms of like how serious people are taking things like voice interfaces or creative uses like you're talking about of speech technology in terms of like practical usage and like real world kind of scale, I guess?

• 9:57Speaker 00

  Yeah. So I think, quick backstory in terms of kind of how long I've been in speech, I started really getting into it in probably 2012, and did academia, research, and that was very fun. And but I got into industry because I like building things that people use. I still like writing papers, which is why, you know, Daniel and I recently wrote this paper with the great folks at Masakane, an African

• 10:27Speaker 00

  Text to speech. But working with Mozilla in particular, the last few years, or collaborating with Mozilla, because I don't work there anymore, but I still keep up ties and collaborate with them. And they're all great folks working on great things. Really, I think some of the best democratization of speech technology is honestly coming out of Mozilla. But in terms of kind of speech tech being a novelty, and or at least the perception of speech tech as a novelty, but actually finding real world applications for it. I think there there used to be this talk about, you know, keyboards are going to be gone in 15 years. You know, like there is, there is this talk that we're never going to type again.

• 11:11Speaker 00

  We're only going to use our voice. And I never really subscribed to that kind of viewpoint, because I think that mixed modality is always makes sense. Sometimes you want to type just because, you know, maybe baby's sleeping in the next room and I'm not going to shout at my computer to like, wake it up. I think in general, the most interesting applications of voice tech and machine learning at large are where they augment and support humans doing human things. I don't think machines taking over completely. Kind of the functionality of a human makes a lot of sense, which is why, for example, let's say self-driving cars.

• 11:56Speaker 00

  I think that the technology underlying self-driving cars is super useful. If I had a self-driving car, I wouldn't let it drive for me around town. But would I let it parallel park for me? Of course, like, that's something that I don't like doing. And so there's functionality where if you take kind of parts of what the the human pipeline is for for whatever task, like going to the grocery store, there's multiple parts of that that the machine can do better than me. But there's also parts of it that I can do better than the machine. So I think a lot of the discussions around machine learning and also

• 12:36Speaker 00

  So voice tech and the kind of the place in humanity and where it's going to be in the next few years. There's a lot of this kind of dichotomy of machines are going to do everything here or humans are. And they're going to completely replace humans there. I don't think that that makes a lot of sense, especially when it comes to even. Uh, you know, voice technology. I think that there's a lot of voice technology that's being used to, let's say, you know, call center technology. Speech to text, speech recognition. I don't see any time in the near future as replacing people who work at call centers. I see it as being very useful to people who work at call centers. They are the customer.

• 13:17Speaker 00

  In that case, right? They're seeing, you know, you're on a call with somebody who's, who's TV is broken. And you got this transcript in real time of what they're saying, but also you're able to run NLP on that. And come to answers faster, like you're getting kind of recommendations on, on how to talk to the client. And I think that this kind of using machines to augment what humans normally do and replace some parts. That are maybe particularly tedious or annoying or whatever, like parallel parking. I think that makes sense. And I think with voice tech, we don't exactly know yet. Which parts are going to be, how it's going to shape out, because it's pretty new technology. I mean.

• 14:02Speaker 00

  Speech recognition and speech synthesis have been around for a while. You know, I think the eighties maybe is when they first got some kind of more mass. Adoption or? Adoption, that's the word, mass adoption. Dragon Naturally Speaking, I don't know if, if you guys remember that I. I do. Yeah, I probably still have it somewhere stuffed in an old bookshelf. Yeah, it's still around. And I mean, I remember having. My parents had that when I was in school and I tried it writing a few papers with it and I was like, uh, not yet, you know? But since then, it's advanced a ton, just in general, the technology, and especially with the speech synthesis in the last.

• 14:43Speaker 00

  Honestly, in the last like five years or less, it's just gotten crazy better. It's uh, you know, five years ago, you would never listen to synthetic speech and be like, is that a human or is that a machine? I can't tell. But now it's like every paper that comes out and every research paper that comes out from the big labs, it's like, wow, this is, that sounds like a human. I can't tell, is that the training data or is that the synthetic speech? You know? And now I, I think the, the big challenge with speech synthesis is getting it to sound more expressive, more emotional, because I think machines and humans can basically do flat speech identical.

• 15:25Speaker 00

  But um, yeah, it's uh, it's going to be interesting to see how it all shapes out. But I, I, I think that there's going to be a lot more creative usage that we're not, that's not predicted by kind of the diehard technologists.

• 16:03Chris Benson?

  So before we dive fully into the technology, I, I want to follow up on something. Um, and cause I, as you were talking your, you know, through that a moment ago, I'm thinking of my own experience and my families and stuff. And, and it's been pretty fascinating for me to see uh, my wife, my daughter, my mother, people that are not, you know, technical currently that are, that are diving into this. And yet I, as a, as a technical person in the AI space, am still tending to default to the keyboard, going back to that. And it's a different user experience that they, I've seen my daughter gravitate to it naturally. What is, before we, we dive fully into the technology, what is it about the, that difference

• 16:46Chris Benson?

  in user experience that seems to make it more accessible? Do you think? The, the idea of speech recognition, natural language processing to. To parse it all and come up with with a good response and then the speech synthesis coming back. It's something that is so natural for my 10 year old daughter. That it's just, you know, it might as well be one of her friends that she's talking to. And so. Could you speak a little bit about what what that experience is and how the rest of us? That maybe are doing it some, but maybe not in such a completely natural setting. How does that evolve over time? What does that look like?

• 17:22Speaker 00

  Yeah, I think so. I think that one reason that we are using. The voice technologies that are built on lots of other technologies like, right? I think Alexa and Siri and Google Home and just any kind of Mycroft. I think that any kind of voice enabled home assistant. Is useful insofar as that as it's immediate. Like you can't sit there and wait, you know, two minutes, not even two minutes. I'm not even going to wait 15 seconds for Siri to talk back to me. Right? You know? I will be walking the dog. So, okay, I will say I'm somebody who's been working in this space for a long time, as I mentioned.

• 18:04Speaker 00

  But I've never been one who's had home assistants. One, because of all my kind of privacy concerns, like I just don't want to have them sitting around. I'm very realistic. Like I know people who work on all the teams, you know, nobody's consciously snooping. And recording audio because one, it would just kill their servers because there's way too much. Way, way too expensive to even stream all that back and recognize it all. And, you know, blah, blah, blah. So I haven't had them for a long time in any case, but I recently got an Apple Watch. And I find that I use it every day when I'm walking the dog.

• 18:41Speaker 00

  Because it's just so convenient because I can't, my dog pulls on a leash and I've, you know, only got one hand that's functional. And so. But if my service is bad and Siri's not replying for like maybe even just 10 seconds, I'm like, forget it, I'll take care of it later. You know, the attention span, I think, of people is in general not super long. And so the technology has gotten faster in general over the years. And that is a, is a big part of making it adoptable. And besides that, there's the kind of the pleasantness of, of talking to a voice that sounds really human.

• 19:24Speaker 00

  Like, I don't know when exactly Apple introduced it, but if I say, hey Siri, at least the voice that I have, sometimes will reply, mm hmm, like a normal, you know, American English speaker would reply, exactly, like, yes, I'm listening. It's not going to say, yes, I'm listening. It's going to, you know, talk like a human, something closer to the movie Her. And so I think that the voice is being high enough quality that it almost sounds like you're talking to a human, not just the quality itself, but what they're saying. It's like that kind of turn of phrases, but also what it's connected to. So the, the home assistants now, it's basically your access to a search engine for the, the internet, right?

• 20:08Speaker 00

  You can ask it kind of fact-based questions and, and get answers that are usually accurate pretty fast. Um, at least they're more accurate than, than a GPT-3's question. But yeah, I think it's the, how fast it is, how, uh, human it sounds and the kind of breadth of functionality it has. You can ask it so many different things and ask it to do things for you, right? Like schedule appointments and blah, blah, blah.

• 20:37Chris Benson?

  Not as a question, but just as a, as a final thing on that topic. It's interesting, coming through COVID, coming through the pandemic era, it's changed the way kids interact. And I've watched my daughter, they'll get on and play Roblox, you know, on online gaming and they're, and they do a cellular conference call on the side as there's all the kids are talking and they're playing in Roblox and they Include these home devices in the conversation and bring things up. And so it's kind of eerie to watch this whole thing happening because you're seeing children leaping forward with this technology in a natural way, very, very rapidly. Multiple times I'll just stop and go, wow, I find that a little bit hot, but and I'm in the space. So anyway.

• 21:21Daniel Whitenack?

  I really appreciated how you brought up this like concept of like how, how people actually use their languages. So like, how, how do humans actually speech or speak, which I think is like a really interesting, like we've sort of done this to ourselves in certain ways because the speech corpora, for the most part that we have created, aren't actually representative of how people use their languages. So things like I recently at ACL, I saw this like amazing paper. I'll link it in the show notes from text to talk from a group at Radbod University in the Netherlands. I'm, I'm not sure if I'm saying that right, but they were talking about like meta communicative

• 22:05Daniel Whitenack?

  devices, meaning like filler things like, mm, uh, yeah, you know, like these sorts of things like you brought up. Like, these are very powerful communicative devices that people use very strategically in their conversation, but are for the most part considered like noise in our AI data, right? And we like clean them out or don't want to have them. And I think the other one that you brought up, Josh, was like emotion and like how you like tune sort of like the emotional aspect of a synthesized voice. And I know that's something you emphasize on the Coqui website. Could you describe maybe like, maybe it's in the context of synthesized speech or speech technology more generally,

• 22:48Daniel Whitenack?

  but like, what are some of your kind of goals as you founded Coqui? And like, how would you like to do some of these things like synthesized voice in like slightly different ways? Or how is your perspective in being in the industry so long informed you about like, oh, there's things like this emotional piece that we really need to think about more deeply?

• 23:10Speaker 00

  I think about emotion and speech way too much, especially the last, last month. It's been really kind of top priority for us because I mean, we've been working for, for a while on getting the voice cloning side of things working. So optimizing models on.

• 23:30Speaker 00

  Speaker similarities, so it sounds like you, like your vocal tract, you know, physically. But getting emotion right is just so hard. And not only getting it right, but there's this kind of, I'm also thinking about, so there's getting the model, you know, the neural network to, to produce speech that sounds appropriately emotional. That's one side of it. But another side that I've been thinking more lately about is how to, from a user's point of view, somebody who is creating new synthetic speech with some neural network, how do they want to interact with that? How do they think about emotions?

• 24:12Speaker 00

  Is it something like a color wheel from Microsoft Word? Where you can say, I want my color to be, you wouldn't even describe it in words. You just, there's, there's a pixel that you point to on the color wheel and you're like, I want that. That's a much better interface, even if you can't describe it in real human words, right? I mean, you can do whatever RGB coordinates, but, uh, and some designers maybe understand those intuitively, but most people in the world do not. So thinking about color and kind of design as thinking about putting emotion into speech. Do we want an emotional color wheel of sorts?

• 24:54Speaker 00

  Do we want to have a dropdown menu that, that says, make this sound more angry, make this sound more sad, make this sound more sarcastic? Or do you want something that's more free form? Like type in a description of somebody, you know, responds in a angry, but sassy, but sarcastic, but also a little bit sad at the end kind of way, you know? It's hard.

• 25:23Daniel Whitenack?

  Yeah, it's almost like there's an arc to how you want it to happen, right? Like even defining like one emotional flavor for a clip is sort of not, not enough in certain ways.

• 25:36Chris Benson?

  It makes the communication authentic, essentially, which creates trust, which affects that user experience.

• 25:43Speaker 00

  So you mean if you're able to put the appropriate emotion

• 25:46Chris Benson?

  Emotion into, yeah. Because at the end, there's a human that that that that model is dealing with.

• 25:52Speaker 00

  It's something that you can think about it in this way that I was describing, like, like a designer that's using Photoshop, right? In terms of using a color palette, you can also think about voice in terms of the stage directions or actor directions, right? I mean, I've spent, I spent a whole day sifting through scripts from movies and TV shows to just understand how do writers express what they're trying to get across in the lines? Because if you look at a movie script, there's, it's not just whatever Batman says this Joker says that. It's looking away from the camera, distantly, thinking about the future solemnly.

• 26:33Speaker 00

  Then they say this and putting that behind a web-based user interface is is a challenge. Even more so, it's a challenge to make a neural network that is that controllable. It's what we were working on. It's exciting to hear. One of my favorite parts of honestly working at Koki is when we've, we've got so many people who are so much smarter than than I am when it comes to speech synthesis and, and, you know, we're, we're working away hacking on something for a week. And then on Friday, we, you know, share some, some new voice clips from the new models. And it's like, wow, that person sounds super angry, you know?

• 27:16Speaker 00

  And it's just a synthesized, it's a synthesized voice talking about whatever, getting the wrong coffee. I don't know. It's, it's, it's one of my favorite things about, about working with this.

• 27:25Daniel Whitenack?

  Yeah. So would, would you say that the kind of the mindset that you have at Koki and kind of one of the things that you're wanting to enable is this sort of easy to access, like from a creator standpoint, this sort of configurable and controllable way to synthesize voices? Would that be a kind of like synopsis of like at least part of what you're, what you're trying to do?

• 27:49Speaker 00

  No, I'd say definitely. I mean, that's, that's one side of it, which is very much the kind of. Business side, customer facing, making synthetic speech for people who are creating content, right? That's one side of it. And another side, which is kind of historically where we came from is the open source research. I mean, I think that we're pretty special in terms of the kind of the voice cloning companies out there. I mean, I wouldn't say we're saying that we're a voice cloning company is not doing it justice. Or so we're synthetic, you know, a speech company, but the amount of work that we do open source, and I think that because we're open source, we've been able to really attract some, some really smart people.

• 28:33Speaker 00

  We've learned from them. We, you know, share and collaborate and, and that is honestly one of the most also refreshing parts of the job is getting people, especially people working on low resource languages, which is something that I did my whole kind of PhD on. And, uh, and that's working on this paper with Daniel, uh, from the Bible TTS, which is coming out of Interspeech, making some of the best synthetic voices for, for a handful of, of African languages is, I mean, the working with everybody on that team was so much fun because it was like, everybody's so motivated and also we're all in such different time zones. It felt like there was this, you know, passing off the torch. Yeah. Yeah.

• 29:17Speaker 00

  I mean, I think you have that, that clip, if you want to play it of kind of what it, what it sounds like.

• 29:21Daniel Whitenack?

  Yeah, this was how's the clip. One of the interesting things about this clip, I think, is it's one of the out of domain clips. So we tried like synthesizing some voices because the voices are out of audio bibles. We tried some Bible related synthesis, but this is actually out of domain. So this is like a news article, but using the same voice.

• 30:10Daniel Whitenack?

  So Josh, I really, it was, as you mentioned, a lot of fun working on the project related to the synthesized African voices. That was so cool. And also seeing the, you know, the actual members of these language communities working on technology for their languages. And I think a lot of that was enabled because of like a variety of open source tools, but certainly kind of centered around some of these that Koki has produced. Did those open source things, were those things that you were working on personally before like the company was like founded? Or was this like a sort of team thing that

• 30:52Daniel Whitenack?

  you started and was always kind of part of a strategy that you were building with Koki?

• 30:59Speaker 00

  Okay, so in terms of the kind of core open source technology that we've been working on at Koki, there's two main sides of it. There's the speech to text and there's the text to speech. Those two projects were projects that we were working on, a founding team, we were all working on for the past almost five years at this point, when we were, we were all working or collaborating with Mozilla. And so it's been going on for a while. And I think, so for this project in particular, there was a couple of other parts that were really helpful that were actually outside of, outside of Koki. So

• 31:41Speaker 00

  one of them, in case people are interested, was the Montreal Forster liner, which is maintained by a very hardworking group of academic folks, which made-

• 31:51Daniel Whitenack?

  It is really nice.

• 31:52Speaker 00

  It's so nice, right? It's nice because it's built on top of Caldi, which is, Caldi, for anybody who's used it, can be a little painful. But the Montreal Forster liner wraps it so nicely that you don't have to worry about all the kind of, how do you put it? All the stuff inside the black box. But yeah, so we, the projects, they were started at Mozilla and the community, the open community grew around them and, and there's, you know, long time collaborators from all over the place in all different kinds of languages. And with Project Common Voice, which we were talking a little bit about, we mentioned before, was really, Common Voice

• 32:32Speaker 00

  was a project that was created to be the data feeder for the speech recognition side of, of the open source project. And that's why there's this really rich, I think, multilingual kind of heritage to the projects, if you want to call it that, because we've been working with kind of traditionally marginalized languages. And those people from those communities, they are so motivated to, to work with the languages. They care so much and they get it. Like, they get that this is important because some of the bigger companies are

• 33:10Speaker 00

  starting to put out more multilingual, multilingual work because of the existence of Common Voice, really. Because before that, there was just no data for, I think Common Voice has all the Celtic languages now. You know, it's got Welsh and, and Gaelic and Gaelic and maybe there's Manks in there too. I mean, there's, there's tons of languages that are just community-driven efforts. At least their, their participation in Common Voice and in speech recognition is a completely community-driven effort. So, yeah, I mean, if, if it weren't for the open source side of things, then, then none of this would, would have been possible.

• 33:51Daniel Whitenack?

  So like, one side of it is like the models, the architectures, the implementations that are driving like the speech to text, text to speech, and like the voice cloning things that, that you're doing. But then you also have kind of pre-built models as well. Like if I'm, if I'm going to your site, like there's a, there's a lot listed there, which like seemingly you could get started with kind of out of the gate. And maybe you could describe that kind of ecosystem a little bit, like what's currently there, how you've seen people use it, maybe even in surprising ways?

• 34:28Speaker 00

  Oh yeah. There's been some funny ways people have So the largest diversity of languages we have is for speech to text, the speech recognition side of things. We set up the code base so that fine tuning to a new language is super easy, even when you have a tiny bit of data. And if you constrain the vocabulary, you can do really cool things, even if you don't have enough data to make a kind of a full blown speech transcription system. So for example, we had a hackathon. Wow, was it this year? I feel like it was earlier this year, where a team put together a voice activated

• 35:10Speaker 00

  3D chessboard that you could, well, you can. It's it's it's open source. It's out there and they got it working for for English, for Turkish, think maybe Hindi. And right now, there's some people who are adapting it for Korean.

• 35:26Daniel Whitenack?

  And that's like, like move on whatever to like, I'm not a like advanced chess player, but I I know the sort of like things I hear on Harry Potter or whatever.

• 35:39Speaker 00

  Yes, yeah, it's exactly that, which is, there's a huge discussion on how to do that well. So now after that, I know how to how to move pieces in chess. But before I did not know how to, you know, say it out loud.

• 35:53Daniel Whitenack?

  You also have to capture Harry Potter's emotion, though, as you're moving the pieces, you know, and all that. You know, that was a very emotional game that they were playing there, so.

• 36:01Speaker 00

  Yeah.

• 36:02Daniel Whitenack?

  We're right on topic here.

• 36:03Speaker 00

  The cool thing about that is the, because the models for speech recognition are so small, I think they're like 46 mega megabytes for the acoustic model. And then the language model is like really small because the vocabulary is so small. So these things you can run just like on your laptop. You know, you don't, you can, you know, turn off the, turn off the wifi and just have it running locally. And actually, right now, the last few weeks, there's been a group of folks, a lot who are working with the Catalan language.

• 36:39Speaker 00

  Who are adapting our speech recognition tools to make it work with Wasm, so that it just runs in the browser like super fast, like everywhere. I mean, it's, and you know, it's just, that's just like a, a group of the, you know, a set, a subset of the, of our open source community who just picked up the tools and are running with it, you know, it's so, and then for the speech synthesis side of things, honestly, one of the easiest places to interact with those models is on Hugging Face. We have a Hugging Face space. I think it's Cokey-AI. And you can just with Gradio, the Gradio app is really nice, I have to admit. You just, you know, type in what you want to say, you click the language in the model you want,

• 37:23Speaker 00

  and then you, you get it. I don't remember exactly how many languages we have for speech synthesis, but it's growing. And after the, the Masakane collaboration, it's six more languages from Sub-Saharan Africa. So it's pretty cool.

• 37:38Chris Benson?

  So I'm, I'm just wondering, as you're, as you're doing some of these, how, I'm going, I'm kind of going back to when you were talking about how you're just thinking about this all the time, you know, and you can't really turn that off. Voices in, in your head, no. Yeah, exactly. Like, like so many topics, what are the types of things that practitioners, as opposed to the users, need to be thinking about as the field at large is moving forward? Because, you know, we've asked these kinds of questions of ourselves and, and, you know, 99% of everyone has the best of intentions as, as do you. But how do we make sure that as we really move the state of the art forward in terms of having things like very, very genuine sounding emotion,

• 38:22Chris Benson?

  you know, in, in different emotions rippling through this? How does the, how do we need to think about that in terms of the effect on, on the users? Cause there's some amazingly positive things and potentially if we screw up, or, or for the very few bad actors out there, there, there could be negative things as well. So it directly affects kind of mental health in a both positive and potentially negative of the end user. So how, like as someone who's thinking about this all the time, how do you frame that? How do you frame moving the field forward in a positive and productive way?

• 38:53Speaker 00

  So I think that I I'm glad you brought this up because it is a huge part of working in this technology. If you ignore it, you say, I'm working in voice technology and I'm just gonna work in my bubble. And I'm not even gonna care about the fact that somebody might be using our speech recognition for illegal surveillance. Right? Like if you ignore that that is a possibility, you are doing yourself and also the the community a disservice. So, oh, there's a lot with this. I would say that so for for one, I'd like to point, hopefully we can get up maybe a link to this. There's a an issue on our GitHub for for text to speech,

• 39:34Speaker 00

  which is an open discussion on ways of mitigating misuse of synthetic speech systems. And that started as kind of, actually, not an issue, it's a GitHub discussion, I think. And it started as this kind of, you know, hey, let's throw some ideas together and and it just evolved. And now it's this kind of growing discussion. And I think being open, having these conversations openly is is really important because we got some feedback. There were ideas that came from, you know, people in the community that I'd never even thought of. You know, there's there's watermarking audio, which is kind of obvious, but there's there's whole layers of that. And then there was somebody who did who's doing their like master's thesis on on this in particular. And they, you know, weighed in.

• 40:15Speaker 00

  So I think that I think one kind of first step is to think about, to not brush it off, to think about what kinds of misuse is possible. Because there's so many different kinds. And if you lump them all together, then it becomes too kind of hard of a problem. And it can just kind of lock up, you get, you know, brain freeze or however you wanna call it. But I like to think about kinds of misuse as basically two two major groups, I think. There's there's people who will misuse technology kind of accidentally, or they don't think they're doing something bad, but it just blows up, which could happen very easily on um social media.

• 40:56Speaker 00

  So The Onion, right, is a satirical newspaper from the United States. Famously, I don't know, like every other month, there's a an article from The Onion that gets taken seriously by people who don't know The Onion. And that can cause real harm, right? It can get people very upset. Because they have a satirical headline and it looks like a real news site, but it's not. And so that is an example of, you know, somebody who's not, they're not trying to spread fake news, but because people don't know the context, because people, it's just shared as a screenshot on social media, it loses all context and it just snowballs into something that it was never intended to be.

• 41:38Speaker 00

  So there's, there's examples like that. Like, you know, you can think of somebody who uses our, our voice cloning to make a clone of, you know, President Trump saying something and they share it on social media. And earlier we had a voice demo, we had a voice cloning demo, which was, you didn't need to sign up to have an account to do it. So anybody could do it. And what we did then, because anybody could do it without having us know who they were, we put basically a watermark, a very audible watermark in the background, which is background music. I mean, it's like a very low hanging fruit, but anybody listening to it would not think,

• 42:19Speaker 00

  oh, this is an actual recording, you know, a secret recording of, of Trump in the Oval Office.

• 42:25Chris Benson?

  You delegitimize it and deliberately, you know, to where any user will pick up on that.

• 42:30Speaker 00

  And so if you're, if you're making something that's like really, really accessible, easy to like the, the barrier to making a mistake and making something that you think is funny and sharing it with your friends and then it blowing up is very low, then I think putting those kind of roadblocks in is important. And roadblocks is, yeah, that's another, if you think about mitigating risk as putting up different kinds of roadblocks, because at some point it's impossible to mitigate all risks, or at least it's impossible to guarantee that somebody or an organization, organization that's motivated enough will not do nefarious things with your code, right? But there are lots of responsible ways to put roadblocks in there. So right now we have this, this

• 43:13Speaker 00

  voice cloning technology, but you have to sign in, you have to have a real email. And that is a way to have some kind of accountability. And on the speech recognition side of things, we have model cards that That explicitly say like, you know, you do not do bad things with this, but it's more specific than do not do bad things. And in terms of research collaborations, I think that especially when you're working with language communities in which you are not a member, it's important to have members of the community working with you so that you know, like, what are the risks? Because the risks for, you know, me living in California, the ones that I can perceive

• 43:55Speaker 00

  are not the same kinds of risks for, uh, you know, I have longstanding collaborators from Makara University in Uganda and we're, we've been working on, uh, radio. How do you say, keyword spotting and, and radio data for ideally to be used by the Ministry of Health, kind of help inform uh, health policies. And there's a whole bunch of risks that we you know, spent a long time talking about, uh, and, and figuring out how to mitigate for that context, because it's just different. And I think somebody brought this up on Twitter. Why it's, it's kind of forefront of my mind. How do you know if you're working with a language technology for

• 44:36Speaker 00

  a, uh, low resource or marginalized language that you're not doing harm to the community? And I think the simple answer is if you're not a part of the community, you have no idea. That's why you really need to be working with people from the community if you want to be working on that technology.

• 44:52Chris Benson?

  I'm glad you addressed it. It's a relief to, you have so much thinking around. I know way more than we can cover. But it's also, uh, it's a relief to know that you've kind of put that thought ahead of, of doing the stuff. And so it's, uh, yeah, thank you for that. Appreciate it.

• 45:06Daniel Whitenack?

  But yeah, I think it's also like, there's really amazing positive things.

• 45:10Chris Benson?

  Absolutely.

• 45:11Daniel Whitenack?

  Come out of involving the community, like this Masa Kana work that, that you and I did, Josh, with that community. Like I don't, I don't speak those languages, but there's like very simple things that like I would have missed in the audio or in the processing that were just completely obvious to them. And like, it made the result so much, so much better, also in terms of the quality of the work. So like just involving the community, like you learn so much, you learn about this like use and, and ethics side of things, but you also generally produce better, better output and better work.

• 45:51Daniel Whitenack?

  There's a lot, a lot to be said for that, and I'm glad you brought it up.

• 45:55Speaker 00

  And actually, I want to say, just like, the way I think about it is, I don't want to be working on a project where I'm, you know, involving the community. I want ideally to be involved. I want them involving me. Like, I don't, you know, the motivation. And I think that with, with this project in particular, I mean, I know I did a lot of the kind of the technical, some of the technical, not even technical side of things, but it was very much a Masakane-driven project. And I think that that is, if there's language communities out there that want to collaborate, I'm like, yes, let's do it. But I definitely want to be the one who's getting involved as opposed to trying to pull other people into a project that might have false pretenses, you know, in the first place.

• 46:39Speaker 00

  Like, I, I could think this is a great project, and I might be able to convince people that it's a great project, and it's their language. But it's not, at the end of the day, I think that if you get the motivation, the impetus going the other direction, that's where real good work is done.

• 46:53Daniel Whitenack?

  Maybe that's a good sort of way to segue to a close as we're coming up on the end here is like, what would you tell kind of people out there? Maybe it's language community members, because now we do have listeners all over the world, like, language community members that want to get involved and sort of like build things with the open source technology that, that Koki is a part of. Or maybe it's people that are creators or curious about this technology and want to get involved. Like, what, what would you tell them in terms of like, joining into this work and helping move it forward in a, in a positive way?

• 47:28Speaker 00

  Yeah, I think there's like a million ways to get involved in a machine learning project, and you do not have to be technical. I think there's people who use the tech, and that's like, that's, that's the obvious one to point to. Like, oh, you know, I'm involved in the project because I'm using it to, you know, I'm using the, the voice cloning software to clone my voice sounding like five different characters in my video game, and I'm using that. That, that's one way. But there's, from the open source side of things, there's so many ways to get involved. Like, documentation is just like the, like, a super low-hanging fruit, you know? Documentation is something that really can make or break an open source project. And you can be...

• 48:09Speaker 00

  Super technical and write super technical documentation, which is useful like API documentation. Or you can be um, somebody who's, you know, writing a kind of best practices playbook on how to use the tech. And honestly, the people who are less technical, maybe they're just starting, they know a little Python. Those people are able to write tutorials and beginner friendly documentation way better than people who have been in it too long. Because if you've been in it too long, you're, you have all of these, you've just like absorbed all of these assumptions that are not intuitive for working with the code. And so I think getting involved with an open source project.

• 48:52Speaker 00

  An easy way to do it is to, you know, join wherever people are talking, whether it's on GitHub, whether it's uh, like we use uh, Gitter, uh, G I T T E R for our chat rooms. And you can pop into the chat rooms and say, hey, this is me, I want to get involved, here's my skills. That's a, that's also a very low hanging fruit. And also, I think, so the Masai Kani community is, I think, like, honestly, I think the best example of this, they've got such an active chat room on Slack. You've got people from all across the, the spectrum of super technical to not technical. But the, the thing that unites everybody is just they love languages, right?

• 49:34Speaker 00

  When I think about where I want us to be, like at Koki for a kind of healthy open source community, I often think about Masai Kani and how they've, they've done a great job. It's just like a, that's the whole reason the project started, like the, the, the speech synthesis project started is because I enjoyed hanging out in those Slack rooms because they were fun. And then we just, you know, started brainstorming and then it just evolved. So yeah, I think that getting involved is, is very easy. Definitely don't have to be technical. And a lot of times, the non-technical people have more to bring uh, because they've got a fresh view on things. Beginner's mind. Yeah.

• 50:11Daniel Whitenack?

  Definitely appreciate that. And I think it's a great way to end. I mean, definitely even with this podcast, it's great to be part of a wider community that's doing amazing things. And you sort of get sucked into these amazing stories and with Masakane or other things. So yeah, really glad that you brought out that side of things and appreciate you taking time to speak with us, Josh. Really excited to see what's happening with Koki and hope to have you on again in another 80 episodes to share all the great things that are happening then. Thanks.

• 50:49Speaker 00

  Yeah, thanks for having me.

• 50:59Speaker 01

  All right, that is Practical AI for this week. If this is your first time listening, subscribe now at practicalai.fm or just search for Practical AI in your favorite podcast app. We're in there. And if you're a longtime listener, please do share the show with your friends. It is the best way you can help Practical AI succeed. Thanks again to Fastly for shipping our shows super fast all around the world to Brake Master Cylinder for the Beats and to you for listening. We appreciate you. That's all for this week. We'll talk to you again next time.