NLP for the world's 7000+ languages

Expanding AI technology to the local languages of emerging markets presents huge challenges. Good data is scarce or non-existent. Users often have bandwidth or connectivity issues. Existing platforms target only a small number of high-resource languages. Our own Daniel Whitenack (data scientist at SIL International) and Dan Jeffries (from Pachyderm) discuss how these and related problems will only be solved when AI technology and resources from industry are combined with linguistic expertise from those on the ground working with local language communities. They have illustrated this approach as they work on pushing voice technology into emerging markets. Featuring: Dan Jeffries – LinkedIn https://www.linkedin.com/in/danjeffries, X https://x.com/Dan_Jeffries1Chris Benson – Website https://chrisbenson.com, GitHub https://github.com/chrisbenson, LinkedIn https://www.linkedin.com/in/chrisbenson, X https://x.com/chrisbensonDaniel Whitenack – Website https://www.datadan.io/, GitHub https://github.com/dwhitena, X https://x.com/dwhitenaShow Notes: The world’s languages https://www.ethnologue.com/guides/how-many-languagesEthnologue https://www.ethnologue.comTTS for Singlish https://github.com/sil-ai/tts-singlishSIL International https://www.sil.org/Pachyderm https://pachyderm.io/Previous episode - Pachyderm’s Kubernetes-based infrastructure for AI https://changelog.com/practicalai/23Upcoming Events: Register for upcoming webinars here https://practicalai.fm/webinars!

Speaker names marked ? are inferred from speaking time and the show’s known hosts — they are not voice-verified.

• 0:00Speaker 00

  Being with your change log is provided by Fastly. Learn more at fastly.com. We move fast and fix things here at Change Law because of Rollbar. Check them out at rollbar.com. And we're hosted on Linode cloud servers. Head to linode.com slash change log.

• 0:17Speaker 00

  Do not underestimate the power of the independent open cloud for developers. Yes, I'm talking about Linode. Linode is our cloud of choice and it's the home of changelog.com. What we love most about Linode is their independence and their commitment to open cloud. Open cloud means being unencumbered by outside investment and maximizing value for the community, not shareholders. And that's exactly what Linode represents. No vendor lock-in, open at every layer. If you want to learn more, head to linode.com slash open. Again, linode.com slash open.

• 1:02Speaker 01

  Welcome to Practical AI, a weekly podcast that makes artificial intelligence practical, productive, and accessible to everyone. This is where conversations around AI, machine learning, and data science happen. Join the community and Slack with us around various topics of the show at changelog.com slash community and follow us on Twitter. We are at Practical AI FM. Okay, here's Daniel and Chris.

• 1:28Chris Benson?

  Welcome to another episode of the Practical AI Podcast. My name is Chris Benson. I'm Principal AI Strategist at Lockheed Martin. And with me as always is my co-host, Daniel Whitenack, who is a data scientist with S I L International. How's it going today, Daniel?

• 1:42Daniel Whitenack?

  It's going pretty good. It's been a busy day of recordings. We're just talking about this before the episode started. I just shoved a bunch of cashews in my mouth as a snack and I think you had some breakfast troubles in between episodes.

• 1:55Chris Benson?

  Yeah, I had not had breakfast and so in the last few minutes I went to get breakfast and as I was putting it together, I have multiple dogs. One of my dogs did something. I think it was an organized thing. One dog was trying to pull me to the side while the other dogs went and got my breakfast. So I'm going to sit here and we're going to do this podcast hungry. That's right. If you know what, we persevere no matter what. We handle it. Exactly. Okay. These are the sacrifices we make for practical AI. That's right. That's right. We're hardcore podcasters. So, you know, it's interesting. I gotta say, I know you live up north, but I'm about to head out today for the Denver, Colorado area, Littleton, where Lockheed Martin has part of its space division. And I looked at the weather for packing and it's like, it's going down to like zero degrees Fahrenheit.

• 2:40Chris Benson?

  Nice. And for listeners who don't know, I'm from Georgia. I'm a southern boy, used to warm weather. I'm quite frightened to get on this plane and go to this place with such frigid temperatures coming. Yeah, yeah. I'm being tough in two ways, both with the dogs and with the weather. I'm just saying. You're leveled up? There you go. I'm ready to go. Okay. Well, you know, we have a pretty interesting episode coming up here. It's going to be a little bit different. Hopefully. I think so. From what our listeners usually hear, typically we'll have either a guest on to talk about what the guest is involved in, or you and I will do what we call our fully connected episodes where we will talk about a topic between the two of us. And we're, we're doing a little bit of a blend of those today. So we're going to address AI with local languages.

• 3:25Chris Benson?

  And today, instead of strictly being the host, you're here representing S I L International, which is a nonprofit in local languages. I'm allowed to speak more than just questions this episode. You're allowed to speak more than just questions on this episode. And also with us today, we have Dan Jeffries, who is the chief technical evangelist at Packyderm. Welcome, Dan. How's it going?

• 3:47Speaker 04

  It's going wonderfully. Thanks for having me on the show.

• 3:49Chris Benson?

  No worries. And for listeners upfront, I'm going to try to go with, for Daniel Whitenack, we're going to say Dan W. And for Daniel Jeffries, we're going to say Dan J. So it'll be a little bit different since we're broken out with Dan's today.

• 4:01Daniel Whitenack?

  Yeah, I suggested that Dan J. go with Pachyderm Dan, but he informed me that he's not completely defined by his employer.

• 4:10Chris Benson?

  Oh, my God.

• 4:11Daniel Whitenack?

  Which I guess I understand.

• 4:12Chris Benson?

  Oh, and you've now put that out on this recording. I can't believe that.

• 4:16Speaker 04

  Well, Dan W. wanted to be fully defined by his employer and says that he has no other outside interests whatsoever. So.

• 4:25Daniel Whitenack?

  I don't know if that's a direct quote, but I'll let it slide through.

• 4:29Chris Benson?

  Okay, boys. Okay, let's get back in our corners now. Okay, we have a conversation to dive into here. I'm going to actually start for a second with Daniel Whitenack, whom our listeners probably know pretty well, but they know you mostly on the Practical AI Host side, and I'd like you to take a moment and talk about S I L International and what you do there. And then we're going to flip over to Dan J. in a moment.

• 4:52Daniel Whitenack?

  Sure, definitely. Yeah, so as you know, I introduced myself as a data scientist with S I L International, which sometimes we joke as kind of S I L International is everywhere, but no one really knows about it. It's an international nonprofit. We actually have people from over 80 countries working in 90 countries. And the vision and mission of S I L is to see people flourish in communities with languages they value most. So we do everything associated with language work, which involves a lot of things. So that involves things like multilingual education and involves things like literacy work, like language development, even like language survey and mapping and other things.

• 5:34Daniel Whitenack?

  But it's also technology related. So one of our products is called Keyman, which is a keyboard for devices like phones and tablets. And it supports over 1,000 languages, which the next biggest keyboard solution doesn't support even nearly that much. We also have a product called the Ethnologue, where we track, you know, what languages are being spoken where by how many people, how many languages there are in the world. We're also involved in the ISO standard process for the ISO standards for languages, little codes that represent languages. But I personally work on things. Related to AI and language. So S I L has actually gathered a lot of data related to

• 6:18Daniel Whitenack?

  languages. We've worked with over 2,000 languages. And so part of my responsibility is to help S I L develop programs and do experiments and research to push AI tasks like translation or sentiment analysis, speech to text, text to speech, these sorts of things, past the languages that are currently supported into the longer tail of languages in the world, those local languages spoken around the world where there currently isn't support for those things. So yeah, that's what I get the privilege of doing.

• 6:53Chris Benson?

  Well, thank you for that unusual introduction for this podcast and kind of bring people in. I got to say, as you finish, having worked with you now all this time on all these episodes, for listeners who don't already know, you are truly a natural language processing expert in AI, and I have learned a lot from you in the time we've been working together.

• 7:10Daniel Whitenack?

  It's been fun.

• 7:10Chris Benson?

  Thank you. As we turned toward Pachyderm, I wanted to actually start off by noting that we have previously had an episode on Pachyderm entitled Pachyderm's Kubernetes-Based Infrastructure for AI. It was episode number 23, and our guest representing Pachyderm on that was Joe Doliner, who everyone calls J D. He's the CEO of Pachyderm. But with us today, as we kind of dive into this story about local languages, Dan J, can you kind of tell us a little bit about yourself and a little bit about how you arrived at Pachyderm?

• 7:38Speaker 04

  Sure. I arrived at Pachyderm through a circuitous path. I've been a technologist for 20 years. I had an IT consulting company for a decade that I sold. And then I spent nine years at Red Hat. I designed some of their early artificial intelligence strategy before anybody was really thinking about it. And I am also a science fiction author with four novels, professional blogger. And one of the things that started to get me very interested in artificial intelligence early was a series of articles that I wrote called Learning AI, If You Suck at Math. And I was taking it from an engineering perspective, trying to look at it

• 8:21Speaker 04

  from, I'm not a data scientist, but I was trying to look at it from someone who'd spent many years setting up huge SaaS infrastructures, gigantic web farms, office-backed infrastructure.

• 8:33Speaker 04

  And trying to figure out whether this stuff was learnable by someone who hadn't studied statistics and enjoyed all those things in school. And so that series of articles proved very popular. It's read by over 2 million people in seven different parts. And I was essentially teaching myself many of the concepts as I was going along. And at that point, I was starting to get talks around the world, both for artificial intelligence and other future technology. And I realized at a certain point in time after my beloved Red Hat was purchased and it was starting to change in terms of its structure.

• 9:12Chris Benson?

  Yep.

• 9:13Speaker 04

  That I wanted to go back to someplace that was very innovative and that was doing fresh things in the industry. And that's when I came upon Pachyderm, which falls very much into the ML Ops side of the house and does essentially version control for data science. So when your models, your data and your code are all changing simultaneously, how do you keep track of all those things and create reproducibility? Because if you run a bunch of tests on a series of a million images and then an administrator comes in and crunches them all down on the backend to a smaller size, then it's going to be nearly impossible to recreate that earlier experiment. So I've been very excited to be with with those folks and everybody there is fantastic. And it's exciting to be in

• 9:58Speaker 04

  this amazing industry now and get to work with incredible people like like you two folks.

• 10:03Chris Benson?

  Fantastic. Well, thank you for that. So, you know, starting off, I know I've learned a lot from Dan W. about local languages myself, just from being associated with them. And we did a keynote together recently at Project Voice. And he has been talking for a while about the partnership between S I L. and Pachyderm on this. And but I'd like to start off by asking, you know, now that you guys have been partnering together and working on this problem for a while, could you actually tell us, kick us off by telling us a bit about what local languages are and why they should matter to the rest of us?

• 10:38Daniel Whitenack?

  Yeah, I can definitely jump in and give my perspective. Something I didn't know when I started working with SIL was really just about the language situation around the world, which is maybe something we should just start out by talking about. So a lot of people don't realize, actually, we track this very, very closely. Right now, there's 7,111 languages spoken around the world. So these are living languages, not like dialects and that sort of thing, but recognized living languages that are spoken around the world. And certain countries, like, let's say, India or Indonesia, other countries, have hundreds of these languages being spoken. Indonesia, for example, it's like about 700.

• 11:20Daniel Whitenack?

  And so what happens as a result of this is that there's some languages that are spoken by a lot of the world. So over half of the world speaks, I think it's 23 of those languages, which is a very small number, right? Compared to 7,000, it sure is. Yeah, the other half of the world speaks the rest of them. And so what happens is that these local language communities, which don't maybe speak one of these higher resource languages, are usually and often marginalized in some way. And so these marginalized language communities value communication in their local languages, but they, they aren't supported in a, in a variety of ways.

• 12:03Daniel Whitenack?

  And this has implications for a lot of things. One of the ways that we think about this is in terms of the United Nations Sustainable Development Goals. So the United Nations developed these 17 goals about sustainable development. And language impacts basically every one of those. So if you think about education or humanitarian assistance, thinking about something like HIV or AIDS or Ebola, how are you going to be able to make progress in those areas where there's extreme language diversity? If you can't get out materials about HIV or other things in a language that people understand and value, that they don't consider something foreign? And this goes across the board in education.

• 12:48Daniel Whitenack?

  There have been studies that have shown if if someone starts their education by going to school, so if they're at home and their family speaks a certain language and then, you know, their mother speaks it, their father speaks it, their whole village speaks it, and then they go to school and the first thing their teacher says is, oh, it's great that you speak that, but you're not going to speak that here. We're going to learn in this other language. Well, immediately right off the bat, they form an association with education as something terrible and hard, and it actually stunts their educational development. Whereas if they start education in their mother tongue, they actually have the same benefits as others in terms of their views of education and their their forward

• 13:31Daniel Whitenack?

  momentum. So, language impacts everything, and that's why we care about these local languages, because they actually make a difference for people's quality of life.

• 13:40Speaker 04

  I think this falls into that AI for good category as well. That's something that's very close to my heart. I started Practical AI Ethics Alliance. It's practical-AI-ethics.org, if anyone's interested in in checking that out. But the basic concepts behind it are, this artificial intelligence is, you know, a dual-use technology, right? It reflects everything that is good and bad about humanity. So, when we're looking at a problem like languages, and I think maybe the most impressive slide and the most impactful slide that I saw at your presentation, Dan W., is where it showed that we were using maybe 100 languages

• 14:24Speaker 04

  for the vast majority of of applications, whether that's translation or speech to text. And there were, you know, 7,000 other languages that half the population was speaking, and we weren't doing anything with those. And that's a surefire way, really, to continue to marginalize people or to ensure that, like you said, that people are going to find education difficult or that even finding basic services incredibly difficult. So, I think it's wonderful that you're working on a side of the house that allows you to make a difference and have an impact in this part of the world, because I feel like... So much of the research sometimes gets poured into, you know, getting people to click on ads

• 15:08Speaker 04

  or all the things that make us money. And those things are certainly important. Economics are incredibly important. But it's also amazing to realize that artificial intelligence can make the world a better place in some ways, right? It almost sounds cliche or it sounds a little, you know, high minded, but it is actually true that certain types of things would never be able to be done without it. I remember seeing a translation for very old Japanese that only maybe a hundred scholars in the world can speak now. And there are tens of thousands or millions of texts in a form of Japanese that's really not used anymore.

• 15:46Speaker 04

  And machine learning is able to augment the ability of those translators to scale what it is they're doing so that those texts don't die out into the pages of history, simply because we don't have anybody interested anymore in being able to translate them. So I think that's where this thing can make a massive impact.

• 16:04Chris Benson?

  Yeah, and you've kind of gotten into the idea of the importance of applying AI to the possibilities. I mean, and is there any way, do either of you have any comments on kind of expanding that a little bit? In terms of, you know, why apply AI to this long tail of languages? You know, you just identified one. Any others that come to mind?

• 16:24Daniel Whitenack?

  Well, I think that, you know, AI and especially, I guess, what I feel like we're on the verge of sort of new possibilities in terms of AI and language. Over the past couple of years, you've definitely seen, like, some people might refer to as an inflection point with a lot of new techniques, a lot of emphasis on transfer learning, a lot of emphasis on usage of monolingual data. And, you know, things that really impact the languages in this sort of long tail of languages, the languages outside of those supported by the major tech platforms. So I think one thing to note is that, you know, whereas used to, we might have been stuck with some of these things like machine translation because of lack of data or something like that,

• 17:07Daniel Whitenack?

  there are brand new possibilities where, you know, what if we could translate? You know, translate these HIV materials or this educational material into all of these different languages. What if we could enable people to be part of the global conversation in their mother tongue? That's a very interesting one for me is, you know, oftentimes I think we feel like, oh, how can we get our great content as Westerners into the languages that, you know, people care about, which isn't, isn't a bad thing. It's great to try to get those materials, those educational materials, scientific materials, you know, entertainment media into local languages. But these people in these local language communities, they have so much to

• 17:52Daniel Whitenack?

  contribute to our understanding of the world, to scientific research, to all sorts of different areas. And so, one of the things that I think AI could enable with things like speech to speech translation, with things like predictive text and other things like that, are enabling those local language communities to be part of a global conversation, not just to be consumers, but to actually contribute in a back and forth sort of way to global conversations around the things that actually impact their life and the things that they can contribute, like politics and education and all those things.

• 18:28Speaker 04

  I think it's interesting that you're talking about it from a two-way street. And I think that that's an amazing way to frame it, because what is it that we can learn? Not just what is it that we can translate from our own content into what other people can consume, but really, what is it that we're missing on the other side of the equation? And historically, if you look at how language has been used, sometimes it's been used as a way to dominate other cultures, or as a way to socialize. It's almost been used as a weapon. The farther that a language can spread, the more that people think in your own way. If you think about something like the Etruscans in the Roman Empire, we know pretty much nothing about the Etruscans, primarily

• 19:09Speaker 04

  because they were completely consumed by the Roman Empire, not in small part due to language. And we've seen this in other parts of the world, but That sort of slash and burn mentality, there's a lot of things that are lost in the same way when we, you know, destroy a huge part of the forest, what medicines get lost that we would never have been able to find, what compounds were hidden in the species of plants that were wiped out. And in the same way, different types of languages allow us to think in a different way. And in fact, we just had the wonderful Super Bowl on the other day and there was a commercial, believe it or not, that stopped me for a second where they said that there were four words in Greek for love. And I went and looked up each of

• 19:54Speaker 04

  the words and they were fascinating in that each one of those words conveyed something very different about the nature of love, from eros, which is more of a passionate, you know, kind of love, to agape, which was more of a selfless kind of love or a love for country. And each of those words convey something very different. And so when we lose these languages, or we assume that we have the words that perfectly convey things, we lose a lot of nuance and meaning. And we lose people's ability to connect in a fluid way with us. So I think it's amazing that you framed it as a two-way conversation. That's very important.

• 20:43Speaker 01

  This episode is brought to you by Brave. We deserve a better internet. That's why the team behind Brave reimagined what a browser could be. Brave is like Chrome, the good parts. Even your extensions will just work. It has built-in add-in tracker blocking, easy anonymization with the Tor network, earn tokens while you browse and use them to tip your favorite creators, and did I mention it's lightning fast? Turns out the web is super fast when you remove all the cruft. Download Brave today using the link in the show notes and give tipping a try on chainsaw.com.

• 21:24Chris Benson?

  So Dan W, what is S I L kind of doing in terms of A I for local languages these days? And what are you interested in doing to tackle in terms of the types of problems and issues that S I L is attending to, specifically on your A I roadmap?

• 21:42Daniel Whitenack?

  Yeah, definitely. So in the longer run, like I say, we want to see this sort of two-way street that we've been talking about in terms of local languages being part of a global conversation. And I think maybe a natural place to start with that is the sorts of A I technology that have already been groundbreaking in our everyday life, maybe as English speakers or other high resource language speakers. And so like if you do a Google search now, right, that's hitting an A I model now integrated BERT into that. If you're writing an email, you've got predictive text along with that that helps you. If you're dealing with a chatbot, you have these things like sentiment analysis

• 22:24Daniel Whitenack?

  and entity recognition. If you're talking to an assistant or a smart speaker or other things, you're using speech to text, you're using maybe those same assistant capabilities, you might be using text to speech. And so these sorts of building block A I technologies are what we're thinking about a lot right now. And how could we take those building blocks, which now only support sort of high resource languages, maybe up to 100, but as Dan J mentioned, that's kind of a drop in the bucket. How do we push those into the longer tail of languages? And what really excites us is, you know, how could we not

• 23:05Daniel Whitenack?

  just do that language by language? Like, oh, we add the next language, and then we add the next language, and then we add the next language. How could we knock out 40 languages at a time, right? And I think those are the things that get us really excited. And so some of the things that contribute to those sorts of advances, I think, are, first of all, multilingual models. So we've seen this shift recently. Into massively multilingual models that support things like Google Translate, where one model actually can process multiple different language pairs. And something people may not know is, you know, I think no one's been able to to challenge me on this, but I think S I L and its partners have access to the most massively multilingual corpus that there is.

• 23:50Daniel Whitenack?

  So we've done work in over 2,000 languages. We have some type of data, whether that be text or audio. And depending what partners we gather together, maybe like 1,200 languages. So there's a lot of data there. And and part of what we're excited about is, you know, what happens if we take the largest multilingual model that there is now, which I think the most multilingual one is around 103. What if we push that to something like 300? You know, how does that affect adding the next language in? Does it make it easier? How should we structure these types of models into language families and other things? So we're exploring those things. At the same time, we're exploring a lot of the low resource machine translation technology

• 24:33Daniel Whitenack?

  that's been developed around transfer learning and fine tuning iterative back translation. There's just a lot of different techniques out there now that allow you to maybe take a high resource language and adapt it to a lower resource language, or even make use of multilingual data. And so those are all the things that we're interested in exploring. First, in terms of experiment and research, and then in terms of making strategic partnerships with tech companies, but also local institutions and governments to pilot out some of these possibilities and actually get them used.

• 25:09Chris Benson?

  And that really begs the question for both of you is, you know, as you talk about these partnerships, what specifically brought S I L and Paco Darm together to tackle these kinds of problems that you've just addressed here? And why, from each of your perspectives, why did that partnership make sense?

• 25:24Daniel Whitenack?

  Sure, I'm very pleased that Paco Darm wanted to work with us. So I'm really happy about that. Thank you to Dan J and the team who wanted to work on this. But I think that despite S I L having a ton of data and a very multilingual corpus and an amazing amount of language information and linguistic expertise. We're not a tech company. We're a nonprofit that has done language related work for a very long time, but. Isn't really an AI company per se and isn't operating a ton of computational infrastructure. So whereas we have a lot of this sort of data and language information, that side of the equation.

• 26:05Daniel Whitenack?

  Part of what we want to do is partner with people that have a lot of expertise on the infrastructure side, on the AI methodology and AI, you know, practical AI training side. And Pachyderm definitely fits into that component. So from my perspective, that's what I was excited about working with Pachyderm is actually building something useful that we can use over time and repeatedly use and scale up. Because this is a large scale problem, right? 7,000 languages, we need something that's going to scale and something that's going to work. And so that's what originally got me thinking of partnership with Pachyderm. And, you know, Dan J can speak from Pachyderm's perspective, but I hope that they were excited about these sort of, you know, AI for good problems.

• 26:48Speaker 04

  Well, we definitely are excited about these AI for good problems. And frankly, we've been looking for a number of these types of things in the field. So folks are out there interested in doing those types of things. We, we want to talk, we're certainly not, you know, DeepMind or OpenAI or have infinitely deep pockets to be able to throw at some of these things. But we do feel it's of tremendous importance for us to help enable projects like this. And frankly, Pachyderm is, is more of the infrastructure side of the house. We recently launched the Pack Hub product, which runs on Google Cloud and allows people to automatically spin up clusters and add GPUs to them and parallelize their resources.

• 27:30Speaker 04

  And those types of things, I think Pachyderm is one of the solutions that people don't realize they need until they start doing data science at scale. And I think we're, we're seeing a development of a canonical stack probably over the next two to five years where the tools become codified that allow data scientists like, like Dan W. to really do their job easier. And if you think about the, the history of how these things have worked, a lot of times data scientists were just passing around a text file between them or maybe FTPing something somewhere and cobbling together infrastructure.

• 28:12Speaker 04

  That's not really going to work as you get this technology out of the hands of the unicorns that have. A billion dollars to just throw at things and create the infrastructure on the fly. So if you think about a company like Google or some of the research foundations doing their own work, they're all building all their pipeline tools and their training visualization tools and their explainer tools, all these types of things. And they're experimenting with lots of different frameworks and libraries. But over time, we're going to start to see more and more standardization. And a problem like being able to version control your data and understand the entire

• 28:53Speaker 04

  data lineage of where things got from point A to point B to point C is incredibly important for being able to reproduce experiments. I read in VentureBeat the other day that something like eighty-seven percent of data science projects never make it to production. And that's a massive number, considering that we've spent upwards of, you know, sixty or seventy billion dollars, I think was the number, or we're spending hundreds of billions of dollars more, or hundreds of millions of dollars more in the coming years. And that means we've wasted that much if we don't improve that. And if we don't improve that, we're in serious trouble. And one of the ways to improve that is by having that level of reproducibility and being able to work across a diverse team.

• 29:36Speaker 04

  And so getting our tools into the hands of people who are doing amazing things is definitely the way to get our name out there, but also really make a difference in the world. And I think both goals are incredibly important.

• 29:48Daniel Whitenack?

  Yeah, and on a practical level, since I'm always interested in keeping this podcast practical, I'll kind of walk through our internal workflow and thought process on this. I mean, if you imagine, let's just take machine translation, for example, which is one of the things that we're working on. Well, I can spin up a Colab notebook and, you know, develop the, kind of pull the data together from a source that we have access to inside of S I L, do some preprocessing on that, do the training, test, you know, do the testing, get the inference bit worked out. But now, if we're really serious about like our goal of

• 30:29Daniel Whitenack?

  Pushing this sort of thing into many languages at once, right? Now I have to think about other sorts of problems. And one of those, you know, on the data side is the data that S I L has access to and that we use is is a big mix of data. So it's partly internal data and most of the time like formatted in sort of maybe non-standard formats as far as A I people are thinking. It might be data from partners. It might be a mix of public data. And so we have all of these sorts of data that we may want to bring together in unique ways. And the combinations of those data for, let's say, if we're targeting 4D languages, they might be different for the different languages, right?

• 31:15Daniel Whitenack?

  And so there's this complicated issue of how do I combine all these things together in a sane sort of way with a bunch of preprocessing? And then I've got the problem of, okay, well, I need to kind of standardize those. I might, those data sets might be updating at certain times. And then I've got to connect all of those data sources with the correct preprocessing, like I said, but then training. And that training needs to happen on GPUs where maybe the preprocessing is happening on CPUs. And then I need to connect those output models. I need to actually export them and optimize them in a way where they can be exported to a certain place where they can actually be used in a in a product.

• 31:54Daniel Whitenack?

  So all of those things for like even, you know, a few languages or 4D languages or whatever we're looking at, that gets complicated fast. And so the ability to track all of that very rigorously, but also be able to scale it as we might want. And do it in a sort of way that isn't like, you know, there, there's only so many technical people at at S I L. So we can write, you know, uh, small bits of code to do these various things. But we're not going to write the whole infrastructure and logic around this. So we needed something that was able to handle those sort of data elements, scaling, preprocessing across lots of data sets, and also scaling our training while utilizing.

• 32:39Daniel Whitenack?

  And GPUs. And so Pachyderm project and the what's available in that project in the pipelining and data management allowed us to do those sorts of things. And it really comes down to the fact that, you know, we want to scale this out. We want to push it to many, many languages. And so to do that, we're going to have to do it reproducibly. And we're going to have to do it over and over and maybe scale it out horizontally as well. So there's a lot that goes into that and very thankful to get help on that front.

• 33:10Chris Benson?

  So, you know, I know Daniel W., when we were doing our keynote together at Project Voice, you had a really interesting example that you as S I L and Pachyderm worked together on around these sorts of problems. And it kind of outlined what both sides, how both sides approached and what you were able to do, including the benefits of doing the activities on Pachyderm. Could you kind of kind of go through that example? And I would also invite Dan J. to pipe in, you know, so you guys can kind of relay that together a bit?

• 33:41Daniel Whitenack?

  Yeah, sure. So the problem we were looking at was text to speech or speech synthesis, and specifically adapting an existing text to speech model to a local language or a local dialect or a local accent. And so you could think of examples, like there's a whole bunch of vernacular Arabics that are spoken around the world. There's a bunch of world Englishes that are different in certain ways. And so we took, for example, Singlish, which is a dialect that's spoken in Singapore. So it's actually a mix of like, English and some other languages that's spoken together. This is kind of an interesting problem, because as a dialect,

• 34:24Daniel Whitenack?

  it has these elements from, you know, at least four or more languages. But it also has various kind of standard accents that go along with it. So there's like an Indian accent, it's English, Chinese accent, it's English. And it's a nice kind of proving ground for some of this adaptation to accents and other things. And so we wanted to create some text to speech models. For Singlish, because for one, these don't exist. And for two, we were able to access some data through one of our partners in Singapore to get some of this data for Singlish and to utilize it to test out these methods. The downside to that is like, there was a lot of processing that had to happen here.

• 35:09Daniel Whitenack?

  So our partners were able to actually get us, I think it was like 800 gigabytes of data. So this was between our partner, Wordly, in Singapore and the government institution there, IMDA, that has gathered a lot of this data. All of that data is formatted in specific ways. Some of it's kind of noisy. It corresponds to a lot of different speakers, like 2,000 different speakers. And so there was a lot of like, how are we going to actually pre-process this? And then how are we going to like, make this efficient? So we're not running these models for like, you know, weeks on end without progress. So that's where we consulted a lot with Pachyderm.

• 35:51Daniel Whitenack?

  And they were able to kind of guide us through like, well, here's how maybe similar people have set up their pipelines in the past and the type of infrastructure that they've used and how they've, they've scaled it. I know, so Dan J, I know we worked through some issues with like, data, like how do we upload that much data? How do we pick out like, you don't want to load 800 gigabytes into memory? So how do you like, access some of that data, but not all of it to like, figure out what you need? So I think these are, you know, from my understanding, these are problems that other people are facing and they're kind of gonna, you know, help us solve some of those. I remember the one problem of like, accessing some data, but not all of it, was kind of key to this whole problem.

• 36:36Speaker 04

  I mean, being able to split up the data and we rely a lot on, I think we made an intelligent choice in going with Kubernetes and Docker early. So we could leverage a lot of the scaling that happens now. If you really think about the history of containers, it really, Google was running billions of these containers even before Docker existed. And so the industry was moving in this, in this direction. And they had originally an internal service called Borg that was then shifted over into the Kubernetes open source project. And that, that really took off and allowed people to kind of build these massive infrastructures that scaled much faster than these virtualization.

• 37:18Speaker 04

  Infrastructures where you ended up having an entire operating system that was built into this little box that you were processing things in. It was very, the virtualization era was very effective. But once we got to, to need lots of ephemeral machines or being able to quickly spin up a thousand different nodes to split up data and process it into little chunks so that you're not trying to load everything into a massive virtual machine and saturating the memory. We spoke with one customer recently, there's a case study coming out where they were doing a lot of language processing and they built their preprocessing tools.

• 37:57Speaker 04

  And they were taking about eight to 10 weeks on the biggest possible node that they could spin up in Google or AWS or Azure. So they were basically, you know, grabbing the most expensive node possible to try to fit everything into, into memory and stack the GPUs in there. And it was taking them about 10 weeks and they were able to parallelize it with Pachyderm and get it down to about six or seven days. So that is a massive increase and that's basically because they're able to split it up across multiple nodes without actually really having to worry about precisely how they split it up.

• 38:39Speaker 04

  Pachyderm does a lot of the heavy lifting for folks in, in the backend and allows it to work across multiple nodes as opposed to having to try to figure out within your own code because you're already worrying about a data center. It's like Dan W is already trying to worry about how do I solve a problem like transfer learning or a noisy data set and how do I clean that up? Or I've got a number of different formats. How do I either use all of those formats or standardize on a different format before I can even do any of my work? The last thing you need to be worrying about is then figuring out how to also be an infrastructure engineer and auto scale lots of different nodes yourself and spin them down and suddenly

• 39:22Speaker 04

  you're forgetting about them and, and then your company gets an AWS bill for, for a million dollars at the end of it. So that's, that's I think where we really make a big difference for, for folks doing this kind of work.

• 39:34Chris Benson?

  Gotcha. I have a kind of a follow-up question to the example itself that you guys have worked on. And that is, you know, if there's one thing that's become clear to me as the person not involved in this, it's what a target-rich environment this is to work on in terms of, you know, 7,000-plus languages and, you know, you're even with as much work as you're working on at this point, it's still, you're only hitting a fairly small fraction, at least at the moment. You have addressed this first example that you've kind of relayed to us. Why that one? Where can this particular approach that you've talked about lead to next? What do you envision as being kind of next steps for extending this?

• 40:14Daniel Whitenack?

  Well, I think one thing to emphasize with this and one other reason why we were really interested in this partnership to enable this sort of thing is Pachyderm is part of the open source community and part of the Kubernetes community. And so, anyone can run a Pachyderm pipeline and anyone can spin up a Kubernetes cluster in the cloud or on premises or wherever. And so, we're thinking about this sort of work as a sort of template, right? So, I created with the help of the Pachyderm team, created this template for training speech models using this pipeline. You could plug in any sort of speech dataset you want, assuming

• 40:55Daniel Whitenack?

  you could pre-process it into the right format. And so, the idea is, we showed this for, let's say, one accent and one speaker of this dataset that we worked with, and we're working on others as well. But I could publish that pipeline on GitHub, which I have, and anyone could pull that down. Anyone could access the IMDA data if they knew where to look, and we put the links in, right? And anyone could generate their own speech model, run the same Pachyderm pipeline on their own Kubernetes cluster, because everything is portable and everything is built on this great open source community. And people could collectively work on this for a greater impact

• 41:40Daniel Whitenack?

  than any one certain person. So, we're not gonna, like you say, This is a target-rich environment. So the only way that we're going to make progress here is if we make these sort of reproducible templates and enable people to run them for their own context and their own data and scale things up that way. So that was another really appealing thing to me about setting this stuff up is internally to S I L, we could rerun this pipeline on any sort of for any language where we have audio data to train our own text to speech, but we're not going to get to them all at once, right? So other people could run this pipeline with their own speech data on their own cluster to create their models.

• 42:25Daniel Whitenack?

  And so by creating this sort of reproducible template, it's actually enabling a different sort of scaling.

• 42:33Speaker 04

  You know, I touch on the open source, the essential part of open source. And I'm admittedly a true believer in open source. I was, like I said, at Red Hat for nine years and saw the early days there where recruiters told me, why are you going into this Linux thing? Solaris is where it's at and where all the money is.

• 42:54Daniel Whitenack?

  Solaris isn't where it's at?

• 42:56Speaker 04

  No. And I told them, I said, it might not exist in 10 years. And they thought, what are you talking about, right? And we spent a lot of time in the early days going in and saying, what is Linux? Why won't it fall over? Why would I bet my future on something like this? And it's been amazing to watch the transition to an open source over the course of those years to become the default model for how things are done. It used to be that it would happen, things would happen in a proprietary world. And then open source would come along and kind of build a commoditized, almost good enough version. But nowadays, everything, including most of artificial intelligence work, starts in open source. And there's a huge advantage, I think, with something like Packer and being completely agnostic

• 43:39Speaker 04

  to the tools that are built on top of it, especially versus some of the pure cloud services that have to, because of limited resources, take a fully opinionated stance on every project. They have to support it. For it to run as opposed to us, which allows a data scientist to really bring whatever tools they need to the project and then publish anything that they create. So we don't just have to explicitly support something like PyTorch. You know, we spoke with another, the group that I mentioned earlier was using a more obscure speech recognition toolkit called CalD, that they had heavily modified themselves. And the chances of something like

• 44:19Speaker 04

  that being supported in one of the cloud providers' choices with the limited resources, even if they are a billion dollar unicorn and they've got a thousand programmers, they still only have so many resources. So something like scikit-learn or PyTorch or TensorFlow and 50 different Python libraries are going to get supported, where something like CalD is not going to get supported in the same way that the languages are not going to get supported across the world because of resources. So allowing people to do things with open source and bring whatever they want to the party, I think allows this kind of collaborative creativity to happen and allows a kind of scaling that wouldn't be allowed to happen

• 45:01Speaker 04

  with smaller projects or being able to move towards languages that might not be represented. And I think those two concepts are intricately interwoven.

• 45:10Daniel Whitenack?

  Yeah, I remember, Chris, when we were talking about like our talk at Project Voice, one of the things that kept coming up is this sort of idea of collaboration for collective impact. And the problems that I have are actually not like I can get data for languages, I can get information for languages, I can get linguistic information for languages, I can get dictionaries, I can get grammars, I can get all of these sorts of resources. But I'm limited in certain areas as related to maybe, you know, some of these more infrastructure related things, I'm related resource wise. I work in an organization that's primarily

• 45:51Daniel Whitenack?

  a language related organization and we're doing a lot with AI now, but we're still kind of figuring out a lot of those things. Whereas you at Lockheed Martin, you have a lot of resources. In terms of computation, you have a lot of AI knowledge, but you might not have those language-related things that I have easy access to. So a great way for us to make an impact in the area of language is in this sort of collaborative way for collective impact, where we're not just kind of siloed in our own world. Like Dan J. was saying, we're not limited to our own sort of implementations, but we can work together, we can open source things, we can bring all of our resources together to solve kind of larger, harder problems.

• 46:35Chris Benson?

  Yeah, you know, I think the collaboration that we did, especially for Project Voice and really all along, as I've been learning from you over time about this, is it really brought home where local language is and how important it is. I know for me personally, I know I've mentioned before, working on humanitarian assistance and disaster relief initiatives at Lockheed Martin. And I know we've talked quite a lot about educational impacts. You know, it's so critical that you be doing this kind of work, what your two teams are involved in, so that we don't leave behind enormous populations of people on the planet and grow that digital divide. It's, I think it's kind of a, a little bit of a stealth issue to many of us, because it's not something we necessarily think about all the time.

• 47:20Chris Benson?

  But if there's one thing that I've come to realize in my own introduction to this has been just how incredibly important this is to everybody going forward. You can't, in a humanitarian assistance or disaster scenario, go in if you can't communicate effectively in the languages of the people that you are trying to work with. And same with education, as you pointed out. So I think if there's anything I'm coming away with, it's that we're at a very special time for kind of the integration of artificial intelligence and language, and that there are so many possibilities that could now be realized. And so I guess I want to finish up by asking you both, you know, what do you think the future is looking like for local languages and AI from this point forward? It's a remarkable moment we're in at this kind of turning point. But what do you see ahead?

• 48:06Chris Benson?

  And like, you know, as an example, how do you think S I L would work with Paquedrum and other organizations out there to enable all the possibilities that we have before us?

• 48:16Daniel Whitenack?

  I have so many ideas. My problem is not lack of ideas. I'm so excited and I'm sure Dan J you probably have a lot of ideas. I'm I mean, maybe you can go first. I don't want to still always be hopping in first. What are you excited about? Dive into it, Dan J. Yeah. Steal my thunder.

• 48:37Speaker 04

  Well, you know, I think actually when we had dinner together the other night at the conference, we talked about a larger misconception in artificial intelligence, which is it seems like everybody is pouring resources into this concept of, you know, generalized artificial intelligence. And that's, that's a noble goal. And I think we get there eventually at some point, maybe even in our lifetimes, maybe not, maybe it proves to be a lot more intractable than we imagine. But you said that you were thinking more in terms of augmentation. And the way that I tend to think about artificial intelligence these days is very much in that augmentation model as well, or that centaur model, right? Where the artificial intelligence

• 49:20Speaker 04

  is helping humans scale their abilities and use their higher order learning and understanding and intuition, the types of things that are still intractable in machines. We see some of this behavior that we might be able to call intuition and something like AlphaGo, as we combine three or four different algorithms. And maybe when we've got 20 algorithms working together, we could mimic it even more. But in the short term, this was really about scaling and about augmentation and about allowing people to do more. And if you think about something like language, where especially when you're working with a language where there aren't necessarily as many speakers, or there aren't as many experts in that field,

• 50:05Speaker 04

  or there isn't as much data, you absolutely have to have augmentation. You have to be able to scale what those folks are doing. And that creates more Leverage, right? If you think about trying to lift a giant rock by yourself, you can only do so much, but if you get a really long pole and put it under that rock, you've got a better chance of lifting it. And that's the way I think of artificial intelligence now in the speech side of the house, right? We need to be able to help all the folks out there if they've only got a thousand experts in a particular field, if the work that you're doing is able to combine a lot of different data sets and look across a thousand languages or a hundred

• 50:47Speaker 04

  languages or 50 languages and find the similarities and therefore make the fact that there is not as much data for that largely irrelevant and still create a very robust translation model that could then make it easier for a thousand different texts to be translated and then a human to go over each of those quick and enhance them versus that person trying to scale themselves to do a thousand different translations and quickly getting burned out. And so I think that's really the wonderful part of this aspect and from the Pachyderm side of the house, being able to scale the infrastructure automatically on the back end for folks to help data scientists do this kind of work so they don't have to also be specialists in machine learning operations and trying to figure out how to

• 51:33Speaker 04

  slice that data setup into a thousand nodes so that they can train it and do pre-prost fully. We're very happy to partner and make a big difference and a small impact in what you're doing as well.

• 51:44Daniel Whitenack?

  Yeah, I think you hit the nail on the head. I think what you got to there at the end around augmentation, but also this sort of idea of leveraging the language information that we have access to and things that are already out there to kind of grease the wheels and get things moving for local languages. I really am fascinated by the concept that like we have all these pre-trained models out there, right? For certain languages and for certain language families, we have these open data sets, plus we have data sets internal to S A L. We also have this Ethnologue resource, which is information about all of the languages of the world and how they're related. I'm just really fascinated by the idea that we

• 52:26Daniel Whitenack?

  could have all of that information together. So what languages exist, what are their populations, but also what languages currently have data and what languages currently have pre-trained models? And that way, when we have a Pachyderm pipeline, let's say, and we want to train a new text-to-speech model, or we want to train a new machine translation model, if we put in the front end, like, oh, I want to train a model for this language, you know, Kabundu, an Angolan language that doesn't currently have any support, what is the closest related language that has either a pre-trained model or has a lot more data? And kind of

• 53:06Daniel Whitenack?

  use tools like AutoML and some of this automation to pull those resources in and sort of augment the development of that next language. Those are the things that really, you know, excite me and fascinate me and excited to dig more into.

• 53:22Chris Benson?

  Well, this has been a truly fascinating conversation. I know I learned a lot. Thank you both for kind of diving deeply into local languages and how AI can impact and move that right along for the benefit of all. Truly an AI for good initiative. I'm pretty excited about it. And so Daniel Whitenack, Daniel Jeffries, thank you both for coming on the show.

• 53:44Daniel Whitenack?

  Thank you.

• 53:45Speaker 04

  Thanks for having us. Really appreciate it.

• 53:50Speaker 01

  Comment on this and every episode of Practical AI on changelog.com. Just pop open your show notes, click the Discuss on Changelog News button, and let your voice be heard. Hey, do you have a friend who'd enjoy the show? Shoot him a quick text or an email, Slack him, WhatsApp, send them a Snapchat, I don't know, write a message in a bottle, throw it into the sea. We don't care how, but we do appreciate the effort. Practical AI is hosted by Daniel Whiteneck and Chris Benson. It's produced by me, Jared Santo. Our music is by The Beat Freak, Breakmaster Cylinder. And we're brought to you by awesome sponsors. Support them, they support the show. We've got Fastly on bandwidth, Linode on hosting, and Rollbar on bugs. Are you receiving our free email every Sunday? If not, you're missing out. It's our editorialized take on this week in the world of software. What's interesting and why?

• 54:35Speaker 01

  Head to changelog.com/weekly to check it out. Subscribe for the price of a free egg roll. Thanks again for listening. We'll talk to you next week.